Search relevance judgments

Step 01

Define and Align

Upload sample data, define your output schema, and provide feedback on the hardest cases.

Output schema

RelevantYes / No: Whether the result satisfies the query’s required constraints.
true
false
Product matchChoose one: How closely the result matches the requested product.
exact
compatible_alternative
wrong_product
Unmet constraintsList of values: Requested attributes or compatibility requirements the result fails.

app.sutro.sh/functions/search-relevance-judgments

Collecting feedback · 1/3

Judge whether a result satisfies the query’s stated product constraints.

Example 1 of 3

Wrong accessory

Model input

Query: laptop charger
Result: Protective laptop sleeve. No charger or power adapter included.

Model output

Relevant

Product match

Unmet constraintsMultiple values

Why? Optional guidance

Step 02

Model Sweep

Compare quality and cost on your examples, then choose a model or use Sutro’s default.

Compare models on expert agreement and cost. The interactive chart uses illustrative measurements.

Illustrative sweep · select Value or Best to use that model in your function.

Step 03

Deployment

Call Sutro Functions like any other software. Swap the model as soon as something more performant is released.

Selected modelWaiting for sweep results.

Real-time

import sutro as so

input_data = {
    "query": "laptop charger",
    "result": "Protective laptop sleeve. No charger or power adapter included."
}

result = so.run_function(
    "search-relevance-judgments",
    input_data,
)

curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/functions/search-relevance-judgments/run" \
  -H "Authorization: Key $SUTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "input": {
    "query": "laptop charger",
    "result": "Protective laptop sleeve. No charger or power adapter included."
  }
}'

Batch

import sutro as so

job_id = so.batch_run_function(
    name="search-relevance-judgments",
    data="search_relevance_judgments.csv",
)
results = so.await_job_completion(job_id)

curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/batch-inference" \
  -H "Authorization: Key $SUTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "search-relevance-judgments",
  "inputs": [
    {
      "query": "laptop charger",
      "result": "Protective laptop sleeve. No charger or power adapter included."
    }
  ]
}'