Result ranking & comparison

Step 01

Define and Align

Review search examples and annotate the intent, constraints, or judgments that need to change. Explain how your relevance policy handles edge cases.

Output schema

Preferred resultChoose one: The better result under the supplied relevance policy; allow neither when both fail.
A
B
tie
neither
Relevance aChoose one: How well result A satisfies the query and its required constraints.
exact
partial
irrelevant
Relevance bChoose one: How well result B satisfies the query and its required constraints.
exact
partial
irrelevant
Deciding factorsChoose any: The relevance criteria that determine the comparison.
intent_match
required_constraints
evidence_quality
ReasoningText: A short explanation grounded in the query and the supplied result details.

app.sutro.sh/functions/result-ranking-comparison

Collecting feedback · 1/3

Compare the search results using the supplied relevance policy, including tie and neither outcomes.

Example 1 of 3

Compatibility beats popularity

Model input

Query: protective sleeve that fits a 13-inch laptop
Ranking policy: Rank results by the requested product function and every mandatory constraint. Popularity or current result position cannot override an explicit incompatibility. Exact satisfies all requirements; partial satisfies the intent but lacks required evidence; irrelevant is the wrong product or explicitly incompatible. Use tie for equally suitable results and neither when both are irrelevant.
Result a: Popular protective laptop sleeve. Compatibility: 17-inch laptops only. Ranked first in the current results.
Result b: Protective sleeve. Compatibility: 13-inch laptops. No popularity information supplied.

Model output

Preferred result

Relevance a

Relevance b

Deciding factorsMultiple values

Reasoning

Why? Optional guidance

Step 02

Model Sweep

Compare models against the same expert annotations and choose your quality–cost tradeoff.

Compare models on expert agreement and cost. The interactive chart uses illustrative measurements.

Illustrative sweep · select Value or Best to use that model in your function.

Step 03

Deployment

Call Sutro Functions like any other software. Swap the model as soon as something more performant is released.

Selected modelWaiting for sweep results.

Real-time

import sutro as so

input_data = {
    "query": "protective sleeve that fits a 13-inch laptop",
    "ranking_policy": "Rank results by the requested product function and every mandatory constraint. Popularity or current result position cannot override an explicit incompatibility. Exact satisfies all requirements; partial satisfies the intent but lacks required evidence; irrelevant is the wrong product or explicitly incompatible. Use tie for equally suitable results and neither when both are irrelevant.",
    "result_a": "Popular protective laptop sleeve. Compatibility: 17-inch laptops only. Ranked first in the current results.",
    "result_b": "Protective sleeve. Compatibility: 13-inch laptops. No popularity information supplied."
}

result = so.run_function(
    "result-ranking-comparison",
    input_data,
)

curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/functions/result-ranking-comparison/run" \
  -H "Authorization: Key $SUTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "input": {
    "query": "protective sleeve that fits a 13-inch laptop",
    "ranking_policy": "Rank results by the requested product function and every mandatory constraint. Popularity or current result position cannot override an explicit incompatibility. Exact satisfies all requirements; partial satisfies the intent but lacks required evidence; irrelevant is the wrong product or explicitly incompatible. Use tie for equally suitable results and neither when both are irrelevant.",
    "result_a": "Popular protective laptop sleeve. Compatibility: 17-inch laptops only. Ranked first in the current results.",
    "result_b": "Protective sleeve. Compatibility: 13-inch laptops. No popularity information supplied."
  }
}'

Batch

import sutro as so

job_id = so.batch_run_function(
    name="result-ranking-comparison",
    data="result_ranking_comparison.csv",
)
results = so.await_job_completion(job_id)

curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/batch-inference" \
  -H "Authorization: Key $SUTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "result-ranking-comparison",
  "inputs": [
    {
      "query": "protective sleeve that fits a 13-inch laptop",
      "ranking_policy": "Rank results by the requested product function and every mandatory constraint. Popularity or current result position cannot override an explicit incompatibility. Exact satisfies all requirements; partial satisfies the intent but lacks required evidence; irrelevant is the wrong product or explicitly incompatible. Use tie for equally suitable results and neither when both are irrelevant.",
      "result_a": "Popular protective laptop sleeve. Compatibility: 17-inch laptops only. Ranked first in the current results.",
      "result_b": "Protective sleeve. Compatibility: 13-inch laptops. No popularity information supplied."
    }
  ]
}'