Preference scoring & ranking

Step 01

Define and Align

Upload sample data, define your output schema, and provide feedback on the hardest cases.

Output schema

PreferredChoose one: The candidate that better satisfies the supplied quality rubric.
A
B
tie
Deciding factorsChoose any: Rubric dimensions that determine the preference.
factual_accuracy
format_compliance
completeness
source_fidelity
ReasoningText: A concise comparison of the candidates against the rubric.

app.sutro.sh/functions/preference-scoring-ranking

Collecting feedback · 1/3

Rank candidate answers against an explicit task-quality rubric.

Example 1 of 3

Constraint satisfaction

Model input

Rubric: Accuracy and the requested two-sentence format.
Request: Write a two-sentence summary.
Answer a: An accurate summary in two sentences.
Answer b: An accurate summary spanning ten paragraphs.

Model output

Preferred

Deciding factorsMultiple values

Reasoning

Why? Optional guidance

Step 02

Model Sweep

Compare quality and cost on your examples, then choose a model or use Sutro’s default.

Compare models on expert agreement and cost. The interactive chart uses illustrative measurements.

Illustrative sweep · select Value or Best to use that model in your function.

Step 03

Deployment

Call Sutro Functions like any other software. Swap the model as soon as something more performant is released.

Selected modelWaiting for sweep results.

Real-time

import sutro as so

input_data = {
    "rubric": "Accuracy and the requested two-sentence format.",
    "request": "Write a two-sentence summary.",
    "answer_a": "An accurate summary in two sentences.",
    "answer_b": "An accurate summary spanning ten paragraphs."
}

result = so.run_function(
    "preference-scoring-ranking",
    input_data,
)

curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/functions/preference-scoring-ranking/run" \
  -H "Authorization: Key $SUTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "input": {
    "rubric": "Accuracy and the requested two-sentence format.",
    "request": "Write a two-sentence summary.",
    "answer_a": "An accurate summary in two sentences.",
    "answer_b": "An accurate summary spanning ten paragraphs."
  }
}'

Batch

import sutro as so

job_id = so.batch_run_function(
    name="preference-scoring-ranking",
    data="preference_scoring_ranking.csv",
)
results = so.await_job_completion(job_id)

curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/batch-inference" \
  -H "Authorization: Key $SUTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "preference-scoring-ranking",
  "inputs": [
    {
      "rubric": "Accuracy and the requested two-sentence format.",
      "request": "Write a two-sentence summary.",
      "answer_a": "An accurate summary in two sentences.",
      "answer_b": "An accurate summary spanning ten paragraphs."
    }
  ]
}'