RAG answer evaluation

Step 01

Define and Align

Upload sample data, define your output schema, and provide feedback on the hardest cases.

Output schema

LabelChoose one: Whether retrieved evidence supports the answer’s claims.
supported
unsupported
partially_supported
Addresses questionYes / No: Whether the answer responds to the question asked.
true
false
Unsupported claimsList of values: Claims that go beyond or contradict the retrieved evidence.

app.sutro.sh/functions/rag-answer-evaluation

Collecting feedback · 1/3

Check whether an answer is supported by the retrieved evidence and addresses the question.

Example 1 of 3

Unsupported promise

Model input

Question: Is delivery guaranteed tomorrow?
Retrieved source: Orders usually arrive within 2–4 days.
Answer: Yes, tomorrow is guaranteed.

Model output

Label

Addresses question

Unsupported claimsMultiple values

Why? Optional guidance

Step 02

Model Sweep

Compare quality and cost on your examples, then choose a model or use Sutro’s default.

Compare models on expert agreement and cost. The interactive chart uses illustrative measurements.

Illustrative sweep · select Value or Best to use that model in your function.

Step 03

Deployment

Call Sutro Functions like any other software. Swap the model as soon as something more performant is released.

Selected modelWaiting for sweep results.

Real-time

import sutro as so

input_data = {
    "question": "Is delivery guaranteed tomorrow?",
    "retrieved_source": "Orders usually arrive within 2–4 days.",
    "answer": "Yes, tomorrow is guaranteed."
}

result = so.run_function(
    "rag-answer-evaluation",
    input_data,
)

curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/functions/rag-answer-evaluation/run" \
  -H "Authorization: Key $SUTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "input": {
    "question": "Is delivery guaranteed tomorrow?",
    "retrieved_source": "Orders usually arrive within 2–4 days.",
    "answer": "Yes, tomorrow is guaranteed."
  }
}'

Batch

import sutro as so

job_id = so.batch_run_function(
    name="rag-answer-evaluation",
    data="rag_answer_evaluation.csv",
)
results = so.await_job_completion(job_id)

curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/batch-inference" \
  -H "Authorization: Key $SUTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "rag-answer-evaluation",
  "inputs": [
    {
      "question": "Is delivery guaranteed tomorrow?",
      "retrieved_source": "Orders usually arrive within 2–4 days.",
      "answer": "Yes, tomorrow is guaranteed."
    }
  ]
}'