Wrong arithmetic
Model input
Task: return 17 + 25. Candidate: 43.
Training-task & verifier evaluation
Verify candidate task outputs against explicit evidence and expected behavior.
Step 01
Define and Align
Upload sample data, define your output schema, and provide feedback on the hardest cases.
Output schema
- VerifiedYes / No
Whether the candidate satisfies the expected task behavior.
- true
- false
- Failure typesChoose any
Correctness failures found; empty when the output passes.
- incorrect_result
- missing_values
- duplicate_handling
- constraint_violation
- Order sensitiveYes / No
Whether the task explicitly requires a particular output order.
- true
- false
Collecting feedback · 1/3
Verify candidate task outputs against explicit evidence and expected behavior.
Validation accuracy
—
Example 1 of 3
3 annotations saved
Sutro can now generalize the rules based on your feedback.
Step 02
Model Sweep
Compare quality and cost on your examples, then choose a model or use Sutro’s default.
Compare models on expert agreement and cost. The interactive chart uses illustrative measurements.
Illustrative sweep · select Value or Best to use that model in your function.
Step 03
Deployment
Call Sutro Functions like any other software. Swap the model as soon as something more performant is released.
Selected modelWaiting for sweep results.
Real-time
import sutro as so
input_data = {
"text": "Task: return 17 + 25. Candidate: 43."
}
result = so.run_function(
"training-task-verifier-evaluation",
input_data,
)curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/functions/training-task-verifier-evaluation/run" \
-H "Authorization: Key $SUTRO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"text": "Task: return 17 + 25. Candidate: 43."
}
}'Batch
import sutro as so
job_id = so.batch_run_function(
name="training-task-verifier-evaluation",
data="training_task_verifier_evaluation.csv",
)
results = so.await_job_completion(job_id)curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/batch-inference" \
-H "Authorization: Key $SUTRO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "training-task-verifier-evaluation",
"inputs": [
{
"text": "Task: return 17 + 25. Candidate: 43."
}
]
}'