Evaluate task completion, tool outcomes, permissions, and side effects across agent workflows.
Choose an example
Evaluate whether an agent completed the requested action and respected its constraints.
Step 01
Choose an example, review the proposed output, and annotate the fields that need to change. Your guidance defines the learned rules.
Whether the agent completed the authorized task.
Any failures demonstrated by the trace; empty when none.
The tool result or trace excerpt supporting the judgment.
Evaluate whether an agent completed the requested action and respected its constraints.
Book a confirmed appointment at 10am.
Request a 10am booking.
status: waitlisted; confirmed: false
Your appointment is booked for 10am.
Sutro can now generalize the rules based on your feedback.
Step 02
Compare quality and cost on your examples, then choose a model or use Sutro’s default.
Compare models on expert agreement and cost. The interactive chart uses illustrative measurements.
Illustrative sweep · select Value or Best to use that model in your function.
Step 03
Call Sutro Functions like any other software. Swap the model as soon as something more performant is released.
import sutro as so
input_data = {
"trace": "[{\"role\": \"user\", \"content\": \"Book a confirmed appointment at 10am.\"}, {\"role\": \"agent\", \"content\": \"Request a 10am booking.\"}, {\"role\": \"tool\", \"content\": \"status: waitlisted; confirmed: false\", \"name\": \"book_appointment\"}, {\"role\": \"agent\", \"content\": \"Your appointment is booked for 10am.\"}]"
}
result = so.run_function(
"agent-evaluations",
input_data,
)curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/functions/agent-evaluations/run" \
-H "Authorization: Key $SUTRO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"trace": "[{\"role\": \"user\", \"content\": \"Book a confirmed appointment at 10am.\"}, {\"role\": \"agent\", \"content\": \"Request a 10am booking.\"}, {\"role\": \"tool\", \"content\": \"status: waitlisted; confirmed: false\", \"name\": \"book_appointment\"}, {\"role\": \"agent\", \"content\": \"Your appointment is booked for 10am.\"}]"
}
}'import sutro as so
job_id = so.batch_run_function(
name="agent-evaluations",
data="agent_evaluations.csv",
)
results = so.await_job_completion(job_id)curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/batch-inference" \
-H "Authorization: Key $SUTRO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "agent-evaluations",
"inputs": [
{
"trace": "[{\"role\": \"user\", \"content\": \"Book a confirmed appointment at 10am.\"}, {\"role\": \"agent\", \"content\": \"Request a 10am booking.\"}, {\"role\": \"tool\", \"content\": \"status: waitlisted; confirmed: false\", \"name\": \"book_appointment\"}, {\"role\": \"agent\", \"content\": \"Your appointment is booked for 10am.\"}]"
}
]
}'Step 01
Upload sample data, define your output schema, and provide feedback on the hardest cases.
Resolution verified against tool outcomes and customer consent.
Whether the required action actually succeeded.
Any issues found in the support workflow.
Evaluate whether a support agent resolved the issue using the permitted tools.
Please reset my password.
Reset the customer’s password.
status: permission_denied; password_changed: false
Your password has been reset.
Sutro can now generalize the rules based on your feedback.
Step 02
Compare quality and cost on your examples, then choose a model or use Sutro’s default.
Compare models on expert agreement and cost. The interactive chart uses illustrative measurements.
Illustrative sweep · select Value or Best to use that model in your function.
Step 03
Call Sutro Functions like any other software. Swap the model as soon as something more performant is released.
import sutro as so
input_data = {
"trace": "[{\"role\": \"user\", \"content\": \"Please reset my password.\"}, {\"role\": \"agent\", \"content\": \"Reset the customer’s password.\"}, {\"role\": \"tool\", \"content\": \"status: permission_denied; password_changed: false\", \"name\": \"reset_password\"}, {\"role\": \"agent\", \"content\": \"Your password has been reset.\"}]"
}
result = so.run_function(
"ai-support-agent-evaluation",
input_data,
)curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/functions/ai-support-agent-evaluation/run" \
-H "Authorization: Key $SUTRO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"input": {
"trace": "[{\"role\": \"user\", \"content\": \"Please reset my password.\"}, {\"role\": \"agent\", \"content\": \"Reset the customer’s password.\"}, {\"role\": \"tool\", \"content\": \"status: permission_denied; password_changed: false\", \"name\": \"reset_password\"}, {\"role\": \"agent\", \"content\": \"Your password has been reset.\"}]"
}
}'import sutro as so
job_id = so.batch_run_function(
name="ai-support-agent-evaluation",
data="ai_support_agent_evaluation.csv",
)
results = so.await_job_completion(job_id)curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/batch-inference" \
-H "Authorization: Key $SUTRO_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "ai-support-agent-evaluation",
"inputs": [
{
"trace": "[{\"role\": \"user\", \"content\": \"Please reset my password.\"}, {\"role\": \"agent\", \"content\": \"Reset the customer’s password.\"}, {\"role\": \"tool\", \"content\": \"status: permission_denied; password_changed: false\", \"name\": \"reset_password\"}, {\"role\": \"agent\", \"content\": \"Your password has been reset.\"}]"
}
]
}'