Agent-trace intent classification

Step 01

Define and Align

Review engineering examples and annotate the judgments or routing fields that need to change. Explain the evidence behind the correction.

Output schema

IntentChoose one: The latest tool-backed action, or an explicit plan when there is no tool invocation.
investigate
edit_code
run_tests
deploy
request_approval
Action statusChoose one: Whether the trace shows a proposal, an invocation without a result, or a returned outcome.
planned
attempted
completed
Target scopeChoose one: The scope established by the actual operation or explicit proposed target.
read_only
local_workspace
remote_environment
unknown
EvidenceText: The trace event that supports the intent, action status, and scope.

app.sutro.sh/functions/agent-trace-intent-classification

Collecting feedback · 1/3

Classify the evidenced action, its status, and its scope from the supplied agent trace.

Example 1 of 3

Planned edits do not override observed reading

Model input

Agent trace

User
Investigate the validation failure and propose a fix.
Agent
Read the validator implementation.
Tool result · read_file
Read validators.py. The relevant function body was returned. No file was modified.
Agent
I will change the validation condition next.

Classification policy: Classify the latest tool-backed action when one is present. A later statement about future work does not override the observed action. If there is no tool invocation, classify the explicit plan or approval request. A returned tool result establishes completed; an invocation without a result establishes attempted; a proposal alone is planned. Do not infer deployment from a mention of publishing.

Model output

Intent

Action status

Target scope

Evidence

Why? Optional guidance

Step 02

Model Sweep

Compare models on the same expert annotations and choose the quality–cost tradeoff for your workflow.

Compare models on expert agreement and cost. The interactive chart uses illustrative measurements.

Illustrative sweep · select Value or Best to use that model in your function.

Step 03

Deployment

Call Sutro Functions like any other software. Swap the model as soon as something more performant is released.

Selected modelWaiting for sweep results.

Real-time

import sutro as so

input_data = {
    "classification_policy": "Classify the latest tool-backed action when one is present. A later statement about future work does not override the observed action. If there is no tool invocation, classify the explicit plan or approval request. A returned tool result establishes completed; an invocation without a result establishes attempted; a proposal alone is planned. Do not infer deployment from a mention of publishing.",
    "trace": "[{\"role\": \"user\", \"content\": \"Investigate the validation failure and propose a fix.\"}, {\"role\": \"agent\", \"content\": \"Read the validator implementation.\"}, {\"role\": \"tool\", \"content\": \"Read validators.py. The relevant function body was returned. No file was modified.\", \"name\": \"read_file\"}, {\"role\": \"agent\", \"content\": \"I will change the validation condition next.\"}]"
}

result = so.run_function(
    "agent-trace-intent-classification",
    input_data,
)

curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/functions/agent-trace-intent-classification/run" \
  -H "Authorization: Key $SUTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "input": {
    "classification_policy": "Classify the latest tool-backed action when one is present. A later statement about future work does not override the observed action. If there is no tool invocation, classify the explicit plan or approval request. A returned tool result establishes completed; an invocation without a result establishes attempted; a proposal alone is planned. Do not infer deployment from a mention of publishing.",
    "trace": "[{\"role\": \"user\", \"content\": \"Investigate the validation failure and propose a fix.\"}, {\"role\": \"agent\", \"content\": \"Read the validator implementation.\"}, {\"role\": \"tool\", \"content\": \"Read validators.py. The relevant function body was returned. No file was modified.\", \"name\": \"read_file\"}, {\"role\": \"agent\", \"content\": \"I will change the validation condition next.\"}]"
  }
}'

Batch

import sutro as so

job_id = so.batch_run_function(
    name="agent-trace-intent-classification",
    data="agent_trace_intent_classification.csv",
)
results = so.await_job_completion(job_id)

curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/batch-inference" \
  -H "Authorization: Key $SUTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "agent-trace-intent-classification",
  "inputs": [
    {
      "classification_policy": "Classify the latest tool-backed action when one is present. A later statement about future work does not override the observed action. If there is no tool invocation, classify the explicit plan or approval request. A returned tool result establishes completed; an invocation without a result establishes attempted; a proposal alone is planned. Do not infer deployment from a mention of publishing.",
      "trace": "[{\"role\": \"user\", \"content\": \"Investigate the validation failure and propose a fix.\"}, {\"role\": \"agent\", \"content\": \"Read the validator implementation.\"}, {\"role\": \"tool\", \"content\": \"Read validators.py. The relevant function body was returned. No file was modified.\", \"name\": \"read_file\"}, {\"role\": \"agent\", \"content\": \"I will change the validation condition next.\"}]"
    }
  ]
}'