CI judges

Step 01

Define and Align

Review engineering examples and annotate the judgments or routing fields that need to change. Explain the evidence behind the correction.

Output schema

VerdictChoose one: Whether the patch meets the criteria, violates them, or lacks enough evidence for a judgment.
pass
fail
needs_review
IssuesChoose any: Issues supported by the supplied patch and CI evidence.
requirement_violation
behavior_regression
insufficient_evidence
Missing evidenceList of values: Specific implementation or test evidence needed before accepting the patch.
ReasoningText: The criterion and source evidence that determine the judgment.

app.sutro.sh/functions/ci-judges

Collecting feedback · 1/3

Judge the supplied patch against its acceptance criteria and the available CI evidence.

Example 1 of 3

Green CI misses a behavior regression

Model input

Acceptance criteria

Apply a 10% discount only to eligible items. Non-eligible item prices must remain unchanged.

Patch

@@ def discounted_price(item):
-    return item.price * (0.9 if item.eligible else 1.0)
+    return item.price * 0.9

CI output

PASS: eligible item priced at 100 returns 90.
Job status: success.
No test for a non-eligible item is supplied.

Review policy

Pass requires the patch to meet all stated acceptance criteria with supporting test evidence. Fail for a demonstrated requirement violation. Use needs_review when implementation or test evidence is missing without a demonstrated violation. A green job alone is not proof of semantic correctness.

Model output

Verdict

IssuesMultiple values

Missing evidenceMultiple values

Reasoning

Why? Optional guidance

Step 02

Model Sweep

Compare models on the same expert annotations and choose the quality–cost tradeoff for your workflow.

Compare models on expert agreement and cost. The interactive chart uses illustrative measurements.

Illustrative sweep · select Value or Best to use that model in your function.

Step 03

Deployment

Call Sutro Functions like any other software. Swap the model as soon as something more performant is released.

Selected modelWaiting for sweep results.

Real-time

import sutro as so

input_data = {
    "review_policy": "Pass requires the patch to meet all stated acceptance criteria with supporting test evidence. Fail for a demonstrated requirement violation. Use needs_review when implementation or test evidence is missing without a demonstrated violation. A green job alone is not proof of semantic correctness.",
    "acceptance_criteria": "Apply a 10% discount only to eligible items. Non-eligible item prices must remain unchanged.",
    "patch": "@@ def discounted_price(item):\n-    return item.price * (0.9 if item.eligible else 1.0)\n+    return item.price * 0.9",
    "ci_output": "PASS: eligible item priced at 100 returns 90.\nJob status: success.\nNo test for a non-eligible item is supplied."
}

result = so.run_function(
    "ci-judges",
    input_data,
)

curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/functions/ci-judges/run" \
  -H "Authorization: Key $SUTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "input": {
    "review_policy": "Pass requires the patch to meet all stated acceptance criteria with supporting test evidence. Fail for a demonstrated requirement violation. Use needs_review when implementation or test evidence is missing without a demonstrated violation. A green job alone is not proof of semantic correctness.",
    "acceptance_criteria": "Apply a 10% discount only to eligible items. Non-eligible item prices must remain unchanged.",
    "patch": "@@ def discounted_price(item):\n-    return item.price * (0.9 if item.eligible else 1.0)\n+    return item.price * 0.9",
    "ci_output": "PASS: eligible item priced at 100 returns 90.\nJob status: success.\nNo test for a non-eligible item is supplied."
  }
}'

Batch

import sutro as so

job_id = so.batch_run_function(
    name="ci-judges",
    data="ci_judges.csv",
)
results = so.await_job_completion(job_id)

curl --fail-with-body -X POST "${SUTRO_API_URL%/}/v1/batch-inference" \
  -H "Authorization: Key $SUTRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "ci-judges",
  "inputs": [
    {
      "review_policy": "Pass requires the patch to meet all stated acceptance criteria with supporting test evidence. Fail for a demonstrated requirement violation. Use needs_review when implementation or test evidence is missing without a demonstrated violation. A green job alone is not proof of semantic correctness.",
      "acceptance_criteria": "Apply a 10% discount only to eligible items. Non-eligible item prices must remain unchanged.",
      "patch": "@@ def discounted_price(item):\n-    return item.price * (0.9 if item.eligible else 1.0)\n+    return item.price * 0.9",
      "ci_output": "PASS: eligible item priced at 100 returns 90.\nJob status: success.\nNo test for a non-eligible item is supplied."
    }
  ]
}'