This article originally appeared on X.

A curve mapping four approaches to decision models from deterministic rules to semantic reasoning as task difficulty increases

Jev, and Jev-like decision models have created heated debates about classifiers, as if all decision problems are of the same type and difficulty. At Sutro, we see a huge range!

Consider the following tasks when looking at a resume:

  1. Did this person work at Google?
  2. Is this person a software engineer?
  3. Would this person be a fit for this JD at our company?
  4. Should we hire this person?

All of them will result in yes/no answers, but require very different modeling, context, and reasoning needs to automate them.

Jev solves certain problems of older-generation models (like needing to train custom classifiers). It also solves problems that older generation models can’t handle.

But it also stops short of solving some of the most complex and important decision problems, at least today.

Level 1: Deterministic Rules/Code

A huge number of decision problems can be “solved” at or near 100% accuracy without ML. You can use regex, exact keyword-matching, filters, fuzzy-joins, splitters, and combinations thereof to fully “solve” your problem.

For example, let’s say you have resumes that look like the following:

Alex Chen — Software Engineer

San Francisco, CA | [email protected] | github.com/alexchen-dev-1995

Experience
Full-Stack Engineer, Acme — 2023–Present
Built Python/React applications serving 100K+ users. Designed APIs and data pipelines on AWS.

Backend Engineer, Beta — 2021–2023
Developed backend services in Go and PostgreSQL.

Education
B.S. Information Sciences, UC Davis

Skills
Python, Go, TypeScript, React, PostgreSQL, AWS

Your task is:

Find people who previously worked at Meta, Google, or Amazon.

The best bet here is to write some code:

def worked_at_big_tech(resume):
    if "Meta" in resume or "Google" in resume or "Amazon" in resume:
        return True
    else:
        return False

Boom. There’s your >98% accurate decision model, free to run at near-infinite scale with near-zero latency. It is simple as that, and you don’t need to get fancier.

Level 2: Learned Keyword Associations

A trickier, but also super common type of problem will be much harder to solve using deterministic code. Instead of exact-matches on unstructured data, we’ll need a model that can make associations, not look for exact matches.

Let’s use the same resume example, with a new task:

Find software engineers or people with CS degrees

Slightly trickier: Alex’s resume doesn’t explicit say that he’s ever worked as a “Software Engineer” and it says he has a degree in “Information Sciences”. Should Alex be returned as a True or False? With what degree of confidence?

The good news: his titles, and education are latent features that will be detected by BERT-like encoder models. If we have several thousand labeled examples mapping:

Resume → True/False

then we can very quickly fine-tune a model like answerdotai/ModernBERT-large and have it learn these associations with relative ease. You’ll want to reserve a held-out to make sure it’s actually generalizing to unseen examples, but it’s likely that it will for a task of this shape.

Level 3: Semantic Policy

This is where we cross over from “things older-generation classifiers can do” to the need for decoder-only language models, or Jev-like Type One models.

Why? Because while latent features may exist, there is an actual a decision policy and decision criteria that must be followed. It’s not just a matter of finding unions of latent features.

Let’s revisit our resume problem for a new task.

Will this candidate be a good fit for a software engineering role at our company?

This is a true decision problem, not a keyword-association task. That said, our task is horribly underspecified, and there is effectively no task to follow. We’ll need to calibrate it against our specific needs. See https://github.com/sutro-sh/jev-align for example.

Is this candidate be a good fit for a software engineering role at WHT?

We build software that helps logistics teams automate warehouse operations.
We’re a 40-person startup with a small engineering team.
Engineers work across the stack and own features end-to-end.

Criteria:

* 3+ years of professional software engineering experience
* Strong Python and/or Go experience
* Experience building production APIs and backend systems
* Comfortable working with React/TypeScript when needed
* Experience with AWS and PostgreSQL
* Evidence of owning projects from design through production

Now that we have good semantic instructions, our model can look at Alex’s resume, and make a calibrated yes/no decision for this task.

This is the big difference between Level 2 and Level 3, and why Jev, and Jev-like models are important. It'll give you a calibrated answer to this very fast, and for very little cost, without all the pain associated with older models too.

It's very possible to use a larger, more expensive/general-purpose LLM for this, but very hard to use a BERT-like model. It likely won’t learn this specific policy and generalize well.

Level 4: Semantic Reasoning, “Agents”

Using Level 1-3 tools, we can now solve a LOT of decision problems! But we cannot solve them all—especially very high-stakes or more complex ones that typically require human judgement.

For example: what if our decision policy contains branching? Negations? What if we want the model to weight certain criteria more than others? Or allow certain criteria to override others? This is now effectively encoding a decision tree in a semantic policy. It will require more than just fast semantic inference—likely autoregressive reasoning to continue looking backwards at what has been inferred so far, and ensure it is traversing the policy correctly.

It may require external context, tool-calls, or on-the-fly code written to solve subcomponents of the problem. Certain problems may need auth to other tools, or access to its own GPUs.

It’s also likely we will want some kind of rationales, or explanations of how it followed the decision policy. Perhaps we may want to know where ambiguity in the policy still lies so it can be updated.

All of this will still just producing a top-level yes/no answer to a question! Would you believe this too is just a classifier?

The task might look like:

Task: Decide if this candidate is a strong fit for our software engineering position: Yes / No.

Our company: We’re a 40-person startup building infrastructure for AI engineering teams.

Rules

* If they have <3 years of professional engineering experience → No, unless they have unusually strong open-source or independent work.
* Must have strong backend experience. Python, Go, or Rust preferred; do not reject solely for using another language.
* If primarily frontend → No, unless they demonstrate substantial backend or infrastructure ownership.
* If technical depth is unclear → inspect their GitHub. Prioritize substantive repositories over contribution counts.
* If independent projects are material → inspect their personal site and linked projects.
* If relevant research is listed → inspect the publications.
* Startup or infrastructure experience is a plus, but neither is required.
* A CS degree is not required. Do not penalize candidates for lacking one.
* Missing evidence is not negative evidence.

This is complex! But it’s more representative, of a human-like, high-stakes decisioning task than our Level 1-3 examples.

For these type of problems you will likely need to reach for a larger, decoder- LLM or build a specific agent harness around the problem.

Conclusion

Maybe at this point you’re pretty convinced not all classification tasks are the same. Maybe not!

But today, much of the complexity still lies in choosing the right tool for the problem. In our ever-increasing world of abstractions, we think that complexity should disappear.

Building a decision model should require only your data and your judgment: examples, annotations, and corrections. The system should do the rest—choosing the right-sized model or combination of models, building the appropriate harness around them, and pulling you back in only when it needs more feedback.

If you’re thinking about decision models, calibrating them, or choosing the right model for a task, reach out at [email protected].

And if you’re building a Level 3 decision problem with Jev, you can try our open-source package, jev-align.