Decision Models: Stop Paying a Chat Model to Choose

Most Agent Calls Choose. They Do Not Write.

Look at what a production agent asks its model. Route this request. Does this message pass policy. Which tool next. Is this answer supported by the source. Which of six categories. None of these needs a paragraph. Each needs one answer from a short list, and a number that says how sure the model is.

Teams have bought these answers from chat models. They ask for JSON, pay for every token generated, parse the text, and get a label with no measure of doubt. In the past two months the labs started selling the narrow tool: a stateless endpoint that scores your options against an input and returns probabilities. It writes nothing and bills no output tokens.

This article makes three claims. Calls with a closed set of answers should leave the chat model. The confidence score turns them into a cascade: act when it is high, escalate when it is not. And that confidence is a claim, not a fact, until you plot it against labeled answers. No vendor has published that test for you.

A Startup Called Jev Started It, and Two Labs Copied It in Eight Weeks

The category has a first maker. TypeSafe AI, a startup that raised a $40 million seed and is led by Diogo Almeida, a former OpenAI researcher, released Jev in mid-September (The Register, September 16). Jev takes a state, as JSON or a string, and answers typed questions with probabilities. TypeSafe calls the class "System One" models and says it trains them with "Reinforcement Learning for Calibrated Decisions." The Register found no independent benchmarks and no named customers at launch.

Adoption was fast anyway. Vercel said on X that about 13% of its AI Gateway teams tried Jev on day one, and Forbes reported on September 19 that Vercel and Cloudflare added it for agent tool selection. One early commenter, Nathan Flurry, called it a smart switch statement: a 2016 classifier with 2026 intelligence. That is a fair description, and it is the point.

OpenAI announced its Decisions API at DevDay on September 29 as a limited preview with no pricing (The New Stack, September 30). Public beta followed on October 6 and general availability on October 22. Google launched the Gemini Decisions API on November 11. TechCrunch reports that OpenAI built its model by reconfiguring Luna for speed and cost, not by training a model for calibrated decisions. Google says Gemini Decision 1 is new and scores the predefined options in one forward pass instead of writing tokens.

The market is a month old and so are its leaderboards. TechCrunch says Liquid's D1 Ultra and Base Lab's Elephant-1 top two decision indices, with Jev at 7th and 3rd and OpenAI at 8th and 14th. Almeida himself tells users to distrust benchmarks and test on their own work. Take his advice.

The Idea Is Old. The Packaging Is New.

A decision model is a language model with its answers cut down to a list. The trick has a long record, and the record explains why confidence is the thing to check.

Before chat models, you classified text by reading the probability the model gave each label word. In 2020, Nogueira, Jiang and Lin trained a sequence-to-sequence model to emit relevance labels as words, then read the underlying logits of those words as relevance probabilities to rank documents. That is a reranker built on label probabilities.

Calibration came as a separate problem. Guo and colleagues showed in 2017 that modern neural networks are poorly calibrated, and that temperature scaling, a one-parameter fit on held-out data, repairs most of it. In 2022, Anthropic's Kadavath team found that larger language models are well calibrated on multiple-choice and true/false questions when the format is right.

Then chat tuning arrived. OpenAI's GPT-4 announcement in 2023 said the pre-trained model was highly calibrated, and that its post-training process reduced the calibration. The assistant layer, built to write well, damaged the one signal a classifier needs.

Our reading, not the vendors': a decision endpoint is the old logprob trick with the assistant layer switched off, sold as an API with the label list, the normalizing and the batching built in. That makes the product sound ordinary. It also tells you where to look for trouble.

The Cascade: Decide First, Escalate on Doubt

Chat models with JSON mode do give you a label. With a strict schema and an enum, the label is guaranteed to be one of your options. What you do not get is a measurement. Any "confidence" field you add is a number the model writes, in the same sampled text as everything else. A decision endpoint returns the distribution over your options. One shared input can carry many questions, so a single call can route, check policy and score urgency together.

Here is the shape. The request below follows OpenAI's guide for a choice question; Google's takes the same form with different field names, so read its reference before you code.

{
  "model": "gpt-6-luna",
  "input": "I was charged twice for my order.",
  "questions": [{
    "type": "choice",
    "name": "department",
    "instructions": "Which department should handle this complaint?",
    "choices": [
      { "value": "billing", "description": "payments, invoices, and refunds" },
      { "value": "technical", "description": "problems using the product" },
      { "value": "shipping", "description": "delivery and tracking" },
      { "value": "other", "description": "requests outside these categories" }
    ]
  }]
}

The answer comes back as a choice, a probability for each option, and a confidence (OpenAI's guide shows 0.93 for this example). The cascade wraps a gate around it.

request
  -> decision model: route (choice), policy (predicate), urgency (score)
  -> gate on confidence and margin, thresholds set from labeled data
       high  -> act in code, no further model call
       mid   -> escalate to a chat model, which also writes the rationale
       low   -> human queue
  -> log: input, question version, full probabilities, threshold, path taken

The money follows from the shape. A chat call pays for input and for every output token, and output length varies. A decision call pays for input only, and you control the input. Output cannot grow without limit, so the worst case sits close to the typical case. Our AI spend piece made that the point of forecasting. Here is the arithmetic for one routing step.

The assumptions: a 2,000-token input (the request plus handler descriptions), 300 tokens of question text, a 150-token JSON reply from the chat model, and a chat price of $2 in and $12 out per million tokens, the GPT-5.6 Terra rate that The Register reports TypeSafe compared against. The 7% escalation rate is a guess. These are list prices, not measurements. The decision call is 28 times cheaper, the cascade 9 times. If you already route with a small cheap model, your gap is smaller. Use your own rates.

Cache hits cost $0.01 per million tokens at OpenAI. OpenAI caches the shared input only, with a 1,024-token minimum, so a routing call whose input changes every time gets little from it unless you mark a stable policy prefix with an explicit breakpoint.

Clive, the agent we run for a client, spends 0.3 to 3.5 cents on each routing step, the figure in that spend piece. We have not run a decision endpoint on it. Hermes, the framework under Clive, had no core support for the endpoint as of v0.23.1 on November 9; one community plugin handles yes/no questions only. For us the first job is the labeled set, not the API call.

Confidence Is a Claim Until You Plot It

The cascade works only if "0.95" means that about 95 in 100 answers at that score are right. That property is calibration. The Psychology of Judgment and Decision Making defines it, and says how much data it takes to see it:

”

"The only way to reliably assess calibration is by comparing accuracy and confidence across hundreds of judgments" — Scott Plous, The Psychology of Judgment and Decision Making (§ "CALIBRATION")

Now read what the vendors say. OpenAI's reference lists confidence and probability as bare numbers with no definition. Its guide makes no calibration claim and tells you to set thresholds from labeled examples in your own application. Its own severity example returns probabilities of 0.1, 0.7 and 0.2 with a confidence of 0.55, so confidence is not the top probability. Predicate answers carry a probability and no confidence field at all. Google puts a 0-to-1 confidence on every answer and says the model has the lowest calibration error across its suite, but the scores sit in images. Langfuse notes that Jev's yes/no type has no confidence field either.

So you have up to three numbers to gate on: the top probability, the margin between the top two, and the vendor's confidence. Test all three.

import numpy as np
 
def reliability(conf, correct, edges=(0.0, 0.5, 0.7, 0.9, 0.97, 1.0001)):
    """Rows: (low, high, n, mean stated confidence, observed accuracy)."""
    conf, correct = np.asarray(conf, float), np.asarray(correct, float)
    rows = []
    for lo, hi in zip(edges[:-1], edges[1:]):
        m = (conf >= lo) & (conf < hi)
        if m.any():
            rows.append((lo, hi, int(m.sum()), conf[m].mean(), correct[m].mean()))
    return rows
 
def best_threshold(conf, correct, cost_wrong, cost_escalate):
    """Expected cost per call at each cut; accepted calls can be wrong,
    rejected calls cost one escalation."""
    conf, correct = np.asarray(conf, float), np.asarray(correct, float)
    best = None
    for t in np.unique(conf):
        take = conf >= t
        cover = take.mean()
        err = 1 - correct[take].mean() if take.any() else 0.0
        cost = cover * err * cost_wrong + (1 - cover) * cost_escalate
        if best is None or cost < best[1]:
            best = (float(t), float(cost), float(cover), float(err))
    return best  # threshold, cost per call, share handled by the model, error rate on that share

The second function is the part no vendor can write for you. Agrawal, Gans and Goldfarb split a decision in two:

”

"In a decision tree, prediction generates the probability that each branch in the tree will occur. Judgment generates the payoffs associated with the outcomes at the ends of each branch." — Ajay Agrawal, Joshua Gans & Avi Goldfarb, Power and Prediction (§ "The Judgment Opportunity")

The endpoint supplies the probability. The payoffs, what a wrong refund approval costs against what a human review costs, are yours, and they set the threshold.

The 300 comes from the rule of three: with zero errors in n clean trials, the 95% upper bound on the error rate is about 3 divided by n. Three hundred clean answers let you say the error rate is under 1%. Hubbard's calibration work gives the matching test for the top band:

”

"If you said you were 100% confident on any answer, you must get it right. Getting even one 100% confident answer wrong is sufficient evidence that you are overconfident." — Douglas W. Hubbard, How to Measure Anything (§ "Calibration Exercise")

Apply it to a score of 0.99. If your top band holds wrong answers at a rate well above 1%, the number is decoration.

What a Decision Model Cannot Do

It gives no reason. Langfuse says Jev gives no reasoning when a trace scores badly, and OpenAI documents no rationale field. An audit that needs a stated reason per answer needs a second path. Log the input, the question version, the full probabilities and the threshold, so you can replay the decision. For escalated cases, have the chat model write the rationale. If you ask it to explain a decision after the fact, label that text a post-hoc account, not the cause. For regulated inputs, OpenAI's guide lists zero data retention and HIPAA use as available to eligible customers; our HIPAA agent design covers what that requires.

The questions are fixed and they are part of the model. OpenAI says choice values and descriptions are part of the prompt, so renaming a value can change results, and it advises a fallback such as "other." Version questions like code and rerun the labeled set on every edit. Langfuse notes Jev cannot abstain: with no "unknown" option, it picks the least wrong answer.

Dependent decisions need separate requests. If one answer picks the next question, you pay a round trip each time. Independent questions on one input fit best, and Google says latency stays nearly flat from 1 to 100 questions.

Long, noisy inputs hurt. Langfuse reports "context rot" in Jev and a 64k-token request limit, with conflicting figures elsewhere. Google accepts up to 256K tokens but ran its latency tests at up to 16K. A decision over a ten-document packet is a retrieval job first.

They can be gamed through the input. Our reading: a closed output space limits what an injected instruction can do, because the attacker can flip a label but cannot make the model call a tool or leak text. Google uses Gemini Decision 1 as a safety layer that watches for adversarial behavior in MCP tools. But a flipped label on a policy gate is still a breach. Keep the input assurance boundary in front of it.

Which of Your Calls Should Move

A call moves when its answer is a choice, code acts on it, and you can check it against labels. The test:

In practice that gives you a short list. Route the request. Gate a policy rule. Pick the next tool. Check that an answer is supported by its source, as a yes/no question with the source as input. Grade passage relevance before a reranker or a long answer. Tag urgency or category. Score a rubric verdict in an LLM-as-judge pipeline, where the agreement caveat above applies. Google reports using it as a model-tier router in Gemini Notebook, and OpenAI lists routing, classifying, severity triage and passage relevance in its guide.

Keep on the chat model: extraction, summaries, multi-step reasoning, and anything a person reads. Measure the move the way the cost-per-completed-task piece teaches. A decision call that is 28 times cheaper but wrong twice as often can cost more per correct answer once errors are repaired. And since a decision model makes the probability explicit where chat text hides it, it fits the pattern in The Probabilistic Stack: put a threshold where a vibe used to be.

What to Do Monday

Limits of this evidence: the prices, speeds and customer figures come from vendor posts and press coverage, not our tests. The worked example is list-price arithmetic with an assumed escalation rate. We have not benchmarked any decision endpoint, and no independent calibration result for OpenAI's or Google's model was published as of November 13, 2026.