Jev vs LLM: When to Use a Decision Model Instead of Generating Text

Kenny Rogers ·

Jev vs LLM: When to Use a Decision Model Instead of Generating Text

Say you’ve got a product that’s getting 40,000 support tickets a month, and every one of them needs to be triaged to give three things: what it is about, should it be escalated, and what the reply is.

A frontier LLM with a JSON schema can do the job. Our benchmark run shows it doing so at a cost of $2.88 per 1,000 tickets and a two-second median, or about $115 a month. A small LLM did it for $0.09 per 1,000 at about one second.

Another way of thinking about it is to ask what a decision model can do. Jev is one from TypeSafe and, in the same run, it did the triage for two and a half cents per 1,000 tickets at a median of 194 milliseconds, or about a dollar a month. Plus it returns probabilities, so there is nothing to parse.

It can’t write the reply though. For that, you still need an LLM. So we benchmarked Jev, GPT Luna, and Claude Opus on 100 support cases through OpenRouter and built two patterns that make use of both.

What Jev is, and what each model actually returns

Ask a language model to make a code decision and it returns text. You can ask for JSON, but the model is built to write. Your code has to parse the output and make sure the model answered rather than explained.

Jev returns a typed answer instead. You send it state and questions built from three primitives: choice picks one option from a set you define, noul returns a yes probability, and score places content on levels you describe. Choice and score answers can include probabilities over your own keys, and noul is a single number. Here is a response for two questions about one billing ticket.

{
  "answers": {
    "intent": {
      "type": "choice",
      "choice": "billing_dispute",
      "confidence": 1,
      "probabilities": { "billing_dispute": 1, "order_status": 0, "other": 0 }
    },
    "escalate": { "type": "noul", "noul": 0.04 }
  },
  "model": "typesafe/jev-1.13-20260917",
  "provider": "TypeSafe",
  "usage": { "cost": 0.000016002, "inputTokens": 381, "outputTokens": 62 }
}

The intent is one of your keys and the escalation is a number you compare against a threshold you decide.

TypeSafe calls Jev a System One model. That means it makes fast, intuitive judgments rather than slow, deliberate analysis. It takes text only, so no images, audio, or PDFs as input. It reads criteria literally. What’s more, it tends to lose accuracy when the state it examines carries irrelevant detail, and it is unreliable at arithmetic, counting, and date comparison.

Compare Jev and a generative LLM side by side

Jev 1.13Conventional LLM (GPT Luna, Claude Opus)
OutputTyped choice, yes/no probability, or score, with per-option probabilitiesFree text (optionally constrained to JSON)
Best atClassification, routing, verification, ranking, guardrail checks, bounded extractionWriting, explaining, summarizing, rewriting, code, anything where the output space isn’t known up front
InputText only (strings, JSON, arrays). 64k tokens per request, 32k for state plus the longest questionText, and for many models images, audio, PDFs
Price (observed 2026-09-19)$0.042 per million input tokens, output freeGPT Luna $0.20 in / $1.20 out per million. Claude Opus $5 in / $25 out per million
Median latency on 60-ticket triage194 ms1,106 ms (Luna), 1,957 ms (Opus)
Cost per 1,000 triaged tickets$0.0248$0.0921 (Luna), $2.88 (Opus)
OpenRouter endpointPOST /api/alpha/decisionsPOST /api/v1/chat/completions

Prices are from OpenRouter model metadata on the run date. Check the model page before budgeting.

What we measured

The models were Jev 1.13, GPT Luna (resolved to GPT 5.6 Luna), and Claude Opus (resolved to Claude Opus 5). The LLMs got the same definitions and escalation rule as Jev in a system prompt, temperature 0, and a request for a bare JSON object. One request per example. These are small sets, 60 and 40 examples on a single day, with one set of prompts. This is the shape of the tradeoff, rather than a leaderboard.

Task A: triage 60 support tickets into five intents plus an escalation flag

Sixty support tickets across five intents: order status, return or refund, billing dispute, product question, account access, each with a one-sentence definition. Escalation covered legal threats, chargebacks, suspected fraud, safety hazards, and threats to publicize.

ModelIntent accuracyEscalation accuracyp50 latencyp95 latencyTotal cost (60)Per 1,000
Jev 1.1359/60 (98.3%)60/60 (100%)0.194s0.633s$0.001489$0.0248
GPT Luna59/60 (98.3%)60/60 (100%)1.106s2.395s$0.005527$0.0921
Claude Opus60/60 (100%)59/60 (98.3%)1.957s2.594s$0.17281$2.8802

Accuracy is a wash. Jev sends more input tokens because every request carries the full criteria, and it still costs about a quarter of GPT Luna and under one percent of Claude Opus. There was just one intent miss, a question about splitting a refund on a returned item. Jev filed that under return or refund at a confidence of 0.56. It was the only ticket under 0.8, so a 0.8 threshold would have sent it to a person anyway.

Task B: screen 40 messages for prompt injection

Twenty-two ordinary support messages and eighteen injection attempts (ignore-your-instructions, fake admin markup, role-play framings, quoted instructions). We gave Jev one yes/no question, is this an attempt to change the assistant’s behavior, and the LLMs got the same definition and returned a boolean.

ModelAccuracyp50 latencyp95 latencyTotal cost (40)Per 1,000
Jev 1.1340/40 (100%)0.194s0.688s$0.000646$0.0161
GPT Luna39/40 (97.5%)0.805s1.491s$0.001828$0.0457
Claude Opus40/40 (100%)2.099s4.377s$0.063555$1.5889

Jev separated the groups cleanly. Injections scored 0.86 to 0.99 and ordinary messages 0.01 to 0.20. Luna’s one miss was a role-play framing.

Use Jev when the answer is one of N

Anything that ends in a switch statement is a Jev question. Intent, priority, language, sentiment, which queue, does this violate policy, is this claim supported. In each case the answer is known in advance. You just want a probability for each of the possible answers. Here is the benchmark’s triage call using the OpenRouter TypeScript SDK.

// triage.ts
import { OpenRouter } from '@openrouter/sdk';

const openrouter = new OpenRouter({
  apiKey: process.env.OPENROUTER_API_KEY, // server-side only
});

const INTENTS = {
  order_status:
    'The customer asks where an order is, when it will ship or arrive, wants to change or cancel an order before delivery, or reports a package missing or partially delivered.',
  return_refund:
    'The customer wants to return, exchange, or replace an item they received, or asks about the status or rules of a return they already started.',
  billing_dispute:
    'The customer says a charge, invoice, tax, discount, or refund amount is wrong, duplicated, unexpected, or unauthorized.',
  product_question:
    "The customer asks about a product's features, compatibility, sizing, materials, stock, warranty, or safety before or after buying, without asking to return it.",
  account_access:
    'The customer cannot log in, needs to change login or account details, or asks to merge, delete, secure, or share an account.',
} as const;

export type Intent = keyof typeof INTENTS;

const ESCALATE =
  'Does the ticket describe any of the following: a threat of legal action, a regulator complaint, or a chargeback; suspected fraud or an account takeover; a safety hazard such as fire, smoke, or injury; or a customer who says this is a repeated failure and threatens to publicize it?';

export type Triage = {
  intent: Intent;
  confidence: number;
  probabilities: Record<string, number>;
  escalateProbability: number;
  costUsd: number;
};

function isIntent(value: string): value is Intent {
  return Object.hasOwn(INTENTS, value);
}

export async function triage(ticket: string): Promise<Triage> {
  const result = await openrouter.alpha.decisions.create({
    decisionsRequest: {
      model: 'typesafe/jev-1.13',
      state: { ticket },
      questions: {
        intent: {
          type: 'choice',
          instructions: 'What is the primary intent of the ticket?',
          criteria: INTENTS,
        },
        escalate: { type: 'noul', instructions: ESCALATE },
      },
    },
  });

  const intent = result.answers.intent;
  const escalate = result.answers.escalate;
  if (intent?.type !== 'choice' || escalate?.type !== 'noul') {
    throw new Error('Unexpected answer types');
  }
  if (!isIntent(intent.choice)) {
    throw new Error(`Unknown intent ${intent.choice}`);
  }
  return {
    intent: intent.choice,
    confidence: intent.confidence ?? 0,
    probabilities: intent.probabilities ?? {},
    escalateProbability: escalate.noul,
    costUsd: requireCost(result.usage.cost),
  };
}

function requireCost(cost: number | undefined): number {
  if (cost === undefined) {
    throw new Error('Response did not include usage.cost');
  }
  return cost;
}

Three things in that triage code to pay attention to:

Ask one small judgment per question and combine the answers in code. Here intent and escalation are separate questions in one request.

Write criteria like a spec. Since Jev is literal, when a case lands in the wrong bucket, fix the criteria text first.

Send only the state each question needs. Irrelevant detail lowers the accuracy of the answer.

Use an LLM when the answer is prose

Now when we need to actually generate a reply, we use an LLM.

// draft-reply.ts
import { OpenRouter } from '@openrouter/sdk';

const openrouter = new OpenRouter({ apiKey: process.env.OPENROUTER_API_KEY });

const REPLY_MODEL = '~openai/gpt-luna-latest';

const REPLY_SYSTEM =
  'You write short, warm replies for the Northwind support team. Use only the facts in the user message. Do not promise refunds, credits, or dates that are not in the facts. Three sentences maximum.';

type Draft = { text: string; model: string; costUsd: number };

export async function draftReply(ticket: string, facts: string): Promise<Draft> {
  const result = await openrouter.chat.send({
    chatRequest: {
      model: REPLY_MODEL,
      messages: [
        { role: 'system', content: REPLY_SYSTEM },
        { role: 'user', content: `Ticket: ${ticket}\n\nFacts: ${facts}` },
      ],
    },
  });
  // chat.send can return a stream when streaming is requested. It isn't here, so reject that case.
  if (result instanceof ReadableStream) {
    throw new Error('Expected a non-streaming response');
  }
  const text = result.choices[0]?.message.content;
  if (typeof text !== 'string') {
    throw new Error('Expected text content');
  }
  const cost = result.usage?.cost;
  if (typeof cost !== 'number') {
    throw new Error('Response did not include usage.cost');
  }
  return { text, model: result.model, costUsd: cost };
}

Pick the model by the writing task. When you need a short reply quick and cheap, GPT Luna delivers, at around $0.0001 per reply in this run. Upgrade to a frontier model when the writing needs real reasoning, long context, or code. An LLM is also the answer wherever you run into some of Jev’s limitations, like input with images, audio, or PDFs, or output that is a document, diff, or plan.

Route with Jev, compute in code, write with an LLM

Zoomed out, the first pattern is simple. Jev triages. The code checks how certain it was. Only when the deliverable is prose does the code call an LLM.

The handler sends anything with escalation at 0.5 or higher to a person. It also sends anything with intent confidence under 0.8. Otherwise, it answers order status from the order system without a model, and it asks the LLM for a draft only for intents that need prose.

// handle.ts
import { triage, type Intent, type Triage } from './triage';
import { draftReply } from './draft-reply';

// These thresholds are application policy. Tune them on your own labeled tickets.
const ROUTE_CONFIDENCE = 0.8;
const ESCALATE_THRESHOLD = 0.5;

type ReplyIntent = Exclude<Intent, 'order_status'>;

type Route =
  | { kind: 'human'; reason: string }
  | { kind: 'deterministic'; intent: 'order_status' }
  | { kind: 'reply'; intent: ReplyIntent };

function chooseRoute(t: Triage): Route {
  if (t.escalateProbability >= ESCALATE_THRESHOLD) {
    return { kind: 'human', reason: `escalation probability ${t.escalateProbability}` };
  }
  if (t.confidence < ROUTE_CONFIDENCE) {
    return { kind: 'human', reason: `intent confidence ${t.confidence}` };
  }
  if (t.intent === 'order_status') {
    return { kind: 'deterministic', intent: t.intent };
  }
  return { kind: 'reply', intent: t.intent };
}

// Stand-in for your order system. Exact data never goes through a model.
function lookupOrder(ticket: string): string {
  const id = ticket.match(/\b\d{5}\b/)?.[0];
  if (id === undefined) {
    return 'Please reply with your five-digit order number and we will check the shipment.';
  }
  return `Order ${id} shipped 2026-09-17 via UPS, tracking 1Z999AA10123456784, estimated delivery 2026-09-21.`;
}

export const FACTS: Record<ReplyIntent, string> = {
  return_refund:
    'Returns are accepted within 30 days of delivery for regular items. Exchanges for another size follow the same 30-day window. Final sale items cannot be returned or exchanged. Prepaid labels are emailed within one business day.',
  billing_dispute: 'A billing specialist will review the charge within one business day.',
  product_question: 'Product specifications are listed on each product page.',
  account_access: 'Password resets are available from the sign-in page.',
};

export type Handled = {
  triage: Triage;
  route: Route;
  text: string;
  costUsd: number; // Jev call plus the LLM call, if one happened
};

// Routes a ticket. A 'reply' route carries an unverified LLM draft; send.ts checks it before anything goes out.
export async function handle(ticket: string): Promise<Handled> {
  const t = await triage(ticket);
  const route = chooseRoute(t);
  switch (route.kind) {
    case 'human':
      return { triage: t, route, text: `queued for a person: ${route.reason}`, costUsd: t.costUsd };
    case 'deterministic':
      return { triage: t, route, text: lookupOrder(ticket), costUsd: t.costUsd };
    case 'reply': {
      const reply = await draftReply(ticket, FACTS[route.intent]);
      return { triage: t, route, text: reply.text, costUsd: t.costUsd + reply.costUsd };
    }
    default:
      return route satisfies never;
  }
}

A short runner prints the Jev signals next to each result.

// run.ts
import { handle } from './handle';

const TICKETS = [
  "Hi, I ordered a pair of running shoes on the 3rd and the tracking page hasn't updated in five days. Where is my package? Order 84721.",
  'The jacket I got is too small. Can I swap it for a large? I got it last Tuesday.',
  "There's an unauthorized $450 charge from your company on my card. I've already called my bank to dispute it and I'm reporting this as fraud.",
  "I paid with a gift card and a credit card, but the refund only went to the credit card. Where's the gift card balance?",
];

for (const ticket of TICKETS) {
  const r = await handle(ticket);
  const { intent, confidence, escalateProbability } = r.triage;
  console.log(`"${ticket}"`);
  console.log(`  intent=${intent} confidence=${confidence} escalate=${escalateProbability} route=${r.route.kind} cost=$${r.costUsd.toFixed(6)}`);
  console.log(`  ${r.text}\n`);
}

Here is the output for four sample tickets.

"Hi, I ordered a pair of running shoes on the 3rd and the tracking page hasn't updated in five days. Where is my package? Order 84721."
  intent=order_status confidence=1 escalate=0.02 route=deterministic cost=$0.000026
  Order 84721 shipped 2026-09-17 via UPS, tracking 1Z999AA10123456784, estimated delivery 2026-09-21.

"The jacket I got is too small. Can I swap it for a large? I got it last Tuesday."
  intent=return_refund confidence=1 escalate=0.01 route=reply cost=$0.000118
  Yes, you can exchange the jacket for a large if it’s a regular item and within 30 days of delivery. If it’s a final sale item, it can’t be exchanged; for eligible exchanges, a prepaid return label is emailed within one business day.

"There's an unauthorized $450 charge from your company on my card. I've already called my bank to dispute it and I'm reporting this as fraud."
  intent=billing_dispute confidence=1 escalate=0.96 route=human cost=$0.000025
  queued for a person: escalation probability 0.96

"I paid with a gift card and a credit card, but the refund only went to the credit card. Where's the gift card balance?"
  intent=return_refund confidence=0.54 escalate=0.02 route=human cost=$0.000025
  queued for a person: intent confidence 0.54

One of the four touched an LLM. The order-status lookup was exact and free. The fraud report never reached a model that could promise anything. Because the confidence for the split-refund ticket, the one Jev misfiled in Task A, landed under 0.8, it went to a person. The jacket reply, finally, is still an unchecked draft here. Checking it is the second pattern.

Verify LLM output with Jev before it ships

The second pattern runs the other way. The LLM drafts. Then Jev checks the draft against policy before it ships. This catches a friendly AI reply promising something against policy.

// verify-draft.ts
import { OpenRouter } from '@openrouter/sdk';

const openrouter = new OpenRouter({ apiKey: process.env.OPENROUTER_API_KEY });

export const POLICY =
  'Returns are accepted within 30 days of delivery for regular items. Final sale items cannot be returned. Refunds go back to the original payment method. Prepaid return labels are emailed within one business day of approval.';

type Label = 'supported' | 'unsupported' | 'declined';

export type Verdict = {
  label: Label;
  confidence: number;
  probabilities: Record<string, number>;
  costUsd: number;
};

function isLabel(value: string): value is Label {
  return value === 'supported' || value === 'unsupported' || value === 'declined';
}

export async function verifyDraft(policy: string, question: string, draft: string): Promise<Verdict> {
  const result = await openrouter.alpha.decisions.create({
    decisionsRequest: {
      model: 'typesafe/jev-1.13',
      state: { policy, customer_question: question, draft_reply: draft },
      questions: {
        support: {
          type: 'choice',
          instructions: 'How does draft_reply relate to policy and customer_question?',
          criteria: {
            supported:
              'The draft answers customer_question, and every fact, number, timeframe, and promise in it appears in policy.',
            unsupported:
              'The draft states a fact, number, timeframe, or promise that policy does not contain or contradicts, or it answers a different question.',
            declined:
              'The draft says policy does not cover the question and adds no facts of its own beyond what policy states.',
          },
        },
      },
    },
  });

  const support = result.answers.support;
  if (support?.type !== 'choice' || !isLabel(support.choice)) {
    throw new Error('Unexpected answer');
  }
  const cost = result.usage.cost;
  if (cost === undefined) {
    throw new Error('Response did not include usage.cost');
  }
  return {
    label: support.choice,
    confidence: support.confidence ?? 0,
    probabilities: support.probabilities ?? {},
    costUsd: cost,
  };
}

Jev gets the policy, the customer question, and the draft, and then it answers one choice question about whether that draft is supported, unsupported, or declined. We ran four drafts through it. The first was from GPT Luna, and the other three were handwritten, each written to hit a specific branch.

Q: "How long until my refund shows up?"
Draft (GPT Luna): "Refunds are sent back to the original payment method, but we don't have a specific timeline for when they will appear. If your return is approved, a prepaid return label will be emailed within one business day."
  supported  confidence=0.09  { supported: 0.39, unsupported: 0.32, declined: 0.29 }

Q: "How long until my refund shows up?"
Draft (fabricated): "Refunds are processed within 5 to 7 business days after we receive the item."
  unsupported  confidence=1.00  { unsupported: 1, supported: 0, declined: 0 }

Q: "Where does my refund go?"
Draft (grounded): "Refunds go back to the original payment method, so it will return to however you paid for the order."
  supported  confidence=1.00  { supported: 1, unsupported: 0, declined: 0 }

Q: "How long until my refund shows up?"
Draft (grounded, wrong question): "Refunds go back to the original payment method, so it will return to however you paid for the order."
  unsupported  confidence=0.48  { unsupported: 0.65, supported: 0.14, declined: 0.21 }

The fabricated timeline comes back unsupported at 1.00, and if the reply had gone out, the customer would have written back on day eight asking where the refund was.

Luna’s draft, now that’s the interesting one, since it states every fact true to the policy. But it half-declines, half-answers the question, and adds a detail nobody asked for. Jev splits the probability three ways, and the 0.8 threshold sends it to a rep for a quick look.

The last two rows use the same grounded draft but two different questions. Note that when the draft answers the question the customer asked, it is supported at 1.00. When it answers a different question, it is unsupported. This is literally what you want from a verifier, and it makes clear why the policy text should use the same words the drafts will.

We first tried two yes or no questions, and the results were muddled, since a draft that correctly says the policy does not cover this is grounded and also not an answer. Mutually exclusive outcomes motivate asking one choice question instead.

Wire it all together

One more file connects the route-first pipeline to the verifier, so nothing the LLM wrote goes out without a supported verdict.

// send.ts
import { FACTS, handle, type Handled } from './handle';
import { verifyDraft, type Verdict } from './verify-draft';

const SEND_CONFIDENCE = 0.8;

type Outcome = Handled & { disposition: 'sent' | 'human_review'; verdict?: Verdict };

export async function process(ticket: string): Promise<Outcome> {
  const handled = await handle(ticket);
  switch (handled.route.kind) {
    case 'human':
      return { ...handled, disposition: 'human_review' };
    case 'deterministic':
      return { ...handled, disposition: 'sent' };
    case 'reply': {
      // Verify against the same facts the draft was written from.
      const verdict = await verifyDraft(FACTS[handled.route.intent], ticket, handled.text);
      const ok = verdict.label === 'supported' && verdict.confidence >= SEND_CONFIDENCE;
      return { ...handled, verdict, costUsd: handled.costUsd + verdict.costUsd, disposition: ok ? 'sent' : 'human_review' };
    }
    default:
      return handled.route satisfies never;
  }
}

If the verifier bounces a reply that, looked at on its own, just looks fine, read the reply against the facts. The facts often miss something the customer asked about. Go look at the full request path.

ticket text
  |
  v
[Jev]  intent (choice) + escalate (noul) .......... 1 request, ~200 ms, ~$0.000025
  |
  v
[code] escalate >= 0.5 or confidence < 0.8 ?  -->  human queue
  |
  v
[code] intent == order_status ?  -->  database lookup, exact reply, no model
  |
  v
[LLM]  draft reply from ticket + facts ............ ~1 s, ~$0.0001 (GPT Luna)
  |
  v
[Jev]  supported / unsupported / declined ......... 1 request, ~200 ms, ~$0.000025
  |
  v
[code] supported and confidence >= 0.8 ?  -->  send
       otherwise                          -->  human review

The ticket that goes all the way through requires two Jev calls and one LLM call. The total cost is about $0.00015 and the total time is 1.5 seconds. The two Jev calls make it possible to use a cheap model for the writing while avoiding the need for a person to read every reply from the model.

Pick your thresholds from your own data

Wherever the threshold occurs in your code, keep in mind that the number doesn’t come from out of nowhere. It comes from you. It’s your policy choice. Jev’s probabilities are calibrated, which basically means take a lot of predictions and look at how often they’re right. So if we see 0.8, then about 80 percent of those predictions turn out right. But not every one of them is right. It’s an average. So treat the 0.8 and 0.5 you see in this post as our numbers, not yours.

If you want to find your own thresholds, a couple of hundred labeled cases should do. Run the model on all of them. For each cutoff, count how many it got right. Lay accuracy against cutoff on a graph. Then set your cutoff at the point where the automatic path matches what your human team would have returned. After you edit the criteria, re-check this. Don’t forget. Rewording one option can shift the probabilities on the others in somewhat surprising ways.

Start with one switch statement

If your codebase has an LLM call that ends in JSON.parse, and then a switch, that’s your first Jev question. Swap it in, keep the LLM for the branch that needs prose, and measure both.

FAQ

What is Jev and how is it different from an LLM?

Jev is TypeSafe’s System One decision model. It takes state plus a typed question, either a choice from a fixed set, a yes or no proposition, or a score on levels you describe, and returns probabilities instead of text. A large language model generates open-ended text that you then have to parse and trust.

Is Jev cheaper than a small LLM for classification?

On 60 support tickets run through OpenRouter on September 19, 2026, Jev cost $0.0248 per 1,000 tickets, GPT Luna cost $0.0921 per 1,000 tickets, and Claude Opus cost $2.88 per 1,000 tickets, while intent accuracy was 59/60 for Jev, 59/60 for GPT Luna, and 60/60 for Claude Opus. Jev 1.13 costs $0.042 per million input tokens with free output. Pricing can change at any time, so be sure to check the model page before you budget.

How do you use Jev and an LLM together?

Two patterns. First, route-first: Jev classifies the request and returns a confidence. Your code checks that confidence against a threshold you choose. Only requests that need prose go to the LLM. Second, verify-after: the LLM drafts first. Jev checks that every claim in the draft is backed by your policy. Only then does the reply go out.

Which OpenRouter endpoint does Jev use?

Jev uses the Decisions API endpoint, POST https://openrouter.ai/api/alpha/decisions with the model ID typesafe/jev-1.13, or openrouter.alpha.decisions.create() in the TypeScript SDK. Conventional LLMs use POST https://openrouter.ai/api/v1/chat/completions, or openrouter.chat.send().

What confidence threshold should I use with Jev?

There is no universal number that will apply to every case. Jev’s probabilities are calibrated across many predictions, so they hold up in aggregate. Any single prediction can still be wrong. So pick your cutoff from your own labeled data and the cost of a mistake. For example, here’s our routing rule: intent confidence below 0.8, or an escalation predicted at or above 0.5, gets sent to a person.

By subscribing you agree to receive the OpenRouter newsletter: model usage data, product updates, and research reports, about one email a week. Unsubscribe anytime via the link in every email. See our Privacy Policy.