Seb-9B-v2 is a 9B decision model that scores 59.16 on the Decision Index, ahead of hosted Jev 1.13 (57.96). It runs in about 22 ms a decision on one GPU or 88 ms on a Mac, and your data never leaves your machine. On public decision rows it edges out Claude Haiku and gpt-6-luna (0.798 vs 0.795 and 0.779, within noise). Put in front of Claude Opus as a first tier, it answered about half the requests itself and kept Opus-level accuracy overall, at an estimated 51% lower cost.
You give it a situation, a question and a few options, and it returns a probability for every option in one forward pass. No text is generated, so there is nothing to parse. It reads images, takes inputs up to 32k tokens, and speaks both a regular chat API and the System One API.
TL;DR
- Quality. 0.801 mean accuracy over 17 public suites (our v1 0.793, hosted Jev 1.13 0.786). BANKING77 0.973, jailbreak detection 0.940, model routing 0.980. On 420 public decision rows it edges out gpt-6-luna and Claude Haiku (medium effort), within noise, and trails Claude Opus.
- Speed. About 22 ms per decision on one RTX PRO 6000 and about 88 ms on an Apple M5 Max, up to roughly 87 decisions a second per GPU.
- Decision Index. 59.16 on the 0.3 public index, against 57.96 for hosted Jev 1.13 and 60.70 for Jev 1.14. Ahead of Jev 1.13 on 22 of 37 benchmarks and in 4 of 5 areas.
Built for the edge
A decision model earns its keep where the data already lives: on a laptop, a gateway, a robot controller, or a server next to the database.
- Local. The weights are about 19 GB in bf16. On an Apple M5 Max, a decision takes about 88 ms (96 ms with 8-bit weights). On one RTX PRO 6000, it takes about 22 ms.
- Private. Nothing leaves your machine, and there is no per-call bill.
- Probabilities, not prose. Every option comes back with a probability so you can set a clear threshold: act when the model is confident, and escalate to a person or a larger model when it is not.
- Images too. Input states can include an image. Held-out image decisions scored 0.886 on 2,451 rows.
- Two ways to call it. You can use a regular chat API (vLLM, with structured choice and logprobs) or the System One API (
POST /v1/systemone: one state, several typed questions, one call). The lightweight server for the second option ships directly in the model repo.
Decisions at the edge: a small model for the calls an application makes all day
on an Apple M5 Max, no API call; ≈ 22 ms on one RTX PRO 6000. Data stays where it is.
answered by Seb at Opus-level accuracy (0.855), the rest escalates: 51% lower cost.
jailbreak detection; phishing 0.895. Probabilities you can threshold.
model routing; BANKING77 intents 0.973, support triage, intent checks.
simulated arm decisions matched the rules; warehouse robot delivered 4 of 4 episodes.
ContractNLI vs hosted Jev 1.13; image decisions 0.886 on 2,451 held-out rows.
Using Seb as the first tier of an LLM stack
The most practical job we found is serving as the front door. Seb reviews every request first. When it is confident, it answers on the spot. When it is uncertain, the request routes to a larger model.
To test this without inventing a synthetic benchmark, we took 420 public decision rows (20 from each of 21 public suites) and collected real, blind answers from four tiers of Claude models: Haiku (medium effort) as low, Sonnet (low effort) as medium, Sonnet as high, and Opus as max. Then we let Seb’s own confidence score guide the routing.
- With a confidence threshold of 0.98, Seb answers 51% of the rows itself, and the combined accuracy is 0.855 (identical to sending every request to Opus) at an estimated 51% lower cost.
- With a threshold of 0.90, it answers 70% locally, combined accuracy is 0.845 (one point lower), and estimated cost is 70% lower.
Answer locally when confident, escalate the rest: Opus-level accuracy at 51% lower cost
A few honest caveats. Cost here is based on a relative list-price ratio (Haiku 1, Sonnet 3, Opus 5), and we did not measure raw token counts. Seb’s local calls are treated as free, which excludes hardware hosting costs. This benchmark also consists of public, mostly straightforward decision rows: 334 of the 420 were solved by the cheapest tier alone. We also asked Seb to pick a tier directly zero-shot, and the gain was small (0.807 at relative cost 1.7). The model has not been trained specifically for tier routing yet, and training a dedicated router on routing data remains the next logical step.
Practical use cases
- Guardrails and triage. Jailbreak detection 0.940, phishing 0.895, model routing 0.980, and BANKING77 banking intents 0.973. Outputs provide probabilities you can threshold directly.
- Documents. On ContractNLI contract questions, it scores 78.0 on the Decision Index, compared to 59.1 for hosted Jev 1.13.
- Robotics. Seb acts as the high-level policy: deciding which object to pick next, where it should go, when to stop because a person walked in, and when to resume. In simulation, 421 of 428 arm decisions matched the written rules, and a warehouse robot delivered all packages in 4 of 4 episodes. Inverse kinematics and a path planner handle physical movement, while the grasp itself is simulated.
A harder scene: five objects, four bins, an unlabeled item that must go to quarantine, slow carrying for fragile and hazardous items, and two people events. Seb got 221 of 228 decisions right across 12 episodes. The misses were all in pick order.
Detailed numbers
Seb-9B-v2 vs hosted Jev and open decision models
| Suite / metric | Seb-9B-v2 | Seb-9B (v1) | Jev 1.13 | Jev 1.14 | Kev 27B | Nimble 9B v3 | Laya |
|---|---|---|---|---|---|---|---|
| Decision Index 0.3, public index37 benchmarks, chance-corrected | 59.16 | — | 57.96 | 60.70 | 56.69 | 57.19 | — |
| Knowledge & reasoning | 39.9 | — | 53.9 | 57.1 | 45.7 | 41.0 | — |
| Language understanding | 62.2 | — | 59.2 | 61.2 | 62.6 | 60.7 | — |
| Retrieval & classification | 63.5 | — | 55.4 | 63.5 | 55.8 | 55.5 | — |
| Tools & automation | 83.1 | — | 75.1 | 73.8 | 72.7 | 84.2 | — |
| Arts & human taste | 48.9 | — | 39.1 | 39.2 | 42.7 | 44.0 | — |
| Mean accuracy, 17 public suitesLaya benchmark suites | 0.801 | 0.792 | 0.786 | — | — | — | 0.694 |
| Calibration error (ECE)lower is better | 0.110 | 0.088 | 0.124 | — | — | — | 0.170 |
| Typed decisions, sealed suite12 suites, pooled | 93.1 | 94.6 | 94.3 | — | — | — | — |
| Trading decisions suite | 0.749 | 0.732 | 0.707 | — | — | — | — |
420 public decision rows across 21 suites, each model answering blind
| Model | Accuracy | Correct | 95% interval | vs Seb-9B-v2 |
|---|---|---|---|---|
| Claude Opus | 359 / 420 | 0.82 – 0.89 | aheadp < 0.001 | |
| Claude Sonnet (low effort) | 343 / 420 | 0.78 – 0.85 | ahead, within noisep = 0.26 | |
| Seb-9B-v2Ours | 335 / 420 | 0.76 – 0.83 | local · 9B open weights≈ 22 ms GPU · 88 ms Mac | |
| Claude Haiku (medium effort) | 334 / 420 | 0.75 – 0.83 | levelp = 1.00 | |
| gpt-6-luna (high) | 327 / 420 | 0.74 – 0.82 | levelp = 0.29 |
The general models are much larger, need a hosted call per decision, and do not return native probabilities. On everyday public decision rows Claude Opus is clearly ahead of Seb-9B-v2 and Claude Sonnet is slightly ahead (within noise). Claude Haiku (medium effort) and gpt-6-luna are level with it.
Where it falls short
- Knowledge and reasoning. Jev is clearly stronger here (GPQA Diamond, MMLU-Pro, and BBH show our largest gaps).
- Calibration. Our v1 model is still better calibrated (ECE 0.088 vs 0.110).
- Public leaderboard. The Decision Index board has moved forward, and newer open entries now score higher on the public index than Seb-9B-v2 (for example, OneJev1.6-9B at 65.3).
- Not a game engine. This is a decision model, not a game engine. It beats a random-move chess bot 7 games out of 10 (the rest end in stalemates) and loses to Stockfish.
- Simulated robotics. The robot evaluations are simulations, and options are provided with the structured facts that an upstream perception stack would supply.
Try it
The model card has full setup details: ironbcc/Seb-9B-v2.
vllm serve ironbcc/Seb-9B-v2 --served-model-name seb \
--logprobs-mode processed_logprobs --max-logprobs 32 --generation-config vllm \
--max-model-len 32768 --dtype bfloat16 --max-num-seqs 128 --enable-prefix-caching
python serve/system1_vllm_backend.py --vllm-url http://127.0.0.1:8000 --served-model seb \
--tokenizer ./seb-9b-v2 --port 8020 --model-name seb-9b-v2Seb-9B-v2 takes inputs of up to 32,768 tokens (set SYSTEM1_MAX_CONTEXT_TOKENS=32768 for the System One server; the default is 1,024). It is released under the Apache-2.0 license, matching the base model. It is intended as a decision aid, not professional advice.