Jev · direct selection
0.94sOne question chooses a number directly from all four clues. Five candidates below are ranked by probability; ties use numeric order, preferring Jev’s returned choice.
TOP FIVE · RANKED
25 ×20 ×26 ✓29 ×85 ×
EXPERIMENT 001 / MODEL BEHAVIOR
Tested · follow-up plannedA better question for Jev.
The model stayed the same. We changed the job.
Try it in the Jev Playground ↗THE FINDING
Let Jev read the clues. Let code do the math.
On the same 20 fresh challenges, Jev’s first-selection validity rose from 30% to 75% when it decoded each clue separately and code combined the constraints. Median response time stayed close to 0.8 seconds.
A small hint-solving test, not a full-game win rate. Abstentions count as unsuccessful.
THE QUESTION
Was the model the problem—or our request?
Our first pilot asked Jev to interpret four cryptic clues, combine their mathematical constraints, and choose among 100 numbers in one decision. It was fast, but its first guess fit every clue on only 5 of 20 challenges.
TypeSafe recommends small, independent decisions that software can combine. We tested that approach.
Read TypeSafe’s design guidance ↗WHAT CHANGED
- Interpret each clueFour independent questions select mathematical properties from the same 83-option vocabulary, plus “unknown.”
- Calculate in codeApply those properties to 0–99 and intersect the surviving sets. An empty intersection means abstain.
- Compare on fresh casesDirect Jev, direct Claude, Jev + code, and Claude + code all receive the same 20 new challenge cases.
ROUND 2 / HELD-OUT COMPARISON
Better structure. Better results.
| Approach | Valid first selection | Exact candidate set | Median time | Abstentions |
|---|---|---|---|---|
| Jev · direct selection | 30%6/20 | Not measured | 0.81s | — |
| Jev + constraint code | 75%15/20 | 75% | 0.82s | 3 |
| Claude · direct selection | 100%20/20 | Not measured | 17.34s | — |
| Claude + constraint code | 90%18/20 | 90% | 7.28s | 1 |
“Valid” means the selected number fits all four clues. The hidden number can be one of several valid candidates. For unordered candidate sets, first selection uses the lowest number; tie-aware metrics are below.
Claude direct reached 100% valid first selections, versus 90% with constraint code, on these same cases. The improvement for Jev does not imply that every model benefits from the same workflow.
INSPECT THE EVIDENCE / ROUND 2
Follow one challenge through all four approaches.
- 01The first numeral of its two-figure inscription is even.
- 02No earlier than pillar twenty does the colonnade reveal it.
- 03The last numeral on its plaque is large, five or greater.
- 04Six masons divide the pile and two blocks remain at the well.
Jev + constraint code
1.07s- ✓Clue 1: tens digit is even (0,2,4,6,8)
- ✓Clue 2: at least 20
- ✓Clue 3: units digit is 5-9
- ✓Clue 4: leaves remainder 2 when divided by 6
SURVIVING CANDIDATES · UNORDERED
26 ✓68 ✓86 ✓
Claude · direct selection
14.93sClaude returns five candidate numbers in its chosen order, directly from all four clues.
TOP FIVE · RANKED
26 ✓68 ✓86 ✓28 ×46 ×
Claude + constraint code
7.65s- ✓Clue 1: tens digit is even (0,2,4,6,8)
- ✓Clue 2: at least 20
- ✓Clue 3: units digit is 5-9
- ✓Clue 4: leaves remainder 2 when divided by 6
SURVIVING CANDIDATES · UNORDERED
26 ✓68 ✓86 ✓
Hidden number: 86All valid candidates: 26 · 68 · 86
KEEPING THE HISTORY
The original result still matters.
Round 1 used a different set of 20 challenges. Claude also used direct number selection then. Its scores should not be treated as a controlled before-and-after comparison with Round 2. The 30% → 75% improvement above compares Jev approaches within Round 2 only.
Round 1: original pilot and challenge replays +
WHAT WE LEARNED
A strong improvement, with limits.
Jev decoded 75 of 80 individual clues correctly. Claude decoded 78 of 80. Three Jev errors produced conflicting constraints that code caught; two produced plausible but incorrect candidate sets.
One wrong interpretation had 0.85 confidence. Confidence alone won’t catch every mistake.
The revised Jev calls used about 3.5× as many reported tokens as direct Jev. Similar latency does not mean similar cost; dollar costs were not measured.
NEXT TEST / PLANNED
Keep alternative interpretations alive.
Try retaining more than one property interpretation for ambiguous clues, and test when a fallback model helps. Choose that policy on development cases, then evaluate it on another untouched sample.
This follow-up has not been run. No improvement is claimed yet.
Method, controls, and limitations +
Two original cases checked the new implementation. Prompts were then frozen. The 20 evaluation cases have hidden numbers excluded from the original pilot, but use existing-bank phrasings that can recur. This is not a test of entirely new language or a claim about model training data.
Both property approaches received identical property descriptions. Jev answered four Choice questions in one API call; Claude returned four IDs. Both used the same code to intersect candidate sets. The answer registry, hidden number, and precomputed solutions were withheld from the models.
Models: jev-1.13.0 and claude-sonnet-4-6. Claude had a 2,000-token output limit without explicitly enabled extended thinking. One request per approach per case. The initial three approaches ran sequentially with rotated order. Claude direct was added in a later batch on the exact same cases, using the original direct-selection prompt and no property vocabulary. All 80 requests succeeded. No retries, confidence thresholds, or fallback model were used.
To account for arbitrary ordering of equally ranked candidates, expected valid-first rates under uniform tie-breaking are 35% for direct Jev, 75% for Jev + code, and 91.67% for Claude + code. These are calculated expectations, not observed game wins. Direct selection had ties at the top-five cutoff in 12/20 original pilot cases.
Latency includes network time. Claude direct ran later, so provider conditions may differ from the other approaches. Samples are small, with no repeated-run uncertainty estimates. We did not test full rounds, gradual hint unlocks, or live gameplay. The results describe these implementations and conditions.