Model trials
Fable 5.2 vs Opus 5.5 (plan: game/trials/fable-editbay.md)
| arm | model | feature | start | end | wall time | tool calls | tokens | compile attempts | checks on first hand-in | quota before | quota after |
|---|---|---|---|---|---|---|---|---|---|---|---|
| A | Fable 5.2 | edit bay | Thu 16:10 | 16:23 | 12.6 min | 34 | 206,622 | 2 (C4456 shadowed local) | build pass, 12/12 tests, bot capture done; writing check skipped (tells.py path wrong in brief); one follow-up commit for dash characters | all 0, Fable 0 | all 1, Fable 1 (a Sonnet worker ran alongside) |
| B | Opus 5.5 | call-sheet menu | Thu 16:27 | 16:39 | 11.6 min | 50 | 193,299 | 1 (clean first time) | build pass, 16/16 tests, bot capture done, tells.py clean | all 1, Fable 1 | all 1, Fable 1 |
Note: arm A ran on a brief with a wrong tells.py path (director error), so its writing check could not run; arm B got the fixed path. Both diffs are now with a blind Opus reviewer.
Blind review (fresh Opus, Thu 16:45), then unblinded
- X (edit bay) = Fable 5.2: correctness 7, fit 8, code 8, tests 8, text 7. Best data layer and the most useful test (shipped JSON checked against the cue tables). Defects: a soft lock after printing the cut, and a ranking rule that lets a wreck outrank a flub, against its own comment.
- Y (call-sheet menu) = Opus 5.5: correctness 7, fit 9, code 7, tests 9, text 7. Finished nearly every point; its defects are edge cases, not dead ends. Also fixed X's dead end with a NEXT button.
- Reviewer verdict: Y slightly ahead relative to brief difficulty, about 60% confident. X had the harder design job.
Result
On this jam-sized feature work, Fable 5.2 was not better than Opus 5.5: same wall time (12.6 vs 11.6 min), about the same tokens (207k vs 193k), one more compile round, and a slightly lower blind score, at twice the price per token. Fable used fewer tool calls (34 vs 50) and wrote the stronger data model. Decision: Opus stays the default for builders; Fable is held for long, hard, many-hour tasks where its claimed strength (long-horizon autonomy) would actually show. Sample size is one feature each, so this is a signal, not a law.
Fable 5.2 at low effort on a long, hard ticket (the brawl rebuild, Fri Nov 13)
| model | ticket | start | end | wall time | tool calls | tokens | result | quota before | quota after |
|---|---|---|---|---|---|---|---|---|---|
| Fable 5.2, effort low | brawl rebuild: branches, heavy's moves, keyed chair and rope, props, seams (6def097) | 03:44 | 05:58 | 2 h 14 min | 188 | 678,871 | build, 36/36 tests, packaged loop clean; props, rope kick and 4 of 6 branches right by the numbers; fist contacts, the bandstand and most main joins still fail. Found two bugs in the take builder that affected every take (crossfades written in frames instead of ticks, so every crossfade was a hard switch; "hold" looped its clip) | all 10, Fable 1 | all 14, Fable 7 |
Comparison, Opus tickets of similar size tonight: the sets build (2 h 11 min, 492k tokens), the longer takes (about 1 h 25 min, 452k), look pass 2 (28 min, 260k). Reading: low effort did not make Fable cheap: 679k tokens, the most of any ticket tonight, and the Fable meter moved 6 points (the all-models meter about 4) in 2 h 14 min. The work was deep and honest: it re-keyed the Blender clips at the right body scales, built an in-game joint log, and caught two builder bugs no Opus ticket had found, which is the long-horizon strength Fable is sold on. It also spent its biggest time on two dead ends (the bandstand and fist contacts) it says a planning step would have avoided. Verdict: Fable earns its price on hard debugging and deep pipeline work, not on routine building; Opus stays the builder, and a planner step goes before any long engine ticket whatever the model.