A/B testing

Compare feature settings across benchmark runs before changing a default. The sweep harness A/Bs any tool-availability dimension: whether the model is offered a given tool. Toggle the env var behind it per run. See Big picture for what each feature does.

Sweepable dimensions

Dimension	On →	Off →
git	git_context available	TSFORGE_NO_GIT_TOOL=1
script	script tool available	TSFORGE_NO_SCRIPT=1
web	web research tools available (TSFORGE_WEB=1)	off

Core uplifts (TTSR, hashline, write diagnostics) are always on and no longer sweepable. They landed as defaults from earlier sweeps. Full flag reference: Environment variables.

Note

Running a sweep drives a real model, so you need an OpenAI-compatible endpoint (the default is a local model at http://localhost:8000/v1; override with TSFORGE_BASE_URL/TSFORGE_MODEL/TSFORGE_API_KEY). The corpus, analysis, and report tooling below ship with the repo and are exercised by the test suite, but the runs themselves need a model.

Seeds

A seed is the task you run the model against. Committed seeds live in evals/corpus/ and ship with the repo, so the examples work on a fresh clone:

• math: single-file pure logic (currency-safe cent arithmetic with half-up rounding).
• slugify: single-file string normalization with Unicode edge cases.
• checkout: multi-file pricing engine (cart + coupons + tax, integer cents).
• auth: multi-file session service (password policy + lockout + token expiry).
• query: multi-file mini-SQL engine (lexer → recursive-descent parser → executor).
• fix-regression (brownfield): an existing repo with git history where a recent commit broke slugify; the fix is a git diff away. Used to exercise git_context.

The first two are warm-ups a strong model one-shots; the next three are multi-file and deliberately harder, so a feature has room to move the pass rate (and the failure breakdown) off the ceiling.

Pick one with TSFORGE_SEED=<name>. A sweep first looks for a local working copy at evals/<name>/, then falls back to the committed evals/corpus/<name>/. To add your own, drop a folder under evals/corpus/ with a <name>.spec.md plus its fixture files. See Spec format.

Brownfield seeds

A greenfield seed is regenerated from scratch (the sweep deletes the task’s files and the model rebuilds them). A brownfield seed instead sets mode: existing in its spec, so the buggy code is kept and the model edits it in place. Such a seed may also ship a setup.sh, which the sweep runs in the run directory after laying down the files. It git inits, makes the commits, and leaves the red working tree. That is how a git_context task has real history (log/blame/show) and a working-tree diff to inspect. Greenfield seeds have no setup.sh and are unaffected.

Running a sweep

bun run eval:sweep accepts TSFORGE_FEATURE_VARIANTS: a comma-separated list of dimensions to sweep (cartesian product).

script on/off

TSFORGE_SEED=checkout \
TSFORGE_TEMPS=0 \
TSFORGE_REPEATS=2 \
TSFORGE_FEATURE_VARIANTS=script \
bun run eval:sweep

Creates four runs: checkout-script=on-t0-... and checkout-script=off-t0-... (two repeats each).

git × script

TSFORGE_SEED=fix-regression \
TSFORGE_TEMPS=0.5 \
TSFORGE_REPEATS=3 \
TSFORGE_FEATURE_VARIANTS=git,script \
bun run eval:sweep

Runs 3 repeats × 2 temps × 4 variants = 24 runs with IDs like fix-regression-git=on,script=off-t0.5-....

git_context on/off

TSFORGE_SEED=fix-regression \
TSFORGE_TEMPS=0 \
TSFORGE_REPEATS=2 \
TSFORGE_FEATURE_VARIANTS=git \
bun run eval:sweep

The git dimension toggles git_context (on → available; off → TSFORGE_NO_GIT_TOOL=1). Pair it with a brownfield seed so there is history to inspect.

Each run directory contains run.log (human transcript) and result.json (structured metrics + feature flags). A sweep also writes evals/runs/sweep-<seed>-<ts>.json with every run record.

Statistical report

bun run eval:report turns a sweep’s JSON into a Markdown table: per-variant pass rate with a 95% Wilson confidence interval, average cycles/ms/quality, turns-to-green (T2G) (the average turns passed runs took to go green, the loop-efficiency signal), and when you name a baseline, a two-proportion z-test of each variant’s pass-rate difference (a * marks p < 0.05).

# newest sweep under evals/runs, comparing every variant to the all-off baseline
TSFORGE_BASELINE="git=off,script=off temp=0" bun run eval:report

# or point at a specific sweep file
bun run eval:report evals/runs/sweep-math-20260613-120000.json

It prints the table and writes it next to the sweep JSON as …​.report.md:

| Variant | Runs | Pass | 95% CI | Cycles | Ms | Quality | vs baseline |
| --- | --- | --- | --- | --- | --- | --- | --- |
| git=off,script=off temp=0 | 10 | 60% | 31%–83% | 6.1 | 41000 | 3.8 | baseline |
| git=on,script=on temp=0   | 10 | 90% | 60%–98% | 4.7 | 33000 | 4.2 | +30% (z=2.13) * |

Wilson intervals (not naive ±) keep the bounds sane at small N, and the z-test tells you whether a pass-rate gap is signal or noise. The bar for “measured wins” before flipping a default is determined by statistical significance.

Why runs failed

Pass rate tells you how often a variant failed; the failure breakdown tells you why. Below the table, the report tallies each failed run by a structured failure class:

### Failure breakdown
- git=off,script=off temp=0: type-error×3, no-progress×1
- git=on,script=on temp=0: type-error×1

Each failed run is classified from its event stream into one of: type-error, lint-rule, hallucinated-import, tool-malformed, edit-reject, degeneration, no-progress, build-fail, browser-fail, route-phantom, or timeout. This turns a sweep from “feature X passes more” into “feature X eliminates the type-error failures.” It points at the next rule, prompt, or fixer to build. The same classifier powers the failure class line in cli-metrics for a single --log run.

Compare edit mechanisms

bun run eval:benchmark reports edit-tool performance across a set of run directories. It helps spot how edit vs edit_lines behave, stale-anchor recovery rates, and token cost across models or seeds:

bun run eval:benchmark evals/checkout-*

Output table compares runs on:

Metric	Meaning
edit / edit_lines calls	tool usage and success rate
Stale-anchor recovery	hashline 3-way merge attempts
Mean tool-args bytes	token-cost proxy
Gate failures	red turns before green
Turns to green	loop efficiency
Pass rate + quality	overall outcome

JSON export

bun run eval:benchmark \
  --json evals/comparison.json \
  evals/checkout-*

Run artifacts

Each run directory contains:

• run.log: live event transcript (renderEvent() pipeline; source of truth for analysis)
• result.json: structured metrics, for example:

{
  "seed": "checkout",
  "runId": "checkout-script=on-t0-20260612-120000-1",
  "temperature": 0,
  "features": { "TSFORGE_NO_SCRIPT": "0" },
  "status": "done",
  "cycles": 5,
  "ms": 42000,
  "quality": 4,
  "tasks": [{ "cycles": 5, "edits": 3, "regressions": 0 }]
}

Events in run.log

bun run eval:benchmark parses these patterns:

Pattern	Meaning
✎ edit	standard edit tool
edit_lines	hashline edit tool
edit_lines ... REJECTED	stale-anchor or parse failure
edit ... REJECTED	out-of-scope or size rejection
snapshot merge	stale-anchor recovery (3-way merge)
turn N: red (K errors)	gate failure
· turn N: GREEN	gate passed
turn N: asking model	turn count

How to read results

Edit success: Higher edit_lines success rate (vs edit rejections) means the hashline mechanism is reducing stale-anchor failures.

Stale recovery: Non-zero recovery counts show the 3-way merge is active; correlate with pass rate.

Turns to green: Lower on a variant means less loop churn.

Token efficiency: Smaller mean args (bytes) at similar success rate is better.

Pass rate + quality: Confirm enabling a feature does not regress outcomes beyond noise.

Landing a new feature

1. Ship the flag with a sensible default.
2. Baseline sweep (small: 2 repeats, temp=0):

TSFORGE_SEED=math TSFORGE_TEMPS=0 TSFORGE_REPEATS=2 \
bun run eval:sweep

3. Disable the feature, same seed/temp/repeats:

TSFORGE_SEED=math TSFORGE_TEMPS=0 TSFORGE_REPEATS=2 \
TSFORGE_<FEATURE>=0 bun run eval:sweep

4. Compare with bun run eval:benchmark or analyze-runs.ts.
5. Document findings in the PR (pass rate, edit success, token delta).
6. Set the default from data and merge.

Debugging incomplete metrics

grep "edit" run.log              # edit tool calls
grep "REJECTED" run.log          # rejections
grep "tool_input_rejected" run.log
grep "repair:" run.log           # L0–L3 repairs
grep "ttsr" run.log              # TTSR interrupts

→ Commands · Repair ladder