Spec runner

Most people use tsforge interactively (tsforge in a terminal). Spec runner is for a different workflow: automated multi-step benchmarks where each step must pass a check before the next one starts.

You typically only need this page if you are running evals or regression tests against tsforge itself.

What a spec is

A spec is a markdown file with frontmatter and numbered tasks. Each task has an accept: shell command that must exit 0 before the runner moves on.

Full format: Spec format.

What the runner does

For each task in order:

1. Run the implement loop (model edits → gate → repair)
2. Stop when that task’s accept: command passes
3. Continue to the next task
4. After all tasks, run the whole-spec verify: command

Outputs include per-task status (done, stuck, interrupted) and a final pass/fail for the spec.

How to run (from a cloned tsforge repo)

bun run eval:spec

TSFORGE_SEED=math TSFORGE_FEATURE_VARIANTS=script \
  bun run eval:sweep

These scripts are for maintainers, not end users of the tsforge npm package. See A/B testing and Commands.

Spec modes

mode:	Use when
scratch	Greenfield: listed files are cleared so the model rebuilds them
existing	Feature work in a repo that already exists

→ Spec format · When the gate fails