Spec format

3 min read

Specs are markdown files that break work into ordered steps. Each step has an accept: command that must pass before the next step runs.

Normal users drive tsforge interactively with tsforge. Specs are for benchmark and regression scripts (bun run eval:spec, bun run eval:sweep). See Spec runner.

File shape

---
id: hello
title: Hello sum
verify: bun test
mode: scratch
---

## Acceptance criteria

A1. sum adds two numbers correctly.

## Tasks

1. [logic] implement sum
     accept: bun test sum.test.ts
     files: sum.ts
     context: sum.test.ts
     fix: bun eslint --fix sum.ts

Frontmatter

Key	Required	Meaning
`id`	yes	stable spec identifier
`title`	yes	human title
`verify`	yes	whole-spec gate after all tasks green
`mode`	no	`scratch` (default) or `existing`

Modes

Mode	Behavior
`scratch`	harness deletes editable `files` to start RED; model regenerates
`existing`	project kept; RED from failing tests or stated goal; edits in place

Sections

Acceptance criteria

Prose under ## Acceptance criteria is attached to every task as intent. The model works from stated goals instead of reverse-engineering from test calls alone.

Tasks

Numbered list under ## Tasks. Each task supports indented fields:

Field	Meaning
`accept:`	shell command that must exit 0 to close the chunk
`files:`	editable scope (comma-separated globs); edits outside are rejected
`context:`	read-only files shown for context (e.g. tests)
`fix:`	optional auto-fix before re-validation (e.g. `eslint --fix`)

Task numbers become stable ids (1, 2, …).

Running specs

Specs are consumed by the loop’s runSpec(): used in eval scripts and benchmarks.

# Example: eval script against a seed spec
bun run eval:spec

# Multi-seed sweep (benchmark harness)
TSFORGE_SEED=math bun run eval:sweep

Benchmark seeds ship as <seed>.spec.md under evals/corpus/ (e.g. evals/corpus/math/, evals/corpus/auth/); the harness copies a seed into a fresh run directory under evals/ for each run. See Spec runner and A/B testing.

Script	Role
`gen-tests.ts`	generate test suite from an untested spec
`eval-spec.ts`	run one spec against the model
`sweep.ts`	cartesian feature-variant sweeps

→ Spec runner · When the gate fails