Test baselines
What already fails on the base branch, so agents fix only the failures they caused, and criteria blocked by the base are re-checked when it moves.
A repository's base branch is not always green. When main already has a
failing test or a type error, an agent that is told "all tests pass" cannot
tell its own failures from the ones that were there before it started.
Yard keeps a test baseline for each repository: the failing tests and
type errors on the workspace's branch, at one commit. yard tests compare
runs a project's checkout and sorts its failures into new, also fail on
main and fixed. Only the new ones are the agent's to fix.
Setting up a repository's tests
Yard does not guess how to run a repository's tests. Each repository gets a test config, kept in Yard (like app config), not in the repository: the commands, the format of each one's output, and an optional setup that runs first.
{
"setup": "bun install --frozen-lockfile",
"commands": [
{ "name": "unit", "run": "bun run test", "format": "bun-test", "timeoutMin": 20 },
{ "name": "typecheck", "run": "bun run typecheck", "format": "tsc", "timeoutMin": 10 },
{ "name": "e2e", "run": "bunx playwright test --reporter=json", "format": "playwright", "timeoutMin": 30 }
]
}
Set it from a terminal or an agent with
yard tests config set <repo> <file.json>, or through the API
(PUT /v1/workspaces/{ws}/repos/{id}/tests/config). Four output formats are
read:
| Format | What is read |
|---|---|
bun-test |
bun test output: each (fail) line under its file, and a file that fails to load as <file> (load error). bun --filter output is read package by package. |
playwright |
The JSON reporter (--reporter=json), or the list reporter's lines. A test that passed on a retry counts as passing. |
tsc |
tsc --noEmit output, plain or --pretty. An error is matched by its code and message, not its line, so a shifted line is the same error. |
junit |
A JUnit XML report on stdout, as vitest run --reporter=junit writes it: each test case with a failure or an error, by its file and title. A file that fails to load reads as <file> (load error). |
A repository without a test config has no baseline, and yard tests says
how to add one.
The baseline
A baseline is taken at the head of the workspace's branch, in a scratch
checkout of the workspace's copy of the repository that is removed
afterwards; node_modules is kept between runs so the setup is quick the
next time. It is stored in Yard with the commit and the time it was taken.
- When it is taken. When the branch moves (a push on GitHub, or a fetch),
with "Refresh test baseline" in the repository block's … menu, or with
yard tests baseline <repo>. One commit is run once, however many of these arrive together, and one runner runs one suite at a time. - Stale. Once the branch has moved past it, a baseline is stale.
yard tests comparetakes a fresh one first and says so. - On the block. The repository's block on a canvas shows the baseline's commit, age and how many failures it has.
A command whose output cannot be read gives no baseline, never "nothing fails": a run killed at the memory limit ("killed: out of memory (limit N MB)"), a timeout, or output with no summary (a run cut short, a run with no tests).
Comparing a project's checkout
In a project, yard tests compare <repo> runs the config's commands in the
project's checkout, as it is (uncommitted changes too), and prints:
atlas · base main@a1b2c3d (baseline 2h old, 14 failing)
New failures (1): e2e/notes.test.ts › notes › keeps a title
Also fail on main (14): …
Fixed vs. main (0)
It exits non-zero when there are new failures or a command gave no result,
and names each such command with the reason. --cached prints the last run
again without running anything; --json prints the whole comparison.
Failures of other commands
A baseline covers the commands of the test config. An agent often runs
others: one journey, one package's unit tests, a script such as
bun run check:overlays, or tests in a repository that has no config.
yard tests base <repo> checks those against the base branch:
yard run -- bun test ./src # in apps/runner, fails with 14
yard tests base ai-workspace # the same command, on main
ai-workspace · ran `bun test ./src` in apps/runner on main@a1b2c3d (here: 14 failed, 3 min ago)
New failures (0)
Also fail on main (14): src/box/docker.test.ts › docker › copies the tools
…
Yard takes the agent's last failing yard run in that repository (or,
with -- <command>, the last run of that command), runs it again at the
head of the workspace's branch in a scratch checkout, in the directory it
ran in, and reads both runs the same way, with the readers that count any
test run. The failures that fail there too were there
before the change; the others are new.
- Setup. The base checkout needs its dependencies: the test config's
setup is used, or
--setup "<install command>"when there is no config (--setup truewhen nothing needs installing). Without either, Yard refuses rather than read a checkout without dependencies as "everything is new". - No result. A run on the base that could not be read (the setup failed, it was killed (for memory or otherwise) or timed out, no test tool was recognised, or more failures than Yard names one by one) gives no result, never "nothing fails there".
- Kept with the run. The result stays with the run in the project's
test runs: the chat's row says "All 14 fail on main@a1b2c3d too" or "2 new,
12 also fail on main", marks each failing test that fails on main too, and
a failed run nobody checked reads "Not checked on the base branch". The
agent's next turn reads the same, and agents are told to call a failure
pre-existing only when
yard tests compareoryard tests basesaid so. - In the pull request. The project's pull request in that repository
has a Failing test runs section: for each command the agent ran
there, its last run when it failed, with the base result ("1 new, 1 also
fails on main@a1b2c3d") and the failing tests split into new and also on
main, or "Not checked on the base branch". Yard writes it when it opens the pull request and rewrites it in place when a run ends or a check on the base comes back; a later run of the command that passes takes it out. The reviewer reads what Yard measured, not what the agent copied. - From the chat. A failed run of the agent's that was not checked
offers Check on main on its row in the chat (Try again on main
after a check gave no result), for anyone who can edit the workspace. It
runs the same check on that run, without waiting for the agent; the row
says "Checking on main…" and then the result. It needs a setup from the
test config, since there is no
--setupto pass from a button (POST /v1/containers/{id}/test-runs/{runId}/base).
It needs runner 0.38.0 or newer, and yard run from a box updated with it
(older ones do not record the directory, and their command runs at the
root of the checkout).
Criteria blocked by the base branch
Some acceptance criteria ("bun run check passes") fail only because of
what is broken on the base branch. The agent runs
yard tests compare <repo> --for "<criterion>". When the run has no new
failures but some that also fail on main, Yard marks that criterion
Blocked by base branch, and the agent leaves it unticked with
Open: blocked by base branch.
In the Plan tab, every unticked criterion shows its Open: reason beside
its checkbox. A blocked one shows a Blocked by base branch badge instead,
which opens to the failures, the base commit and how many fail there.
The open-criteria card of a done project, and the "is done" notification, say
"blocked by base branch (N failures on main)".
When a baseline for a new commit of the branch is ready, Yard looks at every blocked criterion again:
- Its failures are gone from the new base. Yard runs the project's branch
merged onto the new base. Clean, it ticks the criterion in spec.md, removes
its
Open:note and tells the person who started the work ("Unblocked: base branch fixed …"). - Some are still there, or the merged run still fails: the badge shows the new list.
- The project's branch does not merge onto the new base: the badge says "base moved; update from base to re-check".
A criterion that is ticked or edited by hand stops being blocked. Closed projects are not re-checked.
Runner version
Test runs need runner 0.18.0 or newer. An older runner answers "update the runner to use test baselines".
While a baseline or a comparison runs
Each command reports how far it is while it runs, read from its output as
for any test run. The repository block says Taking test baseline at a1b2c3d, e2e: 42 of 118, and yard tests baseline and
yard tests compare print a line such as e2e: 42 of 118, 2 failed, 6 min
whenever it moved on. A comparison's commands are also listed with the
project's test runs. A check on the base (yard tests base) is no baseline:
the repository block does not show it, and the command prints how far it is
while it waits.