Yard Docs
Sign in
On this page

Test baselines

What already fails on the base branch, so agents fix only the failures they caused, and criteria blocked by the base are re-checked when it moves.

Updated November 3, 2026 · 8 min read

A repository's base branch is not always green. When main already has a failing test or a type error, an agent that is told "all tests pass" cannot tell its own failures from the ones that were there before it started.

Yard keeps a test baseline for each repository: the failing tests and type errors on the workspace's branch, at one commit. yard tests compare runs a project's checkout and sorts its failures into new, also fail on main and fixed. Only the new ones are the agent's to fix.

Setting up a repository's tests

Yard does not guess how to run a repository's tests. Each repository gets a test config, kept in Yard (like app config), not in the repository: the commands, the format of each one's output, and an optional setup that runs first.

JSON
{
  "setup": "bun install --frozen-lockfile",
  "commands": [
    { "name": "unit", "run": "bun run test", "format": "bun-test", "timeoutMin": 20 },
    { "name": "typecheck", "run": "bun run typecheck", "format": "tsc", "timeoutMin": 10 },
    { "name": "e2e", "run": "bunx playwright test --reporter=json", "format": "playwright", "timeoutMin": 30 }
  ]
}

Set it from a terminal or an agent with yard tests config set <repo> <file.json>, or through the API (PUT /v1/workspaces/{ws}/repos/{id}/tests/config). Four output formats are read:

Format What is read
bun-test bun test output: each (fail) line under its file, and a file that fails to load as <file> (load error). bun --filter output is read package by package.
playwright The JSON reporter (--reporter=json), or the list reporter's lines. A test that passed on a retry counts as passing.
tsc tsc --noEmit output, plain or --pretty. An error is matched by its code and message, not its line, so a shifted line is the same error.
junit A JUnit XML report on stdout, as vitest run --reporter=junit writes it: each test case with a failure or an error, by its file and title. A file that fails to load reads as <file> (load error).

A repository without a test config has no baseline, and yard tests says how to add one.

The baseline

A baseline is taken at the head of the workspace's branch, in a scratch checkout of the workspace's copy of the repository that is removed afterwards; node_modules is kept between runs so the setup is quick the next time. It is stored in Yard with the commit and the time it was taken.

  • When it is taken. When the branch moves (a push on GitHub, or a fetch), with "Refresh test baseline" in the repository block's … menu, or with yard tests baseline <repo>. One commit is run once, however many of these arrive together, and one runner runs one suite at a time.
  • Stale. Once the branch has moved past it, a baseline is stale. yard tests compare takes a fresh one first and says so.
  • On the block. The repository's block on a canvas shows the baseline's commit, age and how many failures it has.

A command whose output cannot be read gives no baseline, never "nothing fails": a run killed at the memory limit ("killed: out of memory (limit N MB)"), a timeout, or output with no summary (a run cut short, a run with no tests).

Comparing a project's checkout

In a project, yard tests compare <repo> runs the config's commands in the project's checkout, as it is (uncommitted changes too), and prints:

Code
atlas · base main@a1b2c3d (baseline 2h old, 14 failing)
New failures (1):        e2e/notes.test.ts › notes › keeps a title
Also fail on main (14):  …
Fixed vs. main (0)

It exits non-zero when there are new failures or a command gave no result, and names each such command with the reason. --cached prints the last run again without running anything; --json prints the whole comparison.

Failures of other commands

A baseline covers the commands of the test config. An agent often runs others: one journey, one package's unit tests, a script such as bun run check:overlays, or tests in a repository that has no config. yard tests base <repo> checks those against the base branch:

Code
yard run -- bun test ./src            # in apps/runner, fails with 14
yard tests base ai-workspace          # the same command, on main
Code
ai-workspace · ran `bun test ./src` in apps/runner on main@a1b2c3d (here: 14 failed, 3 min ago)
New failures (0)
Also fail on main (14):  src/box/docker.test.ts › docker › copies the tools
                         …

Yard takes the agent's last failing yard run in that repository (or, with -- <command>, the last run of that command), runs it again at the head of the workspace's branch in a scratch checkout, in the directory it ran in, and reads both runs the same way, with the readers that count any test run. The failures that fail there too were there before the change; the others are new.

  • Setup. The base checkout needs its dependencies: the test config's setup is used, or --setup "<install command>" when there is no config (--setup true when nothing needs installing). Without either, Yard refuses rather than read a checkout without dependencies as "everything is new".
  • No result. A run on the base that could not be read (the setup failed, it was killed (for memory or otherwise) or timed out, no test tool was recognised, or more failures than Yard names one by one) gives no result, never "nothing fails there".
  • Kept with the run. The result stays with the run in the project's test runs: the chat's row says "All 14 fail on main@a1b2c3d too" or "2 new, 12 also fail on main", marks each failing test that fails on main too, and a failed run nobody checked reads "Not checked on the base branch". The agent's next turn reads the same, and agents are told to call a failure pre-existing only when yard tests compare or yard tests base said so.
  • In the pull request. The project's pull request in that repository has a Failing test runs section: for each command the agent ran there, its last run when it failed, with the base result ("1 new, 1 also fails on main@a1b2c3d") and the failing tests split into new and also on main, or "Not checked on the base branch". Yard writes it when it opens the pull request and rewrites it in place when a run ends or a check on the base comes back; a later run of the command that passes takes it out. The reviewer reads what Yard measured, not what the agent copied.
  • From the chat. A failed run of the agent's that was not checked offers Check on main on its row in the chat (Try again on main after a check gave no result), for anyone who can edit the workspace. It runs the same check on that run, without waiting for the agent; the row says "Checking on main…" and then the result. It needs a setup from the test config, since there is no --setup to pass from a button (POST /v1/containers/{id}/test-runs/{runId}/base).

It needs runner 0.38.0 or newer, and yard run from a box updated with it (older ones do not record the directory, and their command runs at the root of the checkout).

Criteria blocked by the base branch

Some acceptance criteria ("bun run check passes") fail only because of what is broken on the base branch. The agent runs yard tests compare <repo> --for "<criterion>". When the run has no new failures but some that also fail on main, Yard marks that criterion Blocked by base branch, and the agent leaves it unticked with Open: blocked by base branch.

In the Plan tab, every unticked criterion shows its Open: reason beside its checkbox. A blocked one shows a Blocked by base branch badge instead, which opens to the failures, the base commit and how many fail there. The open-criteria card of a done project, and the "is done" notification, say "blocked by base branch (N failures on main)".

When a baseline for a new commit of the branch is ready, Yard looks at every blocked criterion again:

  • Its failures are gone from the new base. Yard runs the project's branch merged onto the new base. Clean, it ticks the criterion in spec.md, removes its Open: note and tells the person who started the work ("Unblocked: base branch fixed …").
  • Some are still there, or the merged run still fails: the badge shows the new list.
  • The project's branch does not merge onto the new base: the badge says "base moved; update from base to re-check".

A criterion that is ticked or edited by hand stops being blocked. Closed projects are not re-checked.

Runner version

Test runs need runner 0.18.0 or newer. An older runner answers "update the runner to use test baselines".

While a baseline or a comparison runs

Each command reports how far it is while it runs, read from its output as for any test run. The repository block says Taking test baseline at a1b2c3d, e2e: 42 of 118, and yard tests baseline and yard tests compare print a line such as e2e: 42 of 118, 2 failed, 6 min whenever it moved on. A comparison's commands are also listed with the project's test runs. A check on the base (yard tests base) is no baseline: the repository block does not show it, and the command prints how far it is while it waits.

Was this page useful?
Edit this page