Guardrails / Collectors

Test results

The test results the build already wrote, read from whichever report format it left behind and normalized into one shape: the totals across every report, and the cases that failed, errored or were skipped. The passing cases are left out, because a guardrail gates on the others and Buildnote already carries every case as a test event. Nothing identifies a case across reports, so a build writing both a report per class and a merged one counts every case in both.

Facts key	tests
Version	v1
Script	tests.py
Timeout	60 seconds

Inputs

Input	Description	Default	Environment
reports	Comma separated globs matched against every file in the working directory and against bare file names, naming the test reports to read.	TEST-*.xml,*-test-report.xml,junit*.xml,test-results*.xml,*.trx,*-ctrf.json,open-test-report.xml	GUARDRAIL_INPUT_REPORTS
maxFailures	Most cases to carry in failures and in skipped, counted separately. The totals stay whole when cases are dropped.	50	GUARDRAIL_INPUT_MAXFAILURES

A guardrail that declares an input of the same name passes it through, so the guardrail's configuration in buildnote.json is what decides these values.

A guardrail asks for these facts by name, and reads them back the same way:

{
  "collect": ["tests"]
}

tests = guardrail.facts("tests")

Facts

The document tests collects. A path carrying [] is an entry of the list before it, and one carrying [path] is a key of the object before it.

Fact	Meaning
source	report when a test report was read, none when there was none.
reason	Why nothing was read, present only when source is none.
reports	Every report that was read, in path order.
reports[].path	Path of the report, relative to the directory the CLI runs in.
reports[].modified	When the report was last written, ISO 8601, so a stale report is recognisable.
reports[].format	Format it was read as: junit, nunit, trx, otr or ctrf.
totals.tests	Cases across every report that was read, summed rather than deduplicated.
totals.passed	Cases that passed.
totals.failed	Cases that failed an assertion.
totals.errors	Cases that ended in an error rather than a failed assertion.
totals.skipped	Cases that were skipped, ignored or left inconclusive.
totals.durationMs	How long the reports say the cases took, summed, or null when no report stated a duration.
suites	How many report files contributed to the totals.
failures	The failed and errored cases, up to maxFailures.
failures[].name	Name of the case, as the report gave it.
failures[].classname	Class or suite the case belongs to, or null when the report names none.
failures[].status	failed for a failed assertion, error for a case that ended in an error.
failures[].file	File the case was declared in, or null when the report names none.
skipped	The skipped cases, up to maxFailures.
skipped[].name	Name of the case, as the report gave it.
skipped[].classname	Class or suite the case belongs to, or null when the report names none.
skipped[].status	Always skipped.
skipped[].file	File the case was declared in, or null when the report names none.
dropped	Cases left out of failures and skipped because maxFailures was reached.

A collector that cannot finish prints what it had along with an incomplete key saying why, so the facts below the point it stopped are absent. A check that cares reads incomplete before the rest.

Example facts

The example facts collected by tests collector:

{
  "source": "report",
  "reports": [
    {
      "path": "TEST-com.acme.widget.QueueTest.xml",
      "modified": "2026-03-04T10:15:00Z",
      "format": "junit"
    }
  ],
  "totals": {
    "tests": 4,
    "passed": 2,
    "failed": 1,
    "errors": 0,
    "skipped": 1,
    "durationMs": 624
  },
  "suites": 1,
  "failures": [
    {
      "name": "rejects a negative limit",
      "classname": "com.acme.widget.QueueTest",
      "status": "failed",
      "file": "service/src/test/kotlin/QueueTest.kt"
    }
  ],
  "skipped": [
    {
      "name": "drops the oldest under load",
      "classname": "com.acme.widget.QueueTest",
      "status": "skipped",
      "file": "service/src/test/kotlin/QueueTest.kt"
    }
  ],
  "dropped": 0
}

Collected for

Guardrail	Category	Inputs
tests/no-failures	tests	none
tests/no-skipped-growth	tests	none
tests/results-published	tests	none

All collectors