Guardrails / tests

The build published its test results

tests/results-published@v1

The build left a readable test report naming at least as many cases as the team expects, instead of reporting nothing at all.

Id	tests/results-published
Version	v1
Category	tests
Default severity	error
Interpreter	python3
Timeout	30 seconds
Violations tolerated	0
Collects	tests

Collectors

This guardrail doesn't gather anything itself. It relies on the collectors below, which the CLI runs once per build before any check, and reads what they found from GUARDRAIL_FACTS. If a collector collects nothing, this guardrail is skipped, not failed.

Collector	Gathers	Inputs it is given
tests	The test results the build already wrote, read from whichever report format it left behind and normalized into one shape: the totals across every report, and the cases that failed, errored or were skipped. Passing cases are left out, because guardrails gate on the others and Buildnote already records every case as a test event. Nothing identifies a case across reports, so when a build writes both a report per class and a merged report, every case is counted in both.	none

The inputs above are this guardrail's own inputs, passed straight through to the collector. Setting one in buildnote.json changes what is collected, and two guardrails configured the same way share a single collection.

Configuration

{
  "guardrails": {
      "failOn": "error",
      "checks": [
          {
              "use": "tests/results-published@v1",
              "severity": "error",
              "with": {
                  "minTests": "1"
              },
              "exemptions": []
          }
      ]
  }
}

Inputs

Input	Description	Default	Environment variable
minTests	Fewest test cases the reports must contain in total. A report with no cases doesn't count as a test run.	1	GUARDRAIL_INPUT_MINTESTS

How to fix

Before buildnote guardrails runs, write the test reports your test runner already produces into the workspace, and keep them there. Gradle writes JUnit XML to build/test-results/, and most runners take a flag:

./gradlew test
pytest --junitxml=junit.xml
dotnet test --logger trx

A missing report is the violation. If a pipeline's test step went green but left nothing behind, you can't show it ran a single test, and a report with no test cases is the same thing. If a job genuinely has no tests, leave the guardrail off for that job instead of lowering minTests to zero.

More in tests

• tests/coverage-floor. The coverage the build measured is at or above the line and branch floors the team sets. Both floors default to zero, so the guardrail just reports the number until the team chooses one.
• tests/no-failures. Every test case in the reports passed or was skipped, so no failing test gets past the gate.
• tests/no-skipped-growth. The reports contain no more skipped cases than the budget allows. The budget is a fixed number in the configuration, not a comparison with the previous build, so it only catches growth if the team lowers the number as skipped tests come back.

All 4 tests guardrails