Skipped tests stay within budget
tests/no-skipped-growth@v1
The reports contain no more skipped cases than the budget allows. The budget is a fixed number in the configuration, not a comparison with the previous build, so it only catches growth if the team lowers the number as skipped tests come back.
| Id | tests/no-skipped-growth |
| Version | v1 |
| Category | tests |
| Default severity | warning |
| Interpreter | python3 |
| Timeout | 30 seconds |
| Violations tolerated | 0 |
| Collects | tests |
Collectors
This guardrail doesn't gather anything itself. It relies on the collectors below, which the CLI runs once per build before any check, and reads what they found from GUARDRAIL_FACTS. If a collector collects nothing, this guardrail is skipped, not failed.
| Collector | Gathers | Inputs it is given |
|---|---|---|
tests | The test results the build already wrote, read from whichever report format it left behind and normalized into one shape: the totals across every report, and the cases that failed, errored or were skipped. Passing cases are left out, because guardrails gate on the others and Buildnote already records every case as a test event. Nothing identifies a case across reports, so when a build writes both a report per class and a merged report, every case is counted in both. | none |
The inputs above are this guardrail's own inputs, passed straight through to the collector. Setting one in buildnote.json changes what is collected, and two guardrails configured the same way share a single collection.
Configuration
{
"guardrails": {
"failOn": "error",
"checks": [
{
"use": "tests/no-skipped-growth@v1",
"severity": "warning",
"with": {
"maxSkipped": "0"
},
"exemptions": []
}
]
}
}Inputs
| Input | Description | Default | Environment variable |
|---|---|---|---|
maxSkipped | Maximum number of cases the reports may mark as skipped, ignored or inconclusive. | 0 | GUARDRAIL_INPUT_MAXSKIPPED |
How to fix
Re-enable the skipped tests, or delete the ones that are never coming back. A skipped test still costs as much to maintain and tells you nothing.
If a skip is deliberate, set the budget to the number you accept, and lower it as tests are re-enabled:
{
"guardrails": {
"checks": [
{ "use": "tests/no-skipped-growth@v1", "with": { "maxSkipped": "3" } }
]
}
}This guardrail compares against maxSkipped and nothing else. It doesn't look at the previous build, so the number in the configuration is what holds the line, and lowering it is a commit someone reviews.
More in tests
tests/coverage-floor. The coverage the build measured is at or above the line and branch floors the team sets. Both floors default to zero, so the guardrail just reports the number until the team chooses one.tests/no-failures. Every test case in the reports passed or was skipped, so no failing test gets past the gate.tests/results-published. The build left a readable test report naming at least as many cases as the team expects, instead of reporting nothing at all.