Skipped tests stay within budget
tests/no-skipped-growth@v1
The reports name no more skipped cases than the budget allows. The budget is a fixed number this guardrail is configured with rather than a comparison against the previous build, so it catches growth only as far as the team lowers the number when a skip comes back.
| Id | tests/no-skipped-growth |
| Version | v1 |
| Category | tests |
| Default severity | warning |
| Interpreter | python3 |
| Timeout | 30 seconds |
| Violations tolerated | 0 |
| Collects | tests |
Collectors
This guardrail gathers nothing itself. It depends on the collectors below, which the CLI runs once per build before any check, and reads what they found out of GUARDRAIL_FACTS. A collector that collects nothing skips this guardrail rather than failing it.
| Collector | Gathers | Inputs it is given |
|---|---|---|
tests | The test results the build already wrote, read from whichever report format it left behind and normalized into one shape: the totals across every report, and the cases that failed, errored or were skipped. The passing cases are left out, because a guardrail gates on the others and Buildnote already carries every case as a test event. Nothing identifies a case across reports, so a build writing both a report per class and a merged one counts every case in both. | none |
The inputs above are this guardrail's own, passed straight through. Configuring one in buildnote.json changes what is collected, and two guardrails configured the same way share the one collection.
Configuration
{
"guardrails": {
"failOn": "error",
"comment": true,
"checks": [
{
"use": "tests/no-skipped-growth@v1",
"severity": "warning",
"with": {
"maxSkipped": "0"
},
"exemptions": []
}
]
}
}Inputs
| Input | Description | Default | Environment variable |
|---|---|---|---|
maxSkipped | Most cases the reports may name as skipped, ignored or inconclusive. | 0 | GUARDRAIL_INPUT_MAXSKIPPED |
How to fix
Re-enable the skipped tests, or delete the ones that are never coming back. A skip costs what it always cost to maintain and reports nothing.
Where a skip is deliberate, set the budget to the number you accept and lower it as they return:
{
"guardrails": {
"checks": [
{ "use": "tests/no-skipped-growth@v1", "with": { "maxSkipped": "3" } }
]
}
}This compares against maxSkipped and nothing else. It does not read the previous build, so the ratchet is the number in the configuration, and lowering it is a commit somebody reviews.
More in tests
tests/coverage-floor. The coverage the build measured sits at or above the line and branch floors the team sets. Both floors default to zero, so the guardrail reports the number until a team chooses one.tests/no-failures. Every case the test reports name either passed or was skipped, so nothing red was carried past the gate.tests/results-published. The build left a test report that can be read, naming at least the cases the team expects, rather than reporting nothing at all.