CodeshipCloudBees Team6 min readintermediate
When red doesn’t mean broken, your pipeline stops meaning anything
Summary
CloudBees Smart Tests adds a “High‑failure‑rate tests” dashboard and a “Quarantine” mode that lets flaky tests keep running but no longer block the pipeline, with manual or automatic thresholds and automatic recovery when failure rates drop.
- A new dashboard lists tests by failure rate and execution volume, letting teams prioritize noisy tests.
- Quarantine keeps a test’s results recorded but removes its blocking effect; it can be applied per‑test or via a failure‑rate threshold.
- Automatically quarantined tests are re‑enabled when their failure rate falls below the configured threshold; manually quarantined tests require explicit un‑quarantine.
- The feature is positioned as a middle ground between deleting flaky tests and tolerating them, aiming to preserve coverage while restoring CI signal trust.
When flaky tests dominate CI output, engineers start ignoring red builds, which lets real regressions slip through. Providing a systematic way to isolate and later recover flaky tests restores confidence in the test suite without sacrificing coverage.
5/10