flakytest.dev

Software teams working with Python codebases run into flaky tests, tests that fail randomly for no apparent reason. These are common in software development and can be difficult to diagnose and fix. Flaky tests slow down the development process and make it harder to deliver high-quality software, and they create build noise that affects people beyond the person who actually wrote the test.

FlakyTest.dev is a single place to manage, monitor, and report on test suite performance. It implements test ownership, similar to the technique used by GitHub, so that the person responsible for a flaky test is the one notified about it. Teams can mute flaky tests with a single click without code changes, zoom into a single test to see all its runs and judge whether it is worth keeping, view debug information such as standard output and standard error streams for failing tests, and see run statistics to identify tests that are slowing down builds. Setup requires adding one Python dependency and one secret key to a project's CI.