Flaky Test Analysis

Flaky tests, test that sporadically fail but will pass on a retry, are automatically tracked within DeploySentinel. You can manage flaky tests by going to the Analysis page (opens in a new tab) and viewing the Flakiness tab.

To track flakiness, you must have Cypress test retries (opens in a new tab) enabled. On the Flakiness tab, you can view statistics such as the Build Flake Rate, Test Flake Rate, and the number of Flaky Test Cases. These statistics give you a clear understanding of the frequency and severity of flakiness in your tests.

You can click into the test cases listed below the statistics to view specific builds where the test case flaked, and access debug traces of tests that have flaked.

Available Statistics

Build Flake Rate

The number of builds (CI pipeline runs) where at least 1 test case was marked as flaky. This will let you know how often at least 1 flake will pop up in any given test suite run.

This statistic is also graphed as a trend.

Test Flake Rate

The number of times a test case was run and flaked out of all test cases run. (ex. if 3 test cases were always flaky, and the test suite was ran 10 times, it would be 30 flaky test runs). This gives you an idea of how many times tests actually flaked during the analysis time period.

Flaky Test Cases

Number of test cases within a test suite that have flaked at least once during the analysis time period. This can let you know how many test cases within the suite needs to be looked at and potentially triaged.