Some tests fail randomly. If it is hard to fix them for now, what about simply retrying them many times? In this way, we could save a lot of time to rebuild and rerun all tests if one of them fails.
Add a new macro #[flaky_test], which catches errors and runs the test 5 times. Report the error only if all trials fail.
I’m not in favor of this macro.
If a test fails occasionally, it can mean some key constraints are broken occasionally. Such a macro can hide this kind of risks.
I do think it is of high priority to fix flaky tests, not only for developer experience but also to fix potential real bugs in the code.
Flaky marker does not mean that we will not fix them. Instead, we should treat all tests with this marker as bugs and fix them as soon as possible. The marker just helps others who don’t own the tests to speed up development. We should use other platform to track the bugs (including flaky tests) instead of make CI failed randomly.
Another solution is that if a test is failed randomly in master branch, just remove it until someone fixes it. It shouldn’t make noise to every developer.