> even if you don't use @flaky, if one build fails and you run it again and it passes, what do you do?
One's answer to this question is a look into one's soul, at least in terms of engineering. The question is exactly what do you do. In my opinion, good engineers find out why the first one failed and why the second one passed. Either could be a false result. Sometimes the tests failed because the build box ran out of hard drive space, that's a false result. Sometimes a test passes because its condition for passing is incorrect, that's a false result too.
Tests passing is not necessarily a good thing in itself. That is why we have the notion of "testing the test"- a green icon means nothing if it's a lie.
One's answer to this question is a look into one's soul, at least in terms of engineering. The question is exactly what do you do. In my opinion, good engineers find out why the first one failed and why the second one passed. Either could be a false result. Sometimes the tests failed because the build box ran out of hard drive space, that's a false result. Sometimes a test passes because its condition for passing is incorrect, that's a false result too.
Tests passing is not necessarily a good thing in itself. That is why we have the notion of "testing the test"- a green icon means nothing if it's a lie.