Yesterday we discovered a small but annoying performance bug: when getting a list of languages from the JVM we did not cache them but kept on retrieving them, which turned out to be quite slow.
So, like the good guys we are we wanted to have a test that ensured that the error was not reintroduced. We looked at JUnitPerf which seemed to be appropriate - it decorates a unit test and times it. The only problem here was that we were talking about that good performance was 800 ms, and bad performance was 1800 ms. Such timings can randomly fail on a slower computer or a computer that was busy doing something else. We do not like that kind of tests.
So instead my colleague Jimmy came up with a much better idea: Make the test a Runnable, and run it a large number of times. In our case the good performance was 800 ms the first time and virtually nothing after, but the initial bad performance was 1800 ms every time. Then we set the timeout to 5 seconds, giving enough slack for slow, bogged down test machines, but still catching the bug if reintroduced. Good stuff indeedy.