Selenium tests are dreamy: how spectacular that your whole UI can be given a right thrashing on a variety of browsers! But intermittent Selenium test failures… now, they are truly the bane of one’s existence, especially when they hold up progress in a continuous build environment.
So enter: the retry test listener.
In my company’s suite of Selenium tests, there’s a small percentage that always seem to fail intermittently. We mark those tests as retry candidates, and if they do fail, they’ll be given another shot at passing.
To get something similar set up, firstly, implement an ITestListener like so:
public class RetrySeleniumTestListener implements ITestListener {
private static final Logger LOG = LoggerFactory.getLogger(RetrySeleniumTestListener.class);
@Override
public void onTestFailure(final ITestResult result) {
final IRetryAnalyzer retryAnalyzer = result.getMethod().getRetryAnalyzer();
if (retryAnalyzer != null && retryAnalyzer instanceof RetrySeleniumTestCounter && ((RetrySeleniumTestCounter) retryAnalyzer).isOnFirstRun()) {
LOG.info("Test failed on first run and will be marked as skipped: " + result);
// You might ask: why not set this result as skipped in RetrySeleniumTestCounter?
// That's too late and the change in status is ignored.
// Have to do it here instead.
result.setStatus(ITestResult.SKIP);
}
}
/*
Here would be the the implementations of the other methods of ITestListener
*/
}
Next up, implement an IRetryAnalyzer like so:
public class RetrySeleniumTestCounter implements IRetryAnalyzer {
private int retryCount;
@Override
public boolean retry(final ITestResult result) {
if (retryCount < 1) {
retryCount++;
return true;
}
return false;
}
public boolean isOnFirstRun() {
return retryCount == 0;
}
}
And then annotate your intermittently failing test methods and test classes appropriately like so:
@Listeners({ RetrySeleniumTestListener.class })
public class DataExtractsSeleniumTest extends AdmanagerSeleniumTestCase {
@Test(retryAnalyzer = RetrySeleniumTestCounter.class)
public void testCannotAccessFunctionalityForUsers() {
// Contents of intermittently failing test method
}
}
Selenium then knows to apply the test listener to the annotated class, and the retry counter is then applied to the annotated methods.
And voilà, those intermittently failing tests now get a second chance at passing.