Selenium Tests Randomly Failing
Saturday, July 31st, 2010
The Problem
The project I’m currently working on uses a lot of selenium tests to verify the behaviour of the web front end and, I must say, I’ve been quite impressed with it.
Yesterday however after updating my local machine with the latest copy of the project from Clearcase I noticed a lot of failing tests; the worrying (interesting?) thing though was that the failures appeared to be “random”. Tests were passing one run and failing the next, with no changes having being made in the source code and no changes in the initial starting conditions. I was starting to pull my hair out. Curiously the tests ran fine on my colleagues’ and the build machine. My workmate had a look (remember the tests ran fine for him) but he too was getting the random failures on my machine – we were both stumped.
He did unknowingly hit on the problem when he said “Your machine is a lot zippier than mine”. After another hour or so the problem and solution became obvious.
Cause
The Selenium remote control was executing the steps of the test faster than my local app server and Firefox could keep up. In other words it was trying to verify elements on the page before they had rendered.
Solution
To quickly check my hypothesis I ran the Selenium Tests in “slow mode” – all the tests passed!
Now, going forward, I will need to check the test source for all snippets like
browser.click("someButton"); // causes page to load
verifyTrue(browser.isTextPresent("someText"));
and change them to
browser.click("someButton");
browser.waitForPageToLoad("30000");
verifyTrue(browser.isTextPresent("someText"));
You can leave a response, or trackback from your own site.
Tags: java, selenium, testing
Posted in: quick tips