Selenium Tests Randomly Failing

Saturday, July 31st, 2010

The Problem

The project I’m currently working on uses a lot of selenium tests to verify the behaviour of the web front end and, I must say, I’ve been quite impressed with it.

Yesterday however after updating my local machine with the latest copy of the project from Clearcase I noticed a lot of failing tests; the worrying (interesting?) thing though was that the failures appeared to be “random”. Tests were passing one run and failing the next, with no changes having being made in the source code and no changes in the initial starting conditions. I was starting to pull my hair out. Curiously the tests ran fine on my colleagues’ and the build machine. My workmate had a look (remember the tests ran fine for him) but he too was getting the random failures on my machine – we were both stumped.

He did unknowingly hit on the problem when he said “Your machine is a lot zippier than mine”. After another hour or so the problem and solution became obvious.

Cause

The Selenium remote control was executing the steps of the test faster than my local app server and Firefox could keep up. In other words it was trying to verify elements on the page before they had rendered.

Solution

To quickly check my hypothesis I ran the Selenium Tests in “slow mode” – all the tests passed!

Now, going forward, I will need to check the test source for all snippets like

browser.click("someButton"); // causes page to load
verifyTrue(browser.isTextPresent("someText"));

and change them to

browser.click("someButton");
browser.waitForPageToLoad("30000");
verifyTrue(browser.isTextPresent("someText"));

You can leave a response, or trackback from your own site.

Tags: , ,
Posted in: quick tips



Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>