Thursday, 3 January 2008

Automated Dojo testing - DOH & Selenium-RC

Before the 0.9 release, Dojo replaced its old unit-testing framework with a new one named D.O.H. that is miles nicer and cleaner. It'll run in both the browser & under Rhino, supports defferrables and is relatively easy to customise.

At One Track Mind we've been using Selenium-RC to drive unit-testing of JS code since Dojo 0.4 (of our stuff mostly, not Dojo itself) and it wasn't ever particularly clean. The basic idea is to run a suite of unit tests across a series of browser/version/OS combinations and collate the results. So you can find that some irrelevant little change b0rked Opera but only on Windows.

Selenium's API for writing tests (for JS stuff anyway) is pretty awful compared with DOH and we'd done ours in Python, which was then launched from Ant after the Dojo build was done. But what Selenium-RC does do well is fire up browsers on multiple operating systems, do magic proxying, and allow you to interrogate the pages.

A thread on dojo-interest this week sparked me back into looking at it again. I noticed that Selenium-RC now supports running from Javascript. (to be honest, the java bindings would have always worked but someone's actually spelt it out in small words now). Since the Dojo build system is now Javascript-based, why not make the Selenium test runner as well?

Plan:

  • Have a config file defining which browsers to launch, which machines they're on, and what tests to run.
  • Launch each browser via Selenium-RC
  • Run the tests via the normal DOH browser runner.
  • Use Selenium to extract the results from DOH.
  • Collate the results from the various browsers and produce something useful.
So, after a bit of toying I came up with this:
  • Drop seleniumRunner.js, seleniumRunner.config.js, seleniumRunner.sh (or the .bat if you're on Windows), and selenium-java-client-driver.jar into util/doh/ in your Dojo install.
  • Put selenium-server.jar on each test machine, then run java -jar selenium-server.jar -multiWindow so it listens for the browser-control messages.
  • Edit seleniumRunner.config.js and change browsers and rootUrl to match your setup. The rootUrl needs to be reachable from each test machine.
  • run ./seleniumRunner.sh seleniumRunner.config.js from util/doh/ on your workstation
  • It'll load the config, fire up the browsers on each machine, run the unit tests from Dojo core, and print the pass/fail/error stats for each.
  • Each browser is kicked off and monitored in a separate thread (not strictly necessary but too cool to resist doing).
Some issues I ran into:
  • unless I ran the selenium server in multiWindow mode Safari and Firefox would pop up Print dialogs (!?!) whenever the test page was loaded. But Safari never initialised the test page if it was in multiWindow mode. On OSX and Windows. gah.
  • Opera on OSX didn't set up the Selenium proxy properly (localhost:4444 for reference).
  • IE didn't like doing a dojo.connect() via the selenium javascript commands for some reason.
The next step is to improve the reporting. This will probably necessitate writing another browser runner that produces a JSON report (or changing the existing one). Modifying DOH to have an isComplete flag would make life simpler as well. I'd want to know which tests were run on each browser (since not all tests run everywhere), which ones passed and failed and had errors. A hunk of JSON which could be loaded into a Dojo Grid for exploration might make an interesting way of viewing the results.

5 comments:

shaneosullivan said...

This is very cool, keep it up - if all the little gotchas can be worked out this could be a very useful approach

James said...

Just want to agree with Shane: this is neat stuff! I have not used Selenium before, so I cannot offer any help on the issues, but I can see this being very useful.

David Bolter said...

I feel this work is vital. Rob, how is the keyboard event synthesis in Selenium these days?

Lemonk Debri said...

We should always show love to this page and I think it should have a donate button passing nclex

Afzal khan said...

Your music is amazing. You have some very talented artists. I wish you the best of success. Elmendorf Tear Tester