Fumbling toward automation: running JavaScript tests with Watir

A group at work is doing some web-app development, and they’re using a tool called WatiN for end-to-end tests. I’m aware of that but didn’t think much of it.

Last night I went to Brian Marick‘s talk at the odynug meeting, and in passing, he mentioned Watir, which it turns out is infinitely cooler than WatiN, because Watir (a) came first, (b) is in Ruby, and (c) can automate IE, FireFox, and (via a separate library) Chrome.

I’m working on a JavaScript-based video-game development kit, and I spend a lot of time Alt+Tabbing to a browser, Ctrl+Tabbing to the tab with the automated test runner, refreshing the page, waiting (especially in IE) for the tests to finish, and (in Chrome) deciding whether the failures are actual test failures or just Chrome failures. Then it’s Alt+Tab to the next browser in the round-robin to try it again.

It shouldn’t be this hard to run all the tests. And with Watir, it looks like it won’t be. I think I’ll be able to write a Ruby script that

  • finds the Chrome tab that’s already open to the test URL (or opens a new tab);
  • reloads the Screw.Unit test page;
  • waits for the tests to complete;
  • scrapes the HTML to decide whether the tests passed or failed; and
  • repeats with FireFox and then with IE.

It won’t be trivial, because the Watir documentation really has nothing to say about finding stuff in the DOM. They’re heavily oriented toward clicking links and filling in forms, so if you want to manipulate a button or a hyperlink or a text field, they’ve got you covered. A myriad of examples, cheat sheets, and FAQs will get you on your way using methods like text_field and button. But if you want to find an <h3> with a particular CSS class, I wish you a lot of luck. The documentation does not go there. You need a lot of Google searches, a lot of luck, and a lot of lucky guessing.

I did scrape together something useful, and I’m noting it here. My tests currently report their results in an <h3>, which happens to be the only h3 on the page. This snippet of Ruby code will display the contents of that h3 (assuming the variable ie already refers to a Watir::IE instance):

ie.document.getElementsByTagName("h3")["0"].innerHTML

Huh. I get it that getElementsByTagName would return an array, so I would understand — and expect — having to do [0] on it. But [0] gives me an error: “TypeError: can’t convert Fixnum into String”. ["0"] works fine, though. I do not understand why, but as long as it works, I’ll accept that for now.

That’s just a start, though, because that snippet only works for classic Watir (for IE). FireWatir (the FireFox version, which is part of the same install but apparently not a compatible Ruby API) fails with “NoMethodError: undefined method `getElementsByTagName'”, and ChromeWatir (separate install) fails with “NoMethodError: undefined method `document'”.

Ah well. I came up with the above snippet by stealing shamelessly from the code for Watir’s show_spans method. Maybe I can do the same for the FireWatir and ChromeWatir versions of show_spans (if they have it). We shall see.

WEBrick: Web-browser GUI for Ruby apps

I do a fair bit of Ruby hacking. Usually I’m either writing a one-off app, or automating some process I do all the time, and in both cases, I don’t need any fancier GUI than STDOUT (via SciTE’s output pane, of course). But occasionally, I’ll want a script I’ll use more than once — say, to sift through another app’s output, and slice and dice the results different ways. And for those sorts of things, sometimes I’d rather have a nice GUI.

Rather than learn any of the GUI libraries for Ruby, any of which would probably be a lot of work if I wanted to do anything fancy, I decided to try my luck with WEBrick, a lightweight Web server that ships with Ruby. I can still write my little one-off apps, but now they can show their output in HTML.

With WEBrick, you write a Ruby script, and when you run your script, it starts listening for HTTP connections. Stop the script (Ctrl+Break or close the console window), the server stops with it. No Apache to configure. It’s easy. I like easy.

The “it’s all one script” thing also means it’s trivial to load data and cache it in memory. I’m planning to use that to write a really fast grep through our source code, with a Web GUI. (I’ve long since lost count of how many times I’ve written a better grep. I’ve written grep more times than all other programs put together.)

Here’s what I think is a good starting point for WEBrick. It consists of two files: startup and servlet. The startup just starts the Web server. You put the actual logic in the servlet file. The separation is so that the servlet file can get automatically reloaded from disk every time you refresh the page in the browser (idea swiped from here).

startup.rb:

require 'servlet'
require 'webrick'
include WEBrick

server = HTTPServer.new(:Port => 80, :BindAddress => 'localhost')
server.mount '/', Servlet
trap('INT') { server.shutdown }
server.start

servlet.rb:

require 'webrick'
include WEBrick

class Servlet < HTTPServlet::AbstractServlet
  def self.get_instance config, *options
    load __FILE__
    new config, *options
  end
  def do_GET(req, resp)
    resp['Content-Type'] = "text/html"
    resp.body = "Hello, world!"
  end
end

This will run on port 80 (edit the :Port to change this, e.g. if you already have a Web server running), and it will only be accessible on the local machine (remove the :BindAddress => 'localhost' to make it accessible from the network).

Put those files in the same directory, run startup.rb, and browse to http://localhost/. Hello, world! Then just change the code in do_GET, save the source file, and reload the browser — and there’s your new page.

But it’s still the same process, still the same script. So you can save global state in a @@class_variable or a $global_variable or some such, and it’ll still be there the next time you reload the page — even if you’ve changed the source code in the meantime. I like static languages for some things, but try doing that in Delphi or C#.

C# 3.0: Dictionary parameters (like in Ruby)

Eilon Lipton blogged about a way to get Ruby-like dictionary parameters in C# 3.0. (Link via Scott Guthrie‘s post on URL routing.)

Ruby lets you pass a dictionary as the last parameter(s) to a method. I won’t go into details, but the syntax looks like this:

get_html_link("Click me", "Key1" => "value1", "Key2" => "value2", "Key3" => "value3")

(Yes, the idiomatic usage would be to use :key1 instead of a string literal, but the strings are probably less confusing for the non-Ruby-adept.)

With Eilon’s code, you can get some very similar syntax in C#:

GetHtmlLink("Click me", new { Key1 = "value1", Key2 = "value2", Key3 = "value3" });

This code passes an anonymous type (that’s the new {...}), and then uses Reflection to read its properties. Reflection is not something you want to do in tight inner loops, but when readability is the main concern (as it is 95+% of the time), this technique is worth considering.

The place this is useful is when you’re really passing a dictionary of arbitrary values — when the function doesn’t know how many items to expect, or what they’ll be called. If, on the other hand, the function knows it’s expecting three properties called Key1, Key2, and Key3, you’d be better off defining a type with three properties, rather than using an anonymous type. Then the refactoring tools would be on your side, and you wouldn’t have reflection overhead. (The syntax wouldn’t change much, though — you’d just add the type name after new.)

Using anonymous types this way requires some work on the part of the function being called, to do the needed bits of Reflection. Eilon’s article includes sample code, although it leaves something to be desired; for example, he makes a new type for the key-value pairs, when KeyValuePair<> is already out there waiting to be used. And at that, why return a list of key-value pairs, when you could just return a dictionary?

Here’s my contribution. This version has the added advantage that you can pass in either a type with public properties (e.g. an anonymous type), or an IDictionary (generic or non-), and it does the Right Thing™. It returns a dictionary with case-insensitive key lookup, so you could use dictionary["key1"] just as well as dictionary["Key1"] (change it if that’s not what you want). Unit tests are left as an exercise for the reader.

public static IDictionary<string, TValue> ObjectToDictionary<TValue>(object o)
{
var result = new Dictionary<string, TValue>(StringComparer.InvariantCultureIgnoreCase);
if (o == null)
return result;
if (o is IDictionary)
{
foreach (DictionaryEntry entry in (IDictionary) o)
result.Add(entry.Key.ToString(), (TValue) entry.Value);
}
else
{
foreach (PropertyDescriptor prop in TypeDescriptor.GetProperties(o))
result.Add(prop.Name, (TValue) prop.GetValue(o));
}
return result;
}

The above code snippet is public domain; use it how you like.

Generating the grammar document

My Delphi grammar document is built from two pieces: my research, and a tool.

The research

The first step was to do all the research to figure out what the Delphi grammar is. This is not easy. The Delphi 5 documentation included an incomplete, and sometimes wildly inaccurate, BNF grammar. The Delphi 2006 documentation no longer includes the grammar; either the documentation team lost it (along with the docs for the IDE’s regular-expression syntax), or they gave up because it was so far out of date. The language has added loads of features since then: strict private, records with methods, even generics on the way.

So I had a rough sketch to start from, and an undergraduate compiler-design class from ten years ago. The rest — correcting the errors, and filling in the (large) blanks — is trial and error, and a lot of refactoring.

The upshot is, if you see something I’m missing, let me know. Fatih Tolga Ata already put class helpers and generics on my radar — although I can’t really do much with generics yet. Since there is no official (correct) grammar from CodeGear, my main method of discovering the grammar is to type stuff into the IDE and see what compiles (and, often more instructively, what doesn’t), I won’t be able to figure out the generic grammar until I have Highlander.

The tool

As I puzzle out the grammar, I document it in a YAML file. Here’s a snippet from this file:

Atom:
Doc: |
! -> <number>
. -> <stringliteral>
! -> Ident
! -> NIL
. -> '(' Expression ')'
. -> SetLiteral
Block:
Doc: |
. -> BEGIN StatementList END

The ! at the beginning of a line means “I’ve implemented this in my parser”; the . means “I haven’t implemented this yet”. That’s what drives the “(Completed)” and “(In progress)” notations in the grammar document.

I wrote a Ruby script that reads this YAML file and generates the HTML Delphi grammar documentation. That Ruby script is the part that’s cool enough to figure out which rules are fully implemented (shown with a solid underline), which are partially implemented (e.g., Atom, as shown above; shown with a dashed underline), and which ones I haven’t started on yet (no underline). It also figures out the “Likely Targets” — the rules whose dependencies are (fully or partially) done: the ones I can (probably) work on next.

I edit the YAML file frequently — as you can imagine, since it reflects my completion status. And I refer to the generated HTML document just as frequently. So I’ve made the Ruby script part of my Rakefile. It works out fairly well.

Of course, uploading the HTML doc to my Web site happens… a little less frequently. I just uploaded the latest version this morning, but before that, it looks like it had been a little more than a year since my last update. I’ll try to keep it a little more active.

Subversion in Delphi’s Tools menu

At work, we’ve added several items to Delphi’s Tools menu to make Subversion functions easy to get to from the IDE. I set some of them up on my home PC too, because they’re just so convenient. They’re easy to access from the keyboard, too (just hit Alt+T and then C to bring up a commit window), which is a minor miracle given that Delphi still doesn’t support customization of keyboard shortcuts.

We have a few more at work, but here are the “gotta-haves” that I’ve set up on my home machine:

  • Svn Commit: Opens the TortoiseSVN commit window.

  • Svn Diff: Shows diffs for the file currently being edited. (If you’ve configured an external diff viewer like Beyond Compare, this will use it.)

  • Svn Modifications: Opens the TortoiseSVN modifications window, which shows a list of all modified files.

  • Svn Update: Updates your working copy with the latest changes from the repository.

These should be trivial to set up with Delphi’s Tools > Configure Tools dialog, since they have macro expansions for things like the path to the current project. Unfortunately, Delphi’s Configure Tools macro expansions pretty much suck for this. TortoiseSVN requires that the parameter be “slash-parameter name-colon-value”, e.g. /path:C:\svn\DUnitAssertions. Delphi, on the other hand, is incapable of doing a macro expansion that isn’t preceded by a space, so that “colon before the value” thing is a killer. It doesn’t get any better when your paths can include spaces and you need to quote the path.

So I’ve got a script in Ruby, which reads the command-line parameters that Delphi is capable of producing, and translates them into something that TortoiseSVN is capable of understanding. The one I wrote at home looks like this (I saved this in C:\svn\RunTortoise.rb):

COMMAND = ARGV[2]
PATH = ARGV[3..ARGV.length].join(" ")
command = %Q|"c:/program files/tortoisesvn/bin/tortoiseproc.exe" | +
%Q|/command:#{COMMAND} /path:"#{PATH}" /notempfile|
puts command
system command

Then you can set up the Configure Tools items. Mine look like this (these assume that your project file is at the “root” of your Subversion checkout, i.e. that all the files you’re editing are in that directory or its subdirectories):

Title: Svn &Commit
Program: c:\ruby\bin\rubyw.exe
Working dir: (blank)
Parameters: C:\svn\RunTortoise.rb - -- commit $PATH($PROJECT) $SAVEALL
Title: Svn &Diff
Program: c:\ruby\bin\rubyw.exe
Working dir: (blank)
Parameters: C:\svn\RunTortoise.rb - -- diff $EDNAME $SAVEALL
Title: Svn &Modifications
Program: c:\ruby\bin\rubyw.exe
Working dir: (blank)
Parameters: C:\svn\RunTortoise.rb - -- repostatus $PATH($PROJECT) $SAVEALL
Title: Svn &Update
Program: c:\ruby\bin\rubyw.exe
Working dir: (blank)
Parameters: C:\svn\RunTortoise.rb - -- update $PATH($PROJECT) $SAVEALL

Unit tests for parsers

I’m still tinkering with my ANTLR-based Delphi parser, but I’ve also started playing around with Racc. Today, I found out that there are going to be some fundamental differences in how I write unit tests for Racc.

ANTLR generates a recursive descent parser, which means that it can’t handle any sort of ambiguity in the grammar. (Is “at” a keyword or an identifier? “It depends” is not generally an answer ANTLR can accept.) But there’s an upside, too: the code it generates is very easy to understand (and debug, for that matter). If I have a parser rule “expression ::= atom operator atom“, then ANTLR generates a function called expression(), which consists of calls to atom(), operator(), and atom(). There’s some other decorations along the way, but it’s basically that simple.

This makes it really easy to write unit tests for the expression rule in isolation: I just stuff some tokens into the lexer, call parser.expression(), and then make sure all the lexer’s input has been consumed. It’s not really a unit test of expression(), since I’m also testing everything expression() calls, but it’s been working reasonably well so far.

Racc, on the other hand, makes an LALR parser. It should, like yacc, be able to handle all sorts of ambiguity, as long as there’s some way to eventually resolve it. It generates a table-driven parser, basically a big state machine (so I hope I never have to debug it).

And it doesn’t have entry points for individual parser rules. This presents a problem for unit testing.

I see several options here:

  • Don’t write unit tests for the parser. (Fat chance.)
  • Force the state machine into the right state, then throw some tokens at it, and check its final state. The problem with this is, I’m pretty sure there’s more than one starting state for “expression”, since it will show up multiple places in the grammar, each with its own exit conditions. (Which state do you move to if you’re in an expression and you hit a semicolon? Depends entirely on which state you were in when you started…)
  • Feed in a complete source file’s worth of syntax for every test. If I’m testing the expression rule, then each test would prepend “unit Foo; initialization implementation begin” and append “end.” to the expression I’m testing. This might be doable, though it would force me to implement the grammar starting from the unit rule and working down (I had been going the other direction with the ANTLR parser), which would necessitate some really good record-keeping to know how close I was to being done.
  • Whip up some magic tokens that will never be generated by the real tokenizer, but can be injected by tests. Then make the top-level rule able to parse either projects (“program Foo; …” or “library Foo; …”), units (“unit Foo; …”), packages (“package Foo; …”), expressions (“<secret token for expression> 1 + 2”), uses clauses (“<secret token for uses clause> uses Foo, Bar;”), etc. This also would be doable, if I could automate the repetitive parts of the syntax; and it would let me build the grammar from the bottom up, which has its advantages.

Has anyone out there written an LALR grammar test-first? Does anyone have any suggestions, or see anything I hadn’t thought about?

On using Rake.run_tests

Okay, an update to my previous Rake post. Rake does have some nice facilities for running tests, but the code I posted isn’t quite the right way to do it.

When you use rake/runtest, you should put the require inside your test task (inside each test task, if you have more than one). Like so:

# RIGHT
task :test do
  require 'rake/runtest'
  Rake.run_tests "**/*tests.rb"
end

# WRONG
require 'rake/runtest'
task :test do
  Rake.run_tests "**/*tests.rb"
end

The reason: the WRONG version will always (indirectly) require ‘test/unit’, which will always run your tests, even if you go through a code path that doesn’t register any.

Here’s what it looks like if you have the WRONG version and run rake -T (list all the interesting targets to the console, but don’t run any of them). Note that it’s running the tests, even though there aren’t any to run.

C:\svn\dgrok-r>rake -T
(in C:/svn/dgrok-r)
rake all  # Runs everything
Loaded suite c:/Program Files/ruby/bin/rake
Started

Finished in 0.0 seconds.

0 tests, 0 assertions, 0 failures, 0 errors

C:\svn\dgrok-r>

Here’s what it looks like if you use the RIGHT code above. It still runs the tests when you execute the “test” target, but when you’re not doing anything that involves test running, it doesn’t give you all that nonsense about how it ran 0 tests in 0.0 seconds:

C:\svn\dgrok-r>rake -T
(in C:/svn/dgrok-r)
rake all  # Runs everything

C:\svn\dgrok-r>

The coolness that is Rake

Rake is cool enough as a build tool, but wait until you check out its library.

First off, if you use make or ant, you owe it to yourself to look into Rake. The syntax is so much more readable than either, and you have a full-fledged programming language right there in your build script. The docs have a concise introduction to Rakefiles; Martin Fowler has a longer article called “Using the Rake Build Language“.

But wait! There’s more!

So Rake’s documentation is pretty sketchy. Mention is made of some cool libraries that ship with it, but no details. So I decided to RTFS, and I found much of coolness.

Coolness #1: All the coolness comes right out of the box. You don’t even need to require anything from your rakefile (unless you’re doing really fancy stuff). Just write a rakefile containing some tasks, run rake, and you’re off and running.

Coolness #2: They add an ext() method to the String class. Treats the string as a filename; returns a new filename with the extension changed to whatever you specify. Equivalent to Delphi’s ChangeFileExt() or .NET’s Path.ChangeExtension. Something that’s sorely lacking from Ruby out of the box.

Coolness #3: A built-in command to shell out to Ruby. They have a method to shell out, and another one specifically to shell out to another Ruby process.

When you want to shell out (e.g. to run your compiler), you should use the sh() and ruby() methods, which will automatically fail the task if the process returns a nonzero exit code. (By default, the value returned by a task block is ignored; if you want to fail the task, you need to do it explicitly with fail() or raise or some such, or call something like sh() that does it for you.)

rule '.rb' => ['.y'] do |t|
  sh "racc", t.source, "-o", t.name
end
task :test do
  ruby "test_all.rb"
end

Coolness #4: Built-in support for running unit tests. I already wrote a test_all.rb, but if I’d looked at Rake, I wouldn’t have had to. This works just as well:

[Note: It turns out that this sample code isn’t the right way to do it; see this post for corrected code. Still the same number of lines, just better-behaved.]

require 'rake/runtest'
task :test do
  Rake.run_tests "**/*tests.rb"
end

If you follow their convention of having your tests in test/test*.rb, you don’t need to pass an argument to Rake.run_tests at all. I like my tests to be next to the files they’re testing, so the above example is what I prefer.

There’s also a TestTask if you want fancier options, like modifying your $LOAD_PATH before running the test. (TestTask actually launches a new Ruby process via the ruby() method; Rake.run_tests runs them in-process.)

Coolness #5: FileList.

FileList is a list of files, much like an array. But instead of giving it individual filenames, you can give it patterns, like ‘lib/**/*’, and it comes back as an array of matching filenames. (It can take both include and exclude patterns.) It lazy-initializes its contents, so if you create a FileList but then don’t run the task that uses it, it’s basically free. By default, it ignores CVS and .svn directories.

But coolest of all, FileList has shorthand methods: sub(), gsub(), ext(), egrep(). Want to return a new copy of a FileList with all the extensions changed? No sweat. Want to grep through every file in a FileList, and do something with each file whose contents match a given regex? egrep() to the rescue. I may never again have to write six whole lines of code to recursively grep through a directory tree of source files.

Assertions the right way around (in Ruby, anyway)

I’ve always hated the common xUnit convention of writing things like AssertEquals(expectedValue, actualValue). Who came up with the idea of doing things backwards?

Your tests have to say things like “assert that 5 is what you get when you add 2 and 3”. It’s stilted and awkward (unless, perhaps, you’re trying to write test poetry). And it puts the effects before the causes. “I should get this. Oh, by the way, here’s what I’m doing.” Answer first, then question. I mean, what is this, Jeopardy?

This isn’t how things are done in real life. If I put the pizza into the oven for fifteen minutes, take it out, and then preheat the oven, my dinner is not going to taste very good (unless I have something else for dinner). And if I tried to take that pizza home before I paid for it, Wal-Mart would be well within its rights to take a dim view of the situation, even if I did eventually come back and pay for it.

Why not just do it the other way around? “Assert that 2 plus 3 equals 5.” Cause first, then effect: “I should do this calculation, and here’s the result I should get.” Isn’t that nicer?

But most xUnit frameworks put the effect first, so that’s what people are used to, for good or ill. I want to write my own custom assertions that put the cause before the effect, but that would just confuse people (myself among them) who have been putting it the other way around for years.

Well… I was just tinkering in Ruby, and I found what I think is a nice way to flip the parameters, and leave a nice readable clue that that’s how they’re supposed to be:

assert_lexer "*" => [:times, "*"]

Ruby’s => syntax lets me show how things are flowing: the “*” should turn into the array [:times, “*”]. Input on the left, expected output on the right, and visibly so. Neat, huh?

For anyone not familiar with =>, it lets you lump several name => value pairs together at the end of a parameter list. They all get scooped up into a single Hash, which is passed as the method’s final parameter. So the above is equivalent to (but much prettier than):

hash = Hash.new
hash["*"] = [:times, "*"]
assert_lexer(hash)

Racc Tutorial

I’ve done a bit of looking at Racc, a compiler-compiler for Ruby. (It’s vaguely like Yacc, although the syntax is different.)

Unfortunately, Racc has no documentation to speak of. But someone named CMills has put together a PowerPoint introduction to writing parsers in Ruby and Racc (38k download, .tar.gz format). Sketchy but good. I didn’t understand all the slides, but it’s the best Racc documentation I’ve found so far.