DUnitLite 0.5 released

Version 0.5 of DUnitLite is now available. This is a minor enhancement release.

Download it here: DUnitLite 0.5

What’s new in 0.5

  • Should.ReferTo now works with interfaces, and can mix and match between objects and interfaces. It also correctly works around the interface reference-equality problem.
  • Added Given as an alias for Specify. So if you want to document your assumptions at the beginning of your test, you can use Given.That(...) to improve readability.
  • Added Should.[Not.]Be.False and Should.[Not.]Be.True.
  • Added Should.[Not.]Be.Null and Should.[Not.]Be.Assigned. Both work with either objects or interfaces.
  • Added Should.[Not.]Be.GreaterThanOrEqualTo and Should.[Not.]Be.LessThanOrEqualTo, as aliases for AtLeast and AtMost.

DUnit tricks: Getting stack traces for test failures

DUnit has an option to use JCL to generate stack traces. The idea is that, whenever there’s an exception or a test failure, DUnit will show a stack trace right along with the failure message. The only problem is, it doesn’t work.

There’s no problem getting stack traces for unexpected exceptions. This is really useful; you get the address, unit name, method name, and even the line number of the line that threw an exception, together with the call stack that led to that code getting called. Immensely useful.

The problem is, you don’t get the same thing for test failures — even though Fail and CheckEquals, and even DUnitLite’s Specify.That, operate by throwing exceptions (they’ve got their own exception class, ETestFailure). You should be able to get a stack trace that shows the exact line of code that contained the failing assertion. In fact, we’re using older versions of DUnit and JCL at work, and we get stack traces just fine.

Unfortunately, stack traces for test failures are broken by default in the latest versions of DUnit and JCL. But there’s hope — notice that I said “by default”. Below, I’ll tell you how to fix that default.

Enabling DUnit stack tracing

First of all, here’s how to get DUnit to show stack traces in the first place.

You’ll need to download the source code for both DUnit and the Jedi Code Library. (Recent versions of Delphi ship with an older version of DUnit, but I only tested this with the latest version.)

Add the DUnit, JCL, JCL\Common, and JCL\Windows directories to your project’s search path.

Then make the following changes in Project > Options:

  1. On the Directories/Conditionals page, set “Conditional defines” to: USE_JEDI_JCL
  2. On the Linker page, set “Map file” to “Detailed”.

Now write a test that throws an unexpected exception, compile, and run. Here’s a contrived example, but it’ll give you an idea of what it looks like:

Screenshot of the DUnit GUI showing an exception stack trace

Enabing stack tracing for test failures

We don’t get unexpected exceptions in our tests very often. More often, it’s test failures. And when we have more than one assertion in the same test (not the ideal, but it happens a lot), sometimes it’s hard to know which assertion failed. Or rather, it’s always easy to know, if you have stack traces.

The problem is with JCL’s exclusion list. The latest version of JCL keeps a configurable list of exception types that it shouldn’t generate stack traces for. Seems like a reasonable feature. But the JCL and DUnit teams made three important design decisions, at various points in time:

  1. JCL’s exclusion list, by default, contains one class: EAbort.
  2. JCL ignores not just the classes in the exclusion list, but any of their descendant classes as well.
  3. DUnit’s ETestFailure descends from… yep, you guessed it, EAbort.

Put all three together, and stuff doesn’t work.

But, all that said, it’s easy to work around. Just add the following code to your project file, before you call one of the RunRegisteredTests routines:

uses
  ...,
  JclDebug,
  ...;

begin
  ...
  JclDebug.RemoveIgnoredException(EAbort);
  ...
end.

And bask in the goodness that is stack traces.

DUnitLite 0.4: bugfixes and Should.ReferTo

Version 0.4 of DUnitLite is available. It fixes a couple of fairly major bugs (noted below) and adds a minor new feature.

Download it here: DUnitLite 0.4

What is DUnitLite?

For those who don’t remember, DUnitLite is my set of add-ons to DUnit to make assertions more readable, in the spirit of NUnit’s Assert.That(...) assertions (originally from NUnitLite). For example:

Specify.That(Foo, Should.Equal(45));
Specify.That(Bar, Should.Not.Equal(45));
Specify.That(Baz, Should.Be.AtLeast(40));
Specify.That(Quux, Should.Be.OfType(TSpecialQuux));

What’s new in 0.4

  • BUG: When you used TInsulatedTest (or TSpecification), DUnit was unable to count the number of passing tests — it just reported 0 tests run. Fixed.
  • BUG: When you used TInsulatedTest (or TSpecification), the test-suite nodes were labeled “TInsulatedTest” instead of the actual class name. Fixed.
  • NEW: Added Should.ReferTo(...) (reference equality check). Currently only works for objects, not interfaces.

Score one for dogfood

I’m using DUnitLite to write the tests / specifications for Tyler, so I’m quickly running into its bugs and limitations. This is the first time I’ve seriously tried to use DUnitLite in a project (its use of records-with-methods would reliably crash the pre-SR1 Delphi 2006 compiler we had at work at the time, and I never got back to it), so this is the first time I’ve actually noticed these gaping holes.

Fortunately, I wrote DUnitLite, so it’s really easy for me to fix the bugs. Don’t be surprised if you see a few more releases in the near future.

New DUnitLite feature: Test insulation with TInsulatedTest and TSpecification

DUnitLite 0.3 (download) has a new feature: test insulation. Test-case instances live only as long as it takes to run them.

Here’s a quick illustration of the difference. Given this test case:

type
TestFrabjulizer = class(TTestCase)
published
procedure TestFoo;
procedure TestBar;
procedure TestBaz;
end;

With out-of-the-box DUnit, you’d see something like this:

  • Three instances of TestFrabjulizer are created, one for each test.
  • The DUnit GUI is displayed.
  • You select some tests and press Play.
  • The existing TestFrabjulizer instances are executed in turn.
  • You close the program. The TestFrabjulizer instances are freed.

But if instead of TTestCase, you descend from DUnitLite’s new TInsulatedTest, the behavior looks like this instead:

  • The DUnit GUI is displayed.
  • You select some tests and press Play.
  • An instance of TestFrabjulizer is created for TestFoo, executed, and freed.
  • An instance of TestFrabjulizer is created for TestBar, executed, and freed.
  • An instance of TestFrabjulizer is created for TestBaz, executed, and freed.
  • You close the program.

This scheme has several benefits over DUnit’s default behavior:

Interfaces work the way you expect. You can put interface-reference fields on your test case, and they’ll be automatically freed when the test is done — just like the way you’re used to them getting freed when you’re done using any other object.

TestFrabjulizer = class(TTestCase)
strict private
FFrabjulizer: IFrabjulizer;
...
end;

procedure TestFrabjulizer.SetUp;
begin
inherited SetUp;
FFrabjulizer := TFrabjulizer.Create;
// With TTestCase, you need to nil FFrabjulizer in TearDown.
// With TInsulatedTest, you don't: it'll go away automatically.
end;

Inline records and arrays are practical. Much like the previous item, but instead of IFrabjulizer, use TMyBigRecord or array [0..3] of array [0..1023] of Double. With TTestCase, this is a quick way to make your test app run out of memory, especially if you have thousands of tests like we do. With TInsulatedTest, it’s a handy way to simplify your test: no mucking around with GetMem and FreeMem, and the memory is automatically initialized to all zero bytes.

Tests can’t interact with themselves. If you run a test twice in a row, it gets clean instance variables each time, so there’s no chance of problems due to dangling state that you forgot to zero out in SetUp. (Of course, global variables can still screw you up.)

TSpecification

There’s a lot less to say here, but 0.3 also introduces a new TSpecification base class.

uses Specifications;

type
FrabjulizerSpec = class(TSpecification)
...
end;

TSpecification is really just TInsulatedTest, with very little added. Mostly this is to help stick with the “specification” mindset, rather than having a specification that descends from something with “test case” in its name.

TSpecification also, through a type-aliasing trick, makes Specify.That and Should... available to all of its descendants, without those descendant units needing to explicitly use the Specifiers unit. It’s kind of cute. (Specify and Should are, of course, still available outside of TSpecification.)

New DUnitLite feature: String context

When you’re comparing long strings, DUnitLite 0.2 (download) now shows just the context — the first character that’s actually different, plus 30 characters before and after — instead of the entire string. If there’s only a small difference buried in a really long string, this makes the difference much easier to spot.

Specify.That(Query.Sql.Text, Should.Equal(ExpectedSql));

Expected: ...'.TicketID = Tickets.ID'#13#10'WHERE Component = 42 AND Status = 1 A'...
but was: ...'.TicketID = Tickets.ID'#13#10'WHERE Tickets.Component = 42 AND Tick'...

Ellipses (...) are added at the beginning and/or end to show you when you’re seeing a portion of the value, instead of the whole string.

I’d like to do like NUnit does, and show another line with a little arrow ^ pointing to the first character that’s different. Unfortunately, DUnit’s test-failure output isn’t shown in a fixed-width font, so I wouldn’t have much way to get it to line up.

Download DUnitLite 0.2 and give it a spin, and let me know how this feature works for you. I expect it to take some tuning and tweaking before it works reasonably well in most circumstances, and there’s no better way to know what needs tweaking than to use it in the field.

New DUnitLite syntax: Should.Yield

No, I’m not adding Ruby-style blocks, more’s the pity. Just some syntactic sugar.

SpecifyThatInspecting('', Should.Yield(Apostrophes(2)));
SpecifyThatInspecting('a', Should.Yield(Apostrophe + 'a' + Apostrophe));
SpecifyThatInspecting(#13#10, Should.Yield('#13#10'));

The above samples are from the StringInspectorSpecs unit in DUnitLite. They specify parts of the contract for a method called Inspect, which takes a string and returns a string.

One of the requirements for TStringInspector is that when I call the Inspect method, and pass it #13#10, it should return '#13#10'. In other words,

Specify that inspecting #13#10 should yield '#13#10'.

As you can see, Should.Yield is nothing more than an alias for Should.Equal. Use whichever one makes your code most readable.

This is where I expect a lot of the work will be done with DUnitLite-style specifications: with custom assertion methods like SpecifyThatInspecting. Play with it and see what other readable assertions you can come up with. I’m certainly willing to tweak the framework, to add more things along the lines of Should.Yield, if it makes specifications more readable.

(The version with Should.Yield hasn’t been released as a ZIP yet. You can view StringInspectorSpecs.pas, browse all the latest source, anonymously check out the latest code, or wait for the next release.)

Update: This feature is included in the DUnitLite 0.2 download.

Correction: dSpec does have the optional Message parameter

In my last post, I incorrectly said that dSpec doesn’t have the Message parameter. Jody corrected me. Here’s the syntax:

Specify.That(FRate, 'Tax rate').Should.Equal(0.10);

I kinda like this syntax — it puts the label right next to the thing it’s labeling, rather than putting the label last like DUnitLite does now. What do you guys think? Which of the following syntaxes would be best for DUnitLite?

Specify.That(FRate, Should.Equal(0.10), 'Tax rate'); // (a)
Specify.That(FRate, 'Tax rate', Should.Equal(0.10)); // (b)
Specify.That('Tax rate', FRate, Should.Equal(0.10)); // (c)

Currently DUnitLite uses (a). (b) is more like dSpec.

The advantage to (c) would be when your expressions are long enough to line-wrap. If the label is likely to be shorter than the expression being tested (and, therefore, more likely to fit on the first line without wrapping), then this kind of formatting might work really well:

Specify.That('State tax rate',
TInvoiceCalculator.CreateForState(FCustomer.State).TaxRate,
Should.Equal(0.10));

But without wrapping, having a label as the first parameter might get in the way of readability. I’m not sure.

Which syntax seems most readable to you?

Readable behavior-driven tests for DUnit: DUnitLite

DUnitLite (formerly DUnitAssertions*) version 0.1 is now available. (downloads, documentation) It’s open-source (MPL).

DUnitLite is similar to dSpec, except for the placement of parentheses:

// dSpec:
Specify.That(TheAnswer).Should.Equal(42);
// DUnitLite:
Specify.That(TheAnswer, Should.Equal(42));

But I think my approach has a few advantages (which is why I kept working on my version even after I found out about dSpec). As I see it, DUnitLite currently has two things one thing going for it that dSpec doesn’t have:

  • Support for an optional “message” parameter, so that if a test does more than one Specify, you can tell which one it failed on. Just like the optional third parameter to DUnit’s CheckEquals(). dSpec doesn’t currently support this, and it’s not clear (to me) how its syntax could even be extended to support it. Correction: dSpec does support the “message” parameter.

  • Support for enums and records. Okay, “support” is a strong word; you have to write code to support the types you care about — but not much. In DUnit or dSpec, you’d have to write lots of methods to make this work (trust me, I’ve done it many times with DUnit). But in DUnitLite, the duplication has been pulled out. Once you write the code once, the new type is supported everywhere: Specify.That(), Should.Equal(), Should.Not.Equal(), Should.Be.GreaterThan(), Should.Not.Be.Between(), etc. dSpec’s current class structure would make this more work. I think.

There’s also an interesting issue with dSpec’s .Unless() syntax, which can’t be ported to .NET because it relies on a clever deterministic-finalization hack. And if I implemented the same feature in DUnitLite, I think it would work just fine in .NET, just because of the difference in where the parentheses go. But that’s a story for another day, when I haven’t stayed up until after 5:00am trying to release a project.

So anyway. DUnitLite is out there, and it even has a little bit of documentation. It’s an early version (it’s just whatever I had working when I finally got the intellectual-property waiver back from my employer this week), but all of the supported syntax and types should be fully working. Go check it out, and let me know what you think. Enjoy!

* The reason for changing the project’s name is explained in another footnote. “DUnitLite” isn’t that much better, but at least it hasn’t got “Assert” in its name.

Avoiding sentinel values

A “sentinel” is any value that means something special. For example, Math.SameValue takes an epsilon parameter, but zero means “pick an epsilon for me”. So a value of zero doesn’t actually mean zero.

Sentinel values, I have come to realize, are a code smell. They’re saying, “here’s a parameter that has more than one responsibility.”

Last night, I was working on epsilons in DUnitAssertions. TValue, my universal value type, can compare two numbers with an epsilon, and it follows the SameValue convention of “zero means ‘pick for me'”. But I wanted to add another option:

Specify.That(Foo, Should.Equal(45.0).Exactly); // no epsilon

Really, this should pass zero as its epsilon, but that’s already spoken for. So I picked another sentinel (NaN) to mean “exact comparison”, and started writing the tests and making them pass.

It actually took a few minutes before I realized how ridiculous this was. I mean, the body of the method had three completely different code paths, depending on the sentinel values and nothing else. Hello? Polymorphism!

So I’m replacing the epsilon with a comparer. I’ve made an IValueComparer interface with a CompareExtendeds method, and it’s going to have several different implementations. One is the default, which picks an epsilon for you (and never takes an epsilon parameter at all). One is the epsilon-based comparer, which takes an epsilon in its constructor. I probably don’t even need another class for the “exact” comparer, since an epsilon of (really and truly) zero will serve quite nicely. And the sentinel logic will all go away.

I’ll even put another method on IValueComparer for comparing strings, and have a class that does case-insensitive comparisons. I’d been wondering how I was going to plumb the case-sensitivity option into the comparison, and now I know. (Since no single call to Specify.That will compare both a string and a float, this doesn’t pose any duplicate-inheritance-hierarchy problems.) And this will address that nagging doubt I’ve felt all along about passing a numeric epsilon even when I’m comparing strings. That’ll go away entirely; I’ll just be passing a comparer.

Now that I think about it, this is exactly the Strategy pattern. I’m refactoring to patterns and not even realizing it. Cool!

One thing that is nice about sentinel values — as opposed to, say, making several well-named methods — is that sentinel values are easy to plumb through several layers of code. But a strategy object has the same benefit, and it’s more expressive.

So when you see a sentinel value, ask yourself whether a strategy would be better. You might be really pleased with the result.

Footnote: Almost five years ago, I wrote a huge YAGNI at work. It was a function that divided one number by another. That’s it, really. But it took three, count ’em, three optional parameters (that’s five total parameters to divide two numbers), and those three optional parameters were overloaded to bursting with sentinel values. I pulled out all the stops: not just positive and negative infinity, but NaN as well, all had special meanings. Meanings that you could kind of puzzle out, sure, but they were sentinels nonetheless. But it was worth it (so I thought at the time) because this thing was the ultimate in flexibility. It could clamp its results to a range, it could throw exceptions or not, it could return special values when you divided by zero, it even made Julienne fries.

How many lines of code does it take to divide two numbers? Forty-nine. All but nine lines of that was just there to check for sentinels. Those other nine lines were the ones that set the function’s return value.

I looked today. Total number of places that actually called this function? One. Another utility function in the same unit, which I had made less readable when I introduced my YAGNI to it five years ago.

So I did some spring cleaning. And that unit is shorter now.

DUnitAssertions goes behavioral

The more I look at dSpec, the more I like its syntax. Should.Equal and Should.Be.AtLeast are much better than my Tis.EqualTo and Tis.GreaterThanOrEqualTo.

And I like the behavior-driven bent: encouraging the coder to think in terms of specifications (“specify that it should do this”) rather than tests (“test to see if it does this”). I was actually leaning a bit in that direction already — that’s why I picked Expect instead of Check or Assert — but I like Specify better yet.

So I’m stealing it.

None of these are implemented yet (I still need to rip out the operator overloads and do a bit more refactoring before I can move on to this), but here’s how I see the new syntax for DUnitAssertions*:

// All specifications support messages:
Specify.That(Foo, Should.Equal(45), 'Frequency');

// All specifications support Should.Not:
Specify.That(Foo, Should.Not.Equal(45));

// Floating-point comparisons support "ToWithin":
Specify.That(Foo, Should.Equal(45.0)); // default epsilon
Specify.That(Foo, Should.Equal(45.0).ToWithin(0.00001));
Specify.That(Foo, Should.Equal(45.0).Exactly); // no epsilon

// Equality and inequality:
Specify.That(Foo, Should.Equal(45));
Specify.That(Foo, Should.Be.AtLeast(45));
Specify.That(Foo, Should.Be.AtMost(45));
Specify.That(Foo, Should.Be.From(45).To(48).Inclusive);
Specify.That(Foo, Should.Be.From(45).To(48).Exclusive);
Specify.That(Foo, Should.Be.GreaterThan(45));
Specify.That(Foo, Should.Be.LessThan(45));

// Type:
Specify.That(Foo, Should.Be.OfType(TComponent));
Specify.That(Foo, Should.DescendFrom(TComponent));
Specify.That(Foo, Should.Implement(IBar));

// String:
Specify.That(Foo, Should.Contain('Bar'));
Specify.That(Foo, Should.StartWith('Bar'));
Specify.That(Foo, Should.EndWith('Bar'));

// All string comparisons support IgnoringCase:
Specify.That(Foo, Should.Equal('Bar').IgnoringCase);
Specify.That(Foo, Should.Contain('Bar').IgnoringCase);
Specify.That(Foo, Should.StartWith('Bar').IgnoringCase);
Specify.That(Foo, Should.EndWith('Bar').IgnoringCase);

// Numeric:
Specify.That(Foo, Should.Be.Composite);
Specify.That(Foo, Should.Be.Even);
Specify.That(Foo, Should.Be.Negative);
Specify.That(Foo, Should.Be.Odd);
Specify.That(Foo, Should.Be.Positive);
Specify.That(Foo, Should.Be.Prime);

// Specific values:
Specify.That(Foo, Should.Be.False);
Specify.That(Foo, Should.Be.Nil);
Specify.That(Foo, Should.Be.True);
Specify.That(Foo, Should.Be.Zero);

* If it’s going to be for behavior-driven development instead of test-driven development, then it really needs a new name now…