TechEd 2008 notes: Lessons Learned in Programmer (Unit) Testing

This one was a mixed bag. Some of his lessons learned are worth looking at, like the 3A pattern, and he covered classics like inversion of control and avoiding ExpectedException. But he was pretty dogmatic about a couple of things that just didn’t make much sense.

Lessons Learned in Programmer (Unit) Testing: Patterns and Idioms
James Newkirk, CodePlex Product Unit Manager

Been writing software for close to 25 years.
When he was first hired after college:

  • Company made EEs work in manufacturing for 2 years
    • Goal: teach them how to build things that could be manufactured
  • Software people just started writing stuff
    • Never taught how to build things that were testable, maintainable. Up to them to discover that.

JUnit Test Infected: Programmers Love Writing Tests
(Has anyone actually met programmers who love writing tests?)

  • Programmers weren’t writing tests
  • Couldn’t prove that what they had worked
  • James ported JUnit to .NET as NUnit, released as open-source
  • How can we use open-source to enable different kinds of innovation?

“Unit testing” means something to testing people. Some think, if cyclomatic complexity is 5, then I need 5 unit tests. Not the same thing as tests for test-driven development. So we’ll probably not use the term “unit test”.

What is programmer testing? Per Brian Marick

  • Technology vs. Customer
  • Support vs. Critique
  • Support + technology: Programmer tests
  • Support + customer: Customer tests
  • Critique + customer: Exploratory tests (should still happen during the development process)
  • Critique + technology: “ilities” — non-functional. Scalability, usability, performance, etc.

Why do programmer testing?

  • “There is no such thing as done. Much more investment will be spent modifying programs than developing them initially.” [Beck]
    • Whenever we think we’re finished, we’re probably wrong. There can always be more features.
    • Can define “done” locally: “We’re done with this release.”
    • Programmer testing lets me say as a programmer, “I believe that I am done.”
  • “Programs are read more often than they are written.” [Beck, “Implementation Patterns” book]
    • You’re writing a book on your code.
    • The only artifact of any value is the code. It has to be the communication mechanism.
  • “Readers need to understand programs in detail and concept.” [Beck]

Total development cost:

  • Develop (illustrated in green): small. Fun!
  • Extend/Maintain (orange): big. Sucks, especially if you didn’t write the green part!

“I might break something!”

  • Fear lowers our productivity
    • Programmer tests help, because if you broke something, you know. You know the consequences of your actions.
  • Where do I start?
    • Programmer tests are very useful in this understanding phase

Lesson #1: Write tests using the 3A pattern

  • Attributed to Bill Wake (
    • Arrange — Set up the test harness
    • Act — Run the thing you actually want to test
    • Assert — Verify the results
public void TopDoesNotChangeTheStateOfTheStack()
    // Arrange
    Stack<string> stack = new Stack<string>();
    // Act
    string element = stack.Top;
    // Assert
  • Benefits
    • Readability
    • Consistency
  • Liabilities
    • More verbose
    • Might need to introduce local variables
  • Related issues
    • One assert per test?
      • Much agile dogma says there should only be one assert
      • Pragmatic view: test should only test one thing, but if that one thing takes multiple asserts, that’s fine

Lesson #2: Keep your tests close

  • Tests should be as close as possible to the production code
  • Same assembly?
  • Treat them like production code
  • They have to be kept up-to-date
  • Visibility (let you see internal)
    • (I would argue that, if you think you need to test your internal state, what’s really going on is that you’ve got another object that wants to get out.)
  • Liabilities
    • Should you ship your tests?
      • If so, you need to test your tests. Whole new level.
      • Need to make sure your tests don’t modify the state of the system
      • If No, how do you separate the tests from the code when you release?
    • Various tool issues

Lesson #3: ExpectedException leads to uncertainty

public void PopEmptyStack()
    Stack<string> stack = new Stack<string>();
  • Problem #1: where’s the assert? A: it’s in the framework. Violates 3A.
  • Obfuscates the test to some extent.
  • Better:
public void PopEmptyStack()
    Stack<string> stack = new Stack<string>();
    Exception ex = Record.Exception(() => stack.Pop());
  • Makes the exception more explicit.
  • Lets you inspect the exception object.
  • Another possibility is NUnit’s Assert.Throws():
public void PopEmptyStack()
    Stack<string> stack = new Stack<string>();
    Assert.Throws<InvalidOperationException>(delegate {
        stack.Pop(); });
  • Downside: Act and Assert are in the same spot.
  • Other problems with ExpectedException:
    • Don’t know which line of code threw the exception. Test can pass for the wrong reason. (This, IMO, is the most compelling reason to avoid ExpectedException.)
    • ExpectedException forces people to use TearDown. With Record.Exception or Assert.Throws, you can keep the teardown in the test if you like.

Alternatives to ExpectedException

  • Benefits
    • Readability
    • Identify and isolate the code you expect to throw
  • Liabilities
    • Act and Assert are together in Assert.Throws
    • Anonymous delegate syntax leaves something to be desired

Lesson #4: Small fixtures

  • Your code should be a book for people to understand
  • If the fixture has 1,000 tests, it’s hard for someone to know where to start
  • Better: fixtures focused around a single activity
    • Maybe even a fixture for each method
    • Can use nested classes for this. Outer class is named for the class you’re testing, with nested classes for each behavior.
  • Benefits
    • Smaller, more focused test classes
  • Liabilities
    • Potential code duplication
      • May consider duplicating code if it’s for the purpose of communication (but use this carefully)
    • Issues with test runners — not all can deal with nested classes
  • Related issues
    • Do you need SetUp and TearDown?

Lesson #5: Don’t use SetUp or TearDown

(I smell a holy war)

  • If the tests are not all orthogonal, you can end up with a test that doesn’t need anything from the SetUp method.
  • If SetUp takes a long time, you’d be paying the price for every test, even those that don’t need all of it.
  • If there are tests that don’t need all of SetUp, readability suffers.
  • It’s a hacking point. It’s a place where people can add code that you might or might not like.
  • When asked about the duplication, he suggested pulling the duplicated code into a method, and calling it at the beginning of each test. That wouldn’t necessarily be a totally bad idea, if SetUp was the only thing he was trying to get rid of.
  • Benefits
    • Readability (if it removes duplication)
    • Test isolation (but only if it means you get rid of all fixture state, and put absolutely all your state into local variables)
  • Liabilities
    • Duplicated initialization code
    • Things he didn’t mention, that I will (having run into all of them):
      • Duplicated cleanup code
      • Try..finally in every test to make sure your cleanup code gets run
      • Signal/noise ratio: if a test gets too long, it’s much harder to tell what it’s actually trying to accomplish
      • Poor test isolation: if you have any fields on your fixture class, you will forget to initialize some of them in some of the tests, which will introduce test-order dependencies (since NUnit shares the same fixture instance among all the tests in the suite)
  • Related issues
    • Small fixtures

Lesson #6: Don’t use abstract base test classes

He’s apparently specifically referring to the practice of putting tests on a base class, and inherit them in descendants. We use this technique, though sparingly; you do take a big hit in readability, so it really has to be worth having the same tests (including newly-written tests) apply in more than one place.

What’s wrong with base classes?

  • If the tests don’t all apply to all the derived classes, you’d have a problem. (Um, duh. You’d notice when they failed, wouldn’t you?)
  • Removes duplication at the expense of readability. (Fair point.)
  • He wants us to put the test logic into a utility function instead, and call that from the different descendants. (Trouble is, that doesn’t help make sure that all new tests get added to all the fixtures. You don’t use this pattern unless you want all the tests in all the descendants.)
  • Benefits
    • Readability
    • Test isolation
  • Related issues

Lesson #7: Improve testability with Inversion of Control

  • Martin Fowler article: Inversion of Control Containers and the Dependency Injection pattern
  • Dependency Injection
    • Constructor injection
    • Setter injection
  • Don’t want errors to cascade across all parts of the program.
  • Benefits
    • Better test isolation
    • Decoupled class implementation
  • Liabilities
    • Decreases encapsulation
    • Interface explosion
  • Related issues
    • Dependency injection frameworks are overkill for most applications

Side note: Mocks are good for interaction testing, bad for state testing.

Leave a Reply

Your email address will not be published. Required fields are marked *