Visual Studio 2005: Still wondering where the unit-testing tools are
We did manage to trade in our Visual Studio Architect licenses, and get Developer instead. So I sat down to play with the unit-testing tools.
Verdict? The unit-testing support in Visual Studio 2005 is a toy. Nothing more.
Why do I say that? Because they let you write unit tests, but they don’t let you run them.
The unit-testing support is all part of Visual Studio, not of the .NET Framework itself. So that means you can only run your tests inside the Visual Studio IDE. Meaning, you can’t run your tests from the command line. Or as part of an automated build; there’s no way to tie the tests into MSBuild or Ant or NAnt or anything else. If you want to know that your latest official build actually passes all the tests, you have to open up Visual Studio so you can run the tests manually.
Correction: Apparently there’s a command-line tool called MSTask that lets you run the tests, so apparently you can run the tests from an automated build. Doesn’t help with running the tests as you develop in the IDE, though.
And you have to select the tests manually. There is no way to tell the IDE, “Run all the tests.” Oh, you can select a test from a list, and then click a toolbar button to run it. You can even Shift+click and Ctrl+click to select multiple tests, and then click the toolbar button to run them. But if you want to run all the tests, you have to manually select them all in the list. When you write a new test, you have to manually select it, too. And it doesn’t remember your selections the next time you start Visual Studio.
Don’t we use computers because they let us automate things? Just wondering.
You could, alternatively, hit Ctrl+Shift+X, which runs all the tests in the current assembly. But that only works if the code editor is currently on a test file. Meaning, you can write your test, and hit Ctrl+Shift+X to watch it fail; but then when you go edit your production code to make the test pass, and hit Ctrl+Shift+X again, it does nothing. Why, after all, would you want to run a test you’re not editing?
Why, indeed. Good God, what the hell is wrong with these people?
The most fundamental operation for unit tests is running all the tests. That’s the point. Yes, it can be useful to run a subset of the tests, but that’s a feature you add later. The first thing you write in any testing framework is “Run All Tests”. Because it’s something you do dozens of times a day. You do it all the time, because every time you change your code, you want to know you didn’t just break something. If you’re only running a single test after each change, and you get to the end of the day and find out you broke some other test, you have no idea when you broke it. Which is why you don’t run a single test. You run all the tests, unless you have a really good reason not to.
They just don’t get it. They give you tools for writing everything but unit tests. They give you tools for writing database tests, and ASP.NET server tests. And sure, you don’t want to run those dozens of times a day, because they’re slow, and you’d never get anything done. But unit tests are fast. Any unit test worthy of the name is only going to test your code — not the database, not the server, ideally not even the filesystem. A unit test takes maybe a few microseconds to run. You can run thousands of them at a time, dozens of times a day. It can be a lot of work finding a way to test your code without using the database, but it’s usually worth it, because it gives you the chance to run the test All. The. Time.
Unless, of course, you’re using Microsoft tools.
It’s ridiculous. Team System comes with this fancy new version-control system, when you can get a fabulous tool like Subversion for free. It comes with something Microsoft Marketing claims is a unit-testing tool, but it turns out we’re better off sticking with csUnit.
Team System is just looking more and more like a complete waste. Score another point for open source.
January 17th, 2006 at 4:04 pm
Do you think perhaps Microsoft doesn’t know these basic facts about Unit Tests, becuase they don’t perform Unit Tests themselves?
January 20th, 2006 at 3:18 pm
> The most fundamental operation for unit tests is running all the tests. That’s the point. Yes, it can be useful to run a subset of the tests, but that’s a feature you add later. The first thing you write in any testing framework is "Run All Tests". Because it’s something you do dozens of times a day. You do it all the time, because every time you change your code, you want to know you didn’t just break something. If you’re only running a single test after each change, and you get to the end of the day and find out you broke some other test, you have no idea when you broke it. Which is why you don’t run a single test. You run all the tests, unless you have a really good reason not to.
I think this might be because they’re really targeting TESTERS– let’s say offshore testers– rather than developers with this feature.
Definitely a legitimate criticism, but I wonder if the focus is a little different than something like NUnit.
February 9th, 2006 at 3:00 pm
try MSTEST.exe
http://msdn2.microsoft.com/en-us/library/ms182489.aspx
I’m not sure which edition MSTEST comes with.
Maybe you need to buy a "testers edition"?
February 26th, 2006 at 4:26 pm
This article looks like it could hold a lot of answers re VSTS unit tests, MSTest.exe, etc:
http://msdn.microsoft.com/msdnmag/issues/06/03/Bugslayer/default.aspx
Cheers,
Pete.
June 6th, 2006 at 5:38 pm
Running all the unit tests from Visual Studio
June 13th, 2006 at 12:58 pm
Hi,
On the Test toolbar there is button called ‘Test View’ it shows you all the tests in another little window. You can then right click and select all and then do a Run selected to run all the tests.
I’m kinda new but this might be something you already know.
Ming
June 14th, 2006 at 5:50 am
Sure, it’s possible to manually run selected tests. What’s painful is running ALL the tests. You have to manually open up that window, and manually check every test, every time you open Visual Studio. Pretty cumbersome and error-prone for something you do dozens of times a day.
January 26th, 2007 at 3:20 pm
You can set your test project as a startup project and the just hit Run and all tests will be run.
February 5th, 2007 at 6:09 am
ReSharper, day 5: Integrated unit-test runner
August 11th, 2007 at 7:42 am
What a poorly-researched and knee-jerk rant!
My team is happily running VSTS unit tests from our automated command-line MSBuild, without a GUI, including getting full code coverage information out.
It’s really not very hard to discover the mstest.exe command-line, and this is available in the Developer Edition.
We did end up writing a (small) MSBuild task to get the coverage out as XML in the format we wanted, but all the APIs are clearly there to allow you to do this however you want. And we only had to do this because we couldn’t move to the TFS server because we couldn’t switch away from our existing SCM system.
It’s not without its faults but it doesn’t deserve the lazyily-researched invective you pour out in this article.