The team I'm currently working with has spent a lot of money buying MS's team suite and so naturally we wanted to use MSTest (AKA Team System Unit Testing). In theory using MSTest allows you to simply integrate tests into your build process and generate coverage reports etc. Unfortunately the grim reality is somewhat different.
Unit test tools have been around for several years. I've been using NUnit on a daily basis for around four years. I've never really paid much attention to it because it's one of those tools that just works; you write your tests, attribute them as [Test] and they either pass or fail. I keep hearing good things about MbUnit, but I haven't tried it yet because I feel no pain with NUnit and its constant companion, the excellent TestDriven.net. In simple terms, this is how NUnit/TestDriven.net works:
- Attribute any class with TestFixture and any void-void method with Test. Use Asserts to check my expectations.
- I have F8 mapped to run tests, so I hit F8 and TestDriven points NUnit at the test assembly in bin/debug folder (or wherever the project's output path points to).
- NUnit loads the assembly and executes any method attributed with Test and outputs the results to the output window.
And that's it. If my tests need any attendant configuration or need to load any assemblies dynamically I just make sure they are copied to the output path.
As I said before NUnit has been around for a while and MbUnit does the same stuff but with some extra bells and whistles. Any unit testing framework should work in a similar way and should be as simple to use as possible.
Enter MSTest. Now you would have thought that with the great success of NUnit to copy, even Microsoft couldn't make too great a hash of it. But they have. Two major faults make it an unsuitable tool for running unit tests:
- The test runner does not simply reflect over an assembly and run any method attributed with TestMethod, instead you have to create a special test project and your tests have to be present in an XML file in order for the test runner to find them. Why? Why should we need a special test project? There's no conceivable reason I can see. It means you have to have your tests in a separate assembly than the code under test. Now it's my practice to do this, but you shouldn't be forced to do it. Worse though is that it's just too easy for the XML file to get out of sync with the tests and for your tests to mysteriously not get executed. That's not to mention the source control nightmare it causes.
- The test runner runs the test assemblies in some other place that's different each time. Now you can't rely on an assembly manifest to tell you everything that some code might need to run. I'm a big fan of IoC containers which means that much of my application's object graph gets loaded at runtime. I get frequent test failures because my tests can't find assemblies they need to load. Configuration is a similar issue, miscellaneous files don't get copied to the test run location. Now I know that you can do add attributes to every single test or add stuff into the test configuration file to get the test runner to copy certain files to its run directory, but why should I have to do this? Also It doesn't clean up after itself and I run tests every minute of every day, this means it builds up not an insignificant amount of crud on my disk.
I'm just going to mention in passing that the MSTest Asserts are pre-historic and look a bit like NUnit version-a-long-time-ago and that the default unit test project is full of unnecessary files and that the unit test template generates a page of crud that you have to delete before you can start work. That's no so important and I could live with it if the core design wasn't so FUBAR.
MSTest looks like a tool that was designed by someone who'd never really done much unit testing. They'd certainly not worked with NUnit to any degree. I really can't understand the thought process that resulted in such bad design decisions, but it looks like it's designed to be run once a week with an great deal of ceremony. In other words MS have totally misunderstood TDD.
So now I'm lobbying hard to get us to drop MSTest and use NUnit or MbUnit instead. In fact experience with the whole team system is that it's not worth spending the money, you get far better results and much less stress from using a combination of open and third party tools. But that's another post...
Posts
21 comments:
Sorry for the language Mike, but I think MStest is dogs poo. Been where you are and am still battling on a daily basis.
The most important thing to remember is that VSMDI (metadata) is not the be all of it. You can use TestContainers (just run MSTest.EXE /?). Have a look at http://preetsangha.blogspot.com/ for some stuff from my daily batle.
Hi Preet, you've got some great stuff on your blog. It looks like your battle with MSTest has been a lot longer and harder than mine. I'm still hoping I can persuade our team to use NUnit or MbUnit instead. I like your suggestion about using MSTest from the command line and just pointing it at assemblies. Unfortuantely, at the moment my main issue is with the VS test runner. I've been writing tests and running them with testdriven.net, everything's fine, but when the rest of my team try to run them with the builtin test runner and there are failures all over the shop.
Thanks Mike - no where near the level of your posts.
If you can get the team to move do so - I would recommend it. The rapid test-code-test cycle benefits will be self evident.
I've nothing against MS stuff in general, but mstest really is a trial. On a more specific note I did some experimentation with VSMDIs and though it is possible to create these manually its not trivial as thay a have special - not quite UUID - guid in them. For VS test runner I've got into the habbit of checking out (with exclusive lock) the VSMDI before writing tests. This stops VS from creating the wonderful xxx1.vsmdi and then ...2.vsmdi files and messing with your solution files. But of course you lose the ability to allow multiple people in the same test file.
For the case of the deployment directories, you can switch this off (inside the localtestrun.config file), but doing so loses you the code coverage.
Heads you lose tail MSTest win!
On more positive note I think the ability to do bigger tests integrated into Team System is a strong point on MSTest, but we've not got the QA team to do that yet - so I cannot verify it.
Preet, it looks like I've managed to persuade our team to not use MSTest. It looks like we'll be moving to MbUnit. We've just been changing the tests over this morning and it's already a lot more sane, for a start we can the Resharper test runner.
My colleage Iain described our experiences with MSTest as 'the onion of compromise'. You start out with one bad move (deciding to use MSTest) and then make one compromise after another to try and keep it working. As we've been switching to MbUnit we've been able to strip out a ton of 'compromise code'. Nice!
'the onion of compromise' - love it. This is our MS Test in our shop. Anyway sounds like you had a busy morning. I think you've made the right choice! Out of interest why MBUnit? Why not say XUnit?
Preet,
MBUnit is a framework, xUnit is the concept.
I have the book xUnit test patterns but I have yet to read it!
Glad we have stuck with NUnit now (though I have heard lots of good things about MBUnit).
Richard, I've got that book. It's a framework too though:
http://www.codeplex.com/xunit
Didn't know that. What did you think of the book?
I know I'm late to the game, but mileage is everything.
I used to use NUnit, but after using MSTest for the past 2.5 years, I wouldn't go back, even with the bugs in VS2008 (which are minor and only hit if you are doing something specific with AppDomain).
Your config file thing is simple - you don't have to use attribute...in the 'settings' (right-click properties) add files to deploy.
It creates a separate test project because too many Morts out there will just put it in the same project otherwise and all your tests will be deployed with the code. I've worked with a number of these folks.
Having the built in ability to test private and internal methods/properties/fields is the biggest reason I love it. No extra code required. No need to loosen the scope (make things public) on what you're testing.
When a test fails, looking at the test results that gives me a hyper-link to every part of the stack trace where the test failed (or threw an exception) as saved me hundreds if not thousands of hours by now not having to navigate to the file, hit CNTL+G and enter the line number; especially if I have to walk up the stack to see if there was a path taken that shouldn't of been causing the problem.
Having the ability to have tests run automatically when I do a build or check-in (easier than I was ever able to with NUnit - but that's a SCC).
Code coverage gets tied to the tests you write.
Pulling up archives into the GUI of who ran what tests when on what machines. Great from the 'team lead' perspective.
You said in another post that technologies change but methodologies don't - don't let NUnit suck you into the 'one tool' mentality (although I could say the same about myself with MSTest). Love your posts generally - but I think you need to give MSTest another chance and ask someone who's been successful with it for help. It was hard for me to switch from NUnit too - but it was like when I switched from VB to C# - I never looked back once I got past the frustration point.
In response to woonboo, one of the biggest reasons I dislike MSTest is for the feature you like: Testing private code.
Except in the case of legacy code, I have found that testing privates violates encapsulation properties way too much. If you can't get to your code through a public method/property, then the code is unreachable through the clients also.
Internally scoped code can just be made available to the test assembly via "friend assemblies".
I would prefer that my testing tool not encourage me to write code that needs private code to be tested directly.
Just my 2 cents.
MSTest is crap for TDD and the folks who wrote it know it. It wasn't designed for people who practice TDD, it was designed for Microsoft-like teams who have a dev team and a QA team.
MSTest is designed for the QA team.
Or course we don't have one of them either so my impression so far is about the same. MSTest == poop
"Now it's my practice to do this, but you shouldn't be forced to do it."
So you are complaining that MSTest is forcing you to do something... that you were going to do anyway. Jeez, whine much over nothing? ;-)
We're having problem number two. MSTest is attempting to run dyamically referenced assemblies from the folder that the MSTest.exe executable is being run from. In my case this is:
C:/Program Files/Microsoft Visual Studio 9.0/Common7/IDE/
We've verified this by checking the AppDomain.CurrentDomain.BaseDirectory during the execution of the test.
If anyone knows a work-around please post it, you would save me quite a bit of aggravation.
The other problem is this VSMDI file ... this file constantly checks itself in and out everytime a test runs. Then it starts replicating itself like some kind of virus. We are managing this file under TFS ... I've read that this may not be the best idea.
Mike, I hope you don't mind me posting a link to my solution to this. See Here
Preet, thanks for helping out. Of course you very welcome to post links to helpful stuff anytime :)
Preet ... thanks very much that is just what I was looking for. I really really didn't want to create something that dumped all of the files needed into that IDE folder and then clean them up after the tests had run.
We'll give that a try. There is actually talk in my organization about moving away from MSTest to NUnit or Spec Unit due to the amout of problems that we're having.
I now have 4 extra VSMDI files in my solution directory ... it looks something like this:
Project.vsmdi
Project1.vsmdi
Project2.vsmdi
Project3.vsmdi
Project4.vsmdi
Every time I touch a test list this file is automatically checked out, a new one is created and then the IDE asks me if I want to merge
!!!
Ok the thing to do is delete the extra VSMDIs in the Solution Explorer. Then edit the VSMDI to include the tests you've just created.
>> moving away --- I hear rumours VS2010 will have support for XUnit - might want to consider that.
Good post. Amusing factoid: a Bing search for "MSTest is sapping my will to live" (unquoted) doesn't return this page, while a Google search for the same returns it as the first result. Searching for the quoted string returns this post as the first result in both engines.
(Note: I stopped looking through the Bing results after the first 10 pages.)
Thanks for that Stack. In fact this post is currently the 3rd result for 'MSTest' in Google :)
What's this 'Bing' thing you talk of?
Another problem with MSTest i faced is that it does not support inheritance fully. See http://blogs.msdn.com/vstsqualitytools/archive/2007/05/01/known-issues-and-workarounds-for-orcas-beta1-inheritance.aspx
-- Can someone post a work around??
Thanks,
Subbu
Post a Comment