More on MSTest

I recently had a long and considered comment from Woonboo for my post "MSTest is sapping my will to live". Woonboo is a happy user of MSTest. I started writing a reply in the comments, but it was getting so involved I thought it would be nice to promote it to a new post.

Here's Woonboo's comment:

I know I'm late to the game, but mileage is everything.

I used to use NUnit, but after using MSTest for the past 2.5 years, I wouldn't go back, even with the bugs in VS2008 (which are minor and only hit if you are doing something specific with AppDomain).

Your config file thing is simple - you don't have to use attribute...in the 'settings' (right-click properties) add files to deploy.

It creates a separate test project because too many Morts out there will just put it in the same project otherwise and all your tests will be deployed with the code. I've worked with a number of these folks.

Having the built in ability to test private and internal methods/properties/fields is the biggest reason I love it. No extra code required. No need to loosen the scope (make things public) on what you're testing.

When a test fails, looking at the test results that gives me a hyper-link to every part of the stack trace where the test failed (or threw an exception) as saved me hundreds if not thousands of hours by now not having to navigate to the file, hit CNTL+G and enter the line number; especially if I have to walk up the stack to see if there was a path taken that shouldn't of been causing the problem.

Having the ability to have tests run automatically when I do a build or check-in (easier than I was ever able to with NUnit - but that's a SCC).

Code coverage gets tied to the tests you write.

Pulling up archives into the GUI of who ran what tests when on what machines. Great from the 'team lead' perspective.

You said in another post that technologies change but methodologies don't - don't let NUnit suck you into the 'one tool' mentality (although I could say the same about myself with MSTest). Love your posts generally - but I think you need to give MSTest another chance and ask someone who's been successful with it for help. It was hard for me to switch from NUnit too - but it was like when I switched from VB to C# - I never looked back once I got past the frustration point.

Woonboo's points get to the nub of the problem for me: I don't think MSTest was designed for TDD but for some other high-ceremony style of integration testing. I would submit that he is probably not working in that style. Let me take his points one by one to explain what I mean.

"you don't have to use attribute...in the 'settings' (right-click properties) add files to deploy". Yes, but I shouldn't have to do even this. Why can't it just test what's in the target directory? Building a separate run directory for every test and forcing the user to deliberately choose files to deploy is just another symptom of the high-ceremony MSTest approach.

"It creates a separate test project because too many Morts out there will just put it in the same project otherwise and all your tests will be deployed with the code." I agree that it's best to have a separate test project. But any testing framework should be unintrusive. I shouldn't have to have a separate project type. The only reason it's needed by MSTest is because MSTest requires so much configuration to be useful. And you shouldn't rely on project types to force you to correctly organise your source, otherwise we'd have some crazy Microsoft scheme to have the "domain-project", the "data-access" project, the "service-project". Don't suggest that too near to someone from the TFS team!

Now I have to have a rant here about "The Morts won't understand it" argument. I hear this over and over and over again. It usually goes along the lines of "I understand it fine, but the people I work with, or the 'maintenance' people won't". It's the lamest and most often deployed excuse for bad decisions. If the Morts can't understand something, whoever they are, get rid of them. What's the point in employing people who don't know how to do their job? But usually it's not the real reason, the real reason is lack of leadership, and lack of trust. Most advances in development practices actually make things simpler but it requires that you, if you are the team lead, sit down and work with the people you are supposed to be leading.

"Having the built in ability to test private and internal methods/properties/fields is the biggest reason I love it." One of the core reasons for doing TDD is way it drives your design. If you have to access private members in your tests I would submit that your design is not correctly decoupled. I *want* my testing framework to force me to behave like any other client when I'm writing tests.

Now there might be a situation where you have to write tests for some monolithic legacy code base, but MSTest won't help your there, you need to go and talk to Roy Osherove :)

"When a test fails, looking at the test results that gives me a hyper-link to every part of the stack trace where the test failed." You get exactly the same thing with Testdriven.NET. I have "run-tests" mapped to F8 (I've never found a use for bookmarks). "Run-tests" is context specific so if the cursor is currently inside a test method, only that test gets run. So I just hit F8 to run the test(s), the results appear on the console and I can click on any part of the stack trace to go directly to that line of code. I actually dislike fancy test runners. I don't want to have to click here and there to get to stack traces, I'd much rather everything was just thrown onto the console.

Of course running tests on your CI server is essential. I've always found it simple to do that both with Cruise Control and TFS. I don't think MSTest really adds much here and it doesn't make much sense unless you're using TFS.

"Pulling up archives into the GUI of who ran what tests when on what machines. Great from the 'team lead' perspective." OK, I don't get this. Surely what you should be looking at is code coverage. I run tests every few mintues when I'm coding, you probably wouldn't want to look at or store all those test runs. I think this goes back to my first point. A tool that's designed for high ceremony infrequent testing like MSTest would see storing a test run archive as a useful thing, anyone who's done TDD would see it as a huge waste of resources.

I gave MSTest a chance. I've also recently tried to use MBUnit. neither worked as well as TestDriven.NET + NUnit do for me. I must say that MBUnit came very close, and I wouldn't have too much problem with using if I was in a team that has already settled with it. MSTest was just a nightmare from start to finish. As I said to our project manager, if it was an open source tool, it would never have got any traction and would now be sitting unloved in sourceforge. Maybe it suits some people with a very non-agile, non-TDD methodology, but for anyone doing TDD I would stay well clear.

Woonboo. Thanks very much for provoking me to write this. I love a good debate and would really like to hear your reply. Thanks again!

What's an Auto Mocking Container?

An auto mocking container (AMC) sounds pretty scary, but it's a really neat tool if you're writing a lot of unit tests and find yourself forever constructing mock objects. In the same way that an IoC container knits together dependencies at runtime, an AMC can create all your mock objects automatically for your unit tests.

Say you are testing this Reporter class

public class Reporter
{
  private readonly IReportBuilder reportBuilder;
  private readonly IReportSender reportSender;

  public Reporter(IReportBuilder reportBuilder, IReportSender reportSender)
  {
      this.reportBuilder = reportBuilder;
      this.reportSender = reportSender;
  }

  public void SendReports()
  {
      var reports = reportBuilder.GetReports();

      foreach (var report in reports)
      {
          //reportSender.SendReport(report);
      }
  }
}

Using Rhino.Mocks you might do something like this.

[Test]
public void SendReports_ShouldCreateReportsAndSendThem()
{
  // create the mock services that the Reporter requires
  var reportBuilder = MockRepository.GenerateStub<IReportBuilder>();
  var reportSender = MockRepository.GenerateStub<IReportSender>();

  // create the reporter, injecting the mock services
  var reporter = new Reporter(reportBuilder, reportSender);

  // create some reports
  var report1 = new Report();
  var report2 = new Report();
  var reports = new[] {report1, report2};

  // when the reportBuilder mock's GetReports method is called, return the reports
  // we created above
  reportBuilder.Expect(rb => rb.GetReports()).Return(reports);

  // excercise the method under test
  reporter.SendReports();

  // verify that the reporter sent the expected reports
  reportSender.AssertWasCalled(rs => rs.SendReport(report1));
  reportSender.AssertWasCalled(rs => rs.SendReport(report2));
}

After the tenth time you've had to write this kind of code you get quite bored of typing the mock object creation. It's irritating when you're thinking about using a new dependency in an existing class, and you have to add it to both the class under test and the setup for the test.

Here's an example using the AutoMockingContainer from Rhino Tools.

[Test]
public void SendReports_ShouldCreateReportsAndSendThem_WithAMC()
{
  // create a new auto mocking container
  var mocks = new MockRepository();
  var container = new Rhino.Testing.AutoMocking.AutoMockingContainer(mocks);
  container.Initialize();

  // just ask for the reporter
  // the container will automatically create the correct mocks
  var reporter = container.Create<Reporter>();

  // create some reports
  var report1 = new Report();
  var report2 = new Report();
  var reports = new[] { report1, report2 };

  // when the reportBuilder mock's GetReports method is called, return the reports
  // we created above
  container.Get<IReportBuilder>().Expect(rb => rb.GetReports()).Return(reports);

  // expect that the reporter sent the expected reports
  container.Get<IReportSender>().Expect(rs => rs.SendReport(report1));
  container.Get<IReportSender>().Expect(rs => rs.SendReport(report2));

  // excercise the method under test
  reporter.SendReports();

  // assert that expectation were met
  mocks.VerifyAll();
}

As you can see, after we set up the AMC we simply ask it for an instance of the class we wish to test. We don't have to worry about supplying the dependencies because the AMC works out what mocks need to be created and does it for us.

When we setup our expectations we can ask the AMC for the mock objects it's created.

You can get the latest version of the source code for the Rhino Tools AMC by pointing tortoise here:

https://rhino-tools.svn.sourceforge.net/svnroot/rhino-tools/trunk

Or you can download the code and grab the assemblies I've built from the trunk here

http://static.mikehadlow.com/Mike.AutoMockingContainer.zip

MSTest is sapping my will to live

The team I'm currently working with has spent a lot of money buying MS's team suite and so naturally we wanted to use MSTest (AKA Team System Unit Testing). In theory using MSTest allows you to simply integrate tests into your build process and generate coverage reports etc. Unfortunately the grim reality is somewhat different.

Unit test tools have been around for several years. I've been using NUnit on a daily basis for around four years. I've never really paid much attention to it because it's one of those tools that just works; you write your tests, attribute them as [Test] and they either pass or fail. I keep hearing good things about MbUnit, but I haven't tried it yet because I feel no pain with NUnit and its constant companion, the excellent TestDriven.net. In simple terms, this is how NUnit/TestDriven.net works:

1. Attribute any class with TestFixture and any void-void method with Test. Use Asserts to check my expectations.
2. I have F8 mapped to run tests, so I hit F8 and TestDriven points NUnit at the test assembly in bin/debug folder (or wherever the project's output path points to).
3. NUnit loads the assembly and executes any method attributed with Test and outputs the results to the output window.

And that's it. If my tests need any attendant configuration or need to load any assemblies dynamically I just make sure they are copied to the output path.

As I said before NUnit has been around for a while and MbUnit does the same stuff but with some extra bells and whistles. Any unit testing framework should work in a similar way and should be as simple to use as possible.

Enter MSTest. Now you would have thought that with the great success of NUnit to copy, even Microsoft couldn't make too great a hash of it. But they have. Two major faults make it an unsuitable tool for running unit tests:

1. The test runner does not simply reflect over an assembly and run any method attributed with TestMethod, instead you have to create a special test project and your tests have to be present in an XML file in order for the test runner to find them. Why? Why should we need a special test project? There's no conceivable reason I can see. It means you have to have your tests in a separate assembly than the code under test. Now it's my practice to do this, but you shouldn't be forced to do it. Worse though is that it's just too easy for the XML file to get out of sync with the tests and for your tests to mysteriously not get executed. That's not to mention the source control nightmare it causes.
2. The test runner runs the test assemblies in some other place that's different each time. Now you can't rely on an assembly manifest to tell you everything that some code might need to run. I'm a big fan of IoC containers which means that much of my application's object graph gets loaded at runtime. I get frequent test failures because my tests can't find assemblies they need to load. Configuration is a similar issue, miscellaneous files don't get copied to the test run location. Now I know that you can do add attributes to every single test or add stuff into the test configuration file to get the test runner to copy certain files to its run directory, but why should I have to do this? Also It doesn't clean up after itself and I run tests every minute of every day, this means it builds up not an insignificant amount of crud on my disk.

I'm just going to mention in passing that the MSTest Asserts are pre-historic and look a bit like NUnit version-a-long-time-ago and that the default unit test project is full of unnecessary files and that the unit test template generates a page of crud that you have to delete before you can start work. That's no so important and I could live with it if the core design wasn't so FUBAR.

MSTest looks like a tool that was designed by someone who'd never really done much unit testing. They'd certainly not worked with NUnit to any degree. I really can't understand the thought process that resulted in such bad design decisions, but it looks like it's designed to be run once a week with an great deal of ceremony. In other words MS have totally misunderstood TDD.

So now I'm lobbying hard to get us to drop MSTest and use NUnit or MbUnit instead. In fact experience with the whole team system is that it's not worth spending the money, you get far better results and much less stress from using a combination of open and third party tools. But that's another post...

Writing unit tests for an MVC Framework controller sucks

Over the Christmas holidays I had a lot of fun writing a web application using the Castle Project's Monorail. One of the joys of doing Monorail development is the outstanding support for unit testing that it provides. That's not really surprising when you read some of the Castle Project guys' blogs, they are all thoroughly Test Infected, and indeed one of the main driving forces behind adopting any framework that allows a separation of concerns is the ease which such a framework can support Test Driven Development.

During the last couple of weeks I've been embarking on a new commercial project. I've decided to use the new Microsoft MVC Framework simply because it's much easier to justify using a tool when Microsoft are behind it. And looking to the future it's highly likely that most professional MS shops will have MVC Framework skills to support such an application whereas Monorail skills are inevitably always going to be niche.

Monorail and the MVC Framework are quite similar, and having come from a Monorail project, I've found it pretty easy getting up to speed with the Microsoft offering. Some things are nicer: there's the typically good tool support, especially intellisense in the views and the strongly typed view data is an improvement over NVelocity (I haven't tried Brail). The routing engine is very nicely thought out and much better than the afterthought bolted onto Monorail.

The one thing that's really surprised me though is the lack of thought which has gone into support for unit testing controllers. Monorail supplies a BaseControllerTest class that you can inherit from for your controller tests that allows you to mock or access all the controller infrastructure. For example, say I want to test that my controller calls Response.Redirect passing a specific URL.

• Make my controller test inherit from BaseControllerTest
• Override the virtual method BuildResponse and provide a mock response
• Pass my controller to the PrepareController method in BaseControllerTest. This sets up a mock infrastructure for the controller and at some point will call my overriden BuildResponse method.
• In the test itself simply set an expectation for Response.Redirect to be called with the expected URL.

[TestFixture]
public class PosterLoginControllerTests : BaseControllerTest
{
    MockRepository mocks;
    PosterLoginController posterLoginController;

....

    protected override IMockResponse BuildResponse()
    {
        return mocks.CreateMock<IMockResponse>();
    }

    [SetUp]
    public void SetUp()
    {
  ....
  
        PrepareController(posterLoginController);
    }

    [Test]
    public void IndexShouldRedirectLoggedInPosterToAddJobTest()
    {
        string expectedRedirectUrl = "/Job/JobForm.rails";

        using (mocks.Record())
        {
   ....
   
            Response.Redirect(expectedRedirectUrl);
        }

        using (mocks.Playback())
        {
            posterLoginController.Index();
        }
    }

Now you can't do anything like this with the MVC framework. There's no BaseControllerTest and it's impossible to provide mocks for all the hooks in the Controller class. It's not completely decoupled from the infrastructure. The only way to successfully test a controller is to write a test subclass that inherits from the controller under test and overrides all the important methods that deal with the infrastructure. Because you have to write a new one for each controller it's a real overhead with lots of duplicated effort. Here's an example that I've had to write for my current project:

public class TestUserController : UserController
{
    string actualViewName;

    public string ActualViewName
    {
        get { return actualViewName; }
    }

    object actualViewData;

    public object ActualViewData
    {
        get { return actualViewData; }
    }

    Dictionary<string, object> redirectValues = new Dictionary<string, object>();

    public Dictionary<string, object> RedirectValues
    {
        get { return redirectValues; }
    }

    public TestUserController(
        IUserRepository userRepository,
        IRepository<Role> roleRepository,
        IRepository<InstructionType> instructionTypeRepository,
        IAuthorisationBuilder authorisationBuilder)
        : base(
        userRepository,
        roleRepository,
        instructionTypeRepository,
        authorisationBuilder)
    {

    }

    protected override void RenderView(string viewName, string masterName, object viewData)
    {
        this.actualViewName = viewName;
        this.actualViewData = viewData;
    }

    protected override void RedirectToAction(object values)
    {
        PropertyInfo[] properties = values.GetType().GetProperties();
        foreach (PropertyInfo property in properties)
        {
            redirectValues.Add(property.Name, property.GetValue(values, null));
        }
    }

    NameValueCollection form = new NameValueCollection();

    public override NameValueCollection Form
    {
        get
        {
            return form;
        }
    }
}

As you can see it's quite extensive and not something that I relish repeating for each controller. Phil Haack describes injecting mocks into the controller, but then in an update to the post says that it's something that now broken for the CTP and basically admits that the only way of doing it is to write test specific subclasses.

The more I do IoC and dependency injection (which works well with the MVC framework BTW) the more I wish framework builders did the same. It would make all these issues go away. I guess it's a request too far as long as IoC Containers remain a niche thing; supporting them is one thing, requiring them quite another.

Update

John Rayner from Conchango recently pointed out a post on Ayende's blog, showing how you can mock controllers and use extension methods to provide mocks of protected methods on the controller. It's a neat trick and gets over the need to build test sub-classes of your controller. But, it still doesn't remove the main point which is: we shouldn't have to do these crazy tricks; the controller should be testable.