I recently had a long and considered comment from Woonboo for my post "MSTest is sapping my will to live". Woonboo is a happy user of MSTest. I started writing a reply in the comments, but it was getting so involved I thought it would be nice to promote it to a new post.
Here's Woonboo's comment:
I know I'm late to the game, but mileage is everything.
I used to use NUnit, but after using MSTest for the past 2.5 years, I wouldn't go back, even with the bugs in VS2008 (which are minor and only hit if you are doing something specific with AppDomain).
Your config file thing is simple - you don't have to use attribute...in the 'settings' (right-click properties) add files to deploy.
It creates a separate test project because too many Morts out there will just put it in the same project otherwise and all your tests will be deployed with the code. I've worked with a number of these folks.
Having the built in ability to test private and internal methods/properties/fields is the biggest reason I love it. No extra code required. No need to loosen the scope (make things public) on what you're testing.
When a test fails, looking at the test results that gives me a hyper-link to every part of the stack trace where the test failed (or threw an exception) as saved me hundreds if not thousands of hours by now not having to navigate to the file, hit CNTL+G and enter the line number; especially if I have to walk up the stack to see if there was a path taken that shouldn't of been causing the problem.
Having the ability to have tests run automatically when I do a build or check-in (easier than I was ever able to with NUnit - but that's a SCC).
Code coverage gets tied to the tests you write.
Pulling up archives into the GUI of who ran what tests when on what machines. Great from the 'team lead' perspective.
You said in another post that technologies change but methodologies don't - don't let NUnit suck you into the 'one tool' mentality (although I could say the same about myself with MSTest). Love your posts generally - but I think you need to give MSTest another chance and ask someone who's been successful with it for help. It was hard for me to switch from NUnit too - but it was like when I switched from VB to C# - I never looked back once I got past the frustration point.
Woonboo's points get to the nub of the problem for me: I don't think MSTest was designed for TDD but for some other high-ceremony style of integration testing. I would submit that he is probably not working in that style. Let me take his points one by one to explain what I mean.
"you don't have to use attribute...in the 'settings' (right-click properties) add files to deploy". Yes, but I shouldn't have to do even this. Why can't it just test what's in the target directory? Building a separate run directory for every test and forcing the user to deliberately choose files to deploy is just another symptom of the high-ceremony MSTest approach.
"It creates a separate test project because too many Morts out there will just put it in the same project otherwise and all your tests will be deployed with the code." I agree that it's best to have a separate test project. But any testing framework should be unintrusive. I shouldn't have to have a separate project type. The only reason it's needed by MSTest is because MSTest requires so much configuration to be useful. And you shouldn't rely on project types to force you to correctly organise your source, otherwise we'd have some crazy Microsoft scheme to have the "domain-project", the "data-access" project, the "service-project". Don't suggest that too near to someone from the TFS team!
Now I have to have a rant here about "The Morts won't understand it" argument. I hear this over and over and over again. It usually goes along the lines of "I understand it fine, but the people I work with, or the 'maintenance' people won't". It's the lamest and most often deployed excuse for bad decisions. If the Morts can't understand something, whoever they are, get rid of them. What's the point in employing people who don't know how to do their job? But usually it's not the real reason, the real reason is lack of leadership, and lack of trust. Most advances in development practices actually make things simpler but it requires that you, if you are the team lead, sit down and work with the people you are supposed to be leading.
"Having the built in ability to test private and internal methods/properties/fields is the biggest reason I love it." One of the core reasons for doing TDD is way it drives your design. If you have to access private members in your tests I would submit that your design is not correctly decoupled. I *want* my testing framework to force me to behave like any other client when I'm writing tests.
Now there might be a situation where you have to write tests for some monolithic legacy code base, but MSTest won't help your there, you need to go and talk to Roy Osherove :)
"When a test fails, looking at the test results that gives me a hyper-link to every part of the stack trace where the test failed." You get exactly the same thing with Testdriven.NET. I have "run-tests" mapped to F8 (I've never found a use for bookmarks). "Run-tests" is context specific so if the cursor is currently inside a test method, only that test gets run. So I just hit F8 to run the test(s), the results appear on the console and I can click on any part of the stack trace to go directly to that line of code. I actually dislike fancy test runners. I don't want to have to click here and there to get to stack traces, I'd much rather everything was just thrown onto the console.
Of course running tests on your CI server is essential. I've always found it simple to do that both with Cruise Control and TFS. I don't think MSTest really adds much here and it doesn't make much sense unless you're using TFS.
"Pulling up archives into the GUI of who ran what tests when on what machines. Great from the 'team lead' perspective." OK, I don't get this. Surely what you should be looking at is code coverage. I run tests every few mintues when I'm coding, you probably wouldn't want to look at or store all those test runs. I think this goes back to my first point. A tool that's designed for high ceremony infrequent testing like MSTest would see storing a test run archive as a useful thing, anyone who's done TDD would see it as a huge waste of resources.
I gave MSTest a chance. I've also recently tried to use MBUnit. neither worked as well as TestDriven.NET + NUnit do for me. I must say that MBUnit came very close, and I wouldn't have too much problem with using if I was in a team that has already settled with it. MSTest was just a nightmare from start to finish. As I said to our project manager, if it was an open source tool, it would never have got any traction and would now be sitting unloved in sourceforge. Maybe it suits some people with a very non-agile, non-TDD methodology, but for anyone doing TDD I would stay well clear.
Woonboo. Thanks very much for provoking me to write this. I love a good debate and would really like to hear your reply. Thanks again!
Posts
9 comments:
LOL. Well wasn't trying to provoke you; but okay, let's continue the debate.
We don't do TDD - this legacy application makes it impossible - we have a long way to get there...and we inherited something with 0 tests. We do however probably run all of our tests on average at least 3 or 4 times a day - not a drop in the bucket compared to TDD - but not once a week either. The individual 'piece' we're working on we'll probably run 20-30 times a day and while this isn't TDD, it's not painful either.
One of the core reasons for doing TDD is the way it drives your design. If you have to access private members in your tests I would submit that your design is not correctly decoupled.
You missed what I was saying here, or I missed what you were saying - one of the two.
I'm not saying accessing the private members in the test, but testing the private members themselves. What I'm saying is that for me to functionally test:
private Bar DoSomething(Foo f, FoobarType t)
I just create a test directly against it. Do you not have private methods? Or are you testing all the methods that can possibly call DoSomething and varying the inputs to those parent methods to ensure you get 100% coverage?
Or say I want to make sure a private field gets set when some method is called, or change the private field to ensure an exception gets called? Or maybe I'm just lazy and want to use private constants so I don't have to create the same constants in my tests and deal with synchronization? How do you test that your private/internal fields are being set correctly? Or that a private/internal event is firing?
Obviously, if possible we're going to use behavior testing to ensure the method gets called (if possible) and needed (maybe with rhino mocks), but that's testing behavior of the parent method, not the functionality of the target method. I'm curious on your response to this.
At the same time though there's not a chance in heck that I'm going to make private/protected/internal fields/properties/delegates/events/methods/constants public for a testing framework.
So config files; why can't you just test what's in the target directory? That's easy...the config file in my target directory points to a copy of real data...the config file in my test directory points to test data (in some cases). When I'm testing I don't want to use a bloated database used for 'live' (i.e. human) testing. But then in some case I want to use the live data; so I can pick and choose. For mocking that can be especially useful (and oh how I wish I could actually use mocks right now, but that's a different story). I will agree with you though, that by default, the 'App.config' file should be copied over by default - or at least have an option for it. Maybe there is and I just don't know it.
Ok, now employment. It's definitely not an excuse. I have no control over who gets hired - that's way, way, way, way above me.
The Testdriven.NET - must be a feature that wasn't around when I used it last - glad to see it's there.
The looking at archives thing - again, this is a situational thing and I didn't write it well. What has saved my rear several times, is downloading programmer X's code onto my machine who said all his tests passed. Because we can't do TDD (I know, I know, that's the first problem), I may not run his tests right away because they do take 10 or 15 minutes to run. Meanwhile I do some work and maybe later in the day I run my tests. Something is failing now...but what? I can go back and look at the DLLs on the test that passed and see what broke, or more usually, look at the time when it last passed, look at who did the check in after that, and point the finger at who broke the build. Again, I know what you're going to say; this is text-book case for why you do TDD - but introducing TDD on a brown-field application that is severely over-architected isn't a trivial task. Introducing TDD to people who fought vehemently against doing unit testing to begin with only makes it harder. (You know, makes a project take longer, all the extra code to maintain, blah blah, wah, wah...which bring me to my next point).
The people thing: At one point a previous boss of mine had to go so far as to make it a set of checks in the performance review for all developers that tests are written, that tests are run regularly, that broken tests are fixed, and that tests are more than just 'happy path'. Isn't that sad? Now if a previous boss of mine had to put that into the performance review why do you think we need the ability to verify that someone has run tests? Or why I say that most Morts would screw up creating a separate test project or can't/won't figure it out? People don't put things in performance reviews like this unless there's a problem. And believe me, this guy isn't around any longer because he tried to fix some of those problems and some other problems. The person who came after him didn't have much more luck, nor the one after that. Scary huh? You say you hear these types of excuses all the time...well there may be some truth to it and you're just living the good life. I hear all the time about these great places where people get to do TDD, follow Agile principals (I'd love to work in a place that has Scrum actually working), get training, etc. Never seen it in person or met anybody who has...but that doesn't mean I think your making it up. These Morts may seem like Big Foot to you, but the situations you describe sound like Atlantis to me. :)
So some of the other things that really sell me on MSTest - having performance tests built in. I write a test that deals with submitting invalid data, or possibly causes something long running to kick off. Now I want to put some load against that object - let's say 40K hits in a minute (or whatever) - I'll just put my test harness on some various computers and point them all to one place and basically see how my code handles a stressful load. This has pointed out some really bad stored procedures that you wouldn't notice just running tests single threaded, threading issues or cursors problems period have been caught by this, and numerous other things that stand-alone unit tests won't pick up. But the load test...all it is, is a pointer to the unit test. Load testing was something I never did with NUnit, so I'm not sure what's available there to comment.
But you know, I do agree with you in principle with many things - my goal is to get our team more to a TDD mindset, now that we at least got a 'test' mindset or greater as a whole on the team. I know they want it to, but we have some very large hurdles to overcome first, both technically and culturally - and it will be interesting to see how MSTest holds up as we move from slower chunkier tests to more fine grained faster tests (as the legacy architecture is refactored to allow that to happen). I agree with you that the test framework is extremely slow to load and if anything will drive us into NUnit as we more TDD, it would be that...but I guess it's all about frequency, and for now, baby steps. We got a good group of people who recognize the need for change and our doing what they can, but some things have to run their natural course. At my level I'm doing what I can.
On one last note though...you dislike fancy test runners? Come on now! Next your going to tell me that you programming in VI or Notepad. Now that my friend, is a lame excuse! jk.
Good conversation. I've enjoyed it.
I partly agree with woonboo (and partly disagree!). I use MS Test and test drive my code.. and love it! Especially the nice integration with the IDE.
"If you have to access private members in your tests I would submit that your design is not correctly decoupled"
I would definatly agree with that! You shouldnt be unit testing privates!
Im not sure about this:
"the 'App.config' file should be copied over by default"
Is this suggesting that your unit test is dependent on something!? This (to me) removes part of the issolation... would it not be better to create a wrapper (or custom ConfigSection) and inject a mocked version into the class? It starts to get very messy if you test depends on external files and starts to become more of an integration test?
So there's something I missing then since obviously there some heart burn over testing private methods.
How do you folks ensure your test methods are working?
For the config file thing. You know - I'm really on the fence with it. I agree that you don't want coupling - but coupling does not always equal bad. For instance our configuration information for the main frame is in the config file. This was we can switch machines between development/test/training (which is the same machine) and production. So in this case, having certain parts of the config file copied over isn't bad because it won't change on tests.
And whoah before you respond on that because oh yes, there should be some sort of level of indirection so I'm not having to running unit tests against the main-frame in development - but - we're not quite to that level of flexibility yet - Mike has been a real help in giving me some ideas though (along with a few other out there who've been quite helpful) in being able to get there, but we got a ways to go. Also, at some point we do have to test our interface with the main frame - if we don't, we have no 100% guarantee that is works.
Oops...that question should be 'how do you ensure 100% that your private methods are working'...not unit tests
-Sorry, haven't had enough caffeine this morning.
Thanks Woonboo excellent reply.
I don't think I need to add anything, I'd just be repeating myself. We'll just have to agree to disagree, but I'll leave you with the last word.
Woonboo, I didn't see your last comment about testing private members.
I think you should get a copy of Kent Beck's excellent book on Test Driven Development and work through the first section. If you construct your software from fine-grained components that obey the single responsibility principle, you should be able to exercise all their functionality via their public interface. The main point of TDD is that it's not really about testing, it's about design. The minute you start thinking 'I need to be able to test this private method' is the minute that you should start to think 'my design is wrong'.
This isn't pie-in-sky academic nonesense that has no relationship to the real world. Go and get the code for NHibernate, MVCContrib or the Castle Project and read the tests. See how they do it.
I actually got the book. Along with Pragmatic Unit Testing in C#. To be honest, to me it was very academic - great stuff, foundational stuff every developer should know - but academic. No complicated business rules or dealing with brown field applications or interfacing with legacy libraries.
Now I'm going to go into a friendly rant for a minute - but keep in mind - I'm on the same page as you - just at 85% agreement, not 100%.
Talking about banks and 'Reduce' is easy and the ever present ATM machine or simple POS system is elementary - talk about different services that are available to different customers depending on what car they drive, what market the store they're getting there car serviced in, how frequently they used the company and/or that store depending on local, district, and regional managers, what sales are going on, what coupons were brought in, with prices that can vary based upon the sales person or be overriden if the sale person is high enough on the food chain, that has to be reported on, logged, archived, and show visuals of parts and locations of work on the car, make sure an employee isn't overbooked to work on a car, keep track of they're schedule and what they've currently working on (live), manage parts and tie parts to services being entered for the customer (live), plus is built to run on multiple application servers. Give me that in TDD, and it not me a bloated pig or a maintenance nightmare or impossible to extend by anyone other than the original developers - then you'll sell me 100%. Oh, and it has to be user friendly - no 'trees' of menu's or drilling down into separate forms (re: The Inmates are Running the Asylum - great book if you've never read)
Anyway, like I said - 85% with you, not at the 100% and we'll agree with that being the best we can do for now.
So, yeah, I've been trying to get more into NHibernate but time has been a factor because write now I'm very fascinated with functional programming and spend a lot of time in that - it's so cool. The stuff I've been seeing on Fluent NHibernate I'm really looking forward to - I see a lot more in that, that will apply to my current situation then the current NHibernate. MVCContrib doesn't do you much good on Forms applications - we use UIP, not because it's a great implementation of MVC, but because it allow us to use the same views across technologies (in theory)...it's just the view impl that changes (in theory). That's about all the good I can say for UIP, and now I fully expect to be pounced on for either a) saying something good about UIP or b) saying something not cynical about UIP - it will depend on your readers. :)
I've only heard about Castle, I'll put that on my list too.
As always, good chatting with you.
Woonboo you keep on saying that you can't do Unit Testing on Brown Field projects but you can. The thing you have to keep in mind is that you add Unit Tests with new code (Manadatory work), and when you go back and fix a bug or refactor some code you do it then too. Trying to do a blanket add Unit Tests to everything approach never works and the management will never sign off the time.
As for Agile practices, I worked at a company that was VERY waterfall orientated, but we got them to switch over to a Scrum based approach by moving overly slowly. Over time the management saw that we were fire fighting a lot less and encouraged us to migrate to a Scrum based way.
I have to agree that you shouldn't be testing private/protected anything explicitly as it doesn't reflect how the object as a whole is used and so your Unit Tests are really proving the object is good, just snippets of it.
Cooll
Post a Comment