This post is a follow up to A Tale of Unit Testing Technologies. Information about the project repository can be found on my project page.
I've finally managed to slog through the long and slightly tedious process of re-writing the same suite of tests in a number of C# and JavaScript unit testing technologies - the results of which can be seen on the project GitHub page. My conclusion, especially in the .NET world, is that it doesn't really matter that much. I know it's not that much of a brilliant ending but what did you expect? Fireworks?
Anyways before I start amazing you all with more awe inspiring revelations let me run through the remaining technologies (until such times as more are added at least) and summarise their good and bad points. The other technologies I brought on board include,
Some obvious ones there just to round out the comparisons and a few JavaScript based ones.
NUnit
Good old NUnit. Whats not to love? Mark a class as a fixture and mark your methods as a test.
[TestFixture]
public class TagTest
{
[Test]
public void TagNameNormalisationTest()
{
Model.Tag tag = new Model.Tag { Name = "My Tag Name" };
Assert.AreEqual(tag.NormalisedName, "mytagname");
}
}
No point in mentioning too much about NUnit as most people will know the ins and outs of it already. It has VS integration (via TestDriven) and a nice NUnit runner capable of watching changes to DLL's and automatically rerunning the test contained within. One caveat around that is often NUnit hangs on to the DLL reference and requires a restart so Visual Studio can rebuild the DLL which kind of misses the point - but it doesn't happen too often.
The framework itself would lend itself to a bit of betterification through sugar such as Should and Should.Fluent but it does what it says on the tin.
MSTest
Another staple for many people. I'd never really used it too much prior to doing this post because of the general negative opinion of it. I must admit I kind of agree with some of these things. It's pretty much the MS equivalent of NUnit (in terms of features and look and feel)
[TestClass]
public class TagTest
{
[TestMethod]
public void TagNameNormalisationTest()
{
Model.Tag tag = new Model.Tag { Name = "My Tag Name" };
Assert.AreEqual(tag.NormalisedName, "mytagname");
}
}
So why do so many people prefer NUnit over MSTest? Well I think the main problem is that it is quite slow in comparison (from Visual Studio at least) but my main gripe is the amount of flob it generates for each test run and configuration. It even produces solution folders for holding test runs and other files. If I want you to create those I'll ask you - don't force it on me. It also has a very limited set of Assertion methods much like NUnit.
Enough moaning - surely it must have some benefits? Well yes it does of course. The main benefit IMHO is that it has very nice integration into Visual Studio which offers some very clean feedback. This is something that requires TestDriven for NUNit and the other frameworks which may or may not be free depending on your situation.
NSpec
Not to be confused with the first hit on Google which seems to be pretty much a dead project NSpec (.org) is an attempt to bring RSpec into the .NET world.
class describe_Tag : nspec
{
void when_setting_the_tag_name_to_My_Tag_Name()
{
before = () => _tag = new Tag { Name = "My Tag Name" };
it["should normalise the tag name to mytagname"] = () =>
_tag.NormalisedName.should_be("mytagname");
}
private Tag _tag;
}
This is somewhat similar to MSpec I mentioned in the last post. My bugbear with MSpec however was that it didn't feel natural to me. The equivalent MSpec test for the above looks like this,
[Subject("Normalise Tag Name")]
public class when_a_tag_is_created
{
static Tag _tag;
Establish context = () =>
_tag = new Tag();
Because of = () =>
_tag.Name = "New Test Tag";
It should_have_a_normalised_name_of_newtesttag = () =>
_tag.NormalisedName.ShouldEqual("newtesttag");
}
Maybe it's just me but it doesn't read as well as the NSpec version. I even found it easier to write the NSpec version - more natural.
NSpec also has a lot of extra goodies. Currently there is no integration into Visual Studio yet but the runner is simple to enough to spin up from the NuGet Package Manager Console and it has an added bonus of a file watcher. SpecWatchr is similar to NUnit it can watch for changes and automatically run the specs. It differs from NUnit in that it watches changes to the code files rather than the DLL - so need to wait for builds to happen etc. Nice.
One minor gripe with it (bar the lack of VS support - even TestDriven support would be good) is the should syntax. I prefer the Should.Fluent style syntax Should.Be() rather than the current should_be syntax. It's not Ruby and we should accept that :-P.
It's still quite early for NSpec and hopefully it keeps it's momentum as it would be one of my frameworks of choice.
Quixote
Quixote by Rob Conery follows on from his current slew of ultra lightweight solutions to common problems (like Massive and Sugar). Quixote blends the HTML reporting directly with the framework and you write tests right in a Razor file. This is the definition of lightweight.
@using Quixote;
@using CodeSlice.UnitTesting.Model;
<link href='@Url.Content("~/Styles/quixote.css")' rel="stylesheet" type="text/css" />
@TheFollowing.Describes("Tags")
@They.Should("Normalise the Tag Name", () => {
return new Tag { Name = "Test Name" }.NormalisedName.ShouldEqual("testname");
})
The obvious problem with this is that you don't get continuous integration or build support. There is no need to rebuild anything just refresh your page and the tests re-run. Still CI/Build integration would be nice for a framework.
Jasmine
Ah Jasmine. Jasmine is what I wanted out of a JavaScript testing framework. Rather than going the JSpec route of creating a preprocessed DSL Jasmine makes use of JavaScripts dynamic nature and "functions as 1st class citizens" feature to create framework that behaves like RSpec but embraces JavaScript.
describe('Tag', function() {
var tag;
beforeEach(function() {
tag = new Tag();
});
it("should normalise a tag name", function(){
tag.setName('A Tag Name');
expect(tag.getNormalisedName()).toEqual('atagname');
});
});
Out of the box you get an extensible framework (custom matchers etc), mocking and asynchronous support. Ajax faking isn't directly available but I recommend using Sinon for this as it has an amazing FakeServer object that is fully configurable.
Another big advantage of Jasmine is it's build integration. Be it node.js, Java or Ruby you can integrate Jasmine specs into your build process. .NET integration isn't available yet but Jurassic would be a suitable host if anyone wants to make it happen (hint, hint :-P).
QUnit
And finally QUnit. QUnit is the NUnit/JUnit equivalent for the JavaScript world. Well known and mature project with some very nice HTML output.
module('Tag Model Validation');
test('Tag name normalisation', function(){
var tag = new Tag({ name: 'A Tag Name' }),
normalisedName = tag.getNormalisedName();
equals('atagname', normalisedName);
});
Not much else to say on this one right now.... must be running out of steam.
Summing Up
And there you have it. The first real post since the move and it's a bit of a waffly long one. Anyways what would be my recommendations? Well for .NET I'm torn. Currently I'd say NUnit with Should.Fluent extensions. This is a nice combo for creating simple tests that people can execute from within Visual Studio. However NSpec has potential - I'd like to see it grow a bit more and I'll keep watching it. Visual Studio integration would be aces though. JavaScript on the other hand I have a clear cut winner. Jasmine is awesome. It's boosted my code quality and productivity no end on recent projects and with the help of Sinon for mocking and controlling Ajax it'll be a long time before I am this happy with a .NET equivalent tech. Fingers crossed for NSpec on that front.