Thursday, January 14, 2016

Running The KestrelHttpServer On Linux With CoreCLR

Being both a long-time .NET developer and Linux hobbyist, I was very excited about the recent ‘go live’ announcement for CoreCLR on Linux (and Windows and Mac). I thought I’d have a play with a little web server experiment on an Amazon EC2 instance. To start with I tried to get the KesteralHttpServer sample application working which wasn’t as easy as I’d hoped, so this post is a note of the steps you currently need.

So first create a new Ubuntu Server 14.04 AMI:

image[5]

With a t2.micro instance type:

image[11]

Next log in and update:

ssh -i .ssh/mykey.pem ubuntu@-the-ip-address
...
sudo apt-get update
sudo apt-get upgrade

Currently there are two different sets of instructions for installing CoreCLR on Linux. The first one I found (I think linked from Scott Hanselman’s blog) shows how to use the standard Debian package manager to install the new ‘dotnet’ comand line tool. Apparently Kestrel will not currently work with this. The second set of instructions use the existing ‘dnvm’, ‘dnu’ and ‘dnx’ tools. These do work, but you need to get the latest unstable RC2 version of CoreCLR, like this:

First install the Dot Net Version Manager tool:

sudo apt-get install unzip curl
curl -sSL https://raw.githubusercontent.com/aspnet/Home/dev/dnvminstall.sh | DNX_BRANCH=dev sh && source ~/.dnx/dnvm/dnvm.sh
source /home/ubuntu/.dnx/dnvm/dnvm.sh

Next install the latest unstable (thus the '-u' flag) CoreCLR:

sudo apt-get install libunwind8 gettext libssl-dev libcurl4-openssl-dev zlib1g libicu-dev uuid-dev
dnvm upgrade -u -r coreclr

At the time of writing this installed 1.0.0-rc2-16357

image[17]

You also need to follow the instructions to build the libuv library from source. This sounds hairy, but it worked fine for me:

sudo apt-get install make automake libtool curl
curl -sSL https://github.com/libuv/libuv/archive/v1.8.0.tar.gz | sudo tar zxfv - -C /usr/local/src
cd /usr/local/src/libuv-1.8.0
sudo sh autogen.sh
sudo ./configure
sudo make
sudo make install
sudo rm -rf /usr/local/src/libuv-1.8.0 && cd ~/
sudo ldconfig

Next get the KestrelHttpServer source code:

git clone https://github.com/aspnet/KestrelHttpServer.git

Restore the Kestrel packages by running dnu restore in the root of the repository:

cd KestrelHttpServer
dnu restore

Next navigate to the sample app and restore the packages there too:

cd samples/SampleApp/
dnu restore

Now you should be able to run the sample app by typing:

dnx web

image[23]

VoilĂ !

There’s obviously some way to go before this is a straightforward out-of-the-box experience. The team should also try and unify their getting started instructions because there are various different conflicting pages floating around. The Kestrel team were very helpful though in getting this working. Now to do something with my new found Linux web server.

Friday, December 04, 2015

Learn To Code, It’s Harder Than You Think

TL;DR: All the evidence shows that programming requires a high level of aptitude that only a small percentage of the population possess. The current fad for short learn-to-code courses is selling people a lie and will do nothing to help the skills shortage for professional programmers.

This post is written from a UK perspective. I recognise that things may be very different elsewhere, especially concerning the social standing of software developers.

It’s a common theme in the media that there is a shortage of skilled programmers (‘programmers’, ‘coders’, ‘software developers’, all these terms mean the same thing and I shall use them interchangeably). There is much hand-wringing over this coding skills gap. The narrative is that we are failing to produce candidates for the “high quality jobs of tomorrow”. For example, this from The Telegraph:

“Estimates from the Science Council suggest that the ICT workforce will grow by 39 per cent by 2030, and a 2013 report from O2 stated that around 745,000 additional workers with digital skills would be needed to meet demand between now and 2017.

Furthermore, research by City & Guilds conducted last year revealed that three quarters of employers in the IT, Digital and Information Services Sector said that their industry was facing a skills gap, while 47 per cent of employers surveyed said that the education system wasn’t meeting the needs of business.”

Most commentators see the problem as being a lack of suitable training. Not enough programmers are being produced from our educational institutions. For example, here is Yvette Cooper, a senior Labour party politician, in The Guardian:

“The sons and daughters of miners should all be learning coding. We have such huge advantages because of the world wide web being invented as a result of British ingenuity. We also have the English language but what are we doing as a country to make sure we are at the heart of the next technology revolution? Why are we not doing more to have coding colleges and technical, vocational education alongside university education?”

There is also a common belief in the media that there are high barriers to entry to learning to code. This from the Guardian is typical:

“It’s the must-have skill-set of the 21st century, yet unless you’re rich enough to afford the training, or fortunate enough to be attending the right school, the barriers to learning can be high.”

So the consensus seems to be that high barriers to entry and a lack of accessible training mean that only a rich and well educated elite have access to these highly paid jobs. The implication is that there is a large population of people for whom programming would be a suitable career if only they could access the education and training that is currently closed to them.

In response, there are now a number of initiatives to encourage people to take up programming. The UK government created ‘Year of Code’ in 2014:

year-of-code

The message is “start coding this year, it’s easier than you think.” Indeed the executive director of Year of Code, Lottie Dexter, said in a Newsnight interview that people can “pick it up in a day”. Code.org, a “non-profit dedicated to expanding participation in computer science education”, says on its website, “Code.org aims to help demystify that coding is difficult”.

hour-of-code

So is it really that easy to learn how to code and get these high paying jobs? Is it really true that anyone can learn to code? Is it possible to take people off the streets, give them a quick course, and produce professional programmers?

What about more traditional formal education? Can we learn anything about training programmers from universities?

Given the skills shortage one would expect graduates from computer science courses to have very high employment rates. However, it seems that is not the case. The Higher Education Statistics Agency found that computer science graduates have “the unwelcome honour of the lowest employment rate of all graduates.” Why is this? Anecdotally there seems to be a mismatch between the skills the students graduate with and those that employers expect them to have. Or more bluntly, after three years of computer science education they can’t code. A comment on this article by an anonymous university lecturer has some interesting insights:

“Every year it's the same - no more than a third of them [CS students] are showing the sort of ability I would want in anyone doing a coding job. One-third of them are so poor at programming that one would be surprised to hear they had spent more than a couple of weeks supposedly learning about it, never mind half-way through a degree in it. If you really test them on decent programming skills, you get a huge failure rate. In this country it's thought bad to fail students, so mostly we find ways of getting them through even though they don't really have the skills.”

Other research points to similar results. There seems to be a ‘double hump’ in the outcome of any programming course between those who can code and those who can’t.

“In particular, most people can't learn to program: between 30% and 60% of every university computer science department's intake fail the first programming course.”

Remember we are talking about degree level computing courses. These are students who have been accepted by universities to study computer science. They must be self selecting to a certain extent. If the failure rate for programming courses is so high amongst undergraduates it would surely be even higher amongst the general population - the kinds of candidates that the short ‘learn to code’ courses are attempting to attract.

Let’s look at the problem from the other end of the pipeline. Let’s take successful professional software developers and ask them how they learnt to code. One would expect from the headlines above that they had all been to expensive, exclusive coding schools. But here again that seems not to be the case. Here are the results of the 2015 Stack Overflow developers survey. Note that this was a global survey, but I think the results are relevant to the UK too:

SO-dev-survey

Only a third have a computer science or related degree and nearly 42%, the largest group, are self taught. I have done my own small and highly unscientific research on this matter. I run a monthly meet-up for .NET developers here in Brighton, and a quick run around the table produced an even more pronounced majority for the self-taught. For fun, I also did a quick Twitter poll:

mh-self-taught-twitter-pol

76% say they are self taught. Also interesting were the comments around the poll. This was typical:

self-taught-tweet

Even programmers with CS degrees insist that they are largely self taught. Others complained that it was a hard question to answer since the rate of change in the industry means that you never stop learning. So even if you did at some point have formal training, you can’t rely on that for a successful career. Any formal course will be just a small element of the continual learning that defines the career of a programmer.

We are left with a very strange and unexpected situation. Formal education for programmers seems not to work very well and yet the majority of those who are successful programmers are mostly self taught. On the one hand we seem to have people who don’t need any guided education to give them a successful career; they are perfectly capable of learning their trade from the vast sea of online resources available to anyone who wants to use it. On the other hand we have people who seem unable to learn to code even with years of formal training.

This rather puts the lie to the barriers to entry argument. If the majority of current professional software developers are self taught, how can there be barriers to entry? Anyone with access to the internet can learn to code if they have the aptitude for it.

The evidence points to a very obvious conclusion: there are two populations: one that finds programming a relatively painless and indeed enjoyable thing to learn and another that can’t learn no matter how good the teaching. The elephant in the room, the thing that Yvette Cooper, the ‘year of code’ or ‘hour of code’ people seem unwilling to admit is that programming is a very high aptitude task. It is not one that ‘anyone can learn’, and it is not easy, or rather it is easy, but only if you have the aptitude for it. The harsh fact is that most people will find it impossible to get to any significant standard.

If we accept that programming requires a high level of aptitude, it’s fun to compare some of the hype around the ‘learn to code’ movement with more established high-aptitude professions. Just replace ‘coder’ or ‘coding’ with ‘doctor’,  ‘engineer’,  ‘architect’ or ‘mathematician’.

  • “You can pick up Maths in a day.”
  • Start surgery this year, it’s easier than you think!
  • skyscraper.org aims to help demystify that architecture is difficult.
  • “The sons and daughters of miners should all be learning to be lawyers.”

My friend Andrew Cherry put it very well:

free-training-ac

Answer:  only one: software development. You want to be a doctor? Go to medical school for seven years.

Accepting that aptitude is important for a successful career in programming, we can approach the ‘shortage’ problem from a different angle. We can ask how we can persuade talented people to choose programming rather than other high-aptitude professions. The problem is that these individuals have a great deal of choice in their career path and, as I’m going to explain, programming has a number of negative social and career attributes which make them unlikely to choose it.

There’s no doubt that software development is a very attractive career. It’s well paid, mobile, and the work itself is challenging and rewarding. But it has an image problem. I first encountered this at university in the 1990’s. I did a social science degree (yes I’m one of those self taught programmers). Socially, us arts students looked down on people studying computer science, they were the least cool students on the campus - mostly guys, with poor dress sense. If anyone considered them at all it was with a sense of pity and loathing. When towards the end of my degree, I told my then girlfriend, another social science student, that I might choose a career in programming, she exclaimed, “oh no, what a waste. Why would you want to do that?” If you did a pop-quiz at any middle-class gathering in the UK and asked people to compare, say, medicine, law, architecture or even something like accountancy, with software development, I can guarantee that they would rate it as having a lower social status. Even within business, or at least more traditional businesses, software development is seen as a relatively menial middle-brow occupation suitable for juniors and those ill-qualified for middle management. Perversely, all these courses saying ‘learn to code, it’s easy’ just reinforce the perception that software development is not a serious career.

There’s another problem with software development that’s the flip side of the low barriers to entry mentioned above, and that is there is no well established entry route into the profession. Try Googling for ‘how to become a doctor’, or ‘how to become a lawyer’ for example:

how-to-become-a-doctor

There are a well established series of steps to a recognised professional qualification. If you complete the steps, you become a recognised member of one of these professions. I’m not saying it’s easy to qualify as a doctor, but there’s little doubt about how to go about it. Now Google for ‘how to become a software developer’, the results, like this one for example, are full of vague platitudes like ‘learn a programming language’, ‘contribute to an open source project’, ‘go to a local programming group’. No clear career path, no guarantees about when and if you will be considered a professional and get access to those high-paying jobs of the future.

Yes, I made this up, but it makes the point. :)

Now take a high-aptitude individual who has done well at school and finds demanding intellectual tasks relatively straightforward, and offer them a choice: on the one hand, here is a career, let’s take medicine for example, you follow these clearly enumerated steps, which are demanding but you are good at passing exams, and at the end you will have a high-status, high paying job. Or, how about this career: go away, learn some stuff by yourself, we’re not sure exactly what; try and get a junior, low status job, and just learn more stuff – which you can work out somehow – and work your way up. No guarantees that there’s a well paying job at the end of it. Oh, and, by the way, the whole world will think you are a bit of a social pariah while you are about it. Which would you choose?

So could software development follow the example of older professions and establish a professional qualification with high barriers to entry? There are attempts to do this. The British Computer Society (BCS) calls itself ‘the chartered institute for IT’ and seeks establish professional qualifications and standards. The problem is that it’s comprehensively ignored by the software industry. Even if you could get the industry to take a professional body seriously, how would you test people to see if they qualified? What would be on the exam? There are very few established practices in programming and as soon as one seems to gain some traction it gets undermined by the incredibly rapid pace of change. Take Object Oriented programming for example. In the 2000’s, it seemed to be establishing itself as the default technique for enterprise programming, but now many people, including myself, see it as a twenty year diversion and largely a mistake. How quickly would programming standards and qualifications stay up to date with current practice? Not quickly enough I suspect.

However, my main point in this post has been to establish that programming is a high-aptitude task, one than only some people are capable of doing with any degree of success. If the main point of a professional qualification is filter out people who can’t code, does it really matter if what is being tested for is out of date, or irrelevant to current industry practices? Maybe our tentative qualification would involve the completion of a reasonably serious program in LISP? A kind of Glass Bead Game for programmers? The point would be to find out if they can code. They can learn what the current fads are later. The problem still remains how to get industry to recognise the qualification.

In the meantime we should stop selling people a lie. Programming is not easy, it is hard. You can’t learn to code, certainly not to a standard to get a well-paid-job-of-the-future, in just a few weeks. The majority of the population can not learn to code at all, no matter how much training they receive. I doubt very much if the plethora of quick learn-to-code courses will have any impact at all on the skills shortage, or the problem of unskilled low pay and unemployment. Let’s stop pretending that there are artificial barriers to entry and accept that the main barrier to anyone taking it up is their natural aptitude for it. Instead let’s work on improving the social status of the software industry – I think this is in any case happening slowly – and also work on encouraging talented young people to consider it as a viable alternative to some of the other top professions.

Thursday, September 10, 2015

Partial Application in C#

My recent post, C# Program Entirely With Static Methods, got lots of great comments. Indeed, as is often the case, the comments are in many ways a better read than the original post. However, there were several commenters who claimed that C# does not have partial application. I take issue with this. Any language that supports higher order functions, that is, functions that can take functions as arguments and can return functions, by definition, supports partial application. C# supports higher order functions, so it also supports partial application.

Let me explain.

Let’s start by looking at partial application in F#. Here’s a simple function that adds two numbers (you can type this into F# interactive):

>let add a b = a + b;;

Now we call use our ‘add’ function to add two numbers, just as we’d expect:

> add 3 4;;
val it : int = 7

But because F# supports partial application we can also do this:

> let add3 = add 3;;
> add3 4;;
val it : int = 7

We call add with a single argument and it returns a function that takes a single argument which we can then use to add three to any number.

That’s partial application. Of course, if I try this in C# it doesn’t work:

image

Red squiggly line saying “delegate Func has two parameters but is invoked with one argument.

Case proven you say: C# does not support partial application!

But wait!

Let’s look again at the F# add function. This time I’ll include the response from F# interactive:

> let add a b = a + b;;
val add : a:int -> b:int -> int

This shows us the type of the add function. The important bit is: “a:int –> b:int –> int”. This tells us that ‘add’ is a function that takes an int and returns a function that takes an int and returns an int. It is not a function with two arguments. F# is a restrictive language, it only has functions with single arguments. That is a good thing. See Mark Seemann’s post Less is More: Langauge Features for an in depth discussion of how taking features away from a language can make it better. When people say “F# supports partial application” what they really mean is that “F# functions can only have one argument.” The F# compiler understands the syntax ‘let add a b = …’ to mean “I want a function that takes a single argument and returns a function that takes another single argument.”

There’s nothing to stop us from defining our C# function with the same signature as our F# example. Then we can partially apply it in the same way:

image

There you are: partial application in C#. No problem at all.

“But!” You cry, “That’s weird and unusual C#. I don’t want to define all my functions in such a strange way.” In that case, let me introduce you to my friend Curry. It’s not a spicy dish of South Asian origin but the process of turning a function with multiple arguments into a series of higher order functions. We can define a series of overloaded Curry extension methods:

image

We can then use them to turn ‘ordinary’ C# functions with multiple arguments into higher-order functions which we can partially apply:

image

Thinking more about Mark Seemann’s blog post, it would be an interesting exercise to start to take features away from C# whilst keeping syntactic changes to a minimum. If we took away multiple function arguments, classes, interfaces, nullable types, default mutability etc, would we end up with a subset language that would be perfect for functional programming, but still familiar to C# developers? You would of course lose backward compatibility with existing C# code, so the incentive to do it isn’t that great, but it’s a fascinating thought experiment.

Friday, August 07, 2015

C#: Program Entirely With Static Methods

OK, that’s a provocative title to get your attention. This post is really about how one can move to a more functional programming style and remove the need for much of the apparatus of object-oriented programming, including interfaces and classes. In this post, I’m going to take some typical object-oriented C# code and refactor it in a more functional style. I’ll show that the result is more concise and easier to test.

Over the past couple of years I’ve noticed that my C# coding style has changed drastically under the influence of functional programming. Gone are interfaces and instance classes to be replaced by static methods, higher-order functions and closures. It’s somewhat ironic since I spent many years as a cheerleader for object-oriented programming and I considered static methods a code smell.

I guess if I look at my programming career, it has the following progression:

Procedural –> Object-Oriented –> Functional

The OO phase now looks like something of a detour.

C# has all the essential features you need for functional programming – higher-order functions, closures, lambda expressions – that allow you to entirely ditch the OO programming model. This results in more concise, readable and maintainable code. It also has a huge impact on unit testing, allowing one to do away with complex mocking frameworks, and write far simpler tests.

Introducing our object oriented example

Let’s look at an example. First I’ll introduce a highly simplified OO example, a simple service that grabs some customer records from a data-store, creates some reports and then emails them. Then I’ll show the same code refactored in a more functional style using delegates and higher-order static methods.

Let’s look at the object-oriented example first:

Well written object-oriented code is compositional. Concrete classes depend on abstractions (interfaces). These interfaces are consumed as dependencies by classes that rely on them and are usually injected as constructor arguments. This is called Dependency Injection. It’s good practice to compose object instances in a single place in the application - the composition root - usually when the application starts up, or on a significant event, such as an HTTP request. The composition can be hand coded or handed off to an IoC container. The constructed graph is then executed by invoking a method on the root object. This often occurs via an application framework (such as MVC or WebApi) rather than being explicitly invoked by user code.

We are going to get some customer records, create some reports and then email them to our customers. So first we need three interfaces: a data access abstraction, a report building abstraction, and an emailing abstraction:

And here are the implementations. This is not a real program of course, I’ve just coded some dummy customers and the emailer simply writes to the console.

Now we have our service class that depends on the three abstractions and orchestrates the reporting process:

As you can see, we inject the dependencies as constructor arguments, store them in class properties, then invoke methods on them in the code in the RunCustomerReportBatch method. Some people like to store the dependencies in class fields instead. That’s a matter of choice.

Our composition root composes the ReportingService with its dependencies and then returns it for the program to invoke. Don’t forget this is a highly simplified example. Composition is usually far more complex:

To write a unit test for the reporting service we would typically use either hand-crafted mocks, or some kind of mocking framework. Here’s an example unit test using XUnit and Moq:

We first create mocks for ReportingService’s dependencies with the relevant methods stubbed, which we inject as constructor arguments. We then invoke ReportingService and verify that the emailer was invoked as expected.

So that’s our object-oriented example. It’s typical of much well constructed C# code that you will find in the wild. It’s the way I’ve been building software for many years now with much success.

However, this object-oriented code is verbose. About a third of it is simply OO stuff that we have to write repeatedly and mechanically rather than code that is actually solving our problem. This boilerplate includes: the class’ properties (or fields) to hold the dependencies; the assigning of constructor arguments to those properties; writing the class and constructor. We also need complex mocking frameworks simply to test this code. Surely that’s a smell that’s telling us something is wrong?

Enlightenment

Enlightenment begins when you realise that the dependencies and method arguments can actually just be seen as arguments that are applied at different times in the application’s lifecycle. Consider a class with a single method and a single dependency:

We could equally represent this as a static method with two arguments:

But how do we partially apply these arguments? How do we give ‘DoThing’ the IDependency argument at composition time and the ‘string arg’ at the point where it is required by the application logic? Simple: We use a closure. Anything taking a dependency on ‘DoThing’ will ask for an Action<string>, because that is the signature of the ‘Do’ method in our ‘Thing’ class. So in our composition root, we ‘close over’ our previously created IDependency instance in a lambda expression with the signature, Action<string>, that invokes our DoThing static method. Like this:

So the interface is replaced with the built-in Action<T> delegate, and the closure is effectively doing the job of our ‘Thing’ class, the interface’s implementation, but with far fewer lines of code.

Refactoring to functional

OK. Let’s go back to our example and change it to use this new insight. We don’t need the interface definitions. They are replaced by built in delegate types:

ICustomerData becomes Func<IEnumerable<Customer>>

IEmailer becomes Action<string, string>

IReportBuilder becomes Func<Customer, Report>

The classes are replaced with static methods:

Our ReportingService is also replaced with a single static method that takes its dependencies as delegate arguments:

Composition looks like this:

This is functionally equivalent to the object-oriented code above, but it has 57 lines of code as opposed to 95; exactly 60% of the original code.

There’s also a marked simplification of the unit test:

The requirement for a complex mocking framework vanishes. Instead we merely have to set up simple lambda expressions for our stubs. Expectations can be validated with closed over local variables. It’s much easier to read and maintain.

Moving to a functional style of programming is certainly a huge departure from most C# code that you find in the wild and can initially look a little odd to the uninitiated. But it has many benefits, making your code more concise and easier to test and reason about. C# is, surprisingly, a perfectly adequate functional programming language, so don’t despair if for practical reasons you can’t use F#.

The complete code example for this post is on GitHub here: https://github.com/mikehadlow/FunctionalDemo

Friday, June 05, 2015

C#: How to Record What Gets Written to or Read From a Stream

Streams are a very nice abstraction over a read/write loop. We can use them to represent the contents of a file, or a stream of bytes to or from a network socket. They make it easy to read and write large amounts of data without consuming large amounts of memory. Take this little code snippet:

Example.txt may be many GB in size, but this operation will only ever use the amount of memory configured for the buffer. As an aside, the .NET framework’s Stream class’s default buffer size is the maximum multiple of 4096 that is still smaller than the large object heap threshold (85K). This means it likely to be collected at gen zero by the garbage collector, but still gives good performance.

But what if we want to log or view the contents of Example.txt as it’s copied to the output file? Let me introduce my new invention: InterceptionStream. This is simple class that inherits and decorates Stream and takes an additional output stream. Each time the wrapped stream is read from, or written to, the additional output stream gets the same information written to it. You can use it like this:

I could just as well have wrapped the input stream with the InterceptionStream for the same result:

You can use a MemoryStream if you want to capture the log in memory and assign it to a string variable, but of course this negates the memory advantages of the stream copy since we’re now buffering the entire contents of the stream in memory:

Here is the InterceptionStream implementation. As you can see it’s very simple. All the work happens in the Read and Write methods:

Thursday, May 28, 2015

Inject DateTime.Now to Aid Unit Tests

If you have logic that relies on the current system date, it's often difficult to see how to unit test it. But by injecting a function that returns DateTime.Now we can stub the current date to be anything we want it to be.
Let's look at an example. Here we have a simple service that creates a new user instance and saves it in a database:
    public class UserService : IUserService
    {
        private readonly IUserData userData;

        public UserService(IUserData userData)
        {
            this.userData = userData;
        }

        public void CreateUser(string username)
        {
            var user = new User(username, createdDateTime: DateTime.UtcNow);
            userData.SaveUser(user);
        }
    }
Now if I want to write a unit test that checks that the correct created date is set, I have to rely on the assumption that the system date won't change between the creation of the User instance and the test assertions.
    [TestFixture]
    public class UserServiceTests
    {
        private IUserService sut;
        private IUserData userData;

        [SetUp]
        public void SetUp()
        {
            userData = MockRepository.GenerateStub<iuserdata>();
            sut = new UserService(userData);
        }

        [Test]
        public void UserServiceShouldCreateUserWithCorrectCreatedDate()
        {
            User user = null;

            // using Rhino Mocks to grab the User instance passed to the IUserData stub
            userData.Stub(x => x.SaveUser(null)).IgnoreArguments().Callback<user>(x =>
            {
                user = x;
                return true;
            });

            sut.CreateUser("mike");

            Assert.AreEqual(DateTime.UtcNow, user.CreatedDateTime);
        }
    }
But in this case, probably because Rhino Mocks is doing some pretty intensive proxying, a few milliseconds pass between the user being created and my assertions running.
Test 'Mike.Spikes.InjectingDateTime.UserServiceTests.UserServiceShouldCreateUserWithCorrectCreatedDate' failed: 
  Expected: 2015-05-28 09:08:18.824
  But was:  2015-05-28 09:08:18.819
 InjectingDateTime\InjectDateTimeDemo.cs(75,0): at Mike.Spikes.InjectingDateTime.UserServiceTests.UserServiceShouldCreateUserWithCorrectCreatedDate()
The solution is to inject a function that returns a DateTime:
    public class UserService : IUserService
    {
        private readonly IUserData userData;
        private readonly Func<datetime> now;

        public UserService(IUserData userData, Func<datetime> now)
        {
            this.userData = userData;
            this.now = now;
        }

        public void CreateUser(string username)
        {
            var user = new User(username, createdDateTime: now());
            userData.SaveUser(user);
        }
    }
Now our unit test can rely on a fixed DateTime value rather than one that is changing as the test runs:
    [TestFixture]
    public class UserServiceTests
    {
        private IUserService sut;
        private IUserData userData;

        // stub the system date as some arbirary date
        private readonly DateTime now = new DateTime(2015, 5, 28, 10, 46, 33);

        [SetUp]
        public void SetUp()
        {
            userData = MockRepository.GenerateStub<iuserdata>();
            sut = new UserService(userData, () => now);
        }

        [Test]
        public void UserServiceShouldCreateUserWithCorrectCreatedDate()
        {
            User user = null;
            userData.Stub(x => x.SaveUser(null)).IgnoreArguments().Callback<user>(x =>
            {
                user = x;
                return true;
            });

            sut.CreateUser("mike");

            Assert.AreEqual(now, user.CreatedDateTime);
        }
    }
And the test passes as expected.
In our composition root we inject the current system time (here as UTC):
    var userService = new UserService(userData, () => DateTime.UtcNow);
This pattern can be especially useful when we want to test business logic that relies on time passing. For example, say we want to check if an offer has expired; we can write unit tests for the case where the current (stubbed) time is both before and after the expiry time just by injecting different values into the system-under-test. Because we can stub the system time to be anything we want it to be, it makes it easy to test time based busines logic.

Wednesday, April 22, 2015

A Simple Nowin F# Example

In my last post I showed a simple F# OWIN self hosted server without an application framework. Today I want to show an even simpler example that doesn’t reference any of the Microsoft OWIN libraries, but instead uses an open source server implementation, Nowin. Thanks to Damien Hickey for pointing me in the right direction.

The great thing about the Open Web Interface for .NET (OWIN) is that it is simply a specification. There is no OWIN library that you have to install to allow web servers, application frameworks and middlewear built to the OWIN standard to communicate. There is no interface that they must implement. They simply need to provide an entry point for the OWIN application delegate (better know as the AppFunc):

    Func<IDictionary<string , object>, Task>

For simple applications, where we don’t need routing, authentication, serialization, or an application framework, this means we can simply provide our own implementation of the AppFunc and pass it directly to an OWIN web server.

Nowin, by Boris Letocha, is a .NET web server, built directly against the standard .NET socket API. This means it should work on all platforms that support .NET without modification. The author claims that it has equivalent performance to NodeJS on Windows and can even match HttpListener. Although not ready for production, it makes a compelling implementation for simple test servers and stubs, which is how I intend to use it.

To use any OWIN web server with F#, we simply need to provide an AppFunc and since F# lambdas have an implicit cast to System.Func<..> we can simply provide the AppFunc in the form:

    fun (env: IDictionary<string, obj>) -> Task.FromResult(null) :> Task

Let’s see it in action. First create an F# console application and install the Nowin server with NuGet:

    Install-Package Nowin

Now we can host our Nowin server in the application’s entry point:

    [<entrypoint>]
    let main argv = 

        use server = 
            Nowin.ServerBuilder
                .New()
                .SetEndPoint(new IPEndPoint(IPAddress.Any, port))
                .SetOwinApp(fun env -> Task.FromResult(null) :> Task)
                .Build()

        server.Start() 

        printfn "Server listening on http://localhost:%i/ \nhit <enter> to stop." port
        Console.ReadLine() |> ignore

        0

Of course this server does nothing at all. It simply returns the default 200 OK response with no body. To do any useful work you need to read the OWIN environment, understand the request and create a response. To make this easier in F# I’ve created a simple OwinEnvironment type with just the properties I need. You could expand this to encompass whatever OWIN environment properties you need. Just look at the OWIN spec for this.

    type OwinEnvironment = {
        httpMethod: string;
        requestBody: Stream;
        responseBody: Stream;
        setResponseStatusCode: (int -> unit);
        setResponseReasonPhrase: (string -> unit)
    }

Here is a function that takes the AppFunc environment and maps it to my OwinEnvironment type:

    let getOwinEnvironment (env: IDictionary<string , obj>) = {
        httpMethod = env.["owin.RequestMethod"] :?> string;
        requestBody = env.["owin.RequestBody"] :?> Stream;
        responseBody = env.["owin.ResponseBody"] :?> Stream;
        setResponseStatusCode = 
            fun (statusCode: int) -> env.["owin.ResponseStatusCode"] <- statusCode
        setResponseReasonPhrase = 
            fun (reasonPhrase: string) -> env.["owin.ResponseReasonPhrase"] <- reasonPhrase
    }

Now that we have our strongly typed OwinEnvironment, we can grab the request stream and response stream and do some kind of mapping. Here is a function that does this. It also only accepts POST requests, but you could do whatever you like in the body. Note the transform function is where the work is done.

    let handleOwinEnvironment (owin: OwinEnvironment) : unit =
        use writer = new StreamWriter(owin.responseBody)
        match owin.httpMethod with
        | "POST" ->
            use reader = new StreamReader(owin.requestBody)
            writer.Write(transform(reader.ReadToEnd()))
        | _ ->
            owin.setResponseStatusCode 400
            owin.setResponseReasonPhrase "Bad Request"
            writer.Write("Only POST requests are allowed")

Just for completeness, here is a trivial transform example:

    let transform (request: string) : string =
        sprintf "%s transformed" request

Now we can re-visit our console Main function and pipe everything together:

    [<entrypoint>]
    let main argv = 

        use server = 
            Nowin.ServerBuilder
                .New()
                .SetEndPoint(new IPEndPoint(IPAddress.Any, port))
                .SetOwinApp(fun env -> 
                    env 
                    |> getOwinEnvironment 
                    |> handleOwinEnvironment 
                    |> endWithCompletedTask)
                .Build()

        server.Start() 

        printfn "Server listening on http://localhost:%i/ \nhit  to stop." port
        Console.ReadLine() |> ignore

        0

The endWithCompletedTask function, is a little convenience to hide the ugly synchronous Task return code:

    let endWithCompletedTask = fun x -> Task.FromResult(null) :> Task

So as you can see, OWIN and Nowin make it very easy to create small web servers with F#. Next time you just need a simple service stub or test server, consider doing something like this, rather that using a heavyweight server and application framework such as IIS, MVC, WebAPI or WebForms.

You can find the complete code for the example in this Gist https://gist.github.com/mikehadlow/c88e82ee98619f22f174:

Thursday, April 16, 2015

Basic OWIN Self Host With F#

I’m still very much an F# noob, but yesterday I thought I’d use it to write a little stub web service for a project I’m currently working on. I simply want to respond to any POST request to my service. I don’t need routing, or any other ‘web framework’ pieces. I just wanted to use the Microsoft.AspNet.WebApi.OwinSelfHost package to create a little web service that runs inside a console program.

First create a new F# console project. Then install the self host package:

    Microsoft.AspNet.WebApi.OwinSelfHost

Note that this will also install various WebApi pieces which we don’t need here, so we can go ahead and uninstall them:

    uninstall-package Microsoft.AspNet.WebApi.OwinSelfHost
    uninstall-package Microsoft.AspNet.WebApi.Owin
    uninstall-package Microsoft.AspNet.WebApi.Core
    uninstall-package Microsoft.AspNet.WebApi.Client

My requirement is to simply take any POST request to the service, take the post body and transform it in some way (that’s not important here), and then return the result in the response body.

So first, here’s a function that takes a string and returns a string:

    let transform (input: string) =
        sprintf "%s transformed" input

Next we’ll write the OWIN start-up class. This needs to be a class with a single member, Configuration, that takes an IAppBuilder:

    open Owin
    open Microsoft.Owin
    open System
    open System.IO
    open System.Threading.Tasks

    type public Startup() = 
        member x.Configuration (app:IAppBuilder) = app.Use( ... ) |> ignore

We need something to pass into the Use method on IAppBuilder. The Use method looks like this:

    public static IAppBuilder Use(
        this IAppBuilder app,
        Func<IOwinContext, Func<Task>, Task> handler
    )

So we need a handler with the signature Func<IOwinContext, Func<Task>, Task>. Since F# lambdas cast directly to Func<..> delegates, we simply use lots of type annotations and write a function which looks like this:

    let owinHandler = fun (context:IOwinContext) (_:Func) -> 
        handleOwinContext context; 
        Task.FromResult(null) :> Task

Note that this is running synchronously. We’re just returning a completed task.

Now lets look at the handleOwinContext function. This simply takes the IOwinContext, grabs the request, checks that it’s a ‘POST’, and transforms the request stream into the response stream using our transform function:

    let handleOwinContext (context:IOwinContext) =

        use writer = new StreamWriter(context.Response.Body)

        match context.Request.Method with
        | "POST" -> 
            use reader = new StreamReader(context.Request.Body)
            writer.Write(transform(reader.ReadToEnd()))
        | _ ->
            context.Response.StatusCode <- 400
            writer.Write("Only POST")

Now all we need to do is register our Startup type with the OWIN self host in our Program.Main function:

open System
open Microsoft.Owin.Hosting

[]
let main argv = 

    let baseAddress = "http://localhost:8888"

    use application = WebApp.Start<Startup.Startup>(baseAddress)

    Console.WriteLine("Server running on {0}", baseAddress)
    Console.WriteLine("hit <enter> to stop")
    Console.ReadLine() |> ignore
    0

And we’re done. Now let’s try it out with the excellent Postman client, just run the console app and send a POST request to http://localhost:8888/:

Postman_owin_self_host_fsharp

Full source code in this Gist.

Monday, December 15, 2014

The Lava Layer Anti-Pattern

TL:DR Successive, well intentioned, changes to architecture and technology throughout the lifetime of an application can lead to a fragmented and hard to maintain code base. Sometimes it is better to favour consistent legacy technology over fragmentation.

An ‘anti-pattern’ describes a commonly encountered pathology or problem in software development. The Lava Layer (or Lava Flow) anti-pattern is well documented (here and here for example). It’s symptoms are a fragile and poorly understood codebase with a variety of different patterns and technologies used to solve the same problems in different places. I’ve seen this pattern many times in enterprise software. It’s especially prevalent in situations where the software is large, mission critical, long-lived and where there is high staff turn-over. In this post I want to show some of the ways that it occurs and how it’s often driven by a very human desire to improve the software.

To illustrate I’m going to tell a story about a fictional piece of software in a fictional organisation with made up characters, but closely based on real examples I’ve witnessed. In fact, if I’m honest, I’ve been several of these characters at different stages of my career.  I’m going to concentrate on the data-access layer (DAL) technology and design to keep the story simple, but the general principles and scenario can and do apply to any part of the software stack.

Let’s set the scene…

The Royal Churchill is a large hospital in southern England. It has a sizable in-house software team that develop and maintain a suite of applications that support the hospital’s operations. One of these is WidgetFinder, a physical asset management application that is used to track the hospital’s large collection of physical assets; everything from beds to CT scanners. Development on WidgetFinder was started in 2005. The software team that wrote version 1 was lead by Laurence Martell, an developer with may years experience building client server systems based on VB/SQL Server. VB was in the process of being retired by Microsoft, so Laurence decided to build WidgetFinder with the relatively new ASP.NET platform. He read various Microsoft design guideline papers and a couple of books and decided to architect the DAL around the ADO.NET RecordSet. He and his team hand coded the DAL and exposed DataSets directly to the UI layer, as was demonstrated in the Microsoft sample applications. After seven months of development and testing, Version 1 of WidgetFinder was released and soon became central to the Royal Churchill’s operations. Indeed, several other systems, including auditing and financial applications, soon had code that directly accessed WidgetFinders database.

Like any successful enterprise application, a new list of requirements and extensions evolved and budget was assigned for version 2. Work started in 2008. Laurence had left and a new lead developer had been appointed. His name was Bruce Snider. Bruce came from a Java background and was critical of many of Laurence’s design choices. He was especially scornful of the use of DataSets: “an un-typed bag of data, just waiting for a runtime error with all those string indexed columns.” Indeed WidgetFinder did seem to suffer from those kinds of errors. “We need a proper object-oriented model with C# classes representing tables, such as Asset and Location. We can code gen most of the DAL straight from the relational schema.” He asked for time and budget to rewrite WidgetFinder from scratch, but this was rejected by the management. Why would they want to re-write a two year old application that was, as far as they were concerned, successfully doing its job? There was also the problem that many other systems relied on WidgetFinder’s database and they would need to be re-written too.

Bruce decided to write the new features of WidgetFinder using his OO/Code Gen approach and refactor any parts of the application that they had to touch as part of version 2. He was confident that in time his Code Gen DAL would eventually replace the hand crafted DataSet code. Version 2 was released a few months later. Simon, a new recruit on the team asked why some of the DAL was code generated, and some of it hand-coded. It was explained that there had been this guy called Lawrence who had no idea about software, but he was long gone.

A couple of years went by. Bruce moved on and was replaced by Ina Powers. The code gen system had somewhat broken down after Bruce had left. None of the remaining team really understood how it worked, so it was easier just to modify the code by hand. Ina found the code confusing and difficult to reason about. “Why are we hand-coding the DAL in this way? This code is so repetitive, it looks like it was written by an automation. Half of it uses DataSets and the other some half baked Active Record pattern. Who wrote this crap? If you hand code your DAL, you are stealing from your employer. The only sensible solution is an ORM. I recommend that we re-write the system using a proper domain model and NHibernate.” Again the business rejected a rewrite. “No problem, we will adopt an evolutionary approach: write all the new code DDD/NHibernate style, and progressively refactor the existing code as we touch it.” Many months later, Version 3 was released.

Mandy was a new hire. She’d listened to Ina’s description of how the application was architected around DDD with the data access handled by NHibernate, so she was surprised and confused to come across some code using DataSets. She asked Simon what to do. “Yeah, I think that code was written by some guy who was here before me. I don’t really know what it does. Best not to touch it in case something breaks.”

Ina, frustrated by management who didn’t understand the difficulty of maintaining such horrible legacy applications, left for a start-up where she would be able to build software from scratch. She was replaced by Gordy Bannerman who had years of experience building large scale applications. The WidgetFinder users were complaining about it’s performance. Some of the pages took 30 seconds or more to appear. Looking at the code horrified him: Huge Linq statements generating hundreds of individual SQL requests, no wonder it was slow. Who wrote this crap? “ORMs are a horrible leaky abstraction with all kinds of performance problems. We should use a lightweight data-access technology like Dapper. Look at Stack-Overflow, they use it. They also use only static methods for performance, we should do the same.” And so the cycle repeated itself. Version 4 was released a year later. It was buggier than the previous versions. Gordy had dismissed Ina’s love of unit testing. It’s hard to unit test code written mostly with static methods.

Mandy left to be replaced by Peter. Simon introduced him to the WidgetFinder code. “It’s not pretty. A lot of different things have been tried over the years and you’ll find several different ways of doing the same thing depending on where you look. I don’t argue, just get on with trawling through the never ending bug list. Hey, at least it’s a job.”

This is a graphical representation of the DAL code over time. The Y-axis shows the version of the software. It starts with version one at the bottom and ends with version four at the top. The X-axis shows features, the older ones to the left and the newer ones to the right. Each technology choice is coloured differently. red is the hand-coded RecordSet DAL, blue the Active Record code gen, green DDD/NHibernate and Yellow is Dapper/Static methods.

LavaLayer

Each new design and technology choice never completely replaced the one that went before. The application has archaeological layers revealing it’s history and the different technological fashions taken up successively by Laurence, Bruce, Ina and Gordy. If you look along the Version 4 line, you can see that there are four different ways of doing the same thing scattered throughout the code base.

Each successive lead developer acted in good faith. They genuinely wanted to improve the application and believed that they were using the best design and technology to solve the problem at hand. Each wanted to re-write the application rather than maintain it, but the business owners would not allow them the resources to do it. Why should they when there didn’t seem to be any rational business reason for doing so? High staff turnover exacerbated the problem. The design philosophy of each layer was not effectively communicated to the next generation of developers. There was no consistent architectural strategy. Without exposition or explanation, code standing alone needs a very sympathetic interpreter to understand its motivations.

So how should one mitigate against Lava Layer? How can we approach legacy application development in a way that keeps the code consistent and well architected? A first step would be a little self awareness.

We developers should recognise that we suffer from a number of quite harmful pathologies when dealing with legacy code:

  • We are highly (and often overly) critical of older patterns and technologies. “You’re not using a relational database?!? NoSQL is far far better!” “I can’t believe this uses XML! So verbose! JSON would have been a much better choice.”
  • We think that the current shiny best way is the end of history; that it will never be superseded or seen to be suspect with hindsight.
  • We absolutely must ritually rubbish whoever came before us. Better still if they are no longer around to defend themselves. There’s a brilliant Dilbert cartoon for this.
  • We despise working on legacy code and will do almost anything to carve something greenfield out of an assignment, even if it makes no sense within the existing architecture.
  • Rather than try to understand legacy code, how it works and the motivations that created it, we throw up our hands in despair and declare that the whole thing needs to be rewritten.

If you find yourself suggesting a radical change to an existing application, especially if you use the argument that, “we will refactor it to the new pattern over time.” Consider that you may never complete that refactoring, and think about what the application will look like with two different ways of doing the same thing. Will this aid those coming after you, or hinder them? What happens if your way turns out to be sub-optimal? Will replacing it be easy? Or would it have been better to leave the older, but more consistent code in place? Is WidgetFinder better for having four entirely separate ways of getting data from the database to the UI, or would it have been easier to understand and maintain with one? Try and have some sympathy and understanding for those who came before you. There was probably a good reason for why things were done the way they were. Be especially sympathetic to consistency, even if you don’t necessarily agree with the design or technology choices.

Thursday, July 03, 2014

Hire Me

I’m on a sales drive. I want to move away from daily-rate contracting, and focus on full-lifecycle project delivery. I’ve created a new website to help market myself http://mikehadlow.com/. I’m looking for customers who want software written to a specification for a fixed price.

Screen Shot 2014-07-03 at 14.49.11

I’ve been working in IT since 1996, although I’ve played with computers and programming since I was a teenager. Except for the first two years when I had a permanent job, I’ve worked as a daily or hourly rate contractor, with just the occasional foray into fixed-price project work. Looking at my CV I can count 17 different organizations that  I’ve worked for during that time. Some of them where large companies where I was just a small part of a large team. For example, I was one of over a hundred contractors at one particular public sector project. Others were tiny local Brighton companies where I was often the end-to-end developer for a complete system. I’ve had a variety of roles, from being a travelling troubleshooter, driving around the country fixing installs of one particularly nasty system, a bug-fixer for months on end on a huge mission critical system, and a plug-n-play C# programmer on a whole range of different projects. More recently, for the last five years or so, I’ve mostly been hired in an ‘architect’ role. What this means is somewhat vague, but it usually encompasses giving higher-level strategic design direction and getting involved in team structure, process design, and planning. All this experience has given me some very strong opinions about what makes a successful software project. I hope that’s pretty obvious to anyone reading this blog. It’s also given me the confidence to take responsibility for the entire project lifecycle.

During this time I’ve also occasionally done fixed-price projects. The largest of these was a customer relationship management system for a pharmaceutical company, this was a six month project which I worked with a DBA to deliver. I’ve also built a property management system for a legal practice, and a complete eCommerce system that I’ve also maintained for the last six years. I always enjoyed these projects the most. It’s very satisfying to be able to deliver a working system to a client and see it really helping their business. The problem has been finding the work. I’m hoping now that the popularity of this blog and the success of EasyNetQ will provide enough of an audience for me that I’ll be able to do projects full time.

I want to move from being just an element of a project’s delivery, to being the person responsible for it.  Taking responsibility means delivering to a price and time-scale and creating and managing the team that does it. So if you have a requirement for bespoke software and you need a safe pair of hands to deliver it, please get in touch with me at mike@suteki.co.uk and let’s talk.