Moving Suteki Shop from Linq-to-SQL to NHibernate

I’ve finished getting the basic features of Suteki Shop running with NHibernate. Anyone who’s been following this project knows that it was originally written with Linq-to-SQL. Linq-to-SQL worked fine for the initial iterations of Suteki Shop, but there are lots of things I wanted to do with the project that would have been hard or impossible to do without the move.

Linq-to-SQL not a true ORM in the sense that it doesn’t support persistence ignorance, it’s more a object based projection of a relational database. Each class directly represents a table row, including foreign keys. One of the main pain points in the transition was removing all the application code that referenced these foreign keys and replacing them with object references instead. So for example, instead of changing the int value of Contact.CountryId I would change the actual referenced entity, Contact.Country.

The other place where I had to do lots of work was in my Linq queries. The NHibernate trunk now has a native Linq provider written by Steve Strong, so you can write:

var customer = session.Query<Customer>().Where(c => c.Name == "Fred").FirstOrDefault();

The main difference between the Linq-to-SQL provider and the NHiberante one is in the way they treat expressions that they don’t understand. Linq-to-SQL will translate as much of the expression that it can to SQL, get the result, and then allow Linq-to-objects to do the rest. The NHibernate provider will throw an exception saying that it doesn’t understand the expression. There are pros and cons to both approaches. Linq-to-SQL will always give you what you want, but maybe in an inefficient way you weren’t expecting, whereas NHibernate will force you to think about how you structure your queries. There are no surprises, but it’s a mark against persistence ignorance. Right now, I’m liking the NHibernate no surprises approach.

Whilst doing the conversion I ran into a problem with the OfType<T>() Linq extension method. I was using it in several places but NHibernate complained that it didn’t support it. I tweeted about it and almost immediately Steve Strong replied that it should be easy to implement. The next day he committed it to the NHibernate trunk and my queries all worked.

Steve Strong, you are an awesome open source hero!

One of my major regrets with the way that I originally wrote Suteki Shop is binding views to entities, rather than using a view model. At first this can seem like an easy-win, but doing this work showed how can soon lead to all sorts of difficulties. Displaying the view model in the view is not so much of a problem - except in entity loading overhead. The problems mostly manifest themselves at the binding stage. I extended the ASP.NET MVC DefaultModelBinder to work with NHibernate entities, and it mostly works, but I spent far too much time resolving tricky binding issues with complex object graphs. I would have made my life much easier by binding to a nice flat view model and then manually mapping that to my entities.

Persistence ignorance is a dream, a nirvana, where we’d love to be, but no ORM will give you true persistence ignorance. NHibernate is far better in this regard than Linq-to-SQL, but you still have to do things the NHibernate way. Your ORM will hugely influence the way you build your application and although the persistence ignorance ideal should mean you can swap them with little difficulty, moving from Linq-to-SQL to NHibernate was a major piece of work, far more than I expected.

Multi-tenancy part 1: Strategy.

I want my eCommerce application Suteki Shop to be able to flexibly meet the requirements of any customer (within reason). How can we have a single product but enable our customers to have diverse implementations. I think the solution depends on the level of diversity and the number of customers. The right solution for a product with thousands of customers with little customisation is different from a solution for a handful of customers with very diverse requirements.

There are several axis of change to consider:

1. Codebase. Do I have one codebase, or do I maintain a separate codebase for each customer?
2. Application instance. Do I have one application instance to service all my customers, or does each customer have a separate one?
3. Database schemas. Do I have one database schema, or do I have a different schema for each customer?
4. Database instances. Do I have one database instance or separate ones for each customer?

Lots of customers, little customisation

Lets consider the two extremes. First, say I've got a product that I expect to sell to thousands of customers. My business model is premised on selling a cheap service to very many people. It's worth my while not to allow too much customisation because the effort to supply that simply wouldn't be worth it. If someone wants something significantly different from my thousands of other customers, I'll just tell them to look elsewhere. In this case I'd have a single code base, application instance, database schema and database. Effectively a single application will service all my customers.

In this case the main technical challenge will making sure that each user's data is keyed properly so they perceive it as a single application servicing their needs only. It would be a major failure if one customer could see another's data. Think of an on-line email service such as hotmail. Sometimes we want to allow customers to see each other's data, think Facebook, but that interaction needs to be tightly controlled.

Scope for providing diverging customer requirements is very limited in this case. Effectively every customer gets exactly the same features and all you can do is allow them to maybe switch features off and on. The cost of developing a customisable experience with such a setup is high.

The great thing about this single application approach is its scalability. Each extra customer requires no extra development effort. It's the way billion dollar on-line businesses are made.

Few customers, Deep customisation

At the other extreme we might be selling an application to just a handful of clients. Each client has diverse requirements and is willing to spend the money to have them implemented. Do we really have a single application in this case or are we instead delivering bespoke solutions? Here the answer might be to have a common application library but separate codebases for each client. It would then follow that each client needs separate database schemas, databases and application instances.

In this scheme, the cost of developing divergent features is no greater than developing them for bespoke software. The pain comes in deciding what to code in your shared library and what to code in your bespoke applications. Getting that wrong can mean repeating the same code for each customer and seriously violating the DRY principle.

Because we are effectively building bespoke software, each extra customer requires development effort. As our business grows we have to hire more and more developers to service our customers. The problem is that development teams don't scale well and there are high organisational costs involved. Our profit on each instance of our product is likely to decline rather than rise. For this reason there's a limit to the size a company based on this development model can grow to.

So which one is Suteki Shop?

Suteki Shop only has one customer at present. So it's a single bespoke application based on that customer's requirements. In the new year I hope to add a second customer so I'll have to start making decisions about my multi-tenancy strategy. I want to make the right decisions now so that I can provision my third, fourth, tenth or one-hundredth as easily as possible. My customers are likely to have divergent requirements within the constraints of an eCommerce system. The easiest thing for my second customer would be to branch the code, make any changes they want and deploy a second database (with possibly a customised schema) and application instance. But that strategy will not scale to one-hundred customers. Let's make the assumption that I want to service maybe a hundred customers easily, but I'm unlikely to have more than a thousand. Consider the four axis of change above...

1. Codebase. I really don't want to fork my codebase. I'm a firm believer in keeping things DRY. There will be no branching.
2. Application Instance. This is less clear cut. There are obvious benefits from maintaining separate application instances for each customer and this was my initial intention. I can configure and deploy them as required. However, after looking at techniques for provisioning multiple divergent customers on one instance, I believe that the costs can be minimal. Having one instance is much easier to administer and monitor, so I am changing my mind on this point and will aim for one application instance. One application instance can of course be duplicated to scale over a server farm. I think this approach could scale to thousands of customers.
3. Database schemas. In the old days, I would have said that having multiple schemas would cause intense scale pain, but that is changing with modern tools. Using an ORM allows us to defer schema creation until deployment. If I can automate the management of schema creation and change, then having multiple versions is less of a problem. Of course that all depends on having multiple databases....
4. Database instances. If I chose to have one database instance I would have to have one schema which limits the amount of customisation I can do. I would also have to carefully key each customer's data, which is a development overhead. Automating database deployment and maintenance is relatively straightforward  so I think the choice here is clear cut. One database per customer. Also this gives me extremely simple database scalability, I can simply farm out my customers to multiple DBMS's. This approach will scale to hundreds of customers, thousands would be problematic but not impossible.

So now I've got my multi-tenancy strategy, how am I going to implement it? The core technology I'm going to use is component oriented software design using Dependency Injection and an IoC Container. I'm going to follow some of the recommendations that Oren Eini makes in these posts:

Multi Tenancy - Approaches and Applicability

Adaptive Domain Models with Rhino Commons

Components, Implementations and Contextual Decisions

Windsor - IHandlerSelector

Windsor - IModelInterceptersSelector

In part 2, I'm going to show my first faltering steps to making a single instance of Suteki Shop host more than one customer. Stay tuned!

Upgrading Suteki Shop to the MVC Framework Beta

I've just gone through the process of upgrading Suteki Shop, my open source eCommerce application from MVC Framework Preview 4 to the Beta. Yes, I know, I missed out Preview 5 all together, so my experiences are going to be different from someone just moving from Preview 5 to the Beta.

Here's a brief rundown of what was involved:

1. Download and install the MVC Beta
2. Get the latest MvcContrib trunk and build it. It already references the MVC Beta assemblies, so there's no need to copy them in.
3. Copy all the MVC Beta, MvcContrib and Castle assemblies into my 'Dependencies' folder.
4. Make sure all the MVC Beta assemblies are marked as 'Copy Local'. I don't want to have to install them in the GAC on my server.

Now as expected, when I built the solution I got a ton of exceptions. Here's what I needed to do to get it to build:

1. Most HtmlHelper extension methods have been moved to the System.Web.Mvc.Html namespace, so I had to add a using statement to any file which referenced ActionLink.
2. ActionLink<T> now lives in the futures assembly: 'Microsoft.Web.Mvc' so I had to add using statements where appropriate.
3. I had to add my own implementation of 'ReadFromRequest' to my own ControllerBase class as this has been removed from Controller.
4. I had to fully qualify the namespace of my ControllerBase class because there is now a ControllerBase in the MVC Framework.

I also had to make the following changes to the Web.config file:

1. Change the version number of System.Web.Abstractions to 3.5.0.0
2. Change the version number of System.Web.Routing to 3.5.0.0
3. Add the namespace 'System.Web.Mvc.Html'

There were also some changes to the views:

1. Html.Form has been renamed to Html.BeginForm.
2. Html.Checkbox's signature has been changed.

With all the changes made the software now builds and runs. But that's not the end of the story. It works, but it's coded in a very Preview 1 style. I'm planning to do a major refactoring over the Christmas holidays to use all the Beta goodness that been introduced and that will mean changing the way most of the controllers work and re-doing most of the views. Should be fun :P

Jump the Gun

http://www.jumpthegun.co.uk/

Yesterday we went live with the first commercial customer for Suteki Shop: Jump the Gun. They’ve been selling Mod clothing from their shop in the North Laines of Brighton for 15 years. Brighton of course is the spiritual home of the Mods so the shop is a local landmark. It’s really exciting to see Suteki Shop taking real orders from the customers of such a cool business.

Of course the main technical interest with Suteki Shop is that it’s built with the MVC Framework. That’s really nice if you’re a developer since it makes building complex websites that much easier. Suteki Shop’s development mostly took place in the Spring of this year. I worked on it part time while I was also working for a contract client, but I guess total development time was in the region of six weeks. That’s pretty good for a site of Suteki Shop’s complexity. More importantly it hasn’t just been hacked, but built with a scalable component oriented architecture. I am very confident that I’ll be able to build features into the software without countering excessive scale pain.

Probably the main thing that Jump the Gun wanted from their eCommerce site was search engine discoverability. The MVC Framework gives you this by default. The site is totally dynamic but each page has it’s own URL. How about a sky blue Harrington Jacket?

http://www.jumpthegun.co.uk/product/Harrington_Sky

… or a black Fred Perry M12 polo shirt:

http://www.jumpthegun.co.uk/product/M12_Polo_Black

It’s essential to provide these kind of URLs if you want Google to index your product catalogue.

The other way to help search engines is to provide clean HTML. The MVC Framework makes that a sinch. If you’re using Firefox (you’re not???) go to View->Page Style->No Style. Now try to use Suteki Shop. It works fine. You can view the product catalogue categories and buy stuff with no problem. Not so easy if you rely on WebForms and third party controls. It also means that Suteki Shop relies entirely on CSS for its styling. Creating a very different looking shop front should be a simple job for a good designer.

The next thing JtG wanted was a simple to use admin system so that they could have full control over defining the product catalogue and order management. Rather than having an entirely different admin site, editing is built straight into the public web site. Logging in as an administrator gives you this view:

An admin user can define categories and products, including uploading product pictures (which are automatically resized). They can see the results straight away. They can also define the list of countries to ship to and postage rates. The order management system is very simple, it’s a simple list of orders generated by a search form. You can mark orders as dispatched or rejected. The JtG guys, who aren’t in the least bit technical, have found it very easy to use.

Another thing I wanted to provide for Suteki Shop was a simple and easy to use content management system. I’ve been heavily influenced by my experience of blogging. I wanted my customers to edit the content as easily as I write this blog. To that end I’ve implemented the MetaWeblogAPI so that you can edit content with tools such as Windows Live Writer. I wrote about this in more detail here. It makes for a very easy to use CMS. Too easy maybe :)

After spending some time looking at ASP.NET Data Services I now think that this might have been a better way of providing this functionality. Since I’m using LINQ-to-SQL as my ORM it would be very simple to layer the Data Services AtomPub API over it and let blogging tools talk directly to that. In fact it would be easy to provide the entire product catalogue as a RESTfull web service this way.

When the JtG guys said they wanted flash and videos on their site I was initially not keen on the idea. My poor little server, how would it cope, streaming videos all day? Amazon S3 turned out to be a great solution for this kind of high bandwidth content. I can even get Amazon to bill JtG directly for the cost of hosting the video content. It wouldn’t be hard to upload all the static content to S3 and leave the server just hosting the asp.net generated dynamic content. But that’s for the future.

There’s still a lot of tweeking and polishing to do, but on the whole I’m very pleased with the way the software has come together. I wouldn’t dream of attempting something like this with WebForms, it’s a technology that just can’t deliver this kind of site. I’m now talking to a number of other potential customers. It will be very interesting to see how well the design copes with the demands of multi tenancy and what kinds of variation is possible without having to fork the code.

I’d really like to hear from anyone who’s thinking of using Suteki Shop as the basis for their eCommerce site. Of course you don’t have to tell me, the licence is totally permissive. You can take the code and build something entirely your own out of it without mentioning that it’s got anything to do with me.

Suteki Shop at REMIX

Scott Guthrie was kind enough to show a screen shot of the first commercial Suteki Shop implementation at REMIX UK last week. For anyone who was at the keynote speech, I was the 'Mike' he mentioned. I was lucky enough to have a couple of conversations with him during the conference. He's a very down to earth and approachable guy.

Jump the Gun are a legendary Modernist fashion shop in the North Lanes of Brighton. Brighton has, of course, been the spiritual home of the mods ever since their heyday in the 60's so I'm really proud that they've chosen Suteki Shop as their e-commerce platform. It's a great example of a long tail business and they hope to become the world's leading on-line retailer of mod clothing.

We're planning to go live this week and I'll be making a big announcement when that happens, so watch this space!