Suteki Shop is now a sample application on the official MVC Framework site

The MVC Framework guys have been very kind and put Suteki Shop, our open source eCommerce application on the asp.net/mvc website. Just scroll down to Sample Applications.

Jeremy Skinner has been doing some great work refactoring Suteki Shop to use the MVC 1.0 goodness. He’s also been introducing loads of cool stuff from the MvcContrib project including his awesome grid.

We’re constantly evolving the software; there are loads of things I’d still like to do. Here’s a short list:

• Change from Linq-to-SQL to NHibernate. Suteki Shop makes extensive use of LINQ, so I’m very excited about the work that Steve Strong has been doing on LINQ-to-NHibernate.
• Change from MetaWeblog API to Atom pub using WCF Windsor integration. I’m currently looking at this and hope to get it working in the next few weeks.
• Make the multi-tenancy really slick. This works, but we really need to clean up the way assets (style sheets, images etc) are organised.
• Provide a pluggable model for payment providers. Currently Suteki Shop only supports it’s own internal payment system.
• Provide a feed for Google product search.
• Get it working on Mono.
• ___ your patch here ____ :-)

Last Chance to Book Progressive .NET Tutorials with £150 Off!

I will be presenting at the Progressive .NET Tutorials on 11-13th May at Skills Matter - the event consists of 16 sessions on various modern .NET technologies including Castle, NHibernate, OpenRasta, DSLS in C#, Windsor WCF (that’s me), F# and MEF.  If you book on or before 24th April (that's tomorrow!) you will get a discount of £150 off the ticket price.

It’s going to be awesome! You can view the full programme and book online at http://skillsmatter.com/event/open-source-dot-net/progressive-dot-net-exchange/ng-37

See you there!

The sorry tale of the Visual Studio Web Site

The first 1.0 release of ASP.NET introduced a way of building web sites that was very different from classic ASP. Developers that had grown up with script embedded in HTML that executed on demand were suddenly faced with a full blown object oriented web framework. For enterprise developers like me who had been chaffing at the awfulness of VB script, it was a revelation. But for many casual web designers working on simple sites, the new complexity was overwhelming.

Take my friend Roy for example. He’s a successful freelance web designer/developer. Typically he takes requirements from small businesses and knocks out small sites usually with a simple backend database. His programming chops are limited to simple control structures; “what more do you need than an ‘if’ statement?” He has no interest in learning object oriented programming. He sees it as academic and over complex. I have no doubt that Roy would epically fail if he was tasked with building the kind of large scale enterprise applications that I work on. But he doesn’t do that; instead he has many satisfied small business customers that really like his quick, pragmatic and cost-effective approach.

Soon after ASP.NET was released he used it to build a web site for a business idea he had. Rather than his usual ‘few days’ turnaround, he spent weeks trying to get a handle on the new model. He just about managed to make it work, but he was enormously frustrated. He never touched ASP.NET again. To this day he codes his sites using classic ASP. He’s mostly abandoned Visual Studio and instead works with Dreamweaver. As far as he is concerned Microsoft lost their way with ASP.NET.

Many small web design shops felt the same way. At around the same time the LAMP stack was really taking off. PHP was just like classic ASP, but better. If used with a tool like Dreamweaver, the web design experience was excellent. For an ‘if statement’ man like Roy it offers a far more compelling alternative than complex object oriented world of ASP.NET. The word on the street was that PHP rocked and ASP.NET sucked.

Microsoft was listening, and they panicked. With the release of Visual Studio 2005, in a desperate attempt to keep the scriptable template market, they removed the classic ASP.NET web project and replaced it with the abomination known as the ‘web site’. Rather than give the Roys what they really wanted: a better classic ASP; they created a monster.

The ‘web site’ project type tries to pretend that .NET is a scripting environment by implementing on-the-fly page-by-page compilation and removing namespace conventions. It tries to pretend that a site is just a bunch of files by removing the MSBuild project file and doing strange things with the solution file. If you mistakenly chose to use the web site project type for enterprise application development (and you had no choice for a while) you soon found yourself in a world of pain. Just try having a separate unit testing project, or doing CI.

The loud and clear WTFs coming from the enterprise developer community meant that Microsoft quickly resurrected the original Web Application project type, first as a patch and then as part of SP1. But they couldn’t quite bring themselves to admit what a huge mistake web-site was, so there it sits, waiting to snare the naïve developer. Don’t be fooled, if you are building serious web applications use Web Application. If you want scripted templates, use PHP.

Some Fundamentals of Service Oriented Architecture

Most businesses run on a suite of diverse applications. Some have been written in house, some have been written by outside suppliers and some are bought as shrink-wrapped products or COTS (Commercial Off The Shelf). They run on different platforms using different technologies. They range in scale from Excel spreadsheets, created and used by non-technical business users, to large scale enterprise applications based on server farms and created by teams of IT professionals.

For example, one of my recent clients used Sage for their accounts, Salesforce.com for the sales team, and RedDot for content management. They also had a suite of applications written in-house. The older ones were VB/SQL client-server systems, with the more recent web based ones being written with .NET. There were also many business processes based on spreadsheets maintained by individual users. Some processes’ data even lived on whiteboards and paper.

The problems that arise from such a disparate suite of applications are well known; you can find them in most enterprises.

• Users often have to re-key data, copying it from the screen of one application to input fields in another.
• The organisation has more than one source of the same information. For example, a customer’s address might be different on different systems.
• Because there is no single source of information, important details are often missed. A customer might be given preferred status on one system, while they are marked as having a legal dispute with the company on another.
• It is very hard to gather management information. We might not be able to say how many customers with orders above a certain amount always pay late.
• Business processes are manual and add-hoc. They can be lost when a key member of staff moves on. They are often carried out differently at different times by different people; confusing customers, creating inconsistent data and introducing ‘bugs’ into the system.

So what do we do to solve these problems? We try and make the disparate applications talk to each other.

Integration Pathologies

But making the applications talk to each other introduces a whole new set of problems. Most naïve approaches fail primarily for two reasons:

The first is not correctly decoupling the applications. The communication implementation involves the applications knowing far too much about each other’s innards. The most pernicious (and common) form of this is when one application directly accesses another’s database. This reaches an apogee of awfulness when stored procedures execute cross database joins. Multiple applications accessing shared business components can also be a sign of this pathology.

This style of communication quickly becomes a tightly coupled mess. It can very easily stop the possibility of any changes being made to the individual applications as developers recoil against the complications of refactoring all the known (and often unknown) disparate pieces that rely on schemas, stored procedures and other internals. “We can’t touch system-x because who knows what might break”.

In the worst cases, such as when the integration targets the tables of shared databases, business rules can be replicated many times over. This makes it extremely difficult to change them and adds to the forces fossilising the organisation’s software.

The second common pathology is direct application-to-application communication. This is a natural consequence of doing application integration ad-hoc. If we think of an application as a node in a network, it’s easy to see that each additional node requires a new set of connections equal to the number of existing nodes. The task of integration will get successively more complex as we add new systems that require integration.

Each connection requires its own mapping and access to the joined application’s internals. This leads to duplication of effort and business rules.

Before long we find that our integrated applications, rather than creating the wonderful joined up business we envisioned, have made things even worse. The integration effort itself takes up progressively more resources, the tight coupling between applications makes it very hard to change anything and the duplicated business rules and diverse mappings mean that we face an avalanche of poor, inconsistent data.

How can we avoid this?

The industry has built up considerable experience in enterprise integration patterns and SOA. My two favourite sources of wisdom are the book Enterprise Integration Patterns by Hohpe and Woolf and the accumulated advice found in Bill Poole’s excellent blog.

The trick to doing integration successfully is to connect the applications in such a way that they remain decoupled. This is the core secret to doing Service Oriented Architecture well. We decouple applications by hiding them behind well defined service interfaces. We never allow them to interact directly with each other’s internals.

By hiding each application behind a well known interface, we can allow it to change internally without having those changes propagate throughout the organisation. Each application can remain agile and responsive to business needs.

We control the proliferation of connections by making each application talk through a common Enterprise Service Buss (ESB). We then have only one interface to worry about for each application. We make each application talk a single canonical language that is shared throughout the organisation. Now we only have to manage one mapping per application, between the application’s internal representation and the canonical message schema.

The canonical language should consist of messages that are relevant to the business process. We should avoid service interfaces that exhibit CRUD style APIs and instead build an event driven ESB that exchanges coarse grained messages styled as ‘business events’. Changes that are relevant to the organisation are published by applications where the change is sourced, and subscribed to by applications that need to know about the changes.

Messages should be asynchronous and atomic. We need to avoid the situation where one application needs to synchronously call another application to source some data in order to complete an operation. A message should carry all the information needed to complete a business event and messages should not be enrolled in transactions.

We should not concern ourselves with duplicated data between applications, so rather than having a single list of countries, for example, held in a single service, we should be relaxed about duplicate lists held in each application. See Bill Poole’s posts on Centralised vs. Decentralised Data here, here and here.

Integration is hard

Even with all these principles in place, integrating business applications is still a complex task. It is not to be undertaken lightly. We need a relentless business driven focus on integration design and a clear and well communicated vision of how to achieve it. This is doubly hard when we try to disentangle an existing web of poorly designed tightly coupled interactions as described above. However, done well it can be the springboard to a far more flexible organisation.

Agile practices are not just about management.

Although it’s a broad generalisation, Agile software development practices can be broken down into management practices and software design practices. The management practices are things such as short iterations, continuous integration, pair programming, programmer task estimation and co-location. The software design practices are things such as Test Driven Development, the SOLID principles and design patterns.

A common pattern I see repeated in many organisations is that the management principles are understood and adopted, while the software design principles are mostly ignored.

While the management practices may make for saner project management, they won’t do anything in themselves to improve the quality of software being produced or the productivity of the team.

Software development has a core pathology; you can see it on almost any project over a certain size. As the code base increases in complexity, the cost of adding a feature increases and defects proliferate. Often this is accompanied by ever increasing problems with build and deployment and de-motivated and unhappy developers.

Once the software gets part a certain size and age, it becomes almost un-movable and a real hindrance to the business. Often the response is to start from scratch, but this just makes things worse as the new system gradually goes the same way as the old, and in many cases never fully replaces it. Then there is the additional task of integrating the old and new systems, usually as a ‘temporary measure’.

Of course, building a robust scalable Service Oriented Architecture (to integrate the new and old systems) brings its own challenges. A team that failed to build a an initial scalable application will often fail in this task too. Soon more and more of the team’s resources are devoted to making the old and new systems talk to each other. More resources are applied and less is achieved.

The end game is often a static lock, where the business cannot move forward or react to changes in the competitive landscape because the software it relies upon cannot adapt. The software team becomes a large fixed cost that simply drains the bottom line rather than adding any real benefit. Usually the only agility the business retains is by working around their core systems and relying on more primitive forms of record keeping such as the humble spreadsheet or Access database.

I first came across the description ‘Agile’ in relation to software development by reading Bob Martin’s classic, Agile Software Development. Martin spends some of the book discussing Agile management techniques, but the majority of the text is devoted to Agile design practices including several detailed coding case studies.

Agile software design practices are the distillation of the collective experience of the software community over the past decades. A lot of very clever people have been struggling with the problem of how to avoid the pathology I described above. These practices are in no way a complete answer to all the problems we face with building large scale applications; the industry is constantly learning how to do this better; but they do add up to a sizable body of patterns and practices than can make a significant difference to the success of a project.

You can buy a whole library of books on writing better software, and if you are serious about it you should. But as a taster, here is a incomplete list of some Agile software development principles that come to mind as I write this:

Never rewrite the system. See The Big Redesign In The Sky.

Don’t repeat yourself ever. DRY

Refactor relentlessly (see above)

Only code what you need now (You Ain’t Gonna Need It: YAGNI)

Do the simplest thing possible.

Keep methods small (less than 10 lines of code). Keep classes small and with a single responsibility (see SOLID below).

Keep methods simple (low cyclomatic complexity).

Decouple components from each other at every scale (class, service and application).

Only allow components (classes, components, services, applications) to communicate through well defined interfaces.

Decouple your application into well defined tiers. Don’t mix business logic, UI logic and infrastructure.

Follow object oriented principles: Single Responsibility Principle, Open/Closed Principle, Liskov Substitution Principle, Interface Segregation Principle, Dependency Inversion Principle (SOLID)

Use design patterns. Someone has probably solved your problem before.

Use known algorithms and data structures.

Don’t write any code except to make an automated test pass. This is Test Driven Development.

Avoid code generation. But automate anything you can.

Use the right tool for the job: XML is not a good programming language, TSQL is not a good place to encode business rules.

Choosing a Source Control Tool

The one simple answer to choosing a source control tool is easy: don’t choose SourceSafe.

It has many issues including:

• Commits are not atomic. If a file is changed you have no way of linking a set of file changes that implement a particular feature or fix a bug.
• Lacks usable branching support.
• It doesn’t keep a history of file renames. If you rename a file and than get a version previous to the rename, the file will have the new name.
• Permanently deleting a file also deletes its history. This means that versions of the software prior to the deletion are not valid.
• Requires a file share to work and does not perform well over slow connections.
• Randomly corrupts its own database.
• Many other issues, detailed here.

The first point pretty much rules out Source Safe if you want to do continuous integration. Without atomic commits your automated build has no way of knowing when a related set of changes has finished arriving. I’ve seen kludges where the build server waits an interval after the last check-in before kicking off a build, but they are just that; kludges.

Source Safe has historically been the default choice for Microsoft shops. If it’s so bad, what should we be using instead?

There are many source control systems on the market, but I think the choice fundamentally comes down to two contenders: Microsoft’s recommended tool and the most popular tool.

Microsoft recommends the source control system from Team Foundation Server. I’ve been using TFS source control for over a year. Admittedly I haven’t exercised features such as shelving or branching, but my experience has been good.

However TFS is much more than just a source control tool, and unfortunately some of the other features are not good examples of their type. The unit testing tool in particular is very poor and the build server is unnecessarily complex and hard to use. The integration with Visual Studio is too intrusive; it’s hard to work disconnected and you can only be connected to one TFS server at a time, which is a serious limitation.

I haven’t used the integrated bug/task tracking, but that might be a good reason to favour TFS over competing SC tools. If you know you need TFS for its tight integration and management tools, then I think you can be pretty confident about its source control capabilities.

The most popular source control tool, according to this survey is Subversion.

Subversion is the most popular source control tool for a reason; it is robust, mature and has a vibrant supporting tools market, including the excellent windows client, tortoise svn. It has been very successful in the open-source world and powers several huge on-line source repositories, including Source Forge and Google Code.

Its main weakness, when compared to TFS, is that it is only a source control tool and doesn’t come with an integrated build server, test and management tools. You can put together your own suite from commercial and open source suppliers, but their choice and integration is up to you. However, because of Subversion’s popularity, integration points for it are well supplied.

Integrating Subversion authentication with Active Directory can also be tricky, something that is much easier to do with TFS. If you want to integrate easily with your existing developer list, that might be a reason to choose TFS over Subversion. A guide to using Subversion with LDAP can found here. Altassian provide a product called Crowd that can apparently be used for integrating LDAP, but I haven’t had any experience with it.

If tight integration, especially of management tools, is not a priority, it’s hard to argue against Subversion. For that reason I would favour Subversion as the source control tool of choice for most Microsoft shops.

Why you should upgrade to .NET 3.5.

A pathology I’ve noticed in many large organisations is inertia in failing to keep up with recent versions of development tools. It’s not uncommon to find .NET shops still using Visual Studio 2005 and .NET 2.0. Even 1.1 is not unknown.

The usual reasons for not upgrading are down to fear and/or incompatible 3^rd parties.

Fear comes from the worry that newer versions of compilers and runtimes will introduce bugs into the software. This is especially prevalent when teams don’t have a robust testing and release pipeline set up. If you have a fragile and brittle codebase that’s hard to deploy and hard to test, you are naturally reluctant to do anything that might introduce instability. No matter what Microsoft might say about the ease of upgrading, they have a history of breaking things with new versions, so the safe bet is to do nothing.

But if fear of breaking your brittle and unstable software is the main reason you are reluctant to upgrade to the latest tools, there is far more wrong with your development process than simply being behind a few versions of Visual Studio.

Incompatible 3^rd party libraries and frameworks are a more pernicious problem. Since the CLR version hasn’t changed between 2.0 and 3.5 there should not be any problems with mixing older tools. However, if you are building tools or frameworks for customers who refuse to upgrade then that’s a separate problem. But even then, you should be able keep backwards compatibility while moving forward.

But why is it so important to stay reasonably up to date? I think there three main reasons:

• Being left behind by newer tools and the .NET community in general.
• Missing out on newer features.
• Developers don’t like to use old tools.

Being left behind by newer tools and frameworks is probably the most serious problem. As your version of the .NET framework falls further and further behind, it’s harder to find tool vendors or open source projects that can provide compatible versions of their frameworks. An obvious example of this is the popularity of lambda expressions in the APIs of many modern tools. If you’re not using 3.5, you can’t use the tools.

A related problem is that the most readily available documentation usually refers to recent versions. The same goes for blogs, newsgroups and other sources of online help and documentation. Before long you find that every issue you search for leads to information that you can’t use.

Missing out on newer features is another potential source of lost opportunity. Development tools really do get better over time. WCF is a far superior technology to asmx web services; the MVC Framework is a much better way of developing websites than Web Forms; and LINQ is a much better way of writing queries over data structures than for loops and if statements. The list goes on and on.

Lastly it’s de-motivating to be stuck using old tools. All the best programmers want to work with the latest toys, and you do want the best, don’t you? Allowing your development tools to slip further and further behind makes it harder to recruit keen young coders and you risk your current team moving on for worry that their skills are getting legacy.

Some developers don’t like learning new stuff and will resist the adoption of new tools. You really don’t want to be employing such people. If your staff are the main reason you are still running with 1.1, you are in serious trouble.

So how hard is it to upgrade from 2.0 to 3.5?

There are three main parts to the .NET Framework: the Common Language Runtime (CLR), the Base Class Library (BCL) and the compilers. Understanding which has changed with which version makes it easier to predict any problems you might have.

The change from 1.1 to 2.0 was big. The CLR, BCL and compilers all changed. The flagship feature was generics. Migrating from 1.1 to 2.0 was a major upgrade which required some care and planning.

The change from 2.0 to 3.0 was the addition of three major new areas to the BCL: Windows Communication Foundation (WCF), Windows Presentation Foundation (WPF) and Windows Work Flow (WWF). Neither the CLR nor the compilers changed. Upgrading was straightforward. Many commentators complained that it didn’t really justify a point release number and should have been labelled as 2.5.

The change from 3.0 to 3.5 was more important. The CLR didn’t change, it’s still the same as version 2.0, but there were new compilers and language features and extensions to the BCL. The flagship new feature was Language Integrated Query (LINQ). Because the CLR didn’t change the upgrade should not be difficult. The new language features are extensions; version 2.0 code should still compile.

Visual Studio 2008 has a .NET 2.0 mode, so it’s possible to upgrade your development tools first and your software later. 2008 has been around for more than a year now so pretty much all the major add-in vendors (I’m looking at you Resharper) have caught up. There’s no reason not to make the change.

To sum up, unless you have a dependency from a client or supplier that means that you simply have to stay with .NET 2.0, it really makes sense to upgrade. The upgrade is pretty painless but provides many benefits. Even if you have to stick with the 2.0 compilers you can still use Visual Studio 2008 in .NET 2.0 mode.