Functional Programming

I've just read a really good blog on functional programming by defmacro.org (I couldn't find his/her name on the blog anywhere). I didn't know that Alan Turing, Kurt Godel and John von Neumann all worked together at Princeton. I guess it makes sense that those three guys would know each other, but just imagine being able to listen in to their conversations. Hmm, probably completely over my head:) Anyway, there was a fourth guy, who I'd not heard of before, Alonzo Church. He invented an alternative to the Turing machine called lambda calculus. Of course the Turing machine went on to be the basis of pretty much all modern computers, but Alonzo wasn't entirely forgotten. In the late 50's LISP was invented as a programming language that implemented lambda calculus and is still with us. It's never really been mainstream, but the Emacs text editor is written in LISP, for example. One think that I've read in several places about LISP is that it makes some things, like creating domain specific languages much easier. So why is functional programming interesting? And what can you do that you can't do with imperative programming languages like the C* family? Well, it all goes back to the lambda calculus. I don't want to repeat defmacro's excellent article, but you can do all kinds of cool stuff, like mathematically redictable unit testing, simple parallel processing with no chance of race conditions, and even hot deployment where you can drop in code changes to a running program. Also because of the nature of functional programs (they are, in effect, a single function) you can use the lambda calculus to do automated optimisations to your program or do lazy execution. Also, because a functional language's state is always represented by its stack, unlike an imperative language where the state is a combination of the stack and the heap, you can use 'continuations' to halt and restart your program, or even pass your program's state to a separate program (it's also what allows for hot deployment). Now that C# is begining to get some funcional programming constructs (lambda expressions, expression trees, delayed execution), functional programming is going to become a mainstream concern for dot net developers. Joel Spolsky, has a nice post, Can your programming language do this? It shows some stuff that becomes much easier once functions become first class citizens, and there's a post about it on the powershell blog, showing some of the powerfull functional features of that language. If you look at everyone's current favorite language, Ruby, it already does all this stuff, as well as providing a full compliment of object oriented features. Unfortunately, some of the really cool stuff I mentioned above, like parallelisation or automated optimisations, can only be done with 'pure' functional language like Haskell or Erlang. It's making me think that I'd like to dig into a good Haskell book sometime.

Writing web services contract first using WSCF

Web service gurus will always tell you to write web services "contract first". This means defining the schema for your service types, your messages and wsdl definitions before creating your implementation, which is the exact opposite of what you do when you work with Microsoft's tools. The visual studio way of doing it treats the wsdl as a behind-the-scenes artifact that the developer shouldn't need to worry his little head about. If you're using the Visual Studio tool set end-to-end you don't need to be bothered with wsdl or any of the other nasty angle bracket stuff to do with web services. The problem comes when you need to support interoperability with non dot net applications, then attributing your dot net types to produce the correct wsdl schema can be quite tricky. It would be much nicer if you could write your schemas, message definitions and wsdl first and then generate most of the boiler place asmx code from the wsdl. I was recently introduced, by the software architect where I'm contracting at the moment (hello Shelly), to WSCF (Web Service Contract First) which is a free tool that does just that. It's a free download from thinktecture, they're the consultancy headed by Ingo Rammer the remoting guy. It works as a visual studio plug in, all you have to do is write your xsds that define your types and messages (I use xml spy for doing that) and then you just right click on the message xsd file in the solution explorer, and choose 'Create WSDL Interface Description', a nice wizard pops up where you choose some defauls and then whoosh, there's your wsdl. After you've got the wsdl file you can right click on it and choose 'Generate Web Service Code', again a nice wizard pops and before you know it, you've got all your service types and framework generated in C#. Very nice. Thinktecture have also gone to the effort of writing a really nice walk-through for WSCF that gets you up and running really fast, it only took me a couple of hours to produce my first web service with this method yesterday.

Playing with HTTP.SYS

I've been having fun playing with HTTP.SYS and the HttpListener (NET 2.0) class in System.Net. HTTP.SYS is the new kernel mode http listener that replaces winsock in 2003 Server and XP SP2 and the HttpListener class is a nice managed interface for the HTTP.SYS API. It's really easy to write your own web server and do cool things like URL mangling. Basically you just create a new instance of HttpListener and then tell it to hand you any requests that match the urls you give it. It's easy to imgaine writing a RESTfull service (see my previous post on this) using HttpListener. You could just write a windows service to receive any requests, say 'http://mikehadlow.com/restapp/' and then parse the local address part of the url to tell you which object to get from your database. So, say we got the url 'http://mikehadlow.com/restapp/customer/1234', we'd parse the local address 'restapp/customer/1234', grap 'customer' and '1234' and go to our database and 'select * from customer where id = 1234', serialise our customer object as xml and return that xml in the response. You can also host the ASP.NET pipeline in your own application using HttpListener. There's a really good MSDN article on this here. This means you no longer have to worry about having IIS installed on a machine to make your application response to http requests, any app can do it including windows forms apps, console apps and windows services. Here's some code for a little console app, showing how you use HttpListener. It's just a slight re-working of the example code from the help file. Just start up the app and browse to 'http://localhost:1234/something/else/here?somekey=somevalue&someotherkey=someothervalue' and you'll see it echo back the local address and the query string parameters.

using System;
using System.Net;

namespace HttpListenerTest
{
    class Program
    {
        HttpListener _listener = new HttpListener();

        static void Main(string[] args)
        {
            Program program = new Program();
            program.Start();
        }

        public void Start()
        {
            _listener.Prefixes.Add("http://*:1234/");
            _listener.Start();
            Console.WriteLine("Listening, hit enter to stop");
            _listener.BeginGetContext(new AsyncCallback(GetContextCallback), null);
            Console.ReadLine();
            _listener.Stop();
        }

        public void GetContextCallback(IAsyncResult result)
        {
            HttpListenerContext context = _listener.EndGetContext(result);
            HttpListenerRequest request = context.Request;
            HttpListenerResponse response = context.Response;

            System.Text.StringBuilder sb = new System.Text.StringBuilder();

            sb.Append("");
            sb.Append(string.Format("HttpMethod: {0}
", request.HttpMethod));
            sb.Append(string.Format("Uri:        {0}
", request.Url.AbsoluteUri));
            sb.Append(string.Format("LocalPath:  {0}
", request.Url.LocalPath));
            foreach (string key in request.QueryString.Keys)
            {
                sb.Append(string.Format("Query:      {0} = {1}
", key, request.QueryString[key]));
            }
            sb.Append("");

            string responseString = sb.ToString();
            byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
            response.ContentLength64 = buffer.Length;

            using (System.IO.Stream outputStream = response.OutputStream)
            {
                outputStream.Write(buffer, 0, buffer.Length);
            }
            _listener.BeginGetContext(new AsyncCallback(GetContextCallback), null);
        }
    }
}

Resting?

I've just become aware of a new term: REST. It stands for Representational State Transfer. It's kinda like a 'back to basics' movement about web standards and how the internet is developing. As time goes on, protocols have been layered on top of protocols. In the begining was IP, then HTTP got layered on top, subsequently SOAP was layered on top of HTTP and now we have a whole load of WS-* protocols being layered on top of SOAP. The REST people are basically saying that a lot of the new protocols are duplicating stuff that the older lower level protocols do already. Why have a SOAP action when HTTP already gives you CRUD verbs; GET, POST, PUT, DELETE? Why do we need to have identifiers when we already have the URL? A RESTafarian (I didn't make that up) would say you can do web services without SOAP or any of that WS-* stuff. Say I want to get a customer. All I'd need to do is carry out an HTTP GET for this URL: http://www.thecompany.com/enterprise/customer/0123456 That would return me a POX (Plain Old Xml, didn't make that up either) representation of my customer. RESTafarians would also argue that my customer resource should only have stuff about the customer itself, related resources (orders, invoices) should be represented by links to those resources. If it all seems slightly familiar, it is, of course, just how the simple HTML web works and the simplicity of the model is the World Wide web's fundamental strength. It's a persuasive argument and a real challenge to the current dominance (at least in the Microsoft world that I live in) of SOAP RPC style web services. Now, can REST be done with WCF, that's a question!

one line fibonacci

I've been playing with windows powershell (aka monad, aka MSH) recently. It's got some really great functional programming constructs. Here's a one line fibonacci...

PS C:\> 1..10 | foreach{ $i = $j = 1 }{ $k = $i+$j; $i=$j; $j=$k; $k }
2
3
5
8
13
21
34
55
89
144

"1..10" generates a range of numbers from one to ten "|" is a pipe to the next command, "foreach" applies the first statement block once and then the second statement block for each item generated from the previous list.

The Vietnam of Computer Science

Ted Neward has written a great post on one of my favorite subjects: the object relational problem. He calls it 'The Vietnam of Computer Science'. I've thought for a long time that the OR mapping problem is currently the main challenge in developing business applications. I've faced these issues head on when I developed my own data-access-layer code-generator. Funnily enough, although it's such a huge issue in application design, there aren't any really good books on the subject (that I know of). Martin Fowler, Eric Evans and Rocky Lhotka all touch on the issues in their books on enterprise application development, but I've never seen it exhaustively covered anywhere. Ted's post is an excellent summary of the main compromises you have to make, but I'd really like someone to write the definitive book on the subject.

Book Review: Framework Design Guidelines by Krzysztof Cwalina and Brad Abrams.

I've just finished reading Framework Design Guidelines. This book evolved from the design guidelines put together during the development of the .net framework. Both of the authors are program managers on the CLR and .net teams, so you get a great inside look at the thinking behind design decisions made during the development of the base class library. The book is peppered with little side bars where various Microsoft developers comment on the guidelines. These really help to bring the whole thing to life and you get an insight into the arguments that must have gone on over the pros and cons of different approaches. Like they say, designing a good reusable framework is hard and Microsoft have learnt from their past mistakes (did someone say MFC?). I really liked the initial discussion on general framework design where the authors describe how the .net team tried to juggle some of the conflicting requirements that any framework faces; how to keep it simple while providing a powerful tool; how to keep it as open as possible for future enhancements and how to keep it consistent. Having some of the design decisions around some of the best known components laid out in front of you really helps when it comes to understanding and using them. One thing I hadn’t really appreciated, but is obvious when it’s pointed out, is how the framework tries to juggle simplicity with power by using aggregate components for the common simple tasks that offer a façade over a more complex factored API. The book uses the System.IO namespace as an example where the File class provides the high level, easy to use abstraction while the stream classes provide the powerful API needed for more demanding tasks. Of course knowing this is great when you're using .net every day but it's also a real education in how to design good frameworks. Now, you might say, I'm an application developer, not a framework developer so why would I need to know this? But, as Martin Fowler says, following good OO design principles inevitably means that you are constantly designing frameworks, even if you're your only customer. Having said that, one of the best points for me in the book was the discussion on when you shouldn't stick strictly to OO principles, especially when designing components for visual tools to use. Although the book doesn’t once mention test driven development, it did reinforce my love of this way of developing by showing how you should always design from usage scenarios. Always write down how your API is going to be used first. I skimmed a lot of the stuff on naming conventions. If like me you’ve worked in several different enterprises on several different large systems, you’ll have probably discussed naming conventions until you’ve gone slightly silly in the head. But you’ve got to have them and I guess having them here in this book does give you something to point at the next time someone who’s been hiding in a hole for ten years suggests Hungarian. Chapter 7 on exceptions should be required reading for any .net developer. It's the best explanation of why you should use structured exception handling I've yet seen. I wish I'd had this book handy a couple of years ago when I spent hours arguing with an old school developer about the problems with error code return values. Chapter 9 on common design patterns should also be essential reading especially the discussion on the Dispose pattern. The documentation on this in MSDN isn't very good. It stipulates what you should do, but not why you should do it and it's probably why so few developers implement it properly, and that includes me. The authors give an excellent discussion on why should use IDisposable and how you should implement it so there’s no excuse for me doing it incorrectly from now on. So, yes, I really enjoyed this book. It's well designed and written and it works well on two levels, firstly you get an insight into the .net framework, but also it’s a great general good .net practices book. Also it’s well worth checking out Brad Abrams blog.

A simple WSDL object

Wsdl is great, it's like reflection for web services. As I discussed in this post, you can use it to dynamically discover and call web services at runtime. The dot net framework provides a class, 'System.Web.Services.Description.ServiceDescription' that takes a wsdl stream and provides an object model to walk the wsdl. It's really powerfull, but kinda awkward to use if you just want to get a list of web methods with their parameter names and types. Faced with this requirement last week, I put together a simple facade for the ServiceDescription class. You just create a new WebServiceInfo instance using its factory method 'OpenWsdl' and then you can iterate through all the web methods and their parameters and return parameters. Here's the NUnit test which pretty much shows how it works:

[Test]
public void ServiceDescriptionSpike()
{
   string url = "http://localhost:1297/Test/TestService.asmx";

   WebServiceInfo webServiceInfo = WebServiceInfo.OpenWsdl(new Uri(url));

   Console.WriteLine(string.Format("WebService: {0}", webServiceInfo.Url));

   foreach (WebMethodInfo method in webServiceInfo.WebMethods)
   {
       Console.WriteLine(string.Format("\tWebMethod: {0}", method.Name));
       Console.WriteLine("\t\tInput Parameters");

       foreach (Parameter parameter in method.InputParameters)
       {
           Console.WriteLine(
               string.Format("\t\t\t{0} {1}", parameter.Name, parameter.Type));   
       }

       Console.WriteLine("\t\tOutput Parameters");

       foreach (Parameter parameter in method.OutputParameters)
       {
           Console.WriteLine(
               string.Format("\t\t\t{0} {1}", parameter.Name, parameter.Type));
       }
   }
}

And here's the code. As you can see it just wraps the ServiceDescription class, you can see what a pain it is to get this information out of it.

using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Text;
using System.Web.Services.Description;
using System.Net;
/// <summary>
/// Information about a web service
/// </summary>
public class WebServiceInfo
{
   WebMethodInfoCollection _webMethods = new WebMethodInfoCollection();
   Uri _url;
   static Dictionary<string, webserviceinfo=""> _webServiceInfos =
       new Dictionary<string, webserviceinfo="">();

   /// <summary>
   /// Constructor creates the web service info from the given url.
   /// </summary>
   /// <param name="url">
   private WebServiceInfo(Uri url)
   {
       if (url == null)
           throw new ArgumentNullException("url");
       _url = url;
       _webMethods = GetWebServiceDescription(url);
   }

   /// <summary>
   /// Factory method for WebServiceInfo. Maintains a hashtable WebServiceInfo objects
   /// keyed by url in order to cache previously accessed wsdl files.
   /// </summary>
   /// <param name="url">
   /// <returns></returns>
   public static WebServiceInfo OpenWsdl(Uri url)
   {
       WebServiceInfo webServiceInfo;
       if (!_webServiceInfos.TryGetValue(url.ToString(), out webServiceInfo))
       {
           webServiceInfo = new WebServiceInfo(url);
           _webServiceInfos.Add(url.ToString(), webServiceInfo);
       }
       return webServiceInfo;
   }

   /// <summary>
   /// Convenience overload that takes a string url
   /// </summary>
   /// <param name="url">
   /// <returns></returns>
   public static WebServiceInfo OpenWsdl(string url)
   {
       Uri uri = new Uri(url);
       return OpenWsdl(uri);
   }

   /// <summary>
   /// Load the WSDL file from the given url.
   /// Use the ServiceDescription class to walk the wsdl and create the WebServiceInfo
   /// instance.
   /// </summary>
   /// <param name="url">
   private WebMethodInfoCollection GetWebServiceDescription(Uri url)
   {
       UriBuilder uriBuilder = new UriBuilder(url);
       uriBuilder.Query = "WSDL";

       WebMethodInfoCollection webMethodInfos = new WebMethodInfoCollection();

       HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(uriBuilder.Uri);
       webRequest.ContentType = "text/xml;charset=\"utf-8\"";
       webRequest.Method = "GET";
       webRequest.Accept = "text/xml";

       ServiceDescription serviceDescription;

       using (System.Net.WebResponse response = webRequest.GetResponse())
       using (System.IO.Stream stream = response.GetResponseStream())
       {
           serviceDescription = ServiceDescription.Read(stream);
       }

       foreach (PortType portType in serviceDescription.PortTypes)
       {
           foreach (Operation operation in portType.Operations)
           {
               string operationName = operation.Name;
               string inputMessageName = operation.Messages.Input.Message.Name;
               string outputMessageName = operation.Messages.Output.Message.Name;

               // get the message part
               string inputMessagePartName =
                   serviceDescription.Messages[inputMessageName].Parts[0].Element.Name;
               string outputMessagePartName =
                   serviceDescription.Messages[outputMessageName].Parts[0].Element.Name;

               // get the parameter name and type
               Parameter[] inputParameters = GetParameters(serviceDescription, inputMessagePartName);
               Parameter[] outputParameters = GetParameters(serviceDescription, outputMessagePartName);

               WebMethodInfo webMethodInfo = new WebMethodInfo(
                   operation.Name, inputParameters, outputParameters);
               webMethodInfos.Add(webMethodInfo);
           }
       }

       return webMethodInfos;
   }

   /// <summary>
   /// Walk the schema definition to find the parameters of the given message.
   /// </summary>
   /// <param name="serviceDescription">
   /// <param name="messagePartName">
   /// <returns></returns>
   private static Parameter[] GetParameters(ServiceDescription serviceDescription, string messagePartName)
   {
       List<parameter> parameters = new List<parameter>();

       Types types = serviceDescription.Types;
       System.Xml.Schema.XmlSchema xmlSchema = types.Schemas[0];

       foreach (object item in xmlSchema.Items)
       {
           System.Xml.Schema.XmlSchemaElement schemaElement = item as System.Xml.Schema.XmlSchemaElement;
           if (schemaElement != null)
           {
               if (schemaElement.Name == messagePartName)
               {
                   System.Xml.Schema.XmlSchemaType schemaType = schemaElement.SchemaType;
                   System.Xml.Schema.XmlSchemaComplexType complexType = schemaType as System.Xml.Schema.XmlSchemaComplexType;
                   if (complexType != null)
                   {
                       System.Xml.Schema.XmlSchemaParticle particle = complexType.Particle;
                       System.Xml.Schema.XmlSchemaSequence sequence = particle as System.Xml.Schema.XmlSchemaSequence;
                       if (sequence != null)
                       {
                           foreach (System.Xml.Schema.XmlSchemaElement childElement in sequence.Items)
                           {
                               string parameterName = childElement.Name;
                               string parameterType = childElement.SchemaTypeName.Name;
                               parameters.Add(new Parameter(parameterName, parameterType));
                           }
                       }
                   }
               }
           }
       }
       return parameters.ToArray();
   }

   /// <summary>
   /// WebMethodInfo
   /// </summary>
   public WebMethodInfoCollection WebMethods
   {
       get { return _webMethods; }
   }

   /// <summary>
   /// Url
   /// </summary>
   public Uri Url
   {
       get { return _url; }
       set { _url = value; }
   }
}

/// <summary>
/// Information about a web service operation
/// </summary>
public class WebMethodInfo
{
   string _name;
   Parameter[] _inputParameters;
   Parameter[] _outputParameters;

   /// <summary>
   /// OperationInfo
   /// </summary>
   public WebMethodInfo(string name, Parameter[] inputParameters, Parameter[] outputParameters)
   {
       _name = name;
       _inputParameters = inputParameters;
       _outputParameters = outputParameters;
   }

   /// <summary>
   /// Name
   /// </summary>
   public string Name
   {
       get { return _name; }
   }

   /// <summary>
   /// InputParameters
   /// </summary>
   public Parameter[] InputParameters
   {
       get { return _inputParameters; }
   }

   /// <summary>
   /// OutputParameters
   /// </summary>
   public Parameter[] OutputParameters
   {
       get { return _outputParameters; }
   }
}

/// <summary>
/// A collection of WebMethodInfo objects
/// </summary>
public class WebMethodInfoCollection : KeyedCollection<string, webmethodinfo="">
{
   /// <summary>
   /// Constructor
   /// </summary>
   public WebMethodInfoCollection() : base() { }

   protected override string GetKeyForItem(WebMethodInfo webMethodInfo)
   {
       return webMethodInfo.Name;
   }
}

/// <summary>
/// represents a parameter (input or output) of a web method.
/// </summary>
public struct Parameter
{
   /// <summary>
   /// constructor
   /// </summary>
   /// <param name="name">
   /// <param name="type">
   public Parameter(string name, string type)
   {
       this.Name = name;
       this.Type = type;
   }

   /// <summary>
   /// Name
   /// </summary>
   public string Name;
   /// <summary>
   /// Type
   /// </summary>
   public string Type;
}

Dynamically creating a web service proxy class

I'm working on a service integration project these days and so I've been having an interesting time getting to grips with web services and all that stuff. (as you can probably see from some of my recent posts). This code project article by Roman Kiss is really cool, its about creating your own loosely coupled web service bus, but what I really liked about it was the bit about creating web service proxy classes on the fly. Basically wsdl.exe, the command line tool that you can use for generating web service proxy classes, wraps a framework class 'ServiceDescriptionImporter'. It takes a 'ServiceDescription' class with a wsdl file loaded in it and returns a code dom namespace. You can compile any code dom namespace using the CSharpCodeProvider, and you can do it in memory and then directly access the methods and types on the generated assembly. That is so cool, it kind of brings the promise of wsdl as a web service reflection language directly into your project.

I love Regex

I'm no expert, and a relatively recent convert to them, but I love regular expressions. They make so many awkward text manipulation tasks easy. Today I had to write a little function to turn a legacy application date into a dotnet DateTime struct. The legacy application's date is in the format CYYMMDD where C is the century, 0 for 1900, 1 for 2000 etc, YY is the two digit year, MM is the month and DD is the day of the month. It's really easy to parse this into a DateTime struct using the dotnet Regex class:

public static DateTime ToDateTime(string geniusDate)
{
    Regex regex = new Regex(@"^(\d)(\d{2})(\d{2})(\d{2})$");
    Match match = regex.Match(geniusDate);
    if (!match.Success)
    {
        throw new ArgumentException("not a valid geniusDate");
    }

    int century = (int.Parse(match.Groups[1].Value) * 100) + 1900;
    int year = int.Parse(match.Groups[2].Value) + century;
    int month = int.Parse(match.Groups[3].Value);
    int day = int.Parse(match.Groups[4].Value);

    return new DateTime(year, month, day);
}

Also I really like the regular expression search and replace built into visual studio. Here's one that changes a field declaration like:

 string _name;

into a property like:

 string name
 {
  get{ return _name; }
  set{ _name = value;}
 }

Just put this:

:b*<{.*}>:b<_{.*}>;

in the 'find what' field, and this:

\1 \2\n{\nget{ return _\2; }\nset{ _\2 = value;}\n}

in the 'replace with' field. Don't forget to check the 'Use' check box and select 'Regular expressions'. It's a shame that the regular expression syntax is different for the visual studio and the Regex class. The captures are () in Regex, but {} in visual studio. Now, of course it's much easier to type control R, E and let the refactoring tools do this job for you, but it's a good demonstration of how much you can do with search and replace regular expressions.