Sunday, February 11, 2007

Wes Dyer talking about LINQ

More browsing today on MSDN lead me to an interview with Wes Dyer taking about LINQ. He's one of the guys on the C# compiler team. A very bright chap indeed and he gives one of the best explanations of how LINQ works that I've yet seen. The white board session especially in the second half of the video is a fantastic mind expanding ride and shows how LINQ is far far more than just queries. Typically I think Microsoft's marketing have probably muddied the waters again by calling it 'language integrated query' because, although querying is something you can do with it, it's much more like 'functional programming in C#'. For example, Wes showed how you can replace typical looping patterns with LINQ 'queries'. Take this little code snippet to get the even numbers from an array of numbers:
int[] list = int[] {1,2,3,4,5,6,7};
List<int> evenNumbers = new List<int>();
foreach(int n in list)
{
if(n % 2 == 0)
 evenNumbers.Add(n);
}
You can write the same thing with LINQ:
int[] list = int[] {1,2,3,4,5,6,7};
var evenNumbers =
from n in list
where n % 2 == 0
select n;
I think it's much easier to read the intention of the LINQ version. One thing that really struck me was how they use the custom iterators that I talked about in my last post to create the 'where' and 'select' clauses in LINQ. A very bright light bulb went off in my head at that point! Wes Dyer's blog is also well worth checking out.

Friday, February 09, 2007

Creating set based filters with custom iterators

Well it's back to work after a very nice two week break. I wrapped up my last contract with Ace Insurance in Crawley (hi guys!) three weeks ago, and I've been at my new assignment for a week now. I've been very busy getting up to speed with my new role (doing yet more application integration), but I did have time to skim MSDN where I came across this great article by Bill Wagner, titled 'Custom Iterators'. He talks about custom iterators, predicates and generics, all stuff that I'm quite familiar with, but it's the way he puts them together to create aggregate set functions, just like the ones that I've been discovering with functional languages like F#, that makes it really interesting. Custom iterators are new in C# 2.0 and make it really easy to create an instance of IEnumerable. Instead of implementing the entire interface, you just have to write a method that returns IEnumerable and has a yield statement for each item you want your IEnumerable to return. Behind the scenes the C# compiler creates an instance of IEnumerable for you by turning your method into a class with fields for all your local variables. For example, if you define an iterator that returns the numbers one to ten:
IEnumerable<int> OneToTen()
{
   for(int i=1; i<=10; i++)
   {
       yield return i;
   }
}
It will return an object of a class that you can imagine has a private field i and implements the IEnumerable.GetEnumerator() interface. You can then use the IEnumerator that it returns just like any other IEnumerator, for example:
public Test()
{
   IEnumerator<int> oneToTen = OneToTen().GetEnumerator();
   oneToTen.MoveNext();
   Console.WriteLine("{0}", oneToTen.Current);
   oneToTen.MoveNext();
   Console.WriteLine("{0}", oneToTen.Current);
}
Will output:
1
2
Now, the trick that Bill Wagner demonstrates, that hadn't occurred to me before is that you can also pass an instance of IEnumerable to your customer iterator to make a filters, transformers or any other set based operation you can think of. Here's a simple filter that only passes even numbers:
IEnumerable<int> Even(IEnumerable<int> list)
{
   foreach(int number in list)
   {
       if(number % 2 == 0)
           yield return number;
   }
}

public Test()
{
   foreach(int number in Even(OneToTen())
   {
       Console.WriteLine("{0}", number);
   }
}
Will output:
2
4
6
8
10
Bill's next trick to use anonymous delegates and generics to write a generic filter:
delegate bool Test<T>(T value);
IEnumerable<T> Filter<T>(IEnumerable<T> list, Test<T> test)
{
   foreach(T item in list)
   {
       if(test(item))
           yield return item;
   }
}

public Test()
{
   IEnumerable<int> evenNumbers = Filter<int>(OneToTen(), delegate(int number)
   {
       return number % 2 == 0;
   });
   foreach(int number in evenNumbers)
   {
       Console.WriteLine("{0}", number);
   }
}
Will output:
2
4
6
8
10
Very neat, but a bit long winded syntax wise I think. This will all change with C# 3.0's lambda expressions. It's yet another step towards a more functional programming style that's such a powerful addition to the programmer's toolbox.

Wednesday, January 17, 2007

Writing your own XSD.exe

If you spend any time working with Web Services or even just XML, you'll inevitably come into contact with XSD.exe and WSDL.exe, they both generate .net code from XSD type definitions. With XSD.exe, you simply give it the path to an xsd document and it will spit out a .cs file. That file defines types that will serialize to an XML document instance that validates against the xsd. De-serializing your XML to a strongly typed object model is almost always better than fiddling with the XML DOM, but what if you don't like the code that XSD.exe generates? Well, you can easily spin your own XSD.exe since it simply uses the public framework types System.Xml.Serialization.XmlSchemaImporter, System.Xml.Serialization.XmlCodeExporter and CodeDom. For some reason the MSDN documentation on these classes says, "This class supports the .NET Framework infrastructure and is not intended to be used directly from your code.", but don't let that put you off, they're public types and work fine. At a high level the process goes like this, you can follow it with the code sample below:
  1. Load your xsd file into an XmlSchema.
  2. Create an XmlSchemaImporter instance that references your schema. This class is used to generate mappings from XSD types to .net types.
  3. Create a CodeDom CodeNamespace instance where you'll build the syntactic structure of your .net types.
  4. Create an XmlCodeExporter instance with a reference to the CodeNamespace that you use to export your type. This is the class that actually creates the syntactic structure of the .net types in the CodeNamespace.
  5. Create an XmlTypeMapping instance for each type that you wish to export from the XSD.
  6. Call the ExportTypeMapping method on XmlCodeExporter for each XmlTypeMapping object, this creates the types syntax in the CodeNamespace object.
  7. Use a CSharpCodeProvider to output C# source code for the types that were created in CodeNamespace object.
Once the CodeNamespace has been fully populated (after step 6 above) there's an opportunity to make any changes that we wish to the code we output. Note that at this stage, the CodeDom CodeNamespace object represents an IL syntactic structure rather than code in a particular language. We could just as easily generate VB.NET at this point. We can use the CodeDom methods to alter that structure before outputting source code. In the example below I run the RemoveAttributes function to remove some attributes from the type definition.
using System;
using System.IO;
using System.Collections.Generic;
using System.Reflection;
using System.Text;
using System.Xml;
using System.Xml.Serialization;
using System.Xml.Schema;
using System.CodeDom;
using System.CodeDom.Compiler;

using Microsoft.CSharp;

using NUnit.Framework;

namespace XmlSchemaImporterTest
{
  [TestFixture]
  public class XsdToClassTests
  {
      // Test for XmlSchemaImporter
      [Test]
      public void XsdToClassTest()
      {
          // identify the path to the xsd
          string xsdFileName = "Account.xsd";
          string path = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
          string xsdPath = Path.Combine(path, xsdFileName);

          // load the xsd
          XmlSchema xsd;
          using(FileStream stream = new FileStream(xsdPath, FileMode.Open, FileAccess.Read))
          {
              xsd = XmlSchema.Read(stream, null);
          }
          Console.WriteLine("xsd.IsCompiled {0}", xsd.IsCompiled);

          XmlSchemas xsds = new XmlSchemas();
          xsds.Add(xsd);
          xsds.Compile(null, true);
          XmlSchemaImporter schemaImporter = new XmlSchemaImporter(xsds);

          // create the codedom
          CodeNamespace codeNamespace = new CodeNamespace("Generated");
          XmlCodeExporter codeExporter = new XmlCodeExporter(codeNamespace);

          List maps = new List();
          foreach(XmlSchemaType schemaType in xsd.SchemaTypes.Values)
          {
              maps.Add(schemaImporter.ImportSchemaType(schemaType.QualifiedName));
          }
          foreach(XmlSchemaElement schemaElement in xsd.Elements.Values)
          {
              maps.Add(schemaImporter.ImportTypeMapping(schemaElement.QualifiedName));
          }
          foreach(XmlTypeMapping map in maps)
          {
              codeExporter.ExportTypeMapping(map);
          }

          RemoveAttributes(codeNamespace);

          // Check for invalid characters in identifiers
          CodeGenerator.ValidateIdentifiers(codeNamespace);

          // output the C# code
          CSharpCodeProvider codeProvider = new CSharpCodeProvider();

          using(StringWriter writer = new StringWriter())
          {
              codeProvider.GenerateCodeFromNamespace(codeNamespace, writer, new CodeGeneratorOptions());
              Console.WriteLine(writer.GetStringBuilder().ToString());
          }

          Console.ReadLine();
      }

      // Remove all the attributes from each type in the CodeNamespace, except
      // System.Xml.Serialization.XmlTypeAttribute
      private void RemoveAttributes(CodeNamespace codeNamespace)
      {
          foreach(CodeTypeDeclaration codeType in codeNamespace.Types)
          {
              CodeAttributeDeclaration xmlTypeAttribute = null;
              foreach(CodeAttributeDeclaration codeAttribute in codeType.CustomAttributes)
              {
                  Console.WriteLine(codeAttribute.Name);
                  if(codeAttribute.Name == "System.Xml.Serialization.XmlTypeAttribute")
                  {
                      xmlTypeAttribute = codeAttribute;
                  }
              }
              codeType.CustomAttributes.Clear();
              if(xmlTypeAttribute != null)
              {
                  codeType.CustomAttributes.Add(xmlTypeAttribute);
              }
          }
      }
  }
}

Orthogonal Code

As Steve McConnel says in Code Complete, Managing complexity is the fundamental imperative of any software developer, but how do you do that? Well Code Complete spends several hundred pages outlining some techniques and you should also read Bob Martin's Agile Software Development and Martin Fowler's Refactoring. But if you want something that you can digest in your lunch hour, I recently discovered Jeremy Miller's nice series of posts on writing maintainable code, but the one which really struck a chord with me was this one, orthogonal code. Learning how to write code without digging myself into a pit of tangled complexity has been a constant thread in my development as a programmer and he succinctly captures some of the main techniques that I've learnt over the last ten years to avoid doing just that...

Any here's a great quote:

Don't write this post off as just academic hot air, this is about consistently leaving the office on time and protecting your company's code assets because you'll be much more likely to smoothly sustain a forward progress with your existing code.

Thursday, January 04, 2007

Easy impersonation

I've been quite pleasantly surprised recently how easy it is to do impersonation in .net. The trick is a win32 api function that's not covered by the BCL, LogonUser

[System.Runtime.InteropServices.DllImport("advapi32.dll")]
public static extern int LogonUser(
    String lpszUsername, 
    String lpszDomain, 
    String lpszPassword, 
    int dwLogonType, 
    int dwLogonProvider, 
    out  IntPtr phToken);

LogonUser does what it says and logs the given user onto the computer the code is running on. It takes a user name, domain name and a clear text password as well as two constants, a logon type which allows you to specify an interactive user, network user etc and a logon provider (I just used default). It returns a pointer to a token which represents the user and which you can use to create a new WindowsIdentity instance. Once you've got a WindowsIdentity, you can just call the Impersonate method to switch your code execution context to the new identity. Here's a little NUnit test to demonstrate:

[NUnit.Framework.Test]
public void ImpersonationSpike()
{
    string username = "imptest";
    string domain = "mikesMachine"; // this is the machine name
    string password = "imptest";
    IntPtr userToken;

    int hresult = LogonUser(
        username, 
        domain, 
        password, 
        (uint)LogonSessionType.Network, 
        (uint)LogonProvider.Default, 
        out userToken);

    if(hresult == 0)
    {
        int error = System.Runtime.InteropServices.Marshal.GetLastWin32Error();
        Assert.Fail("Error occured: {0}", error);
    }
    WindowsIdentity identity = new WindowsIdentity(userToken);
    Console.WriteLine(identity.Name);

    Console.WriteLine("I am: '{0}'", WindowsIdentity.GetCurrent().Name);
    System.IO.File.WriteAllText(@"c:\Program Files\hello.txt", "Hello");

    WindowsImpersonationContext context = null;
    try
    {
        context = identity.Impersonate();
        Console.WriteLine("Impersonating: '{0}'", WindowsIdentity.GetCurrent().Name);
    }
    finally
    {
        if(context != null)
        {
            context.Undo();
        }
        if(userToken != IntPtr.Zero)
        {
            CloseHandle(userToken);
        }
    }
    Console.WriteLine("I am: '{0}'", WindowsIdentity.GetCurrent().Name);
}

[System.Runtime.InteropServices.DllImport("advapi32.dll")]
public static extern int LogonUser(
    String lpszUsername, 
    String lpszDomain, 
    String lpszPassword, 
    uint dwLogonType, 
    uint dwLogonProvider, 
    out  IntPtr phToken);

[System.Runtime.InteropServices.DllImport("kernel32.dll", SetLastError = true)]
static extern bool CloseHandle(IntPtr handle);

enum LogonSessionType : uint
{
    Interactive = 2,
    Network,
    Batch,
    Service,
    NetworkCleartext = 8,
    NewCredentials
}

enum LogonProvider : uint
{
    Default = 0, // default for platform (use this!)
    WinNT35,     // sends smoke signals to authority
    WinNT40,     // uses NTLM
    WinNT50      // negotiates Kerb or NTLM
}

It should output the following...

mikesMachine\imptest
I am: 'mikesMachine\mike'
Impersonating: 'mikesMachine\imptest'
I am: 'mikesMachine\mike'

1 passed, 0 failed, 0 skipped, took 1.27 seconds.

A few things to note, you have to have pInvoke permission, which might be problem in some web hosting environments. Also, you have to have a cleartext password in order to log on the user you're impersonating, so your user has to supply it, or you have to store it somewhere which is an obvious security risk.

Of course, if you want to impersonate a particular account in ASP.NET you can just use the <identity impersonate="true" username="theUserName" password="ThePassword" />. On Server 2003, you can even more simply just set your application to run in a custom application pool and set the identity in the application pool settings. All this stuff is explained in this msdn article. But this technique here is great if you just want to grab an identity for a single method call or if you need to do impersonation in something other than an ASP.NET application.

Wednesday, January 03, 2007

SQL Authorization Manager

A while back I blogged about AzMan, a tool for managing operation level permissions that's supplied with Windows Server 2003. Recently I've been introduced to (thanks Chris!) a neat extension of the AzMan idea, SQL Authorization Manager (SqlAzMan) that's written by an Italian developer, Andrea Ferendeles. It takes the basic idea of operation level permissions management, but bases it on a SQL Server database rather than Active Directory, which probably makes more sense for most application developers. You can, however, also reference Active Directory users/groups if you want. It's also written entirely in .net 2.0 so you don't have to tackle all that irritating COM stuff like you do with AzMan and like AzMan it's user interface is an mmc snapin. Another neat addition is the time limiting of permissions, so you can allow someone to do an operation only for within a specific period. It comes with a built in RoleProvider, so you can plug it straight into the existing ASP.NET security framework, but to properly leverage the full power of operation level permissions, you can write directly to it's .net API. Here's a little test I wrote to check it out:

using System;
using System.Security.Principal;
using NUnit.Framework;

using NetSqlAzMan;
using NetSqlAzMan.Interfaces;

namespace Ace.Web.Security.SqlAzMan.Test
{
    /// 
    /// This test is a 'spike' to try out the functionality provided by NetSqlAzMan 
    /// see http://sourceforge.net/projects/netsqlazman/
    /// 
    /// To work it requires a NetSqlAzMan database to have been set up with the following properties:
    /// 
    /// Store name:         Test
    /// Application name:   TestApplication
    /// Operation name:     DoSomething
    /// 
    /// The user running the test should be given permission to execute operation 'DoSomething', see
    /// the NetSqlAzMan documentation for details
    /// 
    /// given authorisation to the user executing the test. You will also have to have installed NetSqlAzMan
    /// on the machine being used for testing.
    /// 
    [TestFixture]
    public class SqlAzManSpike
    {
        string _connectionString = "Data Source=(local);Initial Catalog = NetSqlAzManStorage;Integrated Security = SSPI;";
        IAzManStorage _storage;

        [SetUp]
        public void SetUp()
        {
            _storage = new SqlAzManStorage(_connectionString);
        }

        [NUnit.Framework.Test]
        public void CheckPermissionForCurrentUser()
        {
            WindowsIdentity identity = WindowsIdentity.GetCurrent();
            Console.WriteLine("WindowsIdentity = '{0}'", identity.Name);

            // Check if I can do the DoSomething operation
            AuthorizationType authorization = _storage.CheckAccess("Test", "TestApplication", "DoSomething",
                identity, DateTime.Now, true);

            Assert.AreEqual(AuthorizationType.Allow, authorization);
        }

        /// 
        /// This test expects that a SqlAzMan db user exists called 'Domain\user'
        /// 
        [NUnit.Framework.Test]
        public void CheckPermissionForStringUser()
        {
            string username = @"Domain\user";
            IAzManDBUser user = _storage.GetDBUser(username);
            Assert.IsNotNull(user, "user is null");

            Console.WriteLine("Db user = '{0}'", user.UserName);

            // Check if this user can do the DoSomething operation
            AuthorizationType authorization = _storage.CheckAccess("Test", "TestApplication", "DoSomething",
                user, DateTime.Now, true);

            Assert.AreEqual(AuthorizationType.Allow, authorization);
        }
    }
}

It's all very nice. The only complaint I've got is that Andrea hasn't caught all the SQL exceptions and given them more meaningful messages. If you try and check access of an operation that doesn't exist you get a nasty SQL exception rather than a NetSqlAzMan message saying that the given operation doesn't exist.

Tuesday, January 02, 2007

Are you confused by character encodings?

I am. Being a native English speaker leads to a number of pathologies, the the most obvious one is our inability to speak foreign languages. What's the point when everyone speaks English? I only learnt to speak Japanese because I worked in a Japanese school for the JET programme for two years, before that I'd been a language dunce. Three years of secondary school French lessons had left me being only able to order a coffee, tell you my name and complain about the weather. There's a similar tendency with English speaking programmers, a total lack of knowledge about character encodings. After learning about ASCII as a boy I really haven't progressed at all beyond thinking that each character is a byte with all the important ones between 32 and 127. I have a vague awareness of Unicode and other things like UTF-8, but I don't really know what they mean in technical terms. If you're like me, it's well worth reading Joel Spolsky's excellent post on his Joel On Software blog where he has a brief potted history of character encodings and what every programmer should know about them. Joel On Software should be required reading for anyone working in the IT world, not just programmers, so long as you pass his stuff through the coding horror filter :)

Tuesday, December 19, 2006

F# Life

As I said before, I've been getting very into the idea (if not the practice) of functional programming recently, so I was really excited to read about F#. I'm not the only one, there seems to be a general buzz about this new programming language from Microsoft research in Cambridge. The guy who designed it, Don Syme also played a big part in designing the excellent generics implementation in the CLR and has implemented a number of new CLR features to support F# and functional languages in general. I guess in this respect you can see F# as a test bed for pushing the CLR in some exciting new directions. The great thing about F# is that it allows you to leverage all of the .net framework from a functional language with great Visual Studio integration, including an interactive shell that you can use in any project, not just F# ones. Of course, to use the .net libraries, F# has to include object oriented and imperative programming styles as well and while this has the effect of polluting its pure 'functionalness' it also promise to make it a real swiss army knife of programming techniques. Don Syme has got some great examples of F# utilising the power of .net on his blog.

I'm a total novice at functional programming and as I'm still learning about OO programming even after 10 years, I'm sure I'm just starting off on a very long path of discovery. I really don't get a lot of the higher level concepts I've been reading about but some of the simple stuff does make a lot of sense. In particular I'm very excited about the concept of using functions as first class constructs and some of the cool patterns that this allows.

To illustrate how I'm getting on with F# and to show some of the cool stuff that I've found most interesting, I've implemented an F# version of John Conway's game of life. If you've never come across this before it's was one of the first (if not the first) cellular automations and it's great fun to have a play with one of many on line implementations. I've been using the game of life as a great way of trying out new languages for years. It was both my first java program and even one of my first VB ones (all those years ago). The rules are dead simple: there are a grid of cells of arbitrary size and each cell can be either on or off (alive or dead). If a cell has less than two neighbours it dies from loneliness, if it has more than three, it dies from overcrowding. If a dead cell has exactly 3 neighbours it comes alive. You start off with a pattern of cells and then let the generations roll.

OK, let's have a look at the code. I'm going to paste it in segments with some commentary after each, and you can run each segment in F# interactive or put the whole thing in an F# code file and run it through the F# compiler. First let's define some data structures:

// the size of the grid
let size = 10

// make the grid of 10 X 10 cells initialize each to 0
let theGrid = Array.create_matrix size size 0

// a square array of array with the pattern to run
let thePattern = [|
    [|1;1;1|];
    [|1;0;0|];
    [|0;1;0|]|]

The 'let' statement defines a value (which can also be a function). F# programs seem to be just a collection of let statements as you build up a hierarchy of values defined with other values. It's this declarative style that is one of the hardest things to get your head around if you're an imperative programmer of many years like me. I've defined two arrays of arrays here, one for the initial grid and one for the initial pattern. There's a data structure called a 'list' which is lightweight immutable linked list and more in the functional tradition, but I've used a mutable array here, not because I want the mutability, but because you can address the cells by co-ordinate. The pattern I've set here is the famous 'Glider' that should move across the grid.

// define a function for iterating through a matrix (thanks to DeeJay)
let iteri_matrix f = Array.mapi (fun i -> Array.mapi (f i)) 

This function, iteri_matrix, iterates through my array of array applying a function that takes the co-ordinates of the current cell. I got it from a comment to 'a walk with a newbie', on HubFS which is the community site for F#. That post is a great introduction to a lot of good F# techniques, but I couldn't help thinking that the programming style was too imperative. In the same way that you can write procedural code in OO languages, you can write imperative code in F#. You can do it, but it's not particularly elegant.

The 'iteri_matrix' function illustrates one of the core features of functional programming: functions as first class constructs. iteri_matrix takes a function 'f' as its argument and returns a function. The function it returns consists of a static function Array.mapi that for each item (which is an array, since we want to run it over an array of arrays) applies a second Array.mapi that applies the function 'f' that we originally passed in to each item of the array. When you call iteri_matrix passing it a function argument and a matrix, it constructs a new function that will iterate over the matrix with the given function and then applies that function to the matrix. OK, it's a bit mind boggling, but really powerful when you get the hang of it.

// add the pattern to the middle of the grid
let offset = (size - thePattern.Length)/2
let getPatternCell (i,j) =
    if i < 0 || j < 0 then 0 
    elif i < thePattern.Length && j < thePattern.Length then thePattern.(i).(j)
    else 0
let addPatternToGrid grid = grid |> iteri_matrix(fun i j _ -> grid.(i).(j) + getPatternCell(i-offset, j-offset))

Next we add the initial pattern to the centre of the grid. First we define a function to get a pattern cell at a particular co-ordinate. It returns zero if the co-ordinates are out of range because we want to use it to sum the pattern and the grid without having to make the pattern matrix the same size as the grid matrix. Of note here is the use of a 'traditional' if.. elif ... else statement, later on we'll see another way of mapping conditionals called 'pattern matching' (not to be confused with regular expressions!). 

The second function, 'addPatternToGrid', takes a grid and uses iteri_matrix to sum the grid and the pattern (the offset value puts the pattern in the middle of the grid). A really cool feature of F# here is the '|>' operator which secret geek explains very nicely here. It allows you to chain functions very intuitively a bit like the way the pipe operator '|' works in most unix shell environments.

// define neighbours
let prev i = if i = 0 then size - 1 else i - 1
let next i = if i = size - 1 then 0 else i + 1
let neighbours (i,j) = [
    (prev i, prev j);
    (prev i, j);
    (prev i, next j);
    (i, prev j);
    (i, next j);
    (next i, prev j);
    (next i, j);
    (next i, next j)]

The next step is to define the neighbours of a cell. We're using a feature of F# called 'tuple' to define co-ordinates '(i,j)'. It's a really simple data structure that just groups together some values. The functions 'prev' and 'next' just define the previous and next numbers and 'wrap around' the matrix. The function 'neighbours' takes a co-ordinate and returns a list of the co-ordinates of the neighbours.

// sum a list
let rec sum aList =
    match aList with
    | [] -> 0
    | first::newList -> first + sum newList

This function simply sums a list, but it shows several interesting features. The first is that recursion is the preferred way of doing looping in functional languages. We've defined the function with 'let rec' which tells F# that this function is recursive (there must be a good reason for this, but why can't F# just figure that out for itself?) and we get the function to call itself passing the list minus it's first member. 'match aList with' is a match expression that defines a list of possible pattern matches for aList. The first match matches an empty list and returns zero, the second match strips off the first item from the list and then adds it to the sum of the remaining items.

// get the sum of the live neighbours
let cellAddNeighbours (i,j) grid = neighbours (i,j) |> List.map (fun (a,b) -> grid.(a).(b)) |> sum

// calculate neighbour sums for all cells
let addNeighbours grid = grid |> iteri_matrix (fun i j _ -> cellAddNeighbours (i,j) grid)

Next we work out the sum of all a cell's neighbours with the function 'cellAddNeighbours'. This takes a co-ordinate 'tuple' and a grid as arguments. We pass the cell co-ordinate to the neighbours function, get a list of neighbour co-ordinates back, pass that list to a static function of the List class, List.map, which runs a function on each item and returns a new list of results. Here we're just getting the cell value at the neighbour co-ordinate. The list of cell values is then passed to 'sum' which returns a single value of the sum of all the neighbours. The next function, 'addNeighbours', simply runs 'cellAddNeighbours' for each cell in the grid and returns a new grid of the sums. Once again it uses the iteri_matrix function we defined above.

// live or die rules for a single cell:
//      if the cell is alive and has 2 or 3 neighbours, then it lives, otherwise it dies
//      if the cell is dead and has 3 neighbours it comes alive
let cellLiveOrDie cellValue neighbourSum =
    match (cellValue, neighbourSum) with
    | (1,(2 | 3)) -> 1
    | (0, 3) -> 1
    | (_,_) -> 0

// calculate live or die for the whole grid
let liveOrDie grid neighbourSumGrid = grid |> iteri_matrix (fun i j _ -> cellLiveOrDie grid.(i).(j) neighbourSumGrid.(i).(j))

The next function 'cellLiveOrDie' contains the main set of rules for the game of life. Once again we're using pattern matching, but this time on a tuple. This is a really neat feature because it allows you to just list a set of rules for matching each element in a tuple without writing loads of conditional logic. The first rule says 'if the cellValue is 1 and the neighbourSum is 2 or 3 set the cell value to 1'. The second rule says 'if the cellValue is 0 and the neighbourSum is 3 set the cell value to 1. The third rule says, 'for any other combination, set the cell value to 0. The underscore '_' is a really useful bit of syntax that means 'any value'.

The second function here, 'liveOrDie' takes two matrixes, one of the current grid and one containing the neighbour sums that we calculated earlier. Once again it uses iteri_matrix to apply 'cellLiveOrDie' to each cell in the grid.

// print a cell
let printCell cell = 
    if cell = 1 then printf("X ") else printf("_ ")

// print the grid
let printGrid grid = 
    grid |> Array.iter (fun line -> (line |> Array.iter printCell); printf "\n");
    printf "\n\n"

This function 'printCell' prints a cell to the command line and 'printGrid' uses printCell to print the whole grid. There's a tiny bit of imperative programming in 'printGrid': '(line |> Array.iter printCell); printf "\n"', you can use a semicolon to separate statements just like in C and here we're using it to print a new line after each row of the grid.

// do n generations
let rec DoGenerations n grid =
    printGrid grid;
    match n with
    | 0 -> printf "end\n"
    | _ -> grid |> addNeighbours |> liveOrDie grid |> DoGenerations (n-1)

'DoGenerations' is the core loop of the application. Once again we're favoring recursion over looping in order to do the generations. I really like the clarity of the |> operator. Here we see how nice it is to simply chain together our previously defined functions 'addNeighbours' and 'liveOrDie' to create the next generation.

// run 10 generations
do theGrid |> addPatternToGrid |> DoGenerations 10

Finally, here is the program's 'Main()'. The 'do' statement simply runs the given expression. We take the initial grid, add the pre defined pattern and do 10 generations. The output should be something like this:

_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ X X X _ _ _ _ 
_ _ _ X _ _ _ _ _ _ 
_ _ _ _ X _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 


_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ X _ _ _ _ _ 
_ _ _ X X _ _ _ _ _ 
_ _ _ X _ X _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 

........ missing out a few generations here ......

_ _ X _ _ _ _ _ _ _ 
_ X X _ _ _ _ _ _ _ 
_ X _ X _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 


_ X X _ _ _ _ _ _ _ 
_ X _ X _ _ _ _ _ _ 
_ X _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ _ _ 


Monday, December 04, 2006

How to write an XSD

Web services are all about communicating with XML messages. The great benefit of XML is that it is a platform neutral technology. These messages shouldn't have any dependency on a particular web service implementation technology (such as .net or java). Unfortunately many of the implementation toolkits (especially ASP.NET) encourage you to think of web services as Remote Procedure Calls (RPC) which can inject unwanted dependencies on the toolkit and often leads to sub-optimal 'chatty' interfaces. That's why it's always best to define your messages using XSD rather than by getting your implementation toolkit (such as visual studio) to spit out type definitions based on your technology specific types (such as .net classes).

The question then becomes how to write effective XSDs. In this document I'd like to give a few pointers. The example for this demonstration is the following XML document:

<Order 
  xmlns="uri:ace-ina.com:schemas:order" 
  xmlns:prd="uri:ace-ina.com:schemas:product" 
  Id="0">
	<OrderLines>
		<OrderLine Id="0">
			<Product Id="0">
				<prd:Name>Bread</prd:Name>
				<prd:Price>0.79</prd:Price>
			</Product>
			<Quantity>2</Quantity>
			<Total>1.58</Total>
		</OrderLine>
		<OrderLine Id="1">
			<Product Id="2">
				<prd:Name>Milk</prd:Name>
				<prd:Price>0.48</prd:Price>
			</Product>
			<Quantity>1</Quantity>
			<Total>0.48</Total>
		</OrderLine>
	</OrderLines>
	<Total>2.06</Total>
</Order>

It's a simple order with an id and a collection of order lines. Each order line defines a product and gives the quantity and total. The namespace of the order is 'uri:ace-ina.com:schemas:order'. A bit of added complication is introduced by defining the product in a separate namespace: 'uri:ace-ina.com:schemas:product'.

Now let's create an XSD that defines the schema for this XML document. The XSD meta-schema is defined in the namespace: 'http://www.w3.org/2001/XMLSchema', and an XSD's root element is always 'schema', so let's start with that:

<xs:schema 
	xmlns:xs='http://www.w3.org/2001/XMLSchema'>
</xs:schema>

We also want to define the namespace of the target document which in this case is 'uri:ace-ina.com:schemas:order'. We need to include that namespace and reference it in the targetNamespace attribute. To enforce that all the defined elements in the XSD should belong to the target namespace we need to set elementFormDefault to 'qualified'.

<xs:schema 
	xmlns:xs='http://www.w3.org/2001/XMLSchema' 
	xmlns='uri:ace-ina.com:schemas:order' 
	targetNamespace = 'uri:ace-ina.com:schemas:order'
	elementFormDefault='qualified'>
</xs:schema>

Next we should define our types. Think of types in your XSD as entities in the same way as you would think of classes in a .net application or tables in a database. In the order document there are two primary types: 'Order' and 'OrderLine'. 'Product' belongs to a seperate namespace and XSD file and we'll be looking at that later. Types that contain attributes and/or elements are known as 'complex types' and are defined in a 'complexType' element. I like to name complex types '<name of target element>Type'. So let's add two complex types to our XSD, OrderType and OrderLineType:

<xs:schema 
	xmlns:xs='http://www.w3.org/2001/XMLSchema' 
	xmlns='uri:ace-ina.com:schemas:order' 
	targetNamespace = 'uri:ace-ina.com:schemas:order'
	elementFormDefault='qualified'>
	<xs:complexType name="OrderType">
	</xs:complexType>
	<xs:complexType name="OrderLineType">
	</xs:complexType>
</xs:schema>

We can add attributes to our types using the 'attribute' element. Both OrderType and OrderLineType have id attributes which we want to be required integer types:

<xs:schema 
	xmlns:xs='http://www.w3.org/2001/XMLSchema' 
	xmlns='uri:ace-ina.com:schemas:order' 
	targetNamespace = 'uri:ace-ina.com:schemas:order'
	elementFormDefault='qualified'>
	<xs:complexType name="OrderType">
		<xs:attribute name="Id" type="xs:integer" use="required" />
	</xs:complexType>
	<xs:complexType name="OrderLineType">
		<xs:attribute name="Id" type="xs:integer" use="required" />
	</xs:complexType>
</xs:schema>

Child elements can be defined as part of a 'sequence', 'choice' or 'all' containing element. 'Sequence' requires that all its elements exist in the given sequence in the target document, 'choice' allows only one of it's child elements to exist and 'all' requires that all or none of the defined elements exist, but that the order is not important. Repeating elements are not allowed in an 'all' group. minOccurs and maxOccurs are used to define optional and repeating elements. In this case we want to define 'OrderLines' and 'Total' for 'OrderType' and 'Product', 'Quantity' and 'Total' for 'OrderLine'. They are all required non-repeating elements so we don't need to specify minOccurs and maxOccurs (the default for both is '1') and we'll use 'Sequence' for all of them. We need to define the type of each element, both the OrderType and OrderLineType Total are defined as 'double' and Quantity is defined as 'integer'. We'll leave the types of 'OrderLines' and 'Product' until later:

<xs:schema 
	xmlns:xs='http://www.w3.org/2001/XMLSchema' 
	xmlns='uri:ace-ina.com:schemas:order' 
	targetNamespace = 'uri:ace-ina.com:schemas:order'
	elementFormDefault='qualified'>
	<xs:complexType name="OrderType">
		<xs:attribute name="Id" type="xs:integer" use="required" />
		<xs:sequence>
			<xs:element name="OrderLines" type=""/>
			<xs:element name="Total" type="xs:double"/>
		</xs:sequence>
	</xs:complexType>
	<xs:complexType name="OrderLineType">
		<xs:attribute name="Id" type="xs:integer" use="required" />
		<xs:sequence>
			<xs:element name="Product" type=""/>
			<xs:element name="Quantity" type="xs:integer"/>
			<xs:element name="Total" type="xs:double"/>
		</xs:sequence>
	</xs:complexType>
</xs:schema>

Because Total has the same name and type in both OrderType and OrderLineType, we can factor out a global element called Total and reference it from inside OrderType and OrderLineType:

<xs:schema 
	xmlns:xs='http://www.w3.org/2001/XMLSchema' 
	xmlns='uri:ace-ina.com:schemas:order' 
	targetNamespace = 'uri:ace-ina.com:schemas:order'
	elementFormDefault='qualified'>
	<xs:element name="Total" type="xs:double" />
	<xs:complexType name="OrderType">
		<xs:attribute name="Id" type="xs:integer" use="required" />
		<xs:sequence>
			<xs:element name="OrderLines" type=""/>
			<xs:element ref="Total"/>
		</xs:sequence>
	</xs:complexType>
	<xs:complexType name="OrderLineType">
		<xs:attribute name="Id" type="xs:integer" use="required" />
		<xs:sequence>
			<xs:element name="Product" type=""/>
			<xs:element name="Quantity" type="xs:integer"/>
			<xs:element ref="Total"/>
		</xs:sequence>
	</xs:complexType>
</xs:schema>

Now let's consider the OrderLines element in OrderType. In the target document, OrderLines contains a collection of OrderLine types, so we need to create a collection type for OrderLines. We can create a new complex type 'OrderLinesType' with a single repeating element 'OrderLine'. A repeating element is created by setting minOccurs to '0' and maxOccurs to 'unbounded'. We can then set the type of OrderLines to 'OrderLinesType'.

<xs:schema 
	xmlns:xs='http://www.w3.org/2001/XMLSchema' 
	xmlns='uri:ace-ina.com:schemas:order' 
	targetNamespace = 'uri:ace-ina.com:schemas:order'
	elementFormDefault='qualified'>
	<xs:element name="Total" type="xs:double" />
	<xs:complexType name="OrderType">
		<xs:attribute name="Id" type="xs:integer" use="required" />
		<xs:sequence>
			<xs:element name="OrderLines" type="OrderLinesType"/>
			<xs:element ref="Total"/>
		</xs:sequence>
	</xs:complexType>
	<xs:complexType name="OrderLineType">
		<xs:attribute name="Id" type="xs:integer" use="required" />
		<xs:sequence>
			<xs:element name="Product" type=""/>
			<xs:element name="Quantity" type="xs:integer"/>
			<xs:element ref="Total"/>
		</xs:sequence>
	</xs:complexType>
	<xs:complexType name="OrderLinesType">
		<xs:sequence>
			<xs:element name="OrderLine" type="OrderLineType" minOccurs="0" maxOccurs="unbounded"/>
		</xs:sequence>
	</xs:complexType>
</xs:schema>

We're still missing the product type. This is defined in a seperate namespace 'uri:ace-ina.com:schemas:product' in a seperate XSD document:

<xs:schema 
	xmlns:xs="http://www.w3.org/2001/XMLSchema" 
	xmlns="uri:ace-ina.com:schemas:product" 
	targetNamespace="uri:ace-ina.com:schemas:product" 
	elementFormDefault="qualified">
	<xs:complexType name="ProductType">
		<xs:sequence>
			<xs:element name="Name" type="xs:string"/>
			<xs:element name="Price" type="xs:double"/>
		</xs:sequence>
		<xs:attribute name="Id" type="xs:integer" use="required"/>
	</xs:complexType>
</xs:schema>

To reference a schema from another schema with a different namespace we use 'import'. To reference another XSD with the same namespace, use 'include'. Here the namespace is different so we need to add an 'import' element to our Order XSD. We also need to define the product namespace and give it a prefix, since we already have a default namespace (uri:ace-ina.com:schemas:order). We'll use 'prd' here. We can now define the Product element's type as 'prd:ProductType':

<xs:schema 
	xmlns:xs='http://www.w3.org/2001/XMLSchema' 
	xmlns='uri:ace-ina.com:schemas:order' 
	targetNamespace = 'uri:ace-ina.com:schemas:order'
	xmlns:prd='uri:ace-ina.com:schemas:product'
	elementFormDefault='qualified'>
	<xs:import namespace="uri:ace-ina.com:schemas:product" schemaLocation="Product.xsd" />
	<xs:element name="Total" type="xs:double" />
	<xs:complexType name="OrderType">
		<xs:attribute name="Id" type="xs:integer" use="required" />
		<xs:sequence>
			<xs:element name="OrderLines" type="OrderLinesType"/>
			<xs:element ref="Total"/>
		</xs:sequence>
	</xs:complexType>
	<xs:complexType name="OrderLineType">
		<xs:attribute name="Id" type="xs:integer" use="required" />
		<xs:sequence>
			<xs:element name="Product" type="prd:ProductType"/>
			<xs:element name="Quantity" type="xs:integer"/>
			<xs:element ref="Total"/>
		</xs:sequence>
	</xs:complexType>
	<xs:complexType name="OrderLinesType">
		<xs:sequence>
			<xs:element name="OrderLine" type="OrderLineType" minOccurs="0" maxOccurs="unbounded"/>
		</xs:sequence>
	</xs:complexType>
</xs:schema>

The last remaining task is to define our top level global element 'Order' with type 'OrderType':

<xs:schema 
	xmlns:xs='http://www.w3.org/2001/XMLSchema' 
	xmlns='uri:ace-ina.com:schemas:order' 
	targetNamespace = 'uri:ace-ina.com:schemas:order'
	xmlns:prd='uri:ace-ina.com:schemas:product'
	elementFormDefault='qualified'>
	<xs:import namespace="uri:ace-ina.com:schemas:product" schemaLocation="Product.xsd" />
	<xs:element name="Order" type="OrderType" />
	<xs:element name="Total" type="xs:double" />
	<xs:complexType name="OrderType">
		<xs:attribute name="Id" type="xs:integer" use="required" />
		<xs:sequence>
			<xs:element name="OrderLines" type="OrderLinesType"/>
			<xs:element ref="Total"/>
		</xs:sequence>
	</xs:complexType>
	<xs:complexType name="OrderLineType">
		<xs:attribute name="Id" type="xs:integer" use="required" />
		<xs:sequence>
			<xs:element name="Product" type="prd:ProductType"/>
			<xs:element name="Quantity" type="xs:integer"/>
			<xs:element ref="Total"/>
		</xs:sequence>
	</xs:complexType>
	<xs:complexType name="OrderLinesType">
		<xs:sequence>
			<xs:element name="OrderLine" type="OrderLineType" minOccurs="0" maxOccurs="unbounded"/>
		</xs:sequence>
	</xs:complexType>
</xs:schema>

Defining your web service message types in terms of XSD decouples your web service from a particular implementation technology and aids interoperability. Also, understanding the XSD syntax allows you to read and understand WSDL, create your own client proxies and control the serialization between your implementation types and the XSD.

For a much more complete and extensive discussion on writing XSD schemas see the world wide web consortium's XML Schema Part 0: Primer Second Edition

Wednesday, November 29, 2006

XmlDiff

I had to compare two xml files today. This is trickier than you would at first think. Luckily Microsoft has a neat tool, XmlDiff that makes it quite easy. It can compare two files, or two XmlReaders or even fragments. Also it can spit out an xml diffgram so that you can examine and re-apply any changes. Just use it like this...

[Test]
public void XmlDiffTest()
{
    string source = "<root><child1>some text</child1><child2>more text</child2></root>";
    // note some whitespace, child nodes in different order, comments
    string target = "<root> <!-- I'm a comment --> <child2>more text</child2> " + 
        "<child1>some text</child1>  </root>"; 

    XmlReader expected = XmlReader.Create(new StringReader(source));
    XmlReader actual = XmlReader.Create(new StringReader(target));
    StringBuilder differenceStringBuilder = new StringBuilder();
    XmlWriter differenceWriter = XmlWriter.Create(new StringWriter(differenceStringBuilder));

    XmlDiff diff = new XmlDiff(XmlDiffOptions.IgnoreChildOrder |
        XmlDiffOptions.IgnoreComments |
        XmlDiffOptions.IgnoreWhitespace);

    bool areDifferent = diff.Compare(expected, actual, differenceWriter);
    Assert.IsTrue(areDifferent, string.Format(
        "expected response and actual response differ:\r\n{0}", differenceStringBuilder.ToString()));
}