Tuesday, March 27, 2007

How to add a command to a custom VS project type

In this post I'm going to show how to add your own commands to a custom project type's context menu in Visual Studio.

I've been having a lot of fun recently trying to create my own custom project type for VS. In my last post on this subject I described how you can create the most basic custom project type with no real functionality. When you create one of these with the VS SDK's MPF (Managed Package Framework), your project inherits from ProjectNode which seems to be designed to support a buildable project. This makes sense since VS is fundamentally a development tool and the vast majority of project types are going to be things that need to be built. This means that when you right click on your new project, it shows a context menu with commands like 'Set As Startup Project' and 'Debug' and there are hooks in the ProjectNode class to implement this behaviour. However the kind of project I want to create is more like a database project where there's no requirement to actually build anything and the project can't be run or debugged, unfortunately the only way to remove all the build and debug related functionality seems to be to create your own version of ProjectNode (and ProjectFactory too). I haven't explored this yet, so if you walk through this post to create your own context menu items as I describe you'll see those commands still displayed. However, they don't do anything if you select them. OK, so to start off you need to create your new project type following the steps in the last post. But we have to make one change. When the New Package Wizard runs, instead of leaving all the checkboxes blank on the 'Select VSPackage Options' page, you should check the 'Menu Command' box. When the wizard runs, in addtion to all the other project items, it creates a folder called 'CtcComponents' that includes the following files: CommandIds.h Guids.h Resource.h TestMenuPackage.ctc (or whatever you_called_your_project.ctc) If you right click on the ctc file and select properties you'll see that its build action is 'CtcFile'. This file has its own special compiler that builds a binary command table file that VS uses to build all the menus and toolbars you see in the IDE. The CtcFile compiler uses the C++ preprocessor, thus the C style header files, #defines and #includes. When the wizard builds the package project it creates a command for you that appears in the Tools menu. You can see how it's defined by looking in the ctc files. Basically, in the ctc file you define a command set with a unique guid and then a command group that belongs to that command set and a particular VS menu, then define some commands that belong to the command group. To have a command group that appears under your custom project type, you first need to define the command set guid in Guids.h (I just used the one that the wizard created)

// Command set guid for our commands (used with IOleCommandTarget)

// { bdc32849-d202-496c-96b4-837c0254e0e2 }

#define guidTestMenuPackageCmdSet { 0xBDC32849, 0xD202, 0x496C, { 0x96, 0xB4, 0x83, 0x7C, 0x2, 0x54, 0xE0, 0xE2 } }

#ifdef DEFINE_GUID

DEFINE_GUID(CLSID_TestMenuPackageCmdSet,

0xBDC32849, 0xD202, 0x496C, 0x96, 0xB4, 0x83, 0x7C, 0x2, 0x54, 0xE0, 0xE2 );

#endif

The C style syntax is a bit gnarly I agree, but you can get the Toos->Create Guid tool to generate it for you which is nice. Next you define the command group id and command id in the CommandIds.h file:

///////////////////////////////////////////////////////////////////////////////

// Menu Group IDs



#define MyProjectGroup    0x10A0



///////////////////////////////////////////////////////////////////////////////

// Command IDs



#define cmdidDoSomething   0x001

Next you link the command group to the command set and a VS menu in the ctc file under the NEWGROUPS_BEGIN section:

// my project group

guidTestMenuPackageCmdSet:MyProjectGroup, // group name

guidSHLMainMenu:IDM_VS_CTXT_PROJNODE,  // parent group

0x0001;          // priority

The parent group 'guidSHLMainMenu:IDM_VS_CTXT_PROJNODE' tells VS that this command group should appear under project nodes in the solution explorer. OK, so how do find out the name of that constant? It's a case of digging in a file called 'SharedCmdPlace.ctc' that's located at C:\Program Files\Visual Studio 2005 SDK\2007.02\VisualStudioIntegration\Common\Inc on my computer and having a best guess about which one seems to fit the bill. The next step is to link define the command in the ctc file under the BUTTONS_BEGIN section (all menu items are BUTTONs):

// Do Something Fun command

guidTestMenuPackageCmdSet:cmdidDoSomething,  // command

guidTestMenuPackageCmdSet:MyProjectGroup,  // parent group

0x0100,           // priority

guidTestMenuPackageCmdSet:bmpPic1,    // image

BUTTON,           // type

DEFAULTINVISIBLE  DYNAMICVISIBILITY,   // visibility

"Do Something Fun";        // caption

You can see we link the command id to our command group, give it a display priority (which controls where it appears in the command group), an icon (I've just used the one created by the wizard), define the visibility and the caption. The visibility is important. When we defined our command group we told VS that we wanted it to appear under a project node. This means that it will appear under every project node, not only ours. We need to have our command default to being invisible and then enable it only from our project node, that's why the visibility is set to 'DEFAULTINVISIBLE DYNAMICVISIBILITY'. OK, that's the ctc file done, now we can get our project node to enable the command and respond to it when it's clicked. Our project class (referring back to the previous post) inherits from ProjectNode which in turn inherits from HierarchyNode which implements IOleCommandTarget. If you read the documentation on VSPackages you'll see that IOleCommandTarget is the interface for any class that needs to intercept command invocations. IOleCommandTarget defines two methods, QueryStatus and Exec. These are handled by HierarchyNode and you can customise their behaviour by overriding QueryStatusOnNode and ExecCommandOnNode. QueryStatus gives the implementer of IOleCommandTarget a chance to modify the display of the menu item and is invoked whenever the project node becomes active. Because the default visibility of our command is 'DEFAULTINVISIBLE DYNAMICVISIBILITY', we need to enable it when our project is active by overriding QueryStatusOnNode in our project class:

protected override int QueryStatusOnNode(

    Guid guidCmdGroup, uint cmd, IntPtr pCmdText, ref QueryStatusResult result)

{

    if (guidCmdGroup == GuidList.guidTestMenuPackageCmdSet)

    {

        result = QueryStatusResult.SUPPORTED  QueryStatusResult.ENABLED;

        return VSConstants.S_OK;

    }

    return base.QueryStatusOnNode(guidCmdGroup, cmd, pCmdText, ref result);

}

We can find out if the command is one of ours by checking the guidCmdGroup argument against the guid for our command set (I've used the GuidList static class that's generated by the wizard for this). Any other commands we just delegate to the base class. Now if we run our project by hitting F5 and create a new instance of our custom project type we'll see our new command in the context menu of the project node. To respond when a user clicks on our new command we simply override ExecCommandOnNode:

protected override int ExecCommandOnNode(

    Guid guidCmdGroup, uint cmd, uint nCmdexecopt, IntPtr pvaIn, IntPtr pvaOut)

{

    if (guidCmdGroup == GuidList.guidTestMenuPackageCmdSet)

    {

        if (cmd == 13)

        {

            return ShowMessage();

        }

    }

    return base.ExecCommandOnNode(guidCmdGroup, cmd, nCmdexecopt, pvaIn, pvaOut);

}

Once again we look for any of our own commands by checking the guidCmdGroup argument and pick individual commands by checking the 'cmd' argument which holds the command id. This is kinda redundant here because we've only got the one command. ShowMessage is just a private function that pops up a message box in this case, but it's where you'd implement whatever functionality you wanted your command to execute. Again we delegate any other commands to the base class. And that's all there is to it. Now I know I'm going to start sounding like a broken record, but this VSPackage stuff is made far too hard by poor documentation. Needless to say, I spent ages wondering why I couldn't see my command as I debugged through QueryStatusOnNode, and that was after I spent just as long working out that I should be overriding QueryStatusOnNode. As for finding the IDM_VS_CTXT_PROJNODE constant, that was just luck. All the project base stuff really does need some kind of high level overview. Even the best samples are very hard to grep without it.

Sunday, March 25, 2007

Is LINQ the writing on the wall for C#?

C++ was the writing on the wall for C. C is a phenomenally successful low level systems programming language. If you want to write an operating system you use C. I remember first becoming aware of C as a teenage BASIC freak when it was discussed in the computing magazines I used to read. In those days the choice for budding programmers like me was between the ROM based BASIC interpreter, or writing in Assembly language. C was described as a way of getting the speed and power of assembler with the ease of BASIC and it pretty much took over the PC programming world. A lot of business applications were written in C because there was no higher level abstraction that could do the job. Visual Basic took years to get the necessary performance and functionality to write decent enterprise level applications. A lot of people would argue that it never got there. But now, no one would dream of writing a business application with C, we all use modern OO languages like Java or .NET. C++ is an OO language right? So why don’t we build modern business applications with C++? The reason is that C++ is a low level systems programming language with some OO stuff grafted on. You still have to do all your own memory management and worst of all you have to get your head around all that mind boggling pointer stuff that’s frankly beyond the average Mort. For a long time, though, a lot of business applications were built with C++. People were right to believe that OO programming gave them a fantastic tool to get on top of the complexity of large programs, but what they needed was the OO stuff without the C stuff and that’s where, first Java and then C# came into the picture. So why is LINQ the writing on the wall for C#? Well I’m being slightly provocative here, it’s just a thought experiment, but doesn’t this look like a repeat of the whole C + OO = C++ thing. LINQ is giving us functional programming in C#, which isn’t a functional programming language, it’s an imperative language. There’s no doubt that the current trend in programming is away from imperative and towards a more declarative style. SQL’s been doing this it for decades in it’s own tightly bounded domain and now we’re beginning to see general purpose declarative languages becoming more widely known. The problem with grafting some functional stuff on top of an imperative language is that the imperative stuff stops you from getting all the benefits of the functional stuff in the same way that the C in C++ stopped C++ from giving the programmer all the benefits if a language with first class types. Is C#+LINQ another intermediate Frankenstein's monster like C++? When we've all had plenty of practice with LINQ are we going to start thinking that we don't need all that imperative syntax? In ten years time will I be writing, “nobody would dream of writing a business application with an imperative language!”?

REST is the Betamax of Web Services

I while ago I blogged about REST. I was pretty excited about it at the time, it seemed to blow away a lot of the lava layers that have been accumulating around the SOAP protocol with a back to basics, or rather back to HTML approach that leverages a lot of the forgotten functionality of that protocol. OK, so should I be using it when my customers ask me to implement a Service Oriented Architecture? I think the answer has to be no. SOAP has gained far too much market share in the developer tools community. Especially in the Microsoft world, I’d be throwing away all the SOAP support built into the MS toolset and the .NET framework in particular that makes implementing web services with SOAP so easy. Also I’d be loosing any interoperability with other platforms, because with a RESTfull approach you’re basically designing your own protocol. It’s a shame, but sometimes you just have to go with the flow. Of course the world of IT (actually the world generally) is full Betamaxes. If the best tools always won, we’d be programming in Haskell, on BeOS running on a RISC machine.

Friday, March 23, 2007

Building a Visual Studio Custom Project Type

Recently I've been looking at Visual Studio Extensibility and specifically how to create my own custom project types. There are three fundamental ways of extending VS:

  • Macros. This is the easiest way to automate VS and can be as simple as recording a few key strokes or as complex as coding (in VB ugh!) entire extensibility applications against the VS automation model (known as EnvDTE).
  • AddIns. These allow deeper integration with VS and allow you to code in C# and ship your automation code as a compiled assembly.
  • Packages. This is the deepest level of integration and is how Microsoft creates features in VS, including all the languages.

Now there's a common misconception among .net developers (myself included, before I started looking into this) that Visual Studio is itself writen in .net, it's not, it's a evolution of the earlier C++ IDE and is fundamentally a COM framework with plugable COM components called 'Packages'. Apparently many of the VS packages are now implemented in .net, but they still have to deal with the VS COM framework via interop. The only way to do what I want and create a new project type is to use VS Packages. Unfortunately this means dealing with VS in the raw and coding against the gnarly old VS COM framework. Microsoft provide interop classes and a managed framework, the Managed Package Framework (MPF) to help with this, but it's not very well documented and hardly a walk in the park. OK, so after much fustration and effort I've managed to create the most basic non-functional custom project type. It doesn't do anything, but you can select it in the New Project dialogue box and create an instance of it that you can see in solution explorer. The steps below show how I did it. I'm not sure if it's the right way, but it works. First you'll need to download the Visual Studio SDK, currently at version 4. Once that's installed, you can use the new package wizard to create a basic package: 1. Open Visual Studio 2. Select File->New->Project, the new project dialog appears 3. Select Other Project Types->Extensibility->Visual Studio Integration Package 4. Give your package a name 5. Click OK, The Visual Studio Integration Package Wizard Appears, click Next 6. Select C# and Generate a new key file to sign the assembly, click Next 7. Enter some information about your package, this will appear in the VS about dialog, click Next 8. Leave all the following checkboxes blank, click Finish. The wizard does it's stuff and now you've got a working Visual Studio package (not project that comes later!), you can build it and run it by hitting F5. A new instance of VS appears and if you click on Help->About Microsoft Visual Studio, you can see that your package has been registered and appears under 'installed products' with the information you entered in the Wizard. The instance of VS that runs when you hit F5 uses what's known as the 'Experimental Hive' which is simply a copy of the VS registry settings that's created for developing and debugging extensions. You can reset it at any time by using the VsRegEx.exe tool. The part of the MPF that deals with creating new project types isn't provided as part of the MPF assemblies, but as source code, called 'ProjectBase', that you are supposed to include in your project. On my machine, the VS SDK installed it here:

C:\Program Files\Visual Studio 2005 SDK\2007.02\VisualStudioIntegration\Common\Source\CSharp\Project

To include it you have to edit your package's csproj file to include the following nodes:

<!-- This imports the files which makes up the project base classes -->
<PropertyGroup>
<ProjectBasePath>C:\Program Files\Visual Studio 2005 SDK\2007.02\VisualStudioIntegration\Common\Source\CSharp\Project</ProjectBasePath>
</PropertyGroup>
<Import Project="$(ProjectBasePath)\ProjectBase.Files" />

Once you reload the project you'll see that a folder called project base has been added. If you now try and build the project, you'll get a whole load of errors. To fix them, add the following references: EnvDTE Microsoft.VisualStudio.Designer.Interfaces and add the following lines to your AssemblyInfo.cs file:

using System;
...
[assembly: CLSCompliant(false)]

Now you should be able to build the project with the ProjectBase files included. In order for Visual Studio to create an instance of your project you have to create a 'ProjectFactory', you do this by extending the ProjectFactory class in ProjectBase (Microsoft.VisualStudio.Package). Here's my ProjectFactory. Note it must be given a fixed Guid to allow the class to be identified by COM:

using System;
using System.Runtime.InteropServices;
using Microsoft.VisualStudio.Package;
using Microsoft.VisualStudio.Shell;
using IOleServiceProvider = Microsoft.VisualStudio.OLE.Interop.IServiceProvider;

namespace Company.VsPackage1
{
    [Guid("AC959CD1-5A2C-4306-BE72-259804B01F08")]
    public class MyProjectFactory : ProjectFactory
    {
        public MyProjectFactory(Package package)
            : base(package)
        {
        }

        protected override ProjectNode CreateProject()
        {
            // TODO: create project here
            return null;
        }
    }
}

MyProjectFactory now has to be registered by the package. The MPF uses attributes on the package class to create the required registry entries so you need to add the following attribute to the package class created by the wizard. In my case it's called 'VsPackage1' in the file 'VsPkg.cs'. Here you should define the name of your project type and your project file's file extension:

[ProvideProjectFactory(typeof(MyProjectFactory),
"My Project",
"MyProject Files (*.myproj);*.myproj",
"myproj",
"myproj",
@"..\..\Templates\Projects")]

You also have to get the Package to register the project factory type when it initialises by calling RegisterProjectFactory and passing in a new instance of MyProjectFactory that takes a pointer to the package in its constructor:

protected override void Initialize()
{
    Trace.WriteLine (string.Format(CultureInfo.CurrentCulture, "Entering Initialize() of: {0}", this.ToString()));
    this.RegisterProjectFactory(new MyProjectFactory(this));
    base.Initialize();

}

Also at this stage we should change our Pakage to inherit from ProjectPackage (in Microsoft.VisualStudio.Package) rather than Package:

public sealed class VsPackage1 : ProjectPackage

If you hit F5 now to run your package you see that when you go to File->New->Project you'll see 'My Project' as one of the project types available. If you click on 'My Project' there are no projects to select. All the available project types are defined as templates, so the next step is to define a template file. In the ProvideProjectFactory attribute above the last parameter is the path to where VS should look to find your project templates, so create a new folder under your project called 'Templates' and under that another folder called 'Projects'. In that folder create a file called 'My Project.myproj'. Note that the extension should match the extension defined above in the ProvideProjectFactory attribute. Paste this minimal project template into the file:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <Name>"My Project"</Name>
    <SchemaVersion>2.0</SchemaVersion>
    <ProjectGuid>{AC959CD1-5A2C-4306-BE72-259804B01F08}</ProjectGuid>
  </PropertyGroup>
</Project>

Note that the ProjectGuid property should be the same as the MyProjectFactory class' Guid. Now when you hit F5 and then File->New->Project and navigate to 'My Project' you will see the 'My Project' template. If you select it and hit 'OK', you'll get a nasty error and VS will crash out. The next task is do define the project class and get the project factory to create it. Create a class called 'MyProject' and paste in the following code:

using System;
using System.Runtime.InteropServices;
using Microsoft.VisualStudio.Package;

namespace Company.VsPackage1
{
    [CLSCompliant(false)]
    [ComVisible(true)]
    [Guid("7B61EA22-333F-4b4e-A000-FB847CDD4A99")]
    public class MyProject : ProjectNode
    {
        public override Guid ProjectGuid
        {
            get { return typeof(MyProjectFactory).GUID; }
        }

        public override string ProjectType
        {
            get { return this.GetType().Name; }
        }
    }
}

And return a new instance of the project class from the project factory's CreateProject method. We also have to set the site of the project by providing the package's service provider (see my post on service providers for more on this):

protected override ProjectNode CreateProject()
{
    MyProject project = new MyProject();
    project.SetSite((IOleServiceProvider)
        (((IServiceProvider)this.Package).GetService(typeof(IOleServiceProvider)))
        );
    return project;
}

Now when you hit F5 and create a new 'My Project' project it works! You can see a new instance of your project in Solution Explorer under a new solution. You can right click on the solution select Add->New Project and add another instance of your project type to the solution. It took me ages (several days) to work out these few steps to creating my own custom project type. All the walkthroughs I could find only covered building simple single menu command, tool window or editor packages and none covered using the ProjectBase classes. In the end I had to work it out from the samples (mainly the NestedProject and IronPython samples) and a few hints and tips from blogs and the Visual Studio Extensibility forum. I'm expecting a similarly difficult and steep learning curve as I try to add functionality to my project. Getting to grips with buiding packages is also confused by having lots of samples and instructions based on the COM interfaces. I guess you've got to understand that architecture in order to effectively build VS solutions, but when I came to use the MPF I didn't really know how to translate what I'd learnt about the VS architecture into how to code against the MPF. I guess it's good that the VS extensibility team have taken an agile approach and are releasing stuff as they put it together without waiting for it to be fully documented. I'd much rather have the MPF without much documentation than have to code against the raw interop classes and the release cycle is pretty short so I imagine we'll see version 5 of the MPF pretty soon. Just to wrap up I have to mention Carlos Quintero's MZ-Tools site. It's a goldmine of information especially the resources page. Watching the Videos, Webcasts and Screeencasts he lists there is probably the easiest way to get up to speed with VS Packages. I've also recently read Inside Visual Studio .net by Johnson, Skibo and Young which, although it doesn't cover VS Packages is a goldmine of information on the DTE automation API and stuff to do with AddIns. The current version is also avaiable as a free PDF download when you register Visual Studio.

Friday, March 02, 2007

Writing a raw web service using an HttpHandler

The ASP.NET web service infrastructure provides a RPC style abstraction for web serivces. You write a web method that looks like any other C# function and simply add a [WebMethod] attribute. ASP.NET then reflects on the name of the method, your input parameters and return types and creates a web service end point for you. If you use the 'Add Web Reference' facility for the client, you never have to consider what the web service will look like in terms of WSDL or XML, it just looks like a remote procedure call. This is fine in many scenarios, but sometimes what you are interested in is the actual XML message itself and you don't need it deserialized as .net types. It's pretty straightforward to write a web service using an HttpHandler to grab the raw SOAP envelope. The IHttpHandler interface is the core endpoint interface for anything that handles a web request in .net. All you need to do is write a class that implements the interface and then register it in Web.config:
<httpHandlers>
<add verb="*" path="MyServiceEndpoint.asmx" type="MyService.MyHandler, MyHandler" />
</httpHandlers>
IHttpHandler provides a method 'PrcessRequest' that looks like this:
void ProcessRequest(HttpContext context);
You take the HttpContext.Request.InputStream and read the raw SOAP envelope from it. When you're ready to return the response, simply write your SOAP envelope response to the HttpContext.Response.OutputStream. A couple of other things to consider are providing a default information page and your web service's WSDL file. In ASP.NET if you execute an HTTP GET request against your web service, ASP.NET builds a nice web page naming your service and listing the operations. If you append WSDL to the querystring, it builds and returns the WSDL for the web service. You can replicate this behaviour by looking for a GET request (web service requests are typically POST) and then streaming either a descriptive web page or a WSDL document to the response. Here are functions that look for a GET request and stream either the WSDL file or the default page file into the response.
public void ProcessRequest(HttpContext context)
{
switch(context.Request.HttpMethod)
{
 case "GET":
  ProcessGetRequest(context);
  break;
 case "POST":
  ProcessPostRequest(context);
  break;
 default:
  throw new ApplicationException(
   string.Format("HTTP Method '{0}' is not supported.", context.Request.HttpMethod));
}
}

void ProcessGetRequest(HttpContext context)
{
// is this a request for the WSDL file?
string filePath;
if(context.Request.QueryString.ToString().ToUpper() == "WSDL")
{
 filePath = context.Server.MapPath(myWsdlPath);
}
else
{
 filePath = context.Server.MapPath(myDefaultPagePath);
}
RespondWithFile(context, filePath);
}

void RespondWithFile(HttpContext context, string filePath)
{
using(FileStream fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read))
{
 byte[] buffer = new byte[streamBufferSize];
 int bytes;
 while((bytes = fileStream.Read(buffer, 0, streamBufferSize)) > 0)
 {
  context.Response.OutputStream.Write(buffer, 0, bytes);
 }
 context.Response.OutputStream.Flush();
}
}
This technique can be usefull when you're writing stuff to route or map web services. Currently I'm working on a mapping service that takes a request and forwards it on to another web service with a different schema. Using the handler I can simply grab the SOAP envelope and run it through an XSLT to transform it to target schema and then run the response through another XSLT to transform it back to the originating web service's response schema.

Sunday, February 11, 2007

Wes Dyer talking about LINQ

More browsing today on MSDN lead me to an interview with Wes Dyer taking about LINQ. He's one of the guys on the C# compiler team. A very bright chap indeed and he gives one of the best explanations of how LINQ works that I've yet seen. The white board session especially in the second half of the video is a fantastic mind expanding ride and shows how LINQ is far far more than just queries. Typically I think Microsoft's marketing have probably muddied the waters again by calling it 'language integrated query' because, although querying is something you can do with it, it's much more like 'functional programming in C#'. For example, Wes showed how you can replace typical looping patterns with LINQ 'queries'. Take this little code snippet to get the even numbers from an array of numbers:
int[] list = int[] {1,2,3,4,5,6,7};
List<int> evenNumbers = new List<int>();
foreach(int n in list)
{
if(n % 2 == 0)
 evenNumbers.Add(n);
}
You can write the same thing with LINQ:
int[] list = int[] {1,2,3,4,5,6,7};
var evenNumbers =
from n in list
where n % 2 == 0
select n;
I think it's much easier to read the intention of the LINQ version. One thing that really struck me was how they use the custom iterators that I talked about in my last post to create the 'where' and 'select' clauses in LINQ. A very bright light bulb went off in my head at that point! Wes Dyer's blog is also well worth checking out.

Friday, February 09, 2007

Creating set based filters with custom iterators

Well it's back to work after a very nice two week break. I wrapped up my last contract with Ace Insurance in Crawley (hi guys!) three weeks ago, and I've been at my new assignment for a week now. I've been very busy getting up to speed with my new role (doing yet more application integration), but I did have time to skim MSDN where I came across this great article by Bill Wagner, titled 'Custom Iterators'. He talks about custom iterators, predicates and generics, all stuff that I'm quite familiar with, but it's the way he puts them together to create aggregate set functions, just like the ones that I've been discovering with functional languages like F#, that makes it really interesting. Custom iterators are new in C# 2.0 and make it really easy to create an instance of IEnumerable. Instead of implementing the entire interface, you just have to write a method that returns IEnumerable and has a yield statement for each item you want your IEnumerable to return. Behind the scenes the C# compiler creates an instance of IEnumerable for you by turning your method into a class with fields for all your local variables. For example, if you define an iterator that returns the numbers one to ten:
IEnumerable<int> OneToTen()
{
   for(int i=1; i<=10; i++)
   {
       yield return i;
   }
}
It will return an object of a class that you can imagine has a private field i and implements the IEnumerable.GetEnumerator() interface. You can then use the IEnumerator that it returns just like any other IEnumerator, for example:
public Test()
{
   IEnumerator<int> oneToTen = OneToTen().GetEnumerator();
   oneToTen.MoveNext();
   Console.WriteLine("{0}", oneToTen.Current);
   oneToTen.MoveNext();
   Console.WriteLine("{0}", oneToTen.Current);
}
Will output:
1
2
Now, the trick that Bill Wagner demonstrates, that hadn't occurred to me before is that you can also pass an instance of IEnumerable to your customer iterator to make a filters, transformers or any other set based operation you can think of. Here's a simple filter that only passes even numbers:
IEnumerable<int> Even(IEnumerable<int> list)
{
   foreach(int number in list)
   {
       if(number % 2 == 0)
           yield return number;
   }
}

public Test()
{
   foreach(int number in Even(OneToTen())
   {
       Console.WriteLine("{0}", number);
   }
}
Will output:
2
4
6
8
10
Bill's next trick to use anonymous delegates and generics to write a generic filter:
delegate bool Test<T>(T value);
IEnumerable<T> Filter<T>(IEnumerable<T> list, Test<T> test)
{
   foreach(T item in list)
   {
       if(test(item))
           yield return item;
   }
}

public Test()
{
   IEnumerable<int> evenNumbers = Filter<int>(OneToTen(), delegate(int number)
   {
       return number % 2 == 0;
   });
   foreach(int number in evenNumbers)
   {
       Console.WriteLine("{0}", number);
   }
}
Will output:
2
4
6
8
10
Very neat, but a bit long winded syntax wise I think. This will all change with C# 3.0's lambda expressions. It's yet another step towards a more functional programming style that's such a powerful addition to the programmer's toolbox.

Wednesday, January 17, 2007

Writing your own XSD.exe

If you spend any time working with Web Services or even just XML, you'll inevitably come into contact with XSD.exe and WSDL.exe, they both generate .net code from XSD type definitions. With XSD.exe, you simply give it the path to an xsd document and it will spit out a .cs file. That file defines types that will serialize to an XML document instance that validates against the xsd. De-serializing your XML to a strongly typed object model is almost always better than fiddling with the XML DOM, but what if you don't like the code that XSD.exe generates? Well, you can easily spin your own XSD.exe since it simply uses the public framework types System.Xml.Serialization.XmlSchemaImporter, System.Xml.Serialization.XmlCodeExporter and CodeDom. For some reason the MSDN documentation on these classes says, "This class supports the .NET Framework infrastructure and is not intended to be used directly from your code.", but don't let that put you off, they're public types and work fine. At a high level the process goes like this, you can follow it with the code sample below:
  1. Load your xsd file into an XmlSchema.
  2. Create an XmlSchemaImporter instance that references your schema. This class is used to generate mappings from XSD types to .net types.
  3. Create a CodeDom CodeNamespace instance where you'll build the syntactic structure of your .net types.
  4. Create an XmlCodeExporter instance with a reference to the CodeNamespace that you use to export your type. This is the class that actually creates the syntactic structure of the .net types in the CodeNamespace.
  5. Create an XmlTypeMapping instance for each type that you wish to export from the XSD.
  6. Call the ExportTypeMapping method on XmlCodeExporter for each XmlTypeMapping object, this creates the types syntax in the CodeNamespace object.
  7. Use a CSharpCodeProvider to output C# source code for the types that were created in CodeNamespace object.
Once the CodeNamespace has been fully populated (after step 6 above) there's an opportunity to make any changes that we wish to the code we output. Note that at this stage, the CodeDom CodeNamespace object represents an IL syntactic structure rather than code in a particular language. We could just as easily generate VB.NET at this point. We can use the CodeDom methods to alter that structure before outputting source code. In the example below I run the RemoveAttributes function to remove some attributes from the type definition.
using System;
using System.IO;
using System.Collections.Generic;
using System.Reflection;
using System.Text;
using System.Xml;
using System.Xml.Serialization;
using System.Xml.Schema;
using System.CodeDom;
using System.CodeDom.Compiler;

using Microsoft.CSharp;

using NUnit.Framework;

namespace XmlSchemaImporterTest
{
  [TestFixture]
  public class XsdToClassTests
  {
      // Test for XmlSchemaImporter
      [Test]
      public void XsdToClassTest()
      {
          // identify the path to the xsd
          string xsdFileName = "Account.xsd";
          string path = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
          string xsdPath = Path.Combine(path, xsdFileName);

          // load the xsd
          XmlSchema xsd;
          using(FileStream stream = new FileStream(xsdPath, FileMode.Open, FileAccess.Read))
          {
              xsd = XmlSchema.Read(stream, null);
          }
          Console.WriteLine("xsd.IsCompiled {0}", xsd.IsCompiled);

          XmlSchemas xsds = new XmlSchemas();
          xsds.Add(xsd);
          xsds.Compile(null, true);
          XmlSchemaImporter schemaImporter = new XmlSchemaImporter(xsds);

          // create the codedom
          CodeNamespace codeNamespace = new CodeNamespace("Generated");
          XmlCodeExporter codeExporter = new XmlCodeExporter(codeNamespace);

          List maps = new List();
          foreach(XmlSchemaType schemaType in xsd.SchemaTypes.Values)
          {
              maps.Add(schemaImporter.ImportSchemaType(schemaType.QualifiedName));
          }
          foreach(XmlSchemaElement schemaElement in xsd.Elements.Values)
          {
              maps.Add(schemaImporter.ImportTypeMapping(schemaElement.QualifiedName));
          }
          foreach(XmlTypeMapping map in maps)
          {
              codeExporter.ExportTypeMapping(map);
          }

          RemoveAttributes(codeNamespace);

          // Check for invalid characters in identifiers
          CodeGenerator.ValidateIdentifiers(codeNamespace);

          // output the C# code
          CSharpCodeProvider codeProvider = new CSharpCodeProvider();

          using(StringWriter writer = new StringWriter())
          {
              codeProvider.GenerateCodeFromNamespace(codeNamespace, writer, new CodeGeneratorOptions());
              Console.WriteLine(writer.GetStringBuilder().ToString());
          }

          Console.ReadLine();
      }

      // Remove all the attributes from each type in the CodeNamespace, except
      // System.Xml.Serialization.XmlTypeAttribute
      private void RemoveAttributes(CodeNamespace codeNamespace)
      {
          foreach(CodeTypeDeclaration codeType in codeNamespace.Types)
          {
              CodeAttributeDeclaration xmlTypeAttribute = null;
              foreach(CodeAttributeDeclaration codeAttribute in codeType.CustomAttributes)
              {
                  Console.WriteLine(codeAttribute.Name);
                  if(codeAttribute.Name == "System.Xml.Serialization.XmlTypeAttribute")
                  {
                      xmlTypeAttribute = codeAttribute;
                  }
              }
              codeType.CustomAttributes.Clear();
              if(xmlTypeAttribute != null)
              {
                  codeType.CustomAttributes.Add(xmlTypeAttribute);
              }
          }
      }
  }
}

Orthogonal Code

As Steve McConnel says in Code Complete, Managing complexity is the fundamental imperative of any software developer, but how do you do that? Well Code Complete spends several hundred pages outlining some techniques and you should also read Bob Martin's Agile Software Development and Martin Fowler's Refactoring. But if you want something that you can digest in your lunch hour, I recently discovered Jeremy Miller's nice series of posts on writing maintainable code, but the one which really struck a chord with me was this one, orthogonal code. Learning how to write code without digging myself into a pit of tangled complexity has been a constant thread in my development as a programmer and he succinctly captures some of the main techniques that I've learnt over the last ten years to avoid doing just that...

Any here's a great quote:

Don't write this post off as just academic hot air, this is about consistently leaving the office on time and protecting your company's code assets because you'll be much more likely to smoothly sustain a forward progress with your existing code.

Thursday, January 04, 2007

Easy impersonation

I've been quite pleasantly surprised recently how easy it is to do impersonation in .net. The trick is a win32 api function that's not covered by the BCL, LogonUser

[System.Runtime.InteropServices.DllImport("advapi32.dll")]
public static extern int LogonUser(
    String lpszUsername, 
    String lpszDomain, 
    String lpszPassword, 
    int dwLogonType, 
    int dwLogonProvider, 
    out  IntPtr phToken);

LogonUser does what it says and logs the given user onto the computer the code is running on. It takes a user name, domain name and a clear text password as well as two constants, a logon type which allows you to specify an interactive user, network user etc and a logon provider (I just used default). It returns a pointer to a token which represents the user and which you can use to create a new WindowsIdentity instance. Once you've got a WindowsIdentity, you can just call the Impersonate method to switch your code execution context to the new identity. Here's a little NUnit test to demonstrate:

[NUnit.Framework.Test]
public void ImpersonationSpike()
{
    string username = "imptest";
    string domain = "mikesMachine"; // this is the machine name
    string password = "imptest";
    IntPtr userToken;

    int hresult = LogonUser(
        username, 
        domain, 
        password, 
        (uint)LogonSessionType.Network, 
        (uint)LogonProvider.Default, 
        out userToken);

    if(hresult == 0)
    {
        int error = System.Runtime.InteropServices.Marshal.GetLastWin32Error();
        Assert.Fail("Error occured: {0}", error);
    }
    WindowsIdentity identity = new WindowsIdentity(userToken);
    Console.WriteLine(identity.Name);

    Console.WriteLine("I am: '{0}'", WindowsIdentity.GetCurrent().Name);
    System.IO.File.WriteAllText(@"c:\Program Files\hello.txt", "Hello");

    WindowsImpersonationContext context = null;
    try
    {
        context = identity.Impersonate();
        Console.WriteLine("Impersonating: '{0}'", WindowsIdentity.GetCurrent().Name);
    }
    finally
    {
        if(context != null)
        {
            context.Undo();
        }
        if(userToken != IntPtr.Zero)
        {
            CloseHandle(userToken);
        }
    }
    Console.WriteLine("I am: '{0}'", WindowsIdentity.GetCurrent().Name);
}

[System.Runtime.InteropServices.DllImport("advapi32.dll")]
public static extern int LogonUser(
    String lpszUsername, 
    String lpszDomain, 
    String lpszPassword, 
    uint dwLogonType, 
    uint dwLogonProvider, 
    out  IntPtr phToken);

[System.Runtime.InteropServices.DllImport("kernel32.dll", SetLastError = true)]
static extern bool CloseHandle(IntPtr handle);

enum LogonSessionType : uint
{
    Interactive = 2,
    Network,
    Batch,
    Service,
    NetworkCleartext = 8,
    NewCredentials
}

enum LogonProvider : uint
{
    Default = 0, // default for platform (use this!)
    WinNT35,     // sends smoke signals to authority
    WinNT40,     // uses NTLM
    WinNT50      // negotiates Kerb or NTLM
}

It should output the following...

mikesMachine\imptest
I am: 'mikesMachine\mike'
Impersonating: 'mikesMachine\imptest'
I am: 'mikesMachine\mike'

1 passed, 0 failed, 0 skipped, took 1.27 seconds.

A few things to note, you have to have pInvoke permission, which might be problem in some web hosting environments. Also, you have to have a cleartext password in order to log on the user you're impersonating, so your user has to supply it, or you have to store it somewhere which is an obvious security risk.

Of course, if you want to impersonate a particular account in ASP.NET you can just use the <identity impersonate="true" username="theUserName" password="ThePassword" />. On Server 2003, you can even more simply just set your application to run in a custom application pool and set the identity in the application pool settings. All this stuff is explained in this msdn article. But this technique here is great if you just want to grab an identity for a single method call or if you need to do impersonation in something other than an ASP.NET application.