Tuesday, September 13, 2011

Why Write a .NET API For RabbitMQ?

Anyone who reads this blog knows that my current focus is writing a .NET API for RabbitMQ which I’ve named EasyNetQ. This is being paid for by my excellent clients 15Below who build high volume messaging solutions for the airline industry. EasyNetQ is a core part of their strategy moving forwards, so it’s going to get plenty of real-world use.
One question I’ve been asked quite a lot, is ‘why?’. Why am I building a .NET API when one is already available; the C# AMQP client from RabbitHQ? Think of AMQP as the HTTP of messaging. It’s a relatively low-level protocol. You typically wouldn’t build a web application directly against a low-level HTTP API such as System.Net.WebRequest, instead you would use a higher level toolkit such as WCF or ASP.NET MVC. Think of EasyNetQ as the ASP.NET MVC of AMQP.
AMQP is designed to be cross platform and language agnostic. It is also designed to flexibly support a wide range of messaging patterns based on the Exchange/Binding/Queue model. It’s great having this flexibility, but with flexibility comes complexity. It means that you will need to write a significant amount of code in order to implement a RabbitMQ client. Typically this code would include:
  • Implementing messaging patterns such as Publish/Subscribe or Request/Response. Although, to be fair, the .NET client does provide some support here.
  • Implement a routing strategy. How will you design your exchange-queue bindings, and how will you route messages between producers and consumers?
  • Implement message serialization/deserialization. How will you convert the binary representation of messages in AMQP to something your programming language understands?
  • Implement a consumer thread for subscriptions. You will need to have a dedicated consumer loop waiting for messages you have subscribed to. How will you deal with multiple subscribers, or transient subscribers, like those waiting for responses from a request?
  • Implement a versioning strategy for your messages. What happens when your message schema needs to change in response to business requirements?
  • Implement subscriber reconnection. If the connection is disrupted or the RabbitMQ server bounces, how do you detect it and make sure all your subscriptions are rebuilt?
  • Understand and implement quality of service settings. What settings do you need to make to ensure that you have a reliable client.
  • Implement an error handling strategy. What should your client do if it receives a malformed message, or if an unexpected exception is thrown?
  • Implement monitoring tools. How will you monitor your client applications so that you are alerted if there are any problems?
With EasyNetQ, you get all these out-of-the-box. You loose some of the flexibility in exchange for a model based on .NET types for routing, but it saves an awful lot of code.


Mick Delaney said...

What will differeniate your api from say masstransit or nservicebus???

Mike Hadlow said...

Hi Mick,
First let me say how awesome I think MassTransit and NServiceBus are. They are the main influences on the design of EasyNetQ.

The differentiation with NServiceBus is easy to answer: NSB uses MSMQ as its transport, EasyNetQ is built from the ground up as a layer over AMQP. I gather that the NSB transport is pluggable, but the abstraction is built for MSMQ and doesn't have any of the great things that AMQP gives you out-of-the-box.

MassTransit has an AMQP backend, but again it was primarily built for MSMQ (and Tibco?). EasyNetQ is entirely focussed on building the best possible client for AMQP and rabbit with no plans to support any other back-end. This means I can leverage all the native goodness of AMQP and keep the EasyNetQ library as simple as possible. I have a huge amount of respect for Dru and Chris, so I expect they'll do an excellent job with MassTransit's AMQP back-end. In the end, the choice is up to the user.

At this stage, EasyNetQ is still alpha software, so I wouldn't recommend anyone using it in production unless you're willing to get actively involved with the development. But it's being built for a client with some pretty serious messaging requirements, so I expect it to be nicely battle hardened in a year or so. At which point, if you're in the market for a high-level API for RabbitMQ, it should definitely be a consideration.

Gareth said...

Hi Mike,

How are you attempting to prevent duplicate messages being processed, or is that something you leave to the application code?

Mike Hadlow said...

Hi Gareth,

At the moment there's no once-only guarantee in EasyNetQ. Indeed, there's no at-least-once guarantee either, you can loose a publish if the broker goes down in the middle of processing a message. So yes, right now you would have to put checks in your application code if you care about either of these cases.

Unknown said...

I'm not ready to get involved with EasyNetQ yet but I am eagerly watching your progress on it. I've been frustrated with MSMQ many times over the years and RabbitMQ has definitely caught my attention.

Props to 15Below for letting you develop it out in the open!

Anonymous said...

I think this is an excellent example of how you can simplify an API by giving up some flexibility in exchange for simplicity. I recently wrote a blog post on this topic:


Can you give me any feedback based on your own experience?

Anonymous said...

Have you seen greg youngs lightweight one ( pvc -> http://nuget.org/List/Packages/pvc-rabbmitmq )

Jake Scott said...

Hi Mike,

Does AMQP, or the dotnet client for RabbitMQ have support for File Streams?

I am new to messaging and am wondering if I could use them to transfer image files around 4-5MB in size.

Would that be a bad idea?


Mike Hadlow said...

Hi Jake,

This from the RabbitMQ FAQ: "AMQP 0-8 included mention of a "stream" transfer class. However, no one implements it and it's gone in AMQP 0-9-1. It is the responsibility of applications to chunk large messages if they need to."

EasyNetQ doesn't have a 'chunking' API, so the answer to your question is 'no'. Would it be a good idea? I would probably consider holding the document at some location and simply passing a pointer to it in the message.

Jake Scott said...

Hi Mike,

Yeah basically I want to handle image upload, resizing and uploading then to amazon s3. I dont want to do that on my API web servers as that will block them for to long. So I think what I will do is stream the file into a database like MongoDB GridFS (using the new .net 4.5 Request streaming stuff in asp.net which is async :) and then put a message on the queue. Then I have a windows service that processes the queue and does the image resizing and uploading to S3.
The I will delete the original file out of MongoDB

So yeah probably not worth doing chunking :)


Sahil Singh said...

if i make web based wcf then what is localhost,i used my website domain that already virtually directory.but not wcf work .please any suggest me how to make end point in my website.
please Missed call at 9911425805

khanz said...

can i make an sms service using rabbit mq for example i write application of sms based chatroom in .net but the speed of sms will be handeled with rabbit mq. connexted to my gsm modem?

Anonymous said...

Hi Mike,

Have you looked at LMAX Distruptor pattern before? It is highly efficient and removes the need, in many cases, the use of queues, which they prove to be slow.