Wednesday, May 04, 2011

EasyNetQ, a simple .NET API for RabbitMQ

After pondering the results of our message queue shootout, we decided to run with Rabbit MQ. Rabbit ticks all of the boxes, it’s supported (by Spring Source and then VMware ultimately), scales and has the features and performance we need. The RabbitMQ.Client provided by Spring Source is a thin wrapper that quite faithfully exposes the AMQP protocol, so it expects messages as byte arrays.

For the shootout tests spraying byte arrays around was fine, but in the real world, we want our messages to be .NET types. I also wanted to provide developers with a very simple API that abstracted away the Exchange/Binding/Queue model of AMQP and instead provides a simple publish/subscribe and request/response model. My inspiration was the excellent work done by Dru Sellers and Chris Patterson with MassTransit (the new V2.0 beta is just out).

The code is on GitHub here:

https://github.com/mikehadlow/EasyNetQ

The API centres around an IBus interface that looks like this:

/// <summary>
/// Provides a simple Publish/Subscribe and Request/Response API for a message bus.
/// </summary>
public interface IBus : IDisposable
{
/// <summary>
/// Publishes a message.
/// </summary>
/// <typeparam name="T">The message type</typeparam>
/// <param name="message">The message to publish</param>
void Publish<T>(T message);

/// <summary>
/// Subscribes to a stream of messages that match a .NET type.
/// </summary>
/// <typeparam name="T">The type to subscribe to</typeparam>
/// <param name="subscriptionId">
/// A unique identifier for the subscription. Two subscriptions with the same subscriptionId
/// and type will get messages delivered in turn. This is useful if you want multiple subscribers
/// to load balance a subscription in a round-robin fashion.
/// </param>
/// <param name="onMessage">
/// The action to run when a message arrives.
/// </param>
void Subscribe<T>(string subscriptionId, Action<T> onMessage);

/// <summary>
/// Makes an RPC style asynchronous request.
/// </summary>
/// <typeparam name="TRequest">The request type.</typeparam>
/// <typeparam name="TResponse">The response type.</typeparam>
/// <param name="request">The request message.</param>
/// <param name="onResponse">The action to run when the response is received.</param>
void Request<TRequest, TResponse>(TRequest request, Action<TResponse> onResponse);

/// <summary>
/// Responds to an RPC request.
/// </summary>
/// <typeparam name="TRequest">The request type.</typeparam>
/// <typeparam name="TResponse">The response type.</typeparam>
/// <param name="responder">
/// A function to run when the request is received. It should return the response.
/// </param>
void Respond<TRequest, TResponse>(Func<TRequest, TResponse> responder);
}

To create a bus, just use a RabbitHutch, sorry I couldn’t resist it :)

var bus = RabbitHutch.CreateRabbitBus("localhost");

You can just pass in the name of the server to use the default Rabbit virtual host ‘/’, or you can specify a named virtual host like this:

var bus = RabbitHutch.CreateRabbitBus("localhost/myVirtualHost");

The first messaging pattern I wanted to support was publish/subscribe. Once you’ve got a bus instance, you can publish a message like this:

var message = new MyMessage {Text = "Hello!"};
bus.Publish(message);

This publishes the message to an exchange named by the message type.

You subscribe to a message like this:

bus.Subscribe<MyMessage>("test", message => Console.WriteLine(message.Text));

This creates a queue named ‘test_<message type>’ and binds it to the message type’s exchange. When a message is received it is passed to the Action<T> delegate.  If there are more than one subscribers to the same message type named ‘test’, Rabbit will hand out the messages in a round-robin fashion, so you get simple load balancing out of the box. Subscribers to the same message type, but with different names will each get a copy of the message, as you’d expect.

The second messaging pattern is an asynchronous RPC. You can call a remote service like this:

var request = new TestRequestMessage {Text = "Hello from the client! "};

bus.Request<TestRequestMessage, TestResponseMessage>(request, response =>
Console.WriteLine("Got response: '{0}'", response.Text));

This first creates a new temporary queue for the TestResponseMessage. It then publishes the TestRequestMessage with a return address to the temporary queue. When the TestResponseMessage is received, it passes it to the Action<T> delegate. RabbitMQ happily creates temporary queues and provides a return address header, so this was very easy to implement.

To write an RPC server. Simple use the Respond method like this:

bus.Respond<TestRequestMessage, TestResponseMessage>(request => 
new TestResponseMessage { Text = request.Text + " all done!" });

This creates a subscription for the TestRequestMessage. When a message is received, the Func<TRequest, TResponse> delegate is passed the request and returns the response. The response message is then published to the temporary client queue.

Once again, scaling RPC servers is simply a question of running up new instances. Rabbit will automatically distribute messages to them.

The features of AMQP (and Rabbit) make creating this kind of API a breeze. Check it out and let me know what you think.

16 comments:

  1. Wow, this looks really good! I've wanted to check out RabbitMQ for a while, and an API like this would make using it a breeze.

    Is there any kind of transactional semantic around the callbacks?

    I don't know too much about RabbitMQ, or AMQP for that matter, but I'm guessing it uses the pattern where a received message is gone from the queue for a while, until the recipient ACKs...?

    ReplyDelete
  2. I'll need messaging in a month or 2 for my startup.
    was wondering are you deploying it on windows or linux???
    i'm on ec2 so was considering just using SQS.

    ReplyDelete
  3. Hi mookid,

    It's very early days. Released to get feedback. There's no error handling or transaction support yet.

    You are right, AMQP has the concept of an ACK, so you consume your message and then ACK, or NoACK if you don't want it to be removed from the queue.

    Hi Mick,

    We're deploying on Windows, but there's no reason why you couldn't use EasyNetQ against a Linux hosted instance of Rabbit. I have tried EasyNetQ on Mono yet, so I don't know if that will work.

    ReplyDelete
  4. Dave-O4:15 pm

    I love the simplicity of this API - I'd love to see it further developed for things like error handling and allowing use of routing keys etc

    (sorry, posted this under the wrong blog post before!)

    ReplyDelete
  5. Hi Dave-O,

    Yes, this is far from finished. Keep an eye on the github repository, you should see changes come along quite frequently.

    ReplyDelete
  6. Anonymous4:51 am

    Well done. Just starting to explore but already very impressed with your API.

    ReplyDelete
  7. This looks very nice.

    Have you considered using BasicProperties.Type && BasicProperties.ContentType for the type instead of embedding it in the queue name.

    ReplyDelete
  8. Hi Yavor,

    I'm using the .NET type as the routing key so that subscription can be by type.

    Do you know of any particular benefits to populating Type and ContentType? Are they simply information for the consumer post consumption?

    ReplyDelete
  9. I think they are only informational.
    I can't think of any specific benefit of using them. Just makes sense since they are there.

    In my scenario I am working on distributing different message types from a single queue, so that is probably why it is more appealing for me.

    ReplyDelete
  10. Anonymous10:27 am

    This API is awsome! Excellent work! I have a quick question: is it necessary to have console outputs at low level because I am not sure the bus can be embedded in a class library in that case?

    ReplyDelete
  11. Hi Anonymous,

    No, the console outputs are just for illustration. Have I left some in the EasyNetQ code? If so they were just for debugging, you can simply remove them.

    ReplyDelete
  12. Hello Mike,

    The asynchronous RPC part looks nice, but have you thought of a way to do a synchronous request/response?

    I've tried with something like:

    var hasResponded = false;

    bus.Request(request, response => {
    Console.WriteLine("Got response {1}: '{0}'", response.Text, i);
    hasResponded = true;
    });

    do {
    Console.WriteLine("Sleeping {0}", i);
    Thread.Sleep(5);
    } while (!hasResponded);


    But this will fail horribly due to the way the Request method caches the onResponse delegate.

    Removing the caching part makes it work as expected.

    ReplyDelete
  13. I've just tried the pub/sub in a simple example and loved it!!! Congratulations!

    I'm wondering if anyone have a story of using EasyNetQ in an enterprise environment... anyone?

    ReplyDelete
  14. Hi Francisco,

    EasyNetQ was written to provide the messaging infrastructure for 15below.com. We provide integration systems for many of the world's leading airlines, including Virgin, Ryanair, JetBlue and about 30 others. We typically see transaction volumes in the millions across complex business processes. RabbitMQ and EasyNetQ have worked pretty flawlessly in the year or so we've been using them. I'd say that's 90% down to RabbitMQ which has simply been an ultra-reliable black box, but there haven't been any major bugs with EasyNetQ either.

    I hope this helps,
    Mike

    ReplyDelete
  15. This looks great Mike! Can't wait to give it a whirl.

    ReplyDelete
  16. Well documented, help me a lot! Thanks for that!

    ReplyDelete

Note: only a member of this blog may post a comment.