Monday, September 23, 2013

RabbitMQ: AMQP Channel Best Practices

I’ve long been confused with best practices around AMQP channel handling. This post is my attempt to explain my current thoughts, partly in an attempt to elicit feedback.

The conversation between a message broker and a client is two-way. Both the client and the server can initiate ‘communication events’; the client can invoke a method on the server: ‘publish’ or ‘declare’, for example; and the server can invoke a method on the client such as ‘deliver’ or ‘reject’. Because the client needs to receive methods from the server, it needs to keep a connection open for its lifetime. This means that broker connections may last long periods; hours, days, or weeks. Maintaining these connections is expensive, both for the client and the server. In order to have many logical connections without the overhead of many physical TCP/IP connections, AMQP has the concept of ‘channel’ (confusingly represented by the IModel interface in the Java and .NET clients). You can create multiple channels on a single connection and it’s relatively cheap to create and dispose of them.

But what are the recommended rules for handling these channels? Should I create a new one for every operation? Or should I just keep a single one around and execute every method on it?

First some hard and fast rules (note I’m using the RabbitMQ .NET client for reference here, other clients might have different behaviour):

Channels are not thread safe. You should create and use a channel on a single thread. From the .NET client documentation (section 2.10):

“In general, IModel instances should not be used by more than one thread simultaneously: application code should maintain a clear notion of thread ownership for IModel instances.”

Apparently this is not a hard and fast rule with the Java client, just good advice. The java client serializes all calls to a channel.

An ACK should be invoked on the same channel on which the delivery was received. The delivery tag is scoped to the channel and an ACK sent to a different channel from which the delivery was received will cause a channel error. This somewhat contradicts the ‘Channels are not thread safe’ directive above since you will probably ACK on a different thread from one where you invoked basic.consume, but this seems to be acceptable.

Now some softer suggestions:

It seems neater to create a channel for each consumer. I like thinking of the consumer and channel as a unit: when I create a consumer, I also create a channel for it to consume from. When the consumer goes away, so does the channel and vice-versa.

Do not mix publishing and consuming on the same channel. If you follow the suggestion above, it implies that you have dedicated channels for each consumer. It follows that you should create separate channels for server and client originated methods.

Maintain a long running publish channel. My current implementation of EasyNetQ makes creating channels for publishing the responsibility of the user. I now think this is mistake. It encourages the pattern of: create a channel, publish, dispose the channel. This pattern doesn’t work with publisher confirms where you need to keep the channel open at least until you receive an ACK or NACK. And although creating channels is relatively lightweight, there is still some overhead. I now favour the approach of maintaining a single publish channel on a dedicated thread and marshalling publish (and declare) calls to it. This is potentially a bottleneck, but the impressive performance of non-transactional publishing means that I’m not overly concerned about it.

4 comments:

Michael said...

]] Channels are not thread safe.
That's a shame - that's one of the main reasons for using a messaging system in the first place - to easily communicate between threads, whether they be in the same process or machine or not.
Especially with async programming using callbacks or new core async features, your unit or work could get called back on any thread to speed things up. This would force the completion action into a particular thread context - or I suppose thread-local storage of the correct channel to use.
Seems like the Easy wrapper could hide some of this if the user asked for it.

Jonathan Goldman said...

There is one use case for having multiple consumers on a model, and that's if you want to be able to consume from multiple queues and share the same prefetch count. This is needed for work priority with a fixed number of client side threads.

Simon Thorogood said...

Sounds very similar to our .NET / RabbitMQ usage patterns - always reassuring to see others have come to the same conclusion :)

One of the reasons that we switched to a separate long-running publisher thread was that we were finding (on rare occasions) that TCP back-pressure (we think) was bringing our whole publisher application to a halt if the AMQP publish call was on the main processing thread.

Chintan Vadhadiya said...

I have a Problem, Rabbitmq, i am working with 4 channels, but when the internet fails i connect secondary internet it will connect new 4 channels but the 4 old channels will be on that Queue, Old channel should disconnected or how can i make channel unique.