With help from Ann, I did some simple demo programs over the last few months to try to understand RabbitMQ and message queues in general.

The code that we produced, (with a readme document) is here on github

RabbitMQ is a free, open-source cross-platform message queuing system. It is described as robust, fast and scalable though I didn’t really test those features since I just ran it on my laptop and sent a few tens of messages. In production it would run on a server, or on a dedicated cluster if a really large volume is required. It follows a standard called AMQP.

RabbitMQ is written in Erlang and runs on the Erlang virtual machine. This can be installed on Windows, Mac OS X, Linux, or other flavours of unix. If you have any doubts about Erlang or running it on Windows, lay them to rest - it runs fine, and the language that the message queue is written in, or indeed the OS that you use for the message queue server, doesn't matter. There are client libraries for .Net, java, python and other environments, which makes it a potential way to connect different platforms.

I found it easy enough to install the Erlang VM and the RabbitMQ server. For the .Net code, the RabbitMQ .Net client library came via Nuget without any hassle.

I did three samples to work RabbitMQ from C# code. These samples are very simple, in order to concentrate on understanding what RabbitMQ is doing. For more advanced usage of RabbitMQ in .Net, take a look at Mike Hadlow's EasyNetQ library.

Simple send and receive

The first demo is not much more than "hello world" - a sender sends a message onto a queue, and a receiver takes it off. However this exposes some features of the queue:

If no receiver is running, the sender's messages will be kept on the queue, and when a receiver does start, it will get the backlog of messages immediately. This is basic to how message queues work in introducing asynchrony between sender and receiver, and allowing for restarts and recovery from transient errors.

More than one sender works fine - they can all put messages onto the same queue. More than one receiver works fine too - if several receivers are waiting for messages, one of them gets the next message, in a round-robin load-balancing manner. So the queue is not just asynchronous - it's potentially many to many. If need be, the server-side processing can be scaled out by starting a new receiver.

Publish and subscribe

The "publish and subscribe" pattern of the second demo has a different configuration, and not one that I first associated with message queues. The pattern is to broadcast a message to all listeners who want to hear it. This uses an exchange - a central point to which messages are sent, which forwards the message to the attached queues of the receivers. This has the decoupling effect that the sender does not need to know if there is one just listener, or a hundred. Or none. Messages sent while there are no queues attached to the exchange are effectively just discarded.

This also makes use of an interesting supporting feature: in this code, listener queues are created on the fly with a unique, randomly generated name, and are released when the listener stops listening. Queues can be dynamic in RabbitMQ, and can be created and removed as a code task, not an admin task. This was uncommon in my experience with MSMQ. Of cource, RabbitMQ queues can also be well-known, long lived, durable endpoints.

Remote Procedure Call

The third sample demonstrates getting a reply message, like a procedure call, but with the decoupling effects  - the asynchrony and queuing - that a message queue offers. The server replies to a queue that the client has generated. The client sends the reply queue name in the message metadata. As in the first sample, multiple senders, multiple receivers or both work fine.

Conclusion

Message queuing, and RabbitMQ's flexibility in particular, offer a lot of interesting options for building large-scale systems. It's worth noting that other AMQP implementations will have the same flexibility, and there are other simpler options like 0MQ but RabbitMQ is the only one of these that I have tried yet.