29 Dec, 2009
Investigating Message Queueing systems
Posted by Bhavin Turakhia
In my constant quest for creating scalable systems and architecture a robust message queuing system was the missing link. I have begun reviewing some of the available options. This is a rough list of some of the interesting links I came across during this process -
- http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes -Â A detailed comparison of queuing systems and their advantages and disadvantages. Covers RabbitMQ and Apache qpid and several others. The most comprehensive list of message queueing systems I found
- http://stackoverflow.com/questions/731233/activemq-or-rabbitmq-or-zeromq-or -Â A list of links comparing various queueing systems
- http://www.unlimitednovelty.com/2009/04/twitter-blaming-ruby-for-their-mistakes.html -Â A detailed critique on how twitter selected its message queue implementaions with comments from the developers at twitter and why they wrote their own queue. Do read the comments. They have a ton of meat
- http://www.zeromq.org/whitepapers:brokerless -Â An interesting paper on brokerless message queues and the advantages thereof
- http://gojko.net/2009/03/16/qcon-london-2009-upgrading-twitter-without-service-disruptions/ -Â Talks a bit about how twitter uses a messaging framework at the backend
- http://blog.evanweaver.com/articles/2009/03/13/qcon-presentation/ -Â Evan Weaver’s talk on scaling twitter
Watch this space for more info …
http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
The most detailed comparison of queuing systems and their advantages and disadvantages. Covers RabbitMQ and Apache qpid in detail and then several others. The most comprehensive list of message queueing systems.
http://stackoverflow.com/questions/731233/activemq-or-rabbitmq-or-zeromq-or
a list of links comparing various queueing systems
http://www.unlimitednovelty.com/2009/04/twitter-blaming-ruby-for-their-mistakes.html
A detailed critique on how twitter selected its message queue implementaions with comments from the developers at twitter and why they wrote their own queue. Do read the comments. They have a ton of meat
http://gojko.net/2009/03/16/qcon-london-2009-upgrading-twitter-without-service-disruptions/
Talks a bit about how twitter uses a messaging framework at the backend
http://blog.evanweaver.com/articles/2009/03/13/qcon-presentation/
Evan Weaver’s talk on twitters scalability issues and how they were solved
http://www.zeromq.org/whitepapers:brokerless
An interesting paper on differences between brokerless message queues and the advantages thereof
In my quest of selecting an MQ — there were only two major thoughts which helped me arrive at an architecture & implementation of choice:
1. Data Loss & Data type — Essentially, what kind of data is passing through the MQ. If it is okay to lose a few messages here & there, then you may not require persistence & may allow overloaded listeners. For example, implementing the MQ in a payment gateway vs. implementing an MQ for social media.
2. Throughput — How fast? Social media may require instant consumption but okay to be delayed while payment gateway messages cannot be delayed beyond acceptable timeout periods. The loads in a payment gateway can be predicted whereas in social media queues it may have bursts.
2 cents — hope that helps.
Indus
Its worth paying attention to what persistence messages mean. For some messaging implementations – it simply means the message will be persisted if it isn’t consumed – it doesn’t mean you won’t loose messages (often a lot more than a few) on a hard failure.
For messaging systems that are JMS compatible, persistent delivery means ‘guaranteed’ – the send is synchronous and written to disk before its delivered to any consumer(s).
In relation to RabbitMQ we try to keep an up to date set of introductory docs and links to other materials here: http://www.rabbitmq.com/how.html
.. and here: http://delicious.com/alexisrichardson/rabbitmq
I agree with the above comments – you should make sure you fully understand what your requirements are and how they relate to the product’s capabilities and configurations. There are LOTS of different patterns for messaging. You should also be thinking about scale, and location of your server or (possibly) servers if they are in many locations.
Cheers
alexis
@alexis: thanks for the addtl links. we have begun using rabbitmq in a test environment on one of our projects
I think when you think about messaging you should also consider Tibco’s EMS http://www.tibco.com/software/messaging/enterprise-message-service/default.jsp.