29 Dec, 2009

Investigating Message Queueing systems

Posted by

In my constant quest for creating scalable systems and architecture a robust message queuing system was the missing link. I have begun reviewing some of the available options. This is a rough list of some of the interesting links I came across during this process -

Watch this space for more info …
http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
The most detailed comparison of queuing systems and their advantages and disadvantages. Covers RabbitMQ and Apache qpid in detail and then several others. The most comprehensive list of message queueing systems.
http://stackoverflow.com/questions/731233/activemq-or-rabbitmq-or-zeromq-or
a list of links comparing various queueing systems
http://www.unlimitednovelty.com/2009/04/twitter-blaming-ruby-for-their-mistakes.html
A detailed critique on how twitter selected its message queue implementaions with comments from the developers at twitter and why they wrote their own queue. Do read the comments. They have a ton of meat
http://gojko.net/2009/03/16/qcon-london-2009-upgrading-twitter-without-service-disruptions/
Talks a bit about how twitter uses a messaging framework at the backend
http://blog.evanweaver.com/articles/2009/03/13/qcon-presentation/
Evan Weaver’s talk on twitters scalability issues and how they were solved
http://www.zeromq.org/whitepapers:brokerless
An interesting paper on differences between brokerless message queues and the advantages thereof
Tags: , , ,
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Bhavin on December 31, 2009 @ 7:09 am
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Kulbhushan on January 12, 2010 @ 1:11 am
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by yangyang on March 25, 2010 @ 7:31 am

Comments
comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Indus Khaitan
December 29, 2009

In my quest of selecting an MQ — there were only two major thoughts which helped me arrive at an architecture & implementation of choice:

1. Data Loss & Data type — Essentially, what kind of data is passing through the MQ. If it is okay to lose a few messages here & there, then you may not require persistence & may allow overloaded listeners. For example, implementing the MQ in a payment gateway vs. implementing an MQ for social media.

2. Throughput — How fast? Social media may require instant consumption but okay to be delayed while payment gateway messages cannot be delayed beyond acceptable timeout periods. The loads in a payment gateway can be predicted whereas in social media queues it may have bursts.

2 cents — hope that helps.

Indus

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Rob Davies
December 29, 2009

Its worth paying attention to what persistence messages mean. For some messaging implementations – it simply means the message will be persisted if it isn’t consumed – it doesn’t mean you won’t loose messages (often a lot more than a few) on a hard failure.

For messaging systems that are JMS compatible, persistent delivery means ‘guaranteed’ – the send is synchronous and written to disk before its delivered to any consumer(s).

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
alexis
December 31, 2009

In relation to RabbitMQ we try to keep an up to date set of introductory docs and links to other materials here: http://www.rabbitmq.com/how.html

.. and here: http://delicious.com/alexisrichardson/rabbitmq

I agree with the above comments – you should make sure you fully understand what your requirements are and how they relate to the product’s capabilities and configurations. There are LOTS of different patterns for messaging. You should also be thinking about scale, and location of your server or (possibly) servers if they are in many locations.

Cheers

alexis

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Bhavin
December 31, 2009

@alexis: thanks for the addtl links. we have begun using rabbitmq in a test environment on one of our projects

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Kulbhushan
January 12, 2010

I think when you think about messaging you should also consider Tibco’s EMS http://www.tibco.com/software/messaging/enterprise-message-service/default.jsp.

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
yangyang
March 25, 2010

Grassroots Indigenous Multimedia announces the launch of their new Ojibwe Language learning software, Ojibwemodaa.Best Way : Rosetta stone learning Tips. Available in more than 30 languages, the Rosetta stone language-learning solution is used by schools, the consumer will write a message on the Personal Rosetta Stone. Download free language software, play online games and start learning a language now.

Leave a comment

(required)

(required)

Spam protection by WP Captcha-Free