4 Apr, 2010

Selecting a Message Queue – AMQP or ZeroMQ

Posted by

I have spent a better part of my Sunday researching some of the messaging queue options available out there. My research was divided primarily between AMQP queues and ZeroMQ. For those who came in late - iMatix the company behind ZeroMQ was also involved in defining the AMQP specs. iMatix recently announced that they would no longer continue supporting the AMQP spec and instead focus on ZeroMQ – their brokerless queue product

ZeroMQ is a fundamentally different approach to message queues from AMQP. Pieter and folks from iMatix have made several posts on the flaws in the design and philosophy of AMQP (check the resources section below). Infact during the course of my study today these posts were amongst the most interesting material I have read.

Here are some thoughts around considerations on our part with respect to selecting a queue for our application needs -

Ordered Messages

Some queues provide guarantees of ordered delivery, while others do not. Order of delivery may not be required in all applications, and adds complexity. However as an eg – If your messages consist of updates to state, then two messages updating the same entity, may result in inconsistency if their order is switched. These complexities would then have to be resolved in your application if the underlying messaging system does not offer ordering guarantees.

Persistence

If the queue itself does not manage persistence, it will have to be handled by your application. Persistence is required for reliability. A queue that manages all messages and state in memory will lose any undelivered messages incase of a node restart/downtime/crash.

Clustering

Broker based queues may or may not offer clustering. Some message queues offer fail-overs but not clustering. Clustering allows multiple nodes to be used as a single active-active instance, increasing availability.

I also have a short summary on the 2 options I am currently considering -

ZeroMQ

  • Crazy fast
  • Brokerless architecture
  • In-process library
  • Lower latencies
  • Very simple to use
  • No persistence – requiring higher layers to manage persistence

RabbitMQ

  • AMQP compliant
  • Written in erlang
  • Small footprint and seemingly fewer lines of code in comparison to other AMQP compliant queue managers

Resources

Here is a list of the more interesting resources I read through today:


Comments
Pieter Hintjens
April 6, 2010

Thanks for referencing my articles. Regarding ordering, it’s an n-neutral issue. That is, broker or no broker, or five brokers in a federation, the best ordering you can get is per-publisher, unless you are willing to reorder messages in some way, with big impact on performance. Both AMQP and 0MQ give you this per-publisher ordering.

Bhavin Turakhia
April 6, 2010

@pieter: thanks for dropping by :) … and you are right – ordering is an n-neutral issue. have updated that part of my post.

Our biggest concerns are around persistence. We could build persistence in our application, but would much rather abstract it out to the queue. I saw some references to persistence in the mailing list but nothing concrete. I know one of the concerns highlighted by you is that of performance. but many applications (such as ours) dont need millions of messages per second. Additionally with ultra-fast flash drives (the uber cool Fusion-io drives and now the Plaint ones) boasting 100,000+ IOPs the performance hit of an async write operation or even a recovery WAL may not be very high.

What is the current take on persistence support. I would also assume that the same could be handled in the language bindings. are there any versions of the language bindings available there that serve as a wrapper and provide persistence for failure recovery?

Sriram S
April 16, 2010

Hi : You could also consider Tibco EMS.

Sudhakar
April 26, 2010

Not sure whether i could ask here, I was searching for the messaging system using in Gmail or Google Mail.

Read several articles and found, some say Google Mail uses Google Web ToolKit and some say , No they were using a complete difference hidden strategy to show the mails in queue on gmail.

Possible, help me out in finding the system using in GMail.

Akhilesh
April 28, 2010

you can also explore Hornetq,think it’s a potential candidate.

http://www.jboss.org/hornetq

malcolm in St Louis
June 24, 2010

OCI did an evaluation of three types of messaging software.

Socket based BoostASIO, lightweight message quing from ZeroMQ and a fully features OMG DDS implementation known as OpenDDS.

A paper describing the results, with sample code can be found here: http://mnb.ociweb.com/mnb/MiddlewareNewsBrief-201004.html

The results are interesting and somewhat counter-intuitive.

regards Malcolm Spence
Director of Bus. Dev
OCI St.Louis MO
TEL: 1-314-590-0206

Leave a comment

(required)

(required)

Spam protection by WP Captcha-Free