6 May, 2010

RabbitMQ vs Apache ActiveMQ vs Apache qpid

Posted by

We need a simple message queue to ensure asynchronous message passing across a bunch of our server side apps. The message volume is not intended to be very high, latency is not an issue, and order is not important, but we do need to guarantee that the message will be received and that there is no potential for failure irrespective of infrastructure downtime.

Dhruv from my team had taken up the task of researching various persistent message queue options and compiling notes on them. This is a compendium of his notes (disclaimer – this is an outline of our experience, there may be inaccuracies) -

RabbitMQ

General:

  • Some reading on clustering http://www.rabbitmq.com/clustering.html
  • DNS errors cause the DB(mnesia) to crash
  • A RabbitMQ instance won’t scale to LOTS of queues with each queue having fair load since all queues are stored in memory (queue metadata) and also in a clustered setup, each queue’s metadata (but not the queue”s messages) is replicated on each node. Hence, there is the same amount of overhead due to queues on every node in a cluster
  • No ONCE-ONLY semanamntics. Messages may be sent twice by RabbitMQ to the consumer(s)
  • Multiple consumers can be configured for a single queue, and they will all get mutually exclusive messages
  • Unordered; not FIFO delivery
  • Single socket multiple connections. Each socket can have multiple channels and each channel can have multiple consumers
  • No provision for ETA
  • maybe auto-requeue (based on timeout) — needs investigation
  • Only closing connection NACKs a message. Removing the consumer from that channel does NOT. Hence, all queues being listened to on that channel/connetion are closed for the current consumer
  • NO EXPONENTIAL BACKOFF for failed consumers. Failed messages are re-tried almost immediately. Hence an error in the consumer logic that crashes the consumer while consuming a particular message may potentially block the whole queue. Hence, the consumer needs to be programmed well — error free. However, apps are like; well apps…
  • Consumer has to do rate limiting by not consuming messages too fast (if it wants to); no provision for this in RabbitMQ

Persistence:

  • It will use only it’s own DB — you can’t configure mySQL or any such thing

Clustering and Replication:

  • A RabbitMQ cluster is just a set of nodes running the RabbitMQ. No master node is involved.
  • You need to specify hostname of cluster nodes in a cluster manually on the command line or in a config file.
  • Basic load balancing by nodes in a cluster by redirecting requests to other nodes
  • A node can be a RAM node or a disk node. RAM nodes keep their state only in memory (with the exception of the persistent contents of durable queues which are still stored safely on disc). Disk nodes keep state in memory and on disk.
  • Queue metadata shared across all nodes.
  • RabbitMQ brokers tolerate the failure of individual nodes. Nodes can be started and stopped at will
  • It is advisable to have at least 1 disk node in a cluster of nodes
  • You need to specify which nodes are part of a cluster during node startup. Hence, when A is the first one to start, it will think that it is the only one in the cluster. When B is started it will be told that A is also in the cluster and when C starts, it should be told that BOTH A and B are part of the cluster. This is because if A or B go down, C still knows one of the machines in the cluster. This is only required for RAM nodes, since they don’t persist metadata on disk. So, if C is a memory node and it goes down and comes up, it will have to be manually told which nodes to query for cluster membership (since it itself doesn’t store that state locally).
  • Replication needs to be investigated (check addtl resources) however, from initial reading, it seems queue data replication does not exist
  • FAQ: “How do you migrate an instance of RabbitMQ to another machine?”. Seems to be a very manual process.

Transactions:

  • Any number of queues can be involved in a transaction

Addtl Resources

Apache qpid

  • Supports transactions
  • Persistence using a pluggable layer — I believe the default is Apache Derby
  • This like the other Java based product is HIGHLY configurable
  • Management using JMX and an Eclipse Management Console application - http://www.lahiru.org/2008/08/what-qpid-management-console-can-do.html
  • The management console is very feature rich
  • Supports message Priorities
  • Automatic client failover using configurable connection properties -
  • Cluster is nothing but a set of machines have all the queues replicated
  • All queue data and metadata is replicated across all nodes that make up a cluster
  • All clients need to know in advance which nodes make up the cluster
  • Retry logic lies in the client code
  • Durable Queues/Subscriptions
  • Has bindings in many languages
  • For the curious: http://qpid.apache.org/current-architecture.html
  • In our tests -
    • Speed: Non-persistent mode: 5000 messages/sec (receive rate), Persistent mode: 1100 messages/sec (receive rate) (send rate will be typically a bit more, but when you start off with an empty queue, they are almost the same for most queue implementations). However, the interesting bit is that even in transacted mode, I saw a lot of message loss if I crashed the broker (by crash I mean Ctrl+C, not even the more -9 signal type of thing that I usually do). Why I stress this is that apps. can usually hook on to Ctrl+C and save data before quitting, but qpid didn’t think it prudent to do so. Out of 1265 messages sent (and committed), only 1218 were received by the consumer (before the inflicted crash). Even on restarting the broker and consumer, that didn’t change. We observed similar behaviour with RabbitMQ in our tests. However, RabbitMQ docs. mention that you need to run in TRANSACTED mode (not just durable/persistent) for guaranteed delivery. We haven’t run that test yet.

Apache ActiveMQ

  • HIGHLY configurable. You can probably do anything you want it to with it
  • You can choose a message store. 4 are already available
  • Has lots of clustering options:
    • Shared nothing Master-Slave: ACK sent to client when master stores the message
    • Shared Database: Acquires a lock on the DB when any instance tries to access the DB
    • Shared Filesystem: Locks a file when accessing the FS. Issues when using NFS with file-locking; or basically any network based file system since file locking is generally buggy in network file systems
  • Network of brokers: This is an option that allows a lot of flexibility. However, it seems to be a very problematic/buggy way of doing things since people face a lot of issues with this configuration
  • Scaling:
    • A. Default transport is blocking I/O with a thread per connection. Can be changed to use nio
    • Horizontal scaling: Though they mention this, the way to achieve this is by using a network of brokers
    • Patitioning: We all know Mr. Partitioning, don’t we. The client decides where to route packets and hence must maintain multiple open connections to different brokers
  • Allows producer flow-control!!
  • Has issues wrt lost/duplicate messages, but there is an active community that fixes these issues
  • Active MQ crashes fairly frequently, at least once per month, and is rather slow - http://stackoverflow.com/questions/957507/lightweight-persistent-message-queue-for-linux
  • Seems to have bindings in many languages(just like RabbitMQ)
  • Has lots of tools built around it 12. JMS compliant; supports XA transactions: http://activemq.apache.org/how-do-transactions-work.html
  • Less performant as compared to RabbitMQ
  • We were able to perform some tests on Apache Active MQ today, and here are the results:
    • Non persistent mode: 5k messages/sec
    • Persistent mode: 22 messages/sec (yes that is correct)
  • There are multiple persisters that can be configured with ActiveMQ, so we are planning to run another set of tests with MySQL and file as the persisters. However, the current default (KahaDB) is said to be more scalable (and offers faster recoverability) as compared to the older default(file/AMQ Message Store: http://activemq.apache.org/amq-message-store.html).
  • The numbers are fair. Others on the net have observed similar results: http://www.mostly-useless.com/blog/2007/12/27/playing-with-activemq/
  • With MySQL, I get a throughput of 8 messages/sec. What is surprising is that it is possible to achieve much better results using MySQL but ActiveMQ uses the table quite unwisely.
  • ActiveMQ created the tables as InnoDB instead of MyISAM even though it doesn’t seem to be using any of the InnoDB features.
  • I tried changing the tables to MyISAM, but it didn’t help much. The messages table structure has 4 indexes !! Insert takes a lot of time because MySQL needs to update 4 indexes on every insert. That sort of kills performance. However, I don’t know if performance should be affected for small (< 1000) messages in the table. Either ways, this structure won’t scale to millions of messages since everyone will block on this one table.
Tags: ,
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Elden Bishop on May 10, 2010 @ 4:25 pm
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Dhruv on May 10, 2010 @ 11:24 pm
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Dhruv on May 10, 2010 @ 11:32 pm
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Elden Bishop on May 11, 2010 @ 7:48 pm
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Radhakrishnan D on May 12, 2010 @ 2:12 am
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Radhakrishnan D on May 13, 2010 @ 10:38 pm
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Dhruv on May 13, 2010 @ 11:14 pm
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Dhruv on May 13, 2010 @ 11:16 pm
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Dhruv on May 13, 2010 @ 11:21 pm
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Dhruv on May 13, 2010 @ 11:23 pm
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Aaron on May 19, 2010 @ 7:11 pm
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
comment_type == "trackback" || $comment->comment_type == "pingback" || ereg("", $comment->comment_content) || ereg("", $comment->comment_content)) { ?>

Trackbacks & Pingbacks
  • Comment by Anand Gupta on September 6, 2010 @ 3:04 am

Comments
comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Rob Davies
May 7, 2010

The persistent number for ActiveMQ does look really poor – be interested to know your environment – should be 2 orders of magnitude greater with KahaDB.

However, when you compare AMQP with ActiveMQ you are not comparing like with like. The JMS specification demands that for persistent messages, a send will block until it gets a receipt from the broker that the message has been delivered AND stored (which means syncing on disk – which is very slow).

With AMQP – persistent means that the message will be stored – at some point – which in reality means you could loose thousands of messages in event of network or broker failure.

You would need to use transactions with AMQP to get similar results.

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Bhavin Turakhia
May 7, 2010

@rob: thanks. I have merely pasted notes of my colleague. We intend to run a RabbitMQ test with transactions and determine the results :)

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Elden Bishop
May 10, 2010

I don’t understand the numbers you are posting for ActiveMQ. I have been running benchmarks for our company on ActiveMQ 5.3.1 with KahaDB and am seeing roughly 10,000 m/s being hit for persistent messages with transactions off. This is hitting a box over fifty miles away. I am using transactions to allow bulk loading but even the worst case, loading one message at a time, I still hit roughly 200 messages per second.

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Bhavin Turakhia
May 10, 2010

@elden: thanks for your comment. we are definitely not seeing that performance. i will however have dhruv recheck his configs :)

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Dhruv
May 10, 2010

Hello Elden,
You seem to be running your tests with transactions off. Please could you post the exact script/application code that is setting up the environment for the tests?

What we are doing is roughly:
// Set persistence and transactions ON
while (1) {
sendMessage(XXX);
commit();
}

Is that the sort of flow you are seeing?
You would need to set persistence and transactions ON during session creation using code that looks like:

connection.createSession(transacted, Session.AUTO_ACKNOWLEDGE);
and:
producer.setDeliveryMode(DeliveryMode.PERSISTENT);

Also, you mention the distances between your machines, but what do the round trip times look like?

iirc, persistent mode doesn’t guarantee that messages are written when you send them. Only transacted mode does that. However, the semantics could be different from what I expect.

btw, Thanks for taking time out to notify us about discrepencies we may have in our testing :)

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Dhruv
May 10, 2010

Hello Rob,
I am running tests on a Lenovo X200 laptop, which has 2 CPUs (I think 2.3GHz each) with 3GB RAM. The hdd is the one that comes along with the laptop which I think is the 160GB/5400 one. (could be wrong here)
Running mysql which comes along with xampp. No changes made to my.cnf. In fact, ActiveMQ created tables using InnoDB and we went ahead and chnaged them to MyISAM as well, but in either case, we saw similar results.
ActiveMQ is v5.3.1

The tests were run using the ProducerTool.java and ConsumerTool.java files available online. All of the producer, consumer and broker were on the same machine.

I was running tests mainly to compare different queues, so the exact hardware didnt’ matter for me as long as I used the same for all tests.

It is possible that the 2X multiplier you speak of will be achieved if running on more powerful machines.

Note: I was running the tests with the power cable plugged in because the machine becomes slower by anything from 2x to 4x w/o the power plugged in.

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Elden Bishop
May 11, 2010

My test code is bulk loading 100,000 messages in transactional batches. I tested different batch sizes to get an idea of the speed at different sizes. For seeding I am doing something like…
def transactional = true
def session = conn.createSession(transactional, Session.CLIENT_ACK)
def queue = session.createQueue(“test.queue”)
def producer = session.createProducer(queue)
producer.setDeliveryMode(DM.PERSISTENT)
for (i in 0..<100000) {
producer.send(createMessage(session))
if ((i % BATCH_SIZE) == BATCH_SIZE – 1) session.commit()
}

With a batch size of 200 on a very nice server I get 11409 m/s. With a batchsize of "1", which is a commit per message I am seeing around 250 m/s.

My comment about transactions being off was referring to the receive phase, not the send. This is running on a 64bit dual core windows 2003 install with 3GB memory and 150GB RAID 1.

Running all services and clients locally on my desktop is much slower at 110m/s for single message per transaction.

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Radhakrishnan D
May 12, 2010

Hi Bhavin,

You have added this line in activeMQ
“” Less performant as compared to RabbitMQ “”.
Can you tell us under what scenarios activeMQ was less perfomant than rabbitMQ.

I tried running activeMQ on an 8 GB RAM .

i got a rate of 8000 msg / sec for non persistent and
500 / sec for persistent messages.

But the CPU and context switches were much higher.

Could you please eloborate on message losses.
My rabbitmq-broker is dropping messages.
You guys did not come across even a single message drop ?

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
alexis
May 13, 2010

Radha,

This sounds weird. Can you tell us why you think Rabbit is dropping messages, and with what configuration?

alexis

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Radhakrishnan D
May 13, 2010

hi alex,
that problem is solved now .
sorry

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Dhruv
May 13, 2010

Hello Elden,
We were running both the producer and consumer in transactional mode, and the consumer was slower than the producer, so I posted those numbers.
The final configurationn you speak of is roughly the one we are using. Even with that config, you are getting 5x the speed we are seeing.
We were running the producer and consumer at the same time to mesure results. I do vaguely remember running the producer and consumer separately but I don’t recall noting down the numbers. That’s something I’ll probably test again.

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Dhruv
May 13, 2010

@Elden: Coming to think of it, we don’t need to run the consumer in transactional mode for our purposes since the consumer code should be able to process messages twice.

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Dhruv
May 13, 2010

@alexis: The tests we can on RabbitMQ resulted in what we thought was message loss since we were testing with basic_publish() which is an async call. So, it didn’t _really_ result in message loss. We did find a way to get failed publishes, but no way to get successful ones. Does the AMQP standard have any such provision?
So, we tried using transactions which are blocking calls. With RabbitMQ, we get performance pretty much in tune with what we were seeing with ActiveMQ at 20 messages/sec. These messages were about 4 byte messages, so I don’t think that the message payload overhead was significant.

All tests run on a Lenovo X200 with the same configuration as above, with the power plug plugged in.

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Dhruv
May 13, 2010

@alexis: Oh! and in transactional mode, RabbitMQ DIDN”T lose any messages in our tests!! :D
Btw, we ran just the producer in TX mode since that is sufficient. The consumer should be idemponent and prepared to process the same message multiple times.

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
alexis
May 14, 2010

Just a quick note about Radha’s use case:

There is an ‘immediate’ flag in AMQP which affects delivery semantics. Read about this in the AMQP documentation (the 0-9-1 specification is recommended via amqp.org).

Earlier, Radha’s immediate flag was set to true. He then set the immediate flag to false while publishing messages. Then all messages were received.

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Aaron
May 19, 2010

Question about the speed, we’re seeing ~2500 msg/sec non_persistent with a data store so that things can be paged off to disk if the queue grows too large. That’s ~2500 with a couple of producers AND several (>=10) consumers simultaneously. The consumers then also send on to a different queue with non_persistent messages.

When doing Memory only (turn off persistence) with a limit data set without sending on to another queue, we’ve only see ~5300 msg/sec with ONLY the consumers.

We’re not doing any kind of transactions, with AUTO_ACK consumers, asyncSend=true. We definitely appear to be bottlenecking on ActiveMQ, and we’re thinking of benchmarking some others such as RabbitMQ, QPid, or HornetQ.

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?> comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
mbt shoes storm
July 9, 2010

you will like ugg boots, http://www.uggbootscool.com

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Reebok Easytone Shoes
July 13, 2010

The Reebok Easytone Shoes is probably the most athletic and traditional of the Calorie Burning or Reebok EasyTone category, emulating their traditional Reebok EasyTone Trainers. The Reebok ZigTech shoes,

comment_type != "trackback" && $comment->comment_type != "pingback" && !ereg("", $comment->comment_content) && !ereg("", $comment->comment_content)) { ?>
Anand Gupta
September 6, 2010

A very well written comparison of queues i must say. I was looking for it since a long time. However, I would have loved to see ZeroMQ also being included in this comparison. Considering you requirement zeroMQ fits perfectly and would show considerable performance as compared to the queues being compared above.

Leave a comment

(required)

(required)

Spam protection by WP Captcha-Free