[rabbitmq-discuss] Determining Cause of Flow-Control Invocation

[Richard Raseley]

We currently use RabbitMQ as a broker for a subset of our overall logging
platform. Recently, we attempted to add an additional category of logging
that would result in additional load of approximately 2,000 msg/sec with an
average message size of approximately 2KB.

Upon adding this additional category of logging, we noticed a couple things:

(1) Our consumers were not able to keep up with the flow of messages, so
they began to build up in the queue (our developers are working on fixing
this).

(2) We began to see RabbitMQ invoke flow-control in the form of TCP
back-pressure on our producers.

My initial thought was that since we were using durable queues with
persistent messages that when the consumers weren't immediately picking up
the messages, RabbitMQ was attempting to persist them to disk in order to
satisfy the persistence & durability requirements and it couldn't keep up.

But in following up with my developers this morning it seems that they are
in fact no publishing the messages as persistent and being that (based on
my understanding) a non-persistent message published to a durable queue is
not written to disk, this threw my initial theory out the windows (I still
have to verify their claim of non-persistence).

So, my question is - what other things can I look at to determine the cause
of the invocation of flow control. Additional relevant information is as
follows:

(*) 2x RabbitMQ Nodes Running 3.1.3 in a clustered configuration.
(*) Each node has 8x 2GHzCPU cores and 4GB of RAM.
(*) At no time during the window in question did CPU utilization reach >15%
(*) At no time during the window in question did memory utilization reach

50%

(*) In looking at the logs during the window in question I can see events
which I think are associated with the various publishers disconnecting (I
am not sure if our producer code is handling flow-control gracefully) but
nothing that seems to indicate why this happened.

Any help would be greatly appreciated.

Regards,

Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131004/4bd978e7/attachment.htm>

[Michael Klishin]

So, my question is - what other things can I look at to determine the cause of the invocation of flow control

There are 3 cases when RabbitMQ will block publishers (until the underlying problem
is resolved):

* It consumes more RAM than vm_high_memory_watermark
* Node has less disk space left than the configured minimum limit
* RabbitMQ process is out of available file descriptors (ulimit -n)

See http://www.rabbitmq.com/memory.html for more details. RabbitMQ log
should have very visible messages when one of the above happens.

In the first two cases the only solution is to increase capacity of your consumers so that they drain
messages and all alarms unblock. Alternatively, if your data
is temporary in nature, you can set TTL on messages [1].

In the latter case (file descriptors) you should configure open file handles limit for
the Rabbit process or use fewer connections.

1. http://www.rabbitmq.com/ttl.html

MK



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131004/de10e9c4/attachment.pgp>

[Simon MacMullen]

Post by Michael Klishin

So, my question is - what other things can I look at to determine the cause of the invocation of flow control

There are 3 cases when RabbitMQ will block publishers (until the underlying problem
* It consumes more RAM than vm_high_memory_watermark
* Node has less disk space left than the configured minimum limit
* RabbitMQ process is out of available file descriptors (ulimit -n)

I think the OP was talking about this:

http://www.rabbitmq.com/memory.html#per-connection

not this:

http://www.rabbitmq.com/memory.html#memsup

I think so, anyway.

Cheers, Simon

[Richard Raseley]

Michael and Simon,

Thank you for taking the time to respond to my question. Yes, I was talking
about the per-connection flow-control.

The thing that stands out to me is Simon's statement that each queue can
only use 100% of one core. Looking back at our logging, it does seem that
one core was getting close to 100% during this time period (I need to get
more granular data to be sure if it hit it or not) - so this would explain
what we were seeing.

If that is the case, the obvious solution is to separate the load across
multiple queues. I will have to think on that a while, as our current
configuration was 1:1 mapping of queue to error type. Some questions:

(*) When a RabbitMQ broker has multiple queues, will it automatically
disperse them across the available cores? For example if I have 8 queues on
a broker with 8 cores, will it automatically ensure a 1:1 mapping of
queues:cores?

(*) In the situation where I have been using one queue that mapped well to
a logical construct (e.g 1 queue for info, 1 queue for warning, 1 queue for
errors) can you share your own personal opinions on the best way to divide
that up to ensure distribution across all cores (e.g. one queue per type
per core, one queue per producer per type, etc.)?

I have a little more digging to do.

Regards,

Richard

Post by Michael Klishin

Post by Richard Raseley
So, my question is - what other things can I look at to determine the
cause of the invocation of flow control

There are 3 cases when RabbitMQ will block publishers (until the underlying problem
* It consumes more RAM than vm_high_memory_watermark
* Node has less disk space left than the configured minimum limit
* RabbitMQ process is out of available file descriptors (ulimit -n)

http://www.rabbitmq.com/**memory.html#per-connection<http://www.rabbitmq.com/memory.html#per-connection>
http://www.rabbitmq.com/**memory.html#memsup<http://www.rabbitmq.com/memory.html#memsup>
I think so, anyway.
Cheers, Simon
--
Simon MacMullen
RabbitMQ, Pivotal

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131004/02dfa02b/attachment.htm>

[Michael Klishin]

(*) When a RabbitMQ broker has multiple queues, will it automatically disperse them across the available cores? For example if I have 8 queues on a broker with 8 cores, will it automatically ensure a 1:1 mapping of queues:cores?

For all intents and purposes, you can think that the answer is "yes". 8 cores will be used,
but it's a little more complicated than "1 queue per core".

(*) In the situation where I have been using one queue that mapped well to a logical construct (e.g 1 queue for info, 1 queue for warning, 1 queue for errors) can you share your own personal opinions on the best way to divide that up to ensure distribution across all cores (e.g. one queue per type per core, one queue per producer per type, etc.)?

you can implement sharding in the client. That's basically how travis-ci.org worked
at some point (2012). To be more specific, you can have N queues named

info.1
info.2
?
info.N

and have consumers consume from all of them. Publisher can use any hashing
technique you like, e.g. consistent hashing or even basic module-based caching,
since there is no need to rebalance queue contents when you add or remove queues.

The key issue with this is ordering. E.g. if your stream must be ordered (e.g. CI build
log chunks must be), it is trickier but certainly doable. If not, you can parallelize the
stream between many queues pretty trivially.

See http://about.travis-ci.org/blog/2013-08-08-solving-the-puzzle-of-scalable-log-processing/ for
some more insights.

HTH.

MK



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131004/d12a480b/attachment.pgp>

[Michael Klishin]

Post by Michael Klishin
The key issue with this is ordering. E.g. if your stream must be ordered (e.g. CI build
log chunks must be), it is trickier but certainly doable. If not, you can parallelize the
stream between many queues pretty trivially.

To add to this, you can put extra information into message headers to help you
reconstruct the order of chunks.

MK



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131004/2f88aec9/attachment.pgp>

[Richard Raseley]

Michael,

Thank you for your quick reply.

Will you expand more on your statement "[I]t's a little more complicated
than '1 queue per core'."?

Thank you for your suggestion on sharding, I will review the link you
provided.

Regards,

Richard

Post by Michael Klishin

Post by Michael Klishin
The key issue with this is ordering. E.g. if your stream must be ordered

(e.g. CI build

Post by Michael Klishin
log chunks must be), it is trickier but certainly doable. If not, you

can parallelize the

Post by Michael Klishin
stream between many queues pretty trivially.

To add to this, you can put extra information into message headers to help you
reconstruct the order of chunks.
MK

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131004/29055fc9/attachment.htm>

[Michael Klishin]

Will you expand more on your statement "[I]t's a little more complicated than '1 queue per core'."?

You get one Erlang process per queue. Those are scheduled to run in one or more
run queues (VM queues, nothing to do with RabbitMQ queues). Recent Erlang VM
versions detect how many cores are available on start and create a new run queue
per core.

How scheduling is performed (that is, how processes are given time to run
and distributed between run queues) is not so trivial to explain and varies between
SMP and non-SMP systems, see

http://jlouisramblings.blogspot.ru/2013/01/how-erlang-does-scheduling.html

for a fairly concise and approachable overview.

MK



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131004/ea5cb91d/attachment.pgp>

[Richard Raseley]

Michael,

Thank you for that explanation - it was helpful.

Regards,

Richard

Post by Richard Raseley

Post by Richard Raseley
Will you expand more on your statement "[I]t's a little more complicated

than '1 queue per core'."?
You get one Erlang process per queue. Those are scheduled to run in one or more
run queues (VM queues, nothing to do with RabbitMQ queues). Recent Erlang VM
versions detect how many cores are available on start and create a new run queue
per core.
How scheduling is performed (that is, how processes are given time to run
and distributed between run queues) is not so trivial to explain and varies between
SMP and non-SMP systems, see
http://jlouisramblings.blogspot.ru/2013/01/how-erlang-does-scheduling.html
for a fairly concise and approachable overview.
MK

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131004/a07888b4/attachment.htm>

[Simon MacMullen]

Post by Richard Raseley
So, my question is - what other things can I look at to determine the
cause of the invocation of flow control. Additional relevant information

Each queue can only use 100% of one core, so it is possible to hit flow
control without using 100% CPU of all cores (if everything goes through
one queue).

Also: queues with transient messages can start to page to disc if there
is any memory pressure. This is not very easy to observe in current
releases (we aim to make it clearer), but you can look at
backing_queue_status in rabbitmqctl and the HTTP API - if ram_msg_count
+ ram_ack_count != messages then the queue is paging.

This is definitely an area where we aim to improve things. 3.2.0 will
already make paging rather easier to observe and control.

Cheers, Simon