[rabbitmq-users] Significant performance dips every 10 minutes

[Alex Corvin]

I've been trying to track down the cause of a performance issue I'm seeing
in my RabbitMQ cluster. Every 10 minutes (exactly), I'm seeing significant
(often worse than 50%) drops in event rates that my cluster handles. Are
there any processes that run by default within rabbitmq-server that do, for
example, clean up tasks? If so, can these be configured to be less
performance affecting? For a little more info, this is a 3-node VM cluster
with connections from the shovel plugin and a custom python application
using the Pika library.

[Alvaro Videla]

Anything worth noting from the logs? RabbitMQ and Erlang versions being
run? Perhaps the broker is paging to disk when these events happen?

Post by Alex Corvin
I've been trying to track down the cause of a performance issue I'm seeing
in my RabbitMQ cluster. Every 10 minutes (exactly), I'm seeing significant
(often worse than 50%) drops in event rates that my cluster handles. Are
there any processes that run by default within rabbitmq-server that do, for
example, clean up tasks? If so, can these be configured to be less
performance affecting? For a little more info, this is a 3-node VM cluster
with connections from the shovel plugin and a custom python application
using the Pika library.
--
You received this message because you are subscribed to the Google Groups
"rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Alex Corvin]

We're running version 3.4.3 of rabbitmq-server and erlang R14B01. Obviously
this version of erlang is a little dated. Any known issues with it?

Nothing of any note in the logs. I should have noted that we're publishing
messages persistently, and I don't see anything out of the ordinary with
the disk usage behavior of the persistent messages store.

The only strange behavior I've noticed is that when the performance dip
hits, there is a single erlang process that eats up 100% of CPU on a single
core.

For now, I'll continue working to try and identify what that process is
doing.

Post by Alvaro Videla
Anything worth noting from the logs? RabbitMQ and Erlang versions being
run? Perhaps the broker is paging to disk when these events happen?

Post by Alex Corvin
I've been trying to track down the cause of a performance issue I'm
seeing in my RabbitMQ cluster. Every 10 minutes (exactly), I'm seeing
significant (often worse than 50%) drops in event rates that my cluster
handles. Are there any processes that run by default within rabbitmq-server
that do, for example, clean up tasks? If so, can these be configured to be
less performance affecting? For a little more info, this is a 3-node VM
cluster with connections from the shovel plugin and a custom python
application using the Pika library.
--
You received this message because you are subscribed to the Google Groups
"rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Can your app be closing multiple of channels at once? Do you have lots of channels in general?

MK

We're running version 3.4.3 of rabbitmq-server and erlang R14B01. Obviously this version of erlang is a little dated. Any known issues with it?
Nothing of any note in the logs. I should have noted that we're publishing messages persistently, and I don't see anything out of the ordinary with the disk usage behavior of the persistent messages store.
The only strange behavior I've noticed is that when the performance dip hits, there is a single erlang process that eats up 100% of CPU on a single core.
For now, I'll continue working to try and identify what that process is doing.

Anything worth noting from the logs? RabbitMQ and Erlang versions being run? Perhaps the broker is paging to disk when these events happen?

I've been trying to track down the cause of a performance issue I'm seeing in my RabbitMQ cluster. Every 10 minutes (exactly), I'm seeing significant (often worse than 50%) drops in event rates that my cluster handles. Are there any processes that run by default within rabbitmq-server that do, for example, clean up tasks? If so, can these be configured to be less performance affecting? For a little more info, this is a 3-node VM cluster with connections from the shovel plugin and a custom python application using the Pika library.
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Are there any processes that run by default within rabbitmq-server
that do, for example, clean up tasks? If so, can these be configured
to be less performance affecting?

Background GC (interval is self-adjusting, a few minutes by default),
default heartbeat interval is 580 seconds but unless your clients connect all at once,
that's spread in time.

Neither of those is resource-intensive.

RAM and disk limits checks run way more often. So do disk syncs for persistent messages
in durable queues.

Can you have TTL set on queues or messages that is 10 minutes? That may result in lots of
queues and/or messages being deleted at once? 

I'd recommend trying to correlate those events with iostat and vmstat output/metrics.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Alex Corvin]

We run 1 connection/1 channel per instance of our publisher and consumer.
We run multiple instances of the program on a single machine, but no more
than 1 per CPU core. In general, we don't close a channel until program
termination, and we aren't seeing any failure connecting/reconnect attempts.

[Michael Klishin]

Post by Alex Corvin
We run 1 connection/1 channel per instance of our publisher
and consumer. We run multiple instances of the program on a single
machine, but no more than 1 per CPU core. In general, we don't close
a channel until program termination, and we aren't seeing any
failure connecting/reconnect attempts.

It may or may not be https://github.com/rabbitmq/rabbitmq-server/issues/86 then. 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ