[rabbitmq-users] Messages Stuck in Unacked

[Andrew Kowalik]

Stack:
Heroku
Celery
Rabbitmq

On some of our queues we are using acks_late=True in Celery. When True the
worker will not acknowledge until the job is completed. Since these workers
are run on Heroku, its possible for the process to be killed while a job is
running. When this happens, the message remains in unacked. I can even turn
off all workers and the message is still in unacked status. Rabbitmq will
report 0 consumers in this case. Trying to get these messages to be reset
to allow for reprocessing.

Any thoughts? Thanks!

[Michael Klishin]

Post by Andrew Kowalik
On some of our queues we are using acks_late=True in Celery.
When True the worker will not acknowledge until the job is completed.
Since these workers are run on Heroku, its possible for the process
to be killed while a job is running. When this happens, the message
remains in unacked. I can even turn off all workers and the message
is still in unacked status. Rabbitmq will report 0 consumers
in this case. Trying to get these messages to be reset to allow
for reprocessing.
Any thoughts?

We don't cover Celery or Heroku add-on on this list.

As tutorial 2 [1]
should mention, dropped connections (technically: closed channels) will result in unacknowledged messages
being re-queued. Perhaps it takes a while to detect peer loss (shouldn't be if the process
is killed, but who knows), so take a look at using a lower (6-10 seconds) heartbeat timeout
value [2]. Otherwise I'd expect that Celery is trying to be too smart.

 Note that the number of consumers
is irrelevant for deciding whether to re-queue unacknowledged deliveries, all that matters is whether the channel (or its entire connection) is still open.

1. http://www.rabbitmq.com/getstarted.html
2. http://rabbitmq.com/heartbeats.html
--
MK

Staff Software Engineer, Pivotal/RabbitMQ