[rabbitmq-users] Unacknowledged messages lost when client resumes.

[Jesse Bexten]

Ok, I'm going to to my best to explain this issue. We have 2 rabbit
instances running.

One runs locally with our application in order to have a persistent local
store for messages. This local installation has a single queue with a
shovel attached to it. All messages are then shovel to our second rabbit
installation on our queue server. This server has a topic exchange and
several queues that are read by various clients in our company.

The issue we are having is this. We are testing the various failure
scenarios for our queue setup. If we destroy the queue server with vagrant
while messages are being sent, then we have some un-acked messages on our
rabbit instance on the application server. However when we recreate the
queue server, it appears that the unacked messages are not re-sent to the
queue server by the shovel, they are simply lost. We expected the messages
to be re-queued when the server was destroyed.

Our shovel configuration is as follows.

{rabbitmq_shovel,
[ {shovels, [ {say_shovel,
[ {sources,
[ {brokers, [ "amqp://"]}
, {declarations, [ {'exchange.declare',
[ {exchange, <<"shove_it">>}
, {type, <<"topic">>}
, durable
]}
, {'queue.declare',
[{queue,<<"shove_q">>},
{arguments,
[{<<"x-message-ttl">>, long, 60000}]}]}
, {'queue.bind',
[ {exchange, <<"shove_it">>},
{queue, <<"shove_q">>},
{routing_key,<<"#">>}
]}
]}
]}
, {destinations,
[ {broker, "amqp://shovel:hostname:5672/%2F"}
, {declarations, []}
]}
, {queue, <<"shove_q">>}
, {prefetch_count, 10}
, {ack_mode, on_confirm}
, {publish_properties, [ {delivery_mode, 2} ]}
, {add_forward_headers, true}
, {publish_fields, [ {exchange, <<"exchange_name">>}]}
, {reconnect_delay, 5}
]}
]}
]},

[Michael Klishin]

Post by Jesse Bexten
The issue we are having is this. We are testing the various failure
scenarios for our queue setup. If we destroy the queue server
with vagrant while messages are being sent, then we have some
un-acked messages on our rabbit instance on the application
server. However when we recreate the queue server, it appears
that the unacked messages are not re-sent to the queue server
by the shovel, they are simply lost. We expected the messages
to be re-queued when the server was destroyed.

Please use the Shovel terminology instead of "queue server" and "rabbit instance". What node
are you terminating: the one that has messages you'd like to move (upstream)
or the target node (downstream)?

Shovel keeps messages in memory. You use the default (safest) ack_mode [1].
This means that Shovel node failure should re-queue messages upstream.

1. https://www.rabbitmq.com/federation-reference.html
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Jesse Bexten]

We are terminating the downstream node. The shovel is running on the
upstream node. We see unacked messages on the upstream node when the
downstream node is destroyed, but these are not re-queued. When we recreate
the downstream node, the messages appear to be lost. They do not arrive in
the downstream queue.

We are using a static shovel, not federation, so the reference you linked
doesn't appear relevant to me. I have read all the information here

https://www.rabbitmq.com/shovel-static.html

From what I can tell the ack_modes are no_ack, on_publish, and on_confirm.
We are using the on_confirm mode as you can see from the shovel
configuration.

[Michael Klishin]

Post by Jesse Bexten
We are terminating the downstream node. The shovel is running
on the upstream node. We see unacked messages on the upstream
node when the downstream node is destroyed, but these are not
re-queued. When we recreate the downstream node, the messages
appear to be lost. They do not arrive in the downstream queue.

What's in the downstream node log files? Did it even notice the connection loss?
Have you considered lowering the heartbeat? If you pull the plug on a peer, the other
end does not detect this immediately.

http://www.rabbitmq.com/heartbeats.html
http://www.rabbitmq.com/networking.html

Shovel doesn't delete or purge queues when it comes up, so the queue
is likely removed when consumer connection is lost (e.g. auto-deleted or exclusive).
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Jesse Bexten]

Ok, sorry for all the mixup on terminology. It turned out that I was
overlooking the fact that we had a message ttl specified for the source
queue on the shovel. This was causing our messages to expire while
recovering the destination queue server.

Thank you very much for the help and quick answers