[rabbitmq-discuss] Messages unexpectedly moving from Unacked to Ready

[Patrick Long]

We are seeing messages occasionally moving from Unacked back to Ready
whilst we are busy process them

Our windows service dequeues

channel.BasicGet(workQueueName, *false*);

(Note NoAck being set to false.)

Once we finish processing the contents of the message we will call

channel.BasicAck(message.DeliveryTag, false)

Some messages are being moved back to Ready in the test environment even
without us explicitly called BasciAck

The test environment where we are seeing this has a Load Balancer in from
of the 2 node RabbitMq cluster

Does anyone have any idea what would be causing the message to go back to
Ready?

Thanks

[Simon MacMullen]

Post by Patrick Long
Does anyone have any idea what would be causing the message to go back
to Ready?

One of three things (in roughly descending order of likelihood):

* if the client closes its connection / channel without acknowledging
the message

* if the client sends basic.reject or basic.nack with requeue=true for
the message

* if the client calls basic.recover without acknowledging the message

Cheers, Simon

[Simon MacMullen]

Post by Simon MacMullen
* if the client closes its connection / channel without acknowledging
the message

Of course the client might not be the one deciding this, the load
balancer might close the connection for some reason.

Cheers, Simon

[Patrick Long]

Ah so that makes it sound like the connection does not need to be closed by
the specific channel for that to happen

Post by Simon MacMullen
* if the client closes its connection / channel without acknowledging
the message

Of course the client might not be the one deciding this, the load balancer
might close the connection for some reason.
Cheers, Simon
--
Simon MacMullen
RabbitMQ, Pivotal

[Simon MacMullen]

The only way I could see it happen was if one of the other processing
threads looking at the _*same *_queue called close. However they do not
share channels and I assume that closing on a channel different to the
one that retrieved the message would not effect the Unacked state, correct?

That's correct. Closing a channel releases all the unacked messages on
that channel...

Ah so that makes it sound like the connection does not need to be
closed by the specific channel for that to happen

...and closing a connection implicitly closes all the channels on that
connection, releasing all their unacked messages.

Cheers, Simon

[Patrick Long]

The load balancer closing connections would look something like this in the
RMQ logs, yes?

=WARNING REPORT==== 30-Jan-2014::15:25:08 ===
closing AMQP connection <0.31743.12> (10.206.20.1:2750 -> 10.206.20.167:5672
):
connection_closed_abruptly

Those happen between 1 and 3 times a minute. Is this a common problem? We
obviously need load balancing to make use of clustering.

Is there a way of keeping the connection Alive?

Pat

Post by Simon MacMullen

The only way I could see it happen was if one of the other processing
threads looking at the _*same *_queue called close. However they do not
share channels and I assume that closing on a channel different to the
one that retrieved the message would not effect the Unacked state, correct?

That's correct. Closing a channel releases all the unacked messages on
that channel...

Ah so that makes it sound like the connection does not need to be
closed by the specific channel for that to happen

...and closing a connection implicitly closes all the channels on that
connection, releasing all their unacked messages.
Cheers, Simon
--
Simon MacMullen
RabbitMQ, Pivotal

[Simon MacMullen]

Post by Patrick Long
The load balancer closing connections would look something like this in
the RMQ logs, yes?
=WARNING REPORT==== 30-Jan-2014::15:25:08 ===
closing AMQP connection <0.31743.12> (10.206.20.1:2750
<http://10.206.20.1:2750> -> 10.206.20.167:5672
connection_closed_abruptly

Yes.

Post by Patrick Long
Those happen between 1 and 3 times a minute. Is this a common problem?
We obviously need load balancing to make use of clustering.
Is there a way of keeping the connection Alive?

That's down to how the load balancer is configured. If the load balancer
is closing connections because it perceives them to be inactive,
lowering the AMQP heartbeat interval might prevent you from having to
reconfigure the load balancer.

Cheers, Simon

[Laing, Michael]

20 seconds is a good heartbeat interval for most load balancers that I have
tried. -ml

Post by Patrick Long

Post by Patrick Long
The load balancer closing connections would look something like this in
the RMQ logs, yes?
=WARNING REPORT==== 30-Jan-2014::15:25:08 ===
closing AMQP connection <0.31743.12> (10.206.20.1:2750
<http://10.206.20.1:2750> -> 10.206.20.167:5672
connection_closed_abruptly

Yes.
Those happen between 1 and 3 times a minute. Is this a common problem?

Post by Patrick Long
We obviously need load balancing to make use of clustering.
Is there a way of keeping the connection Alive?

That's down to how the load balancer is configured. If the load balancer
is closing connections because it perceives them to be inactive, lowering
the AMQP heartbeat interval might prevent you from having to reconfigure
the load balancer.
Cheers, Simon
--
Simon MacMullen
RabbitMQ, Pivotal
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140130/e8bc0691/attachment.html>

[Patrick Long]

Looks like it was the load balancer (LB), as you suggested, because when we
configured the code to go direct at a specific node in the cluster rather
than via the LB it works as expected.

Wrt the AMQP heartbeat. We are using the official >NET client and I am now
setting the RequestedHeartbeat property, on the factory, to 10 (seconds).

Is that how I control that heartbeat? It was defaulting 0.

Thanks

Patrick

Post by Patrick Long

Post by Patrick Long
The load balancer closing connections would look something like this in
the RMQ logs, yes?
=WARNING REPORT==== 30-Jan-2014::15:25:08 ===
closing AMQP connection <0.31743.12> (10.206.20.1:2750
<http://10.206.20.1:2750> -> 10.206.20.167:5672
connection_closed_abruptly

Yes.
Those happen between 1 and 3 times a minute. Is this a common problem?

Post by Patrick Long
We obviously need load balancing to make use of clustering.
Is there a way of keeping the connection Alive?

That's down to how the load balancer is configured. If the load balancer
is closing connections because it perceives them to be inactive, lowering
the AMQP heartbeat interval might prevent you from having to reconfigure
the load balancer.
Cheers, Simon
--
Simon MacMullen
RabbitMQ, Pivotal

[Matthias Radestock]

Post by Patrick Long
Wrt the AMQP heartbeat. We are using the official >NET client and I am
now setting the RequestedHeartbeat property, on the factory, to 10
(seconds).
Is that how I control that heartbeat? It was defaulting 0.

The hearbeat can be controlled server-side too. See the 'heartbeat'
config item in http://www.rabbitmq.com/configure.html#config-items

Most clients, the .net client included, will take the server's
suggestion when that is lower than their own (or their own is 0 i.e.
infinity).

Matthias.