Discussion:
[rabbitmq-discuss] Blocking Queue - Close connection by client
Claire Fautsch
2014-03-03 08:18:06 UTC
Permalink
Hi,

we have following issue with blocking queues:

Once the memory watermark is reached, the connection from the client gets
blocked by the RabbitMQ server and it is no longer possible to publish
messages.

If this happens, we are for some time still able to publish messages,
however they remain in the socket's write buffer and are not actually
published until the connection is un-blocked again. Once this buffer is
full, publishing blocks completely.

After some research we found out, that with a newer version of the official
Java client, it is possible to use blocked connection notifications to
react on this situation.

Our prefered reaction would be to close the connection, and re-connect to
another broker. Here however comes the problem: It is not possible to close
the connection to the RabbitMQ, as the method to do so, involves a flush on
the socket before the socket is closed and this is blocked as well.

Is there any possibility for a force-closed on the client side? Or is the
only possibility to open a new connection without closing the old one, and
"let it die"?

Re-writing the SocketFrameHandler would of course also be a possibility,
but one we would (if possible) like to avoid. (see also discussion here :
https://github.com/rabbitmq/rabbitmq-java-client/issues/11)


Additionally, when we see blocking connections occurring, we frequently see
following stack trace in our logs, where we are not absolutely sure where
it comes from, or rather why it occurs

com.rabbitmq.client.ShutdownSignalException: connection error; reason:
java.io.EOFException

at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)

at
com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:37)

at
com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:349)

at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:569)

at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:501)

...

Caused by: java.io.EOFException

at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:494)at
java.io.DataInputStream.readUnsignedByte(Unknown Source)

at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)

at
com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)

at
com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:515)


For information, we are currently using Java client,
com.rabbitmq:amqp-client version 3.1.4

and com.zanox.lib.rabbiteasy version:rabbiteasy-core version 1.2.0, for the
connection handling (SingleConnectionFactory) and publishing.

Thanks in advance for any hint or ideas.

Regards

Claire
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140303/0218e162/attachment.html>
Michael Klishin
2014-03-03 09:04:10 UTC
Permalink
After some research we found out, that with a newer version of the official Java client, it is possible to use blocked connection notifications to react on this situation.
Our prefered reaction would be to close the connection, and re-connect to another broker. Here however comes the problem: It is not possible to close the connection to the RabbitMQ, as the method to do so, involves a flush on the socket before the socket is closed and this is blocked as well.
Is there any possibility for a force-closed on the client side? Or is the only possibility to open a new connection without closing the old one, and ?let it die??
Connection#abort is the same as Connection#close but ignores IOExceptions (EOFExceptions included).

This is just a workaround, though. Adding more consumer capacity to avoid frequently blocked connections
would be a better idea.

See also https://www.rabbitmq.com/blog/2014/01/23/preventing-unbounded-buffers-with-rabbitmq/
Re-writing the SocketFrameHandler would of course also be a possibility, but one we would (if possible) like to avoid. (see also discussion here : https://github.com/rabbitmq/rabbitmq-java-client/issues/11)
So far the conclusion is that it will involve fairly significant changes because it?d be necessary to switch to NIO.
Additionally, when we see blocking connections occurring, we frequently see following stack trace in our logs, where we are not absolutely sure where it comes from, or rather why it occurs
com.rabbitmq.client.ShutdownSignalException: connection error; reason: java.io.EOFException
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:37)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:349)
at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:569)
at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:501)
You can inspect the cause exception?s trace (ShutdownSignalException#getCause).

MK

Software Engineer, Pivotal/RabbitMQ
Claire Fautsch
2014-04-10 06:15:41 UTC
Permalink
Hi Michael and all,

Unfortunatelly calling the abort function instead of close, will show
exactly the same behaviour, as there is still a flush at the end (I also
tested this, and indeed not a solution). The only difference is that
Exceptions/Stacktraces are ignroed.
The only option I see so far to avoid hanging here, is to set a timeout to
the close method.

Regards
Claire
Post by Claire Fautsch
Hi,
Once the memory watermark is reached, the connection from the client gets
blocked by the RabbitMQ server and it is no longer possible to publish
messages.
If this happens, we are for some time still able to publish messages,
however they remain in the socket's write buffer and are not actually
published until the connection is un-blocked again. Once this buffer is
full, publishing blocks completely.
After some research we found out, that with a newer version of the
official Java client, it is possible to use blocked connection
notifications to react on this situation.
Our prefered reaction would be to close the connection, and re-connect to
another broker. Here however comes the problem: It is not possible to close
the connection to the RabbitMQ, as the method to do so, involves a flush on
the socket before the socket is closed and this is blocked as well.
Is there any possibility for a force-closed on the client side? Or is the
only possibility to open a new connection without closing the old one, and
"let it die"?
Re-writing the SocketFrameHandler would of course also be a possibility,
https://github.com/rabbitmq/rabbitmq-java-client/issues/11)
Additionally, when we see blocking connections occurring, we frequently
see following stack trace in our logs, where we are not absolutely sure
where it comes from, or rather why it occurs
java.io.EOFException
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
at
com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:37)
at
com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:349)
at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:569)
at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:501)
...
Caused by: java.io.EOFException
at com.rabbitmq.client.impl.ChannelN.close(ChannelN.java:494)at
java.io.DataInputStream.readUnsignedByte(Unknown Source)
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
at
com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)
at
com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:515)
For information, we are currently using Java client,
com.rabbitmq:amqp-client version 3.1.4
and com.zanox.lib.rabbiteasy version:rabbiteasy-core version 1.2.0, for
the connection handling (SingleConnectionFactory) and publishing.
Thanks in advance for any hint or ideas.
Regards
Claire
--
*Claire Fautsch*
Server Developer
cfautsch at goodgamestudios.com

Goodgame Studios
Theodorstr. 42-90, House 9
22761 Hamburg, Germany
Phone: +49 (0)40 219 880 -0

*www.goodgamestudios.com <http://www.goodgamestudios.com>*


Goodgame Studios is a branch of Altigi GmbH
Altigi GmbH, District court Hamburg, HRB 99869
Board of directors: Dr. Kai Wawrzinek, Dr. Christian Wawrzinek, Fabian
Ritter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20140410/dab44a5d/attachment.html>
Michael Klishin
2014-04-14 12:03:56 UTC
Permalink
Post by Claire Fautsch
Unfortunatelly calling the abort function instead of close,
will show exactly the same behaviour, as there is still a flush
at the end (I also tested this, and indeed not a solution). The
only difference is that Exceptions/Stacktraces are ignroed.
The only option I see so far to avoid hanging here, is to set a timeout
to the close method.
Your observations are correct.

N.B. if you have a connection blocked, opening a new one will only help if the
new connection doesn?t publish anything. Which suggests you can separate connections
that consume and publish from the start.?
--
MK

Software Engineer, Pivotal/RabbitMQ

Loading...