[rabbitmq-users] rabbitmq and celery, Operation timed out and Broken pipe errors

[Pim Linders]

Appreciate any help on this as i've been stuck on this for some time.

I'm using rabbitmq with celery (http://www.celeryproject.org/) and running
into the following errors a few minutes after making an initial request

[Errno 60] Operation timed out
[Errno 32] Broken pipe

If I continue to make requests it eventually creates a new connection but
the old connection is never closed. Is there anything I can do to keep the
connection open or prevent it from timing out?

[Michael Klishin]

Post by Pim Linders
[Errno 60] Operation timed out
[Errno 32] Broken pipe
If I continue to make requests it eventually creates a new connection
but the old connection is never closed. Is there anything I can
do to keep the connection open or prevent it from timing out?

We can’t suggest much from just those two messages. We are not Celery experts
either, I’d recommend asking on the Celery list first.

Things to check first:

 * can you obtain full stack traces?
 *  was your cluster in a resource-drive alarm state?
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Pim Linders]

This post might be inappropriate. Click to display it.

[Michael Klishin]

File "/Users/pim/virtualenvs/nearbybuzz/lib/python2.7/site-packages/kombu/entity.py",
line 514, in declare
self.exchange.declare(nowait)
File "/Users/pim/virtualenvs/nearbybuzz/lib/python2.7/site-packages/kombu/entity.py",
line 174, in declare
nowait=nowait, passive=passive,
File "/Users/pim/virtualenvs/nearbybuzz/lib/python2.7/site-packages/amqp/channel.py",
line 613, in exchange_declare
self._send_method((40, 10), args)
File "/Users/pim/virtualenvs/nearbybuzz/lib/python2.7/site-packages/amqp/abstract_channel.py",
line 56, in _send_method
self.channel_id, method_sig, args, content,
File "/Users/pim/virtualenvs/nearbybuzz/lib/python2.7/site-packages/amqp/method_framing.py",
line 221, in write_method
write_frame(1, channel, payload)
File "/Users/pim/virtualenvs/nearbybuzz/lib/python2.7/site-packages/amqp/transport.py",
line 182, in write_frame
frame_type, channel, size, payload, 0xce,
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py",
line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 60] Operation timed out [Errno 32] Broken pipe

What’s in RabbitMQ logs? If I understand the log correctly, it is a socket write timeout.

I’m not sure what values Kombu uses for them but there can be legitimate reasons for that. 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Pim Linders]

Here's what I see in the logs when I make my request, wait a few minutes
and send more request. Eventually another connection is established.

=INFO REPORT==== 7-Jul-2015::16:16:41 ===

accepting AMQP connection <0.1954.0> (96.241.52.177:62175 ->
100.72.154.68:5672)

=INFO REPORT==== 7-Jul-2015::16:24:04 ===

accepting AMQP connection <0.2089.0> (96.241.52.177:62514 ->
100.72.154.68:5672)


Going through the logs I did find

=WARNING REPORT==== 7-Jul-2015::15:59:51 ===

closing AMQP connection <0.1642.0> (52.7.174.94:51321 -> 100.72.154.68:5672
):

connection_closed_abruptly




--
Pim Linders
Data Focus

Post by Pim Linders

Post by Pim Linders
File

"/Users/pim/virtualenvs/nearbybuzz/lib/python2.7/site-packages/kombu/entity.py",

Post by Pim Linders
line 514, in declare
self.exchange.declare(nowait)
File

"/Users/pim/virtualenvs/nearbybuzz/lib/python2.7/site-packages/kombu/entity.py",

Post by Pim Linders
line 174, in declare
nowait=nowait, passive=passive,
File

"/Users/pim/virtualenvs/nearbybuzz/lib/python2.7/site-packages/amqp/channel.py",

Post by Pim Linders
line 613, in exchange_declare
self._send_method((40, 10), args)
File

"/Users/pim/virtualenvs/nearbybuzz/lib/python2.7/site-packages/amqp/abstract_channel.py",

Post by Pim Linders
line 56, in _send_method
self.channel_id, method_sig, args, content,
File

"/Users/pim/virtualenvs/nearbybuzz/lib/python2.7/site-packages/amqp/method_framing.py",

Post by Pim Linders
line 221, in write_method
write_frame(1, channel, payload)
File

"/Users/pim/virtualenvs/nearbybuzz/lib/python2.7/site-packages/amqp/transport.py",

Post by Pim Linders
line 182, in write_frame
frame_type, channel, size, payload, 0xce,
File

"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py",

Post by Pim Linders
line 224, in meth
return getattr(self._sock,name)(*args)
error: [Errno 60] Operation timed out [Errno 32] Broken pipe

What’s in RabbitMQ logs? If I understand the log correctly, it is a socket
write timeout.
I’m not sure what values Kombu uses for them but there can be legitimate
reasons for that.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Post by Pim Linders
Here's what I see in the logs when I make my request, wait a few
minutes and send more request. Eventually another connection
is established.

There are no alarms in the log, and eventually one of the TCP connections
is closed. I’m afraid this is a client-side issue that I can’t explain
(or suggest a workaround for). 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ