[rabbitmq-users] RabbitMQ channel suddenly closing

[Carleen]

I'm having a problem with my channels suddenly shutting down. I have a
shutdown listener, and the error message only says (forgive any errors - I
have to retype this):

connection error; reasons {#method<connection.close>(reply-code=320,
reply-text=CONNECTION_FORCED - broker connection closure with reason
'shutdown', class-id=0, method-id=0), null, ""}

In my code I have a catch for the AlreadyClosedException, and try to create
a new channel like so:

} catch (AlreadyClosedException e) {
Connection connection = channel.getConnection();
channel = connection.createChannel();
}

This seems to have no effect. I still continuously get the
AlreadyClosedException until I restart the broker. The server log (which I
can't post because it's on another network) indicates that rabbitmq is
stopping listening, and then shows the same error message that the shutdown
listener is giving.

Thanks for any help.

[Michael Klishin]

connection error; reasons {#method(reply-code=320,
reply-text=CONNECTION_FORCED - broker connection closure
with reason 'shutdown', class-id=0, method-id=0), null, ""}

This means RabbitMQ was told to shut down (by a human or another program). 

In my code I have a catch for the AlreadyClosedException, and
} catch (AlreadyClosedException e) {
Connection connection = channel.getConnection();
channel = connection.createChannel();
}
This seems to have no effect

You need to register a shutdown hook. connection.close is dispatched
in a different thread.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Carleen]

Do you mean the ShutdownListener? I have registered that. Is there
something I should be doing inside the listener, other than printing the
error message?

I can't see how rabbit was told to shutdown though. We have a cluster of 30
boxes, and this was the only one to have a problem. And it happened in the
middle of the night.

Post by Michael Klishin

connection error; reasons {#method(reply-code=320,
reply-text=CONNECTION_FORCED - broker connection closure
with reason 'shutdown', class-id=0, method-id=0), null, ""}

This means RabbitMQ was told to shut down (by a human or another
program).

In my code I have a catch for the AlreadyClosedException, and
} catch (AlreadyClosedException e) {
Connection connection = channel.getConnection();
channel = connection.createChannel();
}
This seems to have no effect

You need to register a shutdown hook. connection.close is dispatched
in a different thread.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ

[Simon MacMullen]

Post by Carleen
I can't see how rabbit was told to shutdown though. We have a cluster of
30 boxes, and this was the only one to have a problem. And it happened
in the middle of the night.

The server logs will have more information on this. The client will only
know that the server is shutting down, it doesn't get told why.

Cheers, Simon

[Carleen]

Unfortunately the server log only gives me the same message that the client
did.

Post by Simon MacMullen

Post by Carleen
I can't see how rabbit was told to shutdown though. We have a cluster of
30 boxes, and this was the only one to have a problem. And it happened
in the middle of the night.

The server logs will have more information on this. The client will only
know that the server is shutting down, it doesn't get told why.
Cheers, Simon

[Michael Klishin]

Post by Carleen
Unfortunately the server log only gives me the same message
that the client did.

Try correlating that message with syslog. Perhaps the entire OS was shut down/rebooted?
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Carleen]

I know the OS was up because we have other services on that box that didn't
have a problem.

Post by Michael Klishin

Post by Carleen
Unfortunately the server log only gives me the same message
that the client did.

Try correlating that message with syslog. Perhaps the entire OS was shut down/rebooted?
--
MK
Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Post by Carleen
Do you mean the ShutdownListener? I have registered that. Is
there something I should be doing inside the listener, other
than printing the error message?

Your entire connection is closed. I am guessing you'd want to do something other
than logging, or use automatic recovery (which recovers from server-sent connection.close
by default):

https://www.rabbitmq.com/api-guide.html
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Carleen]

I'm currently on an older version of Rabbit that doesn't have the
auto-recover in the connection factory. Can you provide me some sample code
you'd recommend for this shutdown hook and how to restart the connection?

Post by Michael Klishin

Post by Carleen
Do you mean the ShutdownListener? I have registered that. Is
there something I should be doing inside the listener, other
than printing the error message?

Your entire connection is closed. I am guessing you'd want to do something other
than logging, or use automatic recovery (which recovers from server-sent connection.close
https://www.rabbitmq.com/api-guide.html
--
MK
Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Post by Carleen
I'm currently on an older version of Rabbit that doesn't have
the auto-recover in the connection factory. Can you provide
me some sample code you'd recommend for this shutdown hook and
how to restart the connection?

You can use the most recent client with older servers (even 2.x). This is certainly
recommended. 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Yes, and in our next deployment we will upgrade, but for what
I have fielded now, I can't use a newer client.

The docs explain what automatic recovery does. It's not rocket science. You can
recover your connection and channel(s) with fairly low amount of work, so I'd try with
that and see if you need to recover topology (also explained in the Java client docs).
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Carleen]

I'm sorry I'm still not understanding something. The version of the api I'm
using doesn't have

factory.setAutomaticRecoveryEnabled(true);

So what do you recommend I do instead?

Post by Michael Klishin

Yes, and in our next deployment we will upgrade, but for what
I have fielded now, I can't use a newer client.

The docs explain what automatic recovery does. It's not rocket science. You can
recover your connection and channel(s) with fairly low amount of work, so I'd try with
that and see if you need to recover topology (also explained in the Java client docs).
--
MK
Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Post by Carleen
So what do you recommend I do instead?

Automatic recovery in more recent clients uses a shutdown listener to do basically
the following:

 * Recover connection and its state (shutdown listeners)
 * Recover all channels and their state (shutdown listeners, basic.qos prefetch, publisher confirms, etc)
 * Recover topology (queues, exchanges, bindings, consumers) if enabled

I recommending doing the first two items, and, if needed, the last one. You can use
Java client code as an example, except for topology recovery it's fairly  straightforward:

http://hg.rabbitmq.com/rabbitmq-java-client/file/ab27b7201868/src/com/rabbitmq/client/impl/recovery/AutorecoveringConnection.java#l345
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Carleen]

I really appreciate your patience (*smile*), but it's still not clear to me
where to implement this. My version of the API (version 3.2.1) doesn't have
AutorecoveringConnection. Are you telling me to look at its code to
implement somewhere myself? If so, which part of the code and where do I
put it? In the ShutdownListener?

Post by Michael Klishin

Post by Carleen
So what do you recommend I do instead?

Automatic recovery in more recent clients uses a shutdown listener to do basically
* Recover connection and its state (shutdown listeners)
* Recover all channels and their state (shutdown listeners, basic.qos
prefetch, publisher confirms, etc)
* Recover topology (queues, exchanges, bindings, consumers) if enabled
I recommending doing the first two items, and, if needed, the last one. You can use
Java client code as an example, except for topology recovery it's fairly
http://hg.rabbitmq.com/rabbitmq-java-client/file/ab27b7201868/src/com/rabbitmq/client/impl/recovery/AutorecoveringConnection.java#l345
--
MK
Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

My version of the API (version 3.2.1) doesn't have AutorecoveringConnection.
Are you telling me to look at its code to implement somewhere myself? 

 Yes.

If so, which part of the code and where do I put it? In the ShutdownListener?

Yes, this is what automatic recovery does, as can be seen from
http://hg.rabbitmq.com/rabbitmq-java-client/file/ab27b7201868/src/com/rabbitmq/client/impl/recovery/AutorecoveringConnection.java#l345
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Carleen]

In the ShutdownListener, how do I get access to the ConnectionFactory to
restart the connection? I only have the Channel.

Post by Michael Klishin

Post by Carleen
My version of the API (version 3.2.1) doesn't have

AutorecoveringConnection.

Post by Carleen
Are you telling me to look at its code to implement somewhere myself?

Yes.

Post by Carleen
If so, which part of the code and where do I put it? In the

ShutdownListener?
Yes, this is what automatic recovery does, as can be seen from
http://hg.rabbitmq.com/rabbitmq-java-client/file/ab27b7201868/src/com/rabbitmq/client/impl/recovery/AutorecoveringConnection.java#l345
--
MK
Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Post by Carleen
In the ShutdownListener, how do I get access to the ConnectionFactory
to restart the connection? I only have the Channel.

final ConnectionFactory cf = …;
Connection conn = …;

conn.addShutdownListener(new ShutdownListener() {
  public void shutdownCompleted(ShutdownSignalException cause) {
    // access cf here
  }
})
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Carleen]

Ok, I'm onboard now! I just had to figure out how to get the
ConnectionFactory to my class. THANK YOU SO MUCH for all your help!

Carleen

Post by Carleen
In the ShutdownListener, how do I get access to the ConnectionFactory
to restart the connection? I only have the Channel.

final ConnectionFactory cf = 
;
Connection conn = 
;
conn.addShutdownListener(new ShutdownListener() {
public void shutdownCompleted(ShutdownSignalException cause) {
// access cf here
}
})
--
MK
Staff Software Engineer, Pivotal/RabbitMQ