[rabbitmq-users] Problem with Spring-amqp, listener stops with 'Consumer received fatal exception on startup',

Discussion:

Wim Veldhuis

2015-05-08 12:43:58 UTC

Hi,

We are running a number of services using AMQP as messaging middleware. We
quite regularly have to restart services as they become unresponsive to
requests from clients.

We see the following in the logfile:

;2015-05-08 00:35:54,578; DEBUG; servicename=; clusterid=; username=;
msguuid=; org.springframework.amqp.rabbit.core.RabbitAdmin - Declarations
finished
;2015-05-08 00:37:12,015; ERROR; servicename=; clusterid=; username=;
msguuid=; nl.pharmapartners.amqp.connectors.AmqpMessageListenerContainer -
Consumer received fatal exception on startup
;org.springframework.amqp.rabbit.listener.FatalListenerStartupException:
Authentication failure
at
org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:367)
at
org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:963)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.amqp.AmqpAuthenticationException:
com.rabbitmq.client.PossibleAuthenticationFailureException: Possibly caused
by authentication failure
at
org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:57)
at
org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:195)
at
org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:359)
at
org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:309)
at
org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:283)
at
org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:276)
at
org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$600(CachingConnectionFactory.java:69)
at
org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:614)
at
org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createChannel(ConnectionFactoryUtils.java:85)
at
org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:134)
at
org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)
at
org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:363)
... 2 more
Caused by: com.rabbitmq.client.PossibleAuthenticationFailureException:
Possibly caused by authentication failure
at
com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:373)
at
com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:516)
at
com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:545)
at
org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:191)
... 12 more
Caused by: com.rabbitmq.client.AlreadyClosedException: clean connection
shutdown; reason: Attempt to use closed channel
at
com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:223)
at
com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:209)
at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:202)
at
com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:355)
... 15 more
2015-05-08 00:37:12,032; ERROR; servicename=; clusterid=; username=;
msguuid=; nl.pharmapartners.amqp.connectors.AmqpMessageListenerContainer -
Stopping container from aborted consumer

We have no real clue what happens, but it looks like a bug in spring amqp
library. A new connection is started but it seems to get an already closed
connection.

Can somebody explain what is happening and possibly how to fix this ?

We are running RabbitMQ 3.3.3/Erlang 17 with Spring amqp version 1.3.2,
amqp-client 3.2.4.

Regards,
Wim Veldhuis.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+***@googlegroups.com.
To post to this group, send an email to rabbitmq-***@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Gary Russell

2015-05-08 13:06:13 UTC

Permalink

Please note that while the Spring team monitors this group from time to
time, we might miss Spring AMQP messages here; we generally provide support
on Stack Overflow (spring-amqp tag).

That said, what makes you believe this is a Spring AMQP problem; clearly
from the stack trace, Spring AMQP is trying to open a connection and, in
connection.start() there is some problem with a channel on that
connection...

Caused by: com.rabbitmq.client.PossibleAuthenticationFailureException:
Possibly caused by authentication failure
at
com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:373)
at
com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:516)
at
com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:545)
at
org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:191)
... 12 more
Caused by: com.rabbitmq.client.AlreadyClosedException: clean connection
shutdown; reason: Attempt to use closed channel
at
com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:223)
at
com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:209)
at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:202)
at
com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:355)
... 15 more

This is entirely within the rabbitmq-client. While it is trying to
authenticate, it finds the channel is closed. Take a look at the server log
for more information. Notice that the exception is a
PossibleAuthenticationFailureException which is an indication that the
problem occurred while trying to authenticate.

Post by Wim Veldhuis
Hi,
We are running a number of services using AMQP as messaging middleware. We
quite regularly have to restart services as they become unresponsive to
requests from clients.
;2015-05-08 00:35:54,578; DEBUG; servicename=; clusterid=; username=;
msguuid=; org.springframework.amqp.rabbit.core.RabbitAdmin - Declarations
finished
;2015-05-08 00:37:12,015; ERROR; servicename=; clusterid=; username=;
msguuid=; nl.pharmapartners.amqp.connectors.AmqpMessageListenerContainer -
Consumer received fatal exception on startup
Authentication failure
at
org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:367)
at
org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:963)
at java.lang.Thread.run(Thread.java:745)
com.rabbitmq.client.PossibleAuthenticationFailureException: Possibly caused
by authentication failure
at
org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:57)
at
org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:195)
at
org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:359)
at
org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:309)
at
org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:283)
at
org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:276)
at
org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$600(CachingConnectionFactory.java:69)
at
org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:614)
at
org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createChannel(ConnectionFactoryUtils.java:85)
at
org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:134)
at
org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)
at
org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:363)
... 2 more
Possibly caused by authentication failure
at
com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:373)
at
com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:516)
at
com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:545)
at
org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:191)
... 12 more
Caused by: com.rabbitmq.client.AlreadyClosedException: clean connection
shutdown; reason: Attempt to use closed channel
at
com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:223)
at
com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:209)
at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:202)
at
com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:355)
... 15 more
2015-05-08 00:37:12,032; ERROR; servicename=; clusterid=; username=;
msguuid=; nl.pharmapartners.amqp.connectors.AmqpMessageListenerContainer -
Stopping container from aborted consumer
We have no real clue what happens, but it looks like a bug in spring amqp
library. A new connection is started but it seems to get an already closed
connection.
Can somebody explain what is happening and possibly how to fix this ?
We are running RabbitMQ 3.3.3/Erlang 17 with Spring amqp version 1.3.2,
amqp-client 3.2.4.
Regards,
Wim Veldhuis.
--
You received this message because you are subscribed to the Google Groups
"rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

Michael Klishin

2015-05-08 13:10:23 UTC

Permalink

Please post Spring-specific questions to StackOverflow, tagged with spring-amqp. Those projects are maintained by a different team.

Thank you.

MK

Hi,
We are running a number of services using AMQP as messaging middleware. We quite regularly have to restart services as they become unresponsive to requests from clients.
;2015-05-08 00:35:54,578; DEBUG; servicename=; clusterid=; username=; msguuid=; org.springframework.amqp.rabbit.core.RabbitAdmin - Declarations finished
;2015-05-08 00:37:12,015; ERROR; servicename=; clusterid=; username=; msguuid=; nl.pharmapartners.amqp.connectors.AmqpMessageListenerContainer - Consumer received fatal exception on startup
;org.springframework.amqp.rabbit.listener.FatalListenerStartupException: Authentication failure
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:367)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:963)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.PossibleAuthenticationFailureException: Possibly caused by authentication failure
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:57)
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:195)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:359)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:309)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:283)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:276)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$600(CachingConnectionFactory.java:69)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:614)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createChannel(ConnectionFactoryUtils.java:85)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:134)
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:363)
... 2 more
Caused by: com.rabbitmq.client.PossibleAuthenticationFailureException: Possibly caused by authentication failure
at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:373)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:516)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:545)
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:191)
... 12 more
Caused by: com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; reason: Attempt to use closed channel
at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:223)
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:209)
at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:202)
at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:355)
... 15 more
2015-05-08 00:37:12,032; ERROR; servicename=; clusterid=; username=; msguuid=; nl.pharmapartners.amqp.connectors.AmqpMessageListenerContainer - Stopping container from aborted consumer
We have no real clue what happens, but it looks like a bug in spring amqp library. A new connection is started but it seems to get an already closed connection.
Can somebody explain what is happening and possibly how to fix this ?
We are running RabbitMQ 3.3.3/Erlang 17 with Spring amqp version 1.3.2, amqp-client 3.2.4.
Regards,
Wim Veldhuis.
--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
For more options, visit https://groups.google.com/d/optout.

Wim Veldhuis

2015-05-09 13:36:18 UTC

Permalink

My excuses for posting in the wrong group, I will also post in the group
indicated.

Just to give some additional information, the problem is the service is
running for quite a while, and then suddenly stops. Other services using
the same credentials running on the same system have no problem. We see
this in more services we run. Restarting the service 'cures' the problem,
everything works again, but it involves manual interception of the problem.'
Also the original exeption is that we are trying to use a closed channel,
and we are wondering why the channel might have been closed. It feels, for
us at least, that shutting down all listeners when one channel fails is a
bit strange. Why shutdown everything if there is only one channel with a
problem ?

Since posting we found that we are running a cached listener with a
cachesize of one (default), prefetch of 1 and possibly 5 concurrent
threads. Not a very good configuration I think, but that is a different
issue.

Regards,
Wim.

Gary Russell

2015-05-09 20:06:53 UTC

Permalink

Spring AMQP changed authentication errors to be fatal after in 1.2.1, as a
result of user requests (no point in keep trying if the credentials are
bad).

However, it appears there is a race condition in the rabbitmq-client in
that if a connection is closed right after it opens (before
authentication), it is reported as a PossibleAuthenticationFailureException.

We could add an option to treat PossibleAuthenticationFailureException as
non-fatal (and keep trying). Feel free to open up a JIRA issue
<https://jira.spring.io/browse/AMQP> and we'll take a look.

Michael Klishin

2015-05-09 20:18:17 UTC

Permalink

Post by Gary Russell
However, it appears there is a race condition in the rabbitmq-client
in that if a connection is closed right after it opens (before
authentication), it is reported as a PossibleAuthenticationFailureException.

That's not really a race condition: the protocol dictates that authentication
failures must result in a closed TCP connection. RabbitMQ has an extension that
reports them using a separate method, but that was only added in 3.x.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ