[rabbitmq-users] Unable to delete a specific queue after RabbitMQ cluster recovery

[Kirill Lutarevich]

I have a cluster of 2 servers (3.4.1) where one node just failed and left
the cluster. I recovered it by restarting the server and it rejoined the
cluster, all queues (durable) got synchronized except one. That queue was
still inaccessible by applications on both publisher and subscriber side. I
rerouted them to a different queue and tried to delete that one but all
delete requests (via UI or rabbitmqadmin) hanged. I can still see it in
Admin UI and it shows as not synchronized, binding removed and ? ? ? in
Messages section. How can I delete that queue and create a new one with the
original name?

[Jean-Sébastien Pédron]

(...) I rerouted them to a different queue and tried to delete that
one but all delete requests (via UI or rabbitmqadmin) hanged.

Hi!

Could you please post your log files (both "normal" and -sasl), as well
as the output from "rabbitmqctl report"?

The output of "rabbitmqctl report" contains informations such as
hostnames, IP addresses, resources names, etc. If you don't want to
disclose them, feel free to replace sensitive data and/or send the
report privately to me.

[Kirill Lutarevich]

Hi
Considering the names of RabbitMQ hosts (rabbitmqc and rabbitmqd) I
attached normal log files (current sasl logs are empty on both)

"rabbitmqctl report" hangs on both machines after showing the information
below (after printing out *Channels:*)

*rabbitmqc:*

*<***@rabbitmqc.2.17888.308> 10.5.31.185:54226 -> 10.5.15.18:5672
5672 54226 10.5.15.18 10.5.31.185 false
PLAIN {0,9,1} User workflow 580 131072 0
[{"product","RabbitMQ"},{"version","3.4.1.0"},{"platform",".NET"},{"copyright","Copyright
(C) 2007-2014 GoPivotal, Inc."},{"information","Licensed under the MPL.
See
http://www.rabbitmq.com/"},{"capabilities",[{"publisher_confirms",true},{"exchange_exchange_bindings",true},{"basic.nack",true},{"consumer_cancel_notify",true},{"connection.blocked",true},{"authentication_failure_close",true}]}]
1437667989861 664 9 654 9 0 running 1*
*<***@rabbitmqc.2.4417.309> 10.5.25.5:50019 -> 10.5.15.18:5672
5672 50019 10.5.15.18 10.5.25.5 false
PLAIN {0,9,1} User workflow 580 131072 0
[{"product","RabbitMQ"},{"version","3.4.1.0"},{"platform",".NET"},{"copyright","Copyright
(C) 2007-2014 GoPivotal, Inc."},{"information","Licensed under the MPL.
See
http://www.rabbitmq.com/"},{"capabilities",[{"publisher_confirms",true},{"exchange_exchange_bindings",true},{"basic.nack",true},{"consumer_cancel_notify",true},{"connection.blocked",true},{"authentication_failure_close",true}]}]
1437668170375 632 9 646 9 0 running 1*
*<***@rabbitmqc.2.5730.309> 10.5.25.5:50021 -> 10.5.15.18:5672
5672 50021 10.5.15.18 10.5.25.5 false
PLAIN {0,9,1} User workflow 580 131072 0
[{"product","RabbitMQ"},{"version","3.4.1.0"},{"platform",".NET"},{"copyright","Copyright
(C) 2007-2014 GoPivotal, Inc."},{"information","Licensed under the MPL.
See
http://www.rabbitmq.com/"},{"capabilities",[{"publisher_confirms",true},{"exchange_exchange_bindings",true},{"basic.nack",true},{"consumer_cancel_notify",true},{"connection.blocked",true},{"authentication_failure_close",true}]}]
1437668182739 624 9 644 9 0 running 1*

*Channels:*

*rabbitmqd:*

*<***@rabbitmqc.2.30196.307> 10.5.25.5:49999 -> 10.5.15.18:5672
5672 49999 10.5.15.18 10.5.25.5 false
PLAIN {0,9,1} User workflow 580 131072 0
[{"product","RabbitMQ"},{"version","3.4.1.0"},{"platform",".NET"},{"copyright","Copyright
(C) 2007-2014 GoPivotal, Inc."},{"information","Licensed under the MPL.
See
http://www.rabbitmq.com/"},{"capabilities",[{"publisher_confirms",true},{"exchange_exchange_bindings",true},{"basic.nack",true},{"consumer_cancel_notify",true},{"connection.blocked",true},{"authentication_failure_close",true}]}]
1437667797708 648 9 650 9 0 running 1*
*<***@rabbitmqc.2.17888.308> 10.5.31.185:54226 -> 10.5.15.18:5672
5672 54226 10.5.15.18 10.5.31.185 false
PLAIN {0,9,1} User workflow 580 131072 0
[{"product","RabbitMQ"},{"version","3.4.1.0"},{"platform",".NET"},{"copyright","Copyright
(C) 2007-2014 GoPivotal, Inc."},{"information","Licensed under the MPL.
See
http://www.rabbitmq.com/"},{"capabilities",[{"publisher_confirms",true},{"exchange_exchange_bindings",true},{"basic.nack",true},{"consumer_cancel_notify",true},{"connection.blocked",true},{"authentication_failure_close",true}]}]
1437667989861 664 9 654 9 0 running 1*
*<***@rabbitmqc.2.4417.309> 10.5.25.5:50019 -> 10.5.15.18:5672
5672 50019 10.5.15.18 10.5.25.5 false
PLAIN {0,9,1} User workflow 580 131072 0
[{"product","RabbitMQ"},{"version","3.4.1.0"},{"platform",".NET"},{"copyright","Copyright
(C) 2007-2014 GoPivotal, Inc."},{"information","Licensed under the MPL.
See
http://www.rabbitmq.com/"},{"capabilities",[{"publisher_confirms",true},{"exchange_exchange_bindings",true},{"basic.nack",true},{"consumer_cancel_notify",true},{"connection.blocked",true},{"authentication_failure_close",true}]}]
1437668170375 632 9 646 9 0 running 1*
*<***@rabbitmqc.2.5730.309> 10.5.25.5:50021 -> 10.5.15.18:5672
5672 50021 10.5.15.18 10.5.25.5 false
PLAIN {0,9,1} User workflow 580 131072 0
[{"product","RabbitMQ"},{"version","3.4.1.0"},{"platform",".NET"},{"copyright","Copyright
(C) 2007-2014 GoPivotal, Inc."},{"information","Licensed under the MPL.
See
http://www.rabbitmq.com/"},{"capabilities",[{"publisher_confirms",true},{"exchange_exchange_bindings",true},{"basic.nack",true},{"consumer_cancel_notify",true},{"connection.blocked",true},{"authentication_failure_close",true}]}]
1437668182739 624 9 644 9 0 running 1*

*Channels:*

BTW what;s the meaning of these 3 number references like <0.6693.194>?

Thanks

Post by Jean-Sébastien Pédron

(...) I rerouted them to a different queue and tried to delete that
one but all delete requests (via UI or rabbitmqadmin) hanged.

Hi!
Could you please post your log files (both "normal" and -sasl), as well
as the output from "rabbitmqctl report"?
The output of "rabbitmqctl report" contains informations such as
hostnames, IP addresses, resources names, etc. If you don't want to
disclose them, feel free to replace sensitive data and/or send the
report privately to me.
--
Jean-Sébastien Pédron
Pivotal / RabbitMQ

[Jean-Sébastien Pédron]

Post by Kirill Lutarevich
Hi
Considering the names of RabbitMQ hosts (rabbitmqc and rabbitmqd) I
attached normal log files (current sasl logs are empty on both)
"rabbitmqctl report" hangs on both machines after showing the
information below (after printing out /Channels:/)

It probably hangs because it fails to query informations for the broken
queue.

In the log from rabbitmqc, you have a lot of:

=ERROR REPORT==== 23-Jul-2015::06:44:52 ===
Discarding message ... from <0.252.0> to <0.560.0> in an old incarnation
(3) of this node (2)

This means the PID for the broken queue is now invalid. The queue's
process is gone, but somehow, the event remained unnoticed or was not
fired at all.

What version of Erlang and RabbitMQ are you using?

Post by Kirill Lutarevich
BTW what;s the meaning of these 3 number references like <0.6693.194>?

These are Erlang PIDs (they do not map to system PIDs).

For instance, your corrupted queue process had the PID <0.560.0>.

[Kirill Lutarevich]

RabbitMQ 3.4.1, Erlang R16B03.

Is it possible to kill that faulty Erlang process?

Post by Jean-Sébastien Pédron

Post by Kirill Lutarevich
Hi
Considering the names of RabbitMQ hosts (rabbitmqc and rabbitmqd) I
attached normal log files (current sasl logs are empty on both)
"rabbitmqctl report" hangs on both machines after showing the
information below (after printing out /Channels:/)

It probably hangs because it fails to query informations for the broken
queue.
=ERROR REPORT==== 23-Jul-2015::06:44:52 ===
Discarding message ... from <0.252.0> to <0.560.0> in an old incarnation
(3) of this node (2)
This means the PID for the broken queue is now invalid. The queue's
process is gone, but somehow, the event remained unnoticed or was not
fired at all.
What version of Erlang and RabbitMQ are you using?

Post by Kirill Lutarevich
BTW what;s the meaning of these 3 number references like <0.6693.194>?

These are Erlang PIDs (they do not map to system PIDs).
For instance, your corrupted queue process had the PID <0.560.0>.
--
Jean-Sébastien Pédron
Pivotal / RabbitMQ

[Michael Klishin]

Post by Kirill Lutarevich
Is it possible to kill that faulty Erlang process?

rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup({resource, <<"/">>, queue, <<"a-queue">>}), Pid = erlang:element(7, Q), exit(Pid, please_crash).'

where / and a-queue are replaced with the actual vhost
and queue names.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Kirill Lutarevich]

I ran
rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup({resource,
<<"workflow">>, queue, <<"clustering">>}), Pid = erlang:element(7, Q),
exit(Pid, please_crash).'

and it didn't do anything. The disabled queue is still on the admin UI.

Post by Michael Klishin

Post by Kirill Lutarevich
Is it possible to kill that faulty Erlang process?

rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup({resource, <<"/">>,
queue, <<"a-queue">>}), Pid = erlang:element(7, Q), exit(Pid,
please_crash).'
where / and a-queue are replaced with the actual vhost
and queue names.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Post by Michael Klishin
I ran
rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup({resource,
<<"workflow">>, queue, <<"clustering">>}), Pid = erlang:element(7,
Q), exit(Pid, please_crash).'
and it didn't do anything. The disabled queue is still on the admin UI.

what did it output? Was anything logged in the SASL log? 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Kirill Lutarevich]

The output said "true" and nothing in the sasl log. Should I have run that
command on both servers in cluster or only on 'rabbitmqc' that reported
errors in the "standard" log?

Post by Michael Klishin

Post by Michael Klishin
I ran
rabbitmqctl eval '{ok, Q} = rabbit_amqqueue:lookup({resource,
<<"workflow">>, queue, <<"clustering">>}), Pid = erlang:element(7,
Q), exit(Pid, please_crash).'
and it didn't do anything. The disabled queue is still on the admin UI.

what did it output? Was anything logged in the SASL log?
--
MK
Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

Post by Kirill Lutarevich
The output said "true" and nothing in the sasl log. Should I have
run that command on both servers in cluster or only on 'rabbitmqc'
that reported errors in the "standard" log?

It's sufficient to run this just once, if the queue master process was indeed
killed there should be a SASL error log message and the it will be restarted. 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Kirill Lutarevich]

I ran it on a single server (rabbitmqc), it returned 'true' and there were
no records in sasl log. I attached a screenshot from Admin UI showing the
current status of that queue (clustering) compared to another queue
(summarizer). All other working queues look exactly like 'summarizer'.

Post by Michael Klishin

Post by Kirill Lutarevich
The output said "true" and nothing in the sasl log. Should I have
run that command on both servers in cluster or only on 'rabbitmqc'
that reported errors in the "standard" log?

It's sufficient to run this just once, if the queue master process was indeed
killed there should be a SASL error log message and the it will be
restarted.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ

[Gavin M. Roy]

I recently ran into this behavior/issue as well after a partition and ended
up forcing the node with the stuck queue definition out of the cluster,
resetting the database on that node, and adding it back.

Post by Kirill Lutarevich
I ran it on a single server (rabbitmqc), it returned 'true' and there were
no records in sasl log. I attached a screenshot from Admin UI showing the
current status of that queue (clustering) compared to another queue
(summarizer). All other working queues look exactly like 'summarizer'.

Post by Michael Klishin

Post by Kirill Lutarevich
The output said "true" and nothing in the sasl log. Should I have
run that command on both servers in cluster or only on 'rabbitmqc'
that reported errors in the "standard" log?

It's sufficient to run this just once, if the queue master process was indeed
killed there should be a SASL error log message and the it will be
restarted.
--
MK
Staff Software Engineer, Pivotal/RabbitMQ
--

You received this message because you are subscribed to the Google Groups
"rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Jean-Sébastien Pédron]

Post by Michael Klishin

Post by Kirill Lutarevich
The output said "true" and nothing in the sasl log. Should I have
run that command on both servers in cluster or only on 'rabbitmqc'
that reported errors in the "standard" log?

It's sufficient to run this just once, if the queue master process was indeed
killed there should be a SASL error log message and the it will be restarted.

To me, the queue process is already gone. Apparently, the "Discarding
message ... in an old incarnation of this node" message happens when a
given node was restarted in a very short time and this restart remained
unnoticed (ie. the queue process, <0.560.0>, belonged to an previous
instance of the node). I still don't understand how this could ever happen.

As I'm not sure about what I say, could you please run the following
one-liner and post the output?

rabbitmqctl eval "lists:filter(fun({gs2_state, _, _, T, _, _, _, _, _,
_}) when is_tuple(T) -> element(1, T) =:= q; (_) -> false end, [catch
sys:get_state(P, 100) || P <- processes()])."

(pay attention to line wrapping, it must be a single line when you
execute it)

This command will list all running queue processes and show their state
(including the queue name). You can run it on all your nodes. The
command may take some time to return.

[Kirill Lutarevich]

*rabbitmqctl eval "lists:filter(fun({gs2_state, _, _, T, _, _, _, _, _, _})
when is_tuple(T) -> element(1, T) =:= q; (_) -> false end,
[catch sys:get_state(P, 100) || P <- processes()])."* returned [] on both
servers and the disabled queue is still there.

To make things worse one of the nodes (rabbitmqd) in cluster failed today
at exactly 11:54 UTC (see the attached logs). I'm starting to think the
issue is much bigger than one disabled queue.

Post by Michael Klishin

Post by Michael Klishin

Post by Kirill Lutarevich
The output said "true" and nothing in the sasl log. Should I have
run that command on both servers in cluster or only on 'rabbitmqc'
that reported errors in the "standard" log?

It's sufficient to run this just once, if the queue master process was

indeed

Post by Michael Klishin
killed there should be a SASL error log message and the it will be

restarted.
To me, the queue process is already gone. Apparently, the "Discarding
message ... in an old incarnation of this node" message happens when a
given node was restarted in a very short time and this restart remained
unnoticed (ie. the queue process, <0.560.0>, belonged to an previous
instance of the node). I still don't understand how this could ever happen.
As I'm not sure about what I say, could you please run the following
one-liner and post the output?
rabbitmqctl eval "lists:filter(fun({gs2_state, _, _, T, _, _, _, _, _,
_}) when is_tuple(T) -> element(1, T) =:= q; (_) -> false end, [catch
sys:get_state(P, 100) || P <- processes()])."
(pay attention to line wrapping, it must be a single line when you
execute it)
This command will list all running queue processes and show their state
(including the queue name). You can run it on all your nodes. The
command may take some time to return.
--
Jean-Sébastien Pédron
Pivotal / RabbitMQ

[Jean-Sébastien Pédron]

/rabbitmqctl eval "lists:filter(fun({gs2_state, _, _, T, _, _, _, _, _,
_}) when is_tuple(T) -> element(1, T) =:= q; (_) -> false end, [catch
sys:get_state(P, 100) || P <- processes()])."/ returned [] on both
servers and the disabled queue is still there.

The command was not meant to fix anything, just grab more informations.
It must not apply correctly to RabbitMQ 3.4.x. Let's simplify it:

rabbitmqctl eval "[catch sys:get_state(P, 100) || P <- processes()])."

It does no filtering so the output will be much bigger.

To make things worse one of the nodes (rabbitmqd) in cluster failed
today at exactly 11:54 UTC (see the attached logs). I'm starting to
think the issue is much bigger than one disabled queue.

Could you please post the normal and -sasl logs from this node?

[Kirill Lutarevich]

*rabbitmqctl eval "[catch sys:get_state(P, 100) || P <- processes()])."*
returned *Error: syntax error before: ')'*

I attached the logs from the failed node

Post by Jean-Sébastien Pédron

/rabbitmqctl eval "lists:filter(fun({gs2_state, _, _, T, _, _, _, _, _,
_}) when is_tuple(T) -> element(1, T) =:= q; (_) -> false end, [catch
sys:get_state(P, 100) || P <- processes()])."/ returned [] on both
servers and the disabled queue is still there.

The command was not meant to fix anything, just grab more informations.
rabbitmqctl eval "[catch sys:get_state(P, 100) || P <- processes()])."
It does no filtering so the output will be much bigger.

To make things worse one of the nodes (rabbitmqd) in cluster failed
today at exactly 11:54 UTC (see the attached logs). I'm starting to
think the issue is much bigger than one disabled queue.

Could you please post the normal and -sasl logs from this node?
--
Jean-Sébastien Pédron
Pivotal / RabbitMQ

[Jean-Sébastien Pédron]

Post by Jean-Sébastien Pédron
rabbitmqctl eval "[catch sys:get_state(P, 100) || P <- processes()])."
returned /Error: syntax error before: ')'/

Sorry... Here is the corrected line:
rabbitmqctl eval "[catch sys:get_state(P, 100) || P <- processes()]."

Post by Jean-Sébastien Pédron
I attached the logs from the failed node

Thanks!

[Jean-Sébastien Pédron]

/rabbitmqctl eval "[catch sys:get_state(P, 100) || P <- processes()])."/
returned /Error: syntax error before: ')'/

The name of the broken queue is "clustering", right?

[Kirill Lutarevich]

Yes, the name of the queue is 'clustering'. Your command is super scary :)
meaning that it disables the whole cluster. I had to restart both servers
after running this command. I attached the output though.

Post by Jean-Sébastien Pédron

/rabbitmqctl eval "[catch sys:get_state(P, 100) || P <- processes()])."/
returned /Error: syntax error before: ')'/

The name of the broken queue is "clustering", right?
--
Jean-Sébastien Pédron
Pivotal / RabbitMQ

[Jean-Sébastien Pédron]

Post by Kirill Lutarevich
Yes, the name of the queue is 'clustering'. Your command is super scary
:) meaning that it disables the whole cluster. I had to restart both
servers after running this command. I attached the output though.

Yeah, the command queries the state of all running processes, so it's
pretty expensive. Sorry, I should have warned you.

However, the timeout of 100 ms in this command may be too short, because
the log doesn't contain any queue processes.

Anyway, you can remove the queue record using the attached script. It
won't remove any associated files however, because their names are
stored in the queue process which is probably gone.

You just need to run it on one node; no need to repeat the action on
other nodes.

The syntax:
./remove-mia-queue <vhost> <queue>

Example:
./remove-mia-queue workflow clustering

[Kirill Lutarevich]

Ok, do you know if the cluster crash today (see log files I sent earlier)
is related to that problem with disabled queue?

Post by Jean-Sébastien Pédron

Post by Kirill Lutarevich
Yes, the name of the queue is 'clustering'. Your command is super scary
:) meaning that it disables the whole cluster. I had to restart both
servers after running this command. I attached the output though.

Yeah, the command queries the state of all running processes, so it's
pretty expensive. Sorry, I should have warned you.
However, the timeout of 100 ms in this command may be too short, because
the log doesn't contain any queue processes.
Anyway, you can remove the queue record using the attached script. It
won't remove any associated files however, because their names are
stored in the queue process which is probably gone.
You just need to run it on one node; no need to repeat the action on
other nodes.
./remove-mia-queue <vhost> <queue>
./remove-mia-queue workflow clustering
--
Jean-Sébastien Pédron
Pivotal / RabbitMQ

[Jean-Sébastien Pédron]

Post by Kirill Lutarevich
Ok, do you know if the cluster crash today (see log files I sent
earlier) is related to that problem with disabled queue?

I don't know yet. I quickly read it but nothing obvious explained the
problem. I need more time.