[rabbitmq-discuss] Using Shovel to Copy Messages for Testing

[David Legg]

Hi All,

I'd like to get everyone's opinion on how best to do something.

At the moment I have a requirement to copy messages verbatim from a queue on a RabbitMQ server to another, primarily for testing and troubleshooting purposes. The messages on the original RabbitMQ server must remain intact and copied before any other consumer running on the queue deals with and acknowledges them. I'm looking at the Shovel plugin at the moment to do this, but can anyone see any pitfalls?

Thanks in advance,

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130620/b1136570/attachment.htm>

[David Legg]

OK, so no opinions there.

Things are somewhat complicated because there are external publishers which I cannot change that publish to the default queue with the queue name as the routing key. If they were publishing to the direct exchange I could bind another queue with the same routing key and route messages to two queues. Is there any possible way of doing this with the default exchange as I haven't found one so far?


From: David Legg <david.legg at smithelectric.com<mailto:david.legg at smithelectric.com>>
Date: Thursday, 20 June 2013 14:57
To: "rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>" <rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>>
Subject: Using Shovel to Copy Messages for Testing

Hi All,

I'd like to get everyone's opinion on how best to do something.

At the moment I have a requirement to copy messages verbatim from a queue on a RabbitMQ server to another, primarily for testing and troubleshooting purposes. The messages on the original RabbitMQ server must remain intact and copied before any other consumer running on the queue deals with and acknowledges them. I'm looking at the Shovel plugin at the moment to do this, but can anyone see any pitfalls?

Thanks in advance,

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130624/805dd9f1/attachment.htm>

[Michael Klishin]

2013/6/24 David Legg <David.Legg at smithelectric.com>

Post by David Legg
If they were publishing to the direct exchange I could bind another queue
with the same routing key and route messages to two queues. Is there any
possible way of doing this with the default exchange as I haven't found one
so far?

The default exchange is a direct exchange.

[David Legg]

Yes, I know it's direct. What I want is to bind the same routing key to it in the same way as other exchanges and distribute to multiple queues.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20130624/d62fb810/attachment.htm>

[Michael Klishin]

2013/6/24 David Legg <David.Legg at smithelectric.com>

Post by David Legg
What I want is to bind the same routing key to it in the same way as other
exchanges and distribute to multiple queues

You should be able to do anything you can do with a direct exchange.

[Simon MacMullen]

Post by Michael Klishin
2013/6/24 David Legg <David.Legg at smithelectric.com
<mailto:David.Legg at smithelectric.com>>
What I want is to bind the same routing key to it in the same way as
other exchanges and distribute to multiple queues
You should be able to do anything you can do with a direct exchange.

That's not correct if the direct exchange in question is the default
exchange - the default exchange is quite a lot more magical than other
direct exchanges. Really it can be thought of as a way to express
"publish to queue" in an API which assumes publishing to exchanges.

So to the OP, I'm afraid you can't do that.

The best you can do would be use the firehose
http://www.rabbitmq.com/firehose.html to grab the messages as they are
published. But that will rewrite message headers (not bodies). Other
than that you'd need to modify the publisher.

Cheers, Simon

[David Legg]

Yer, I thought so. Thanks Simon.


Yes, it is the default exchange I'm talking about. You can't just
arbitrarily bind a queue to the default exchange with a routing key like
you can with others. Alas, changing the publishers is not going to be
possible right now.

Firehose looks like the only way I'll be able to approach this.

Post by Simon MacMullen
That's not correct if the direct exchange in question is the default
exchange - the default exchange is quite a lot more magical than other
direct exchanges. Really it can be thought of as a way to express
"publish to queue" in an API which assumes publishing to exchanges.
So to the OP, I'm afraid you can't do that.
The best you can do would be use the firehose
http://www.rabbitmq.com/firehose.html to grab the messages as they are
published. But that will rewrite message headers (not bodies). Other
than that you'd need to modify the publisher.
Cheers, Simon
--
Simon MacMullen
RabbitMQ, Pivotal

[Matthias Reik]

Sticking out my head here, but ...

... if you can't change your publishers, can you change your consumers?
If so, can't you reroute your messages from the default exchange to
another exchange
and do the processing then against that other exchange?

Cheers
Maze

Post by David Legg
Yer, I thought so. Thanks Simon.
Yes, it is the default exchange I'm talking about. You can't just
arbitrarily bind a queue to the default exchange with a routing key like
you can with others. Alas, changing the publishers is not going to be
possible right now.
Firehose looks like the only way I'll be able to approach this.

Post by Simon MacMullen
That's not correct if the direct exchange in question is the default
exchange - the default exchange is quite a lot more magical than other
direct exchanges. Really it can be thought of as a way to express
"publish to queue" in an API which assumes publishing to exchanges.
So to the OP, I'm afraid you can't do that.
The best you can do would be use the firehose
http://www.rabbitmq.com/firehose.html to grab the messages as they are
published. But that will rewrite message headers (not bodies). Other
than that you'd need to modify the publisher.
Cheers, Simon
--
Simon MacMullen
RabbitMQ, Pivotal

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss