[rabbitmq-discuss] Large number of connections

[Lawrence Kesteloot]

Hello,

I need to have a large number of connections to the message broker,
perhaps 10,000. There is typically (on Unix) a limit of 1024 open file
descriptors per process. The number of connections may blow up by a
factor of 10 later, I'm not sure yet. How would RabbitMQ deal with
this? Multiple instances of the broker? Also, each connection will
have its own queue. Can RabbitMQ deal with tens of thousands of
queues? The total message throughput will be on the order of the
number of connections per second (several thousand per second).

Thanks,

Lawrence

[Tony Garnock-Jones]

Hi Lawrence,

Post by Lawrence Kesteloot
I need to have a large number of connections to the message broker,
perhaps 10,000. There is typically (on Unix) a limit of 1024 open file
descriptors per process.

Indeed. You will need to engage in kernel (or at least rlimit) tuning to
raise the limit. (You may also need to increase the limit the erlang VM
places on the maximum number of processes; "+P 1000000" argument to the VM)

I've had upwards of 4000 simultaneous connections to a single node
before, but haven't done any serious testing of it with so many connections.

Post by Lawrence Kesteloot
Multiple instances of the broker?

Clustering will help spread the load, so yes, maybe.

Post by Lawrence Kesteloot
Also, each connection will
have its own queue. Can RabbitMQ deal with tens of thousands of
queues? The total message throughput will be on the order of the
number of connections per second (several thousand per second).

It can deal with tens or even hundreds of thousands of queues. The
throughput may be a limiting factor, but again clustering may help there.

Regards,
Tony

[Lawrence Kesteloot]

Hi Tony,

Thanks for the talk on Wednesday.

I'd rather not hack the kernel. If RabbitMQ can already handle the queues
and messages, could I instead have a dozen multiplexer processes that do
nothing but multiplex 1000 TCP AMQP connections into a single one? I don't
know if the AMQP protocol makes this possible. It might need to keep track
of virtual connections within the single connection.

Lawrence

Post by Tony Garnock-Jones
Hi Lawrence,

Post by Lawrence Kesteloot
I need to have a large number of connections to the message broker,
perhaps 10,000. There is typically (on Unix) a limit of 1024 open file
descriptors per process.

Indeed. You will need to engage in kernel (or at least rlimit) tuning to
raise the limit. (You may also need to increase the limit the erlang VM
places on the maximum number of processes; "+P 1000000" argument to the VM)
I've had upwards of 4000 simultaneous connections to a single node before,
but haven't done any serious testing of it with so many connections.
Multiple instances of the broker?
Clustering will help spread the load, so yes, maybe.
Also, each connection will

Post by Lawrence Kesteloot
have its own queue. Can RabbitMQ deal with tens of thousands of
queues? The total message throughput will be on the order of the
number of connections per second (several thousand per second).

It can deal with tens or even hundreds of thousands of queues. The
throughput may be a limiting factor, but again clustering may help there.
Regards,
Tony

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090410/3e9ec6ae/attachment.htm

[Valentino Volonghi]

Post by Lawrence Kesteloot
Hi Tony,
Thanks for the talk on Wednesday.
I'd rather not hack the kernel.

That's not hacking the kernel... If you want to setup a server to
seriously handle
many thousands of connections you _have to_ set it up so that it is as
fast as possible
to handle that traffic. The default parameters in the kernel are for
desktop
and minimal RAM usage, on servers this doesn't make sense at all
because you
already know how much traffic you are going to deal with and what the
server is
going to be working on (DB, reverse proxy, load balancer, app server
etc).

Notice also that every connection uses a port and you only have room
for 65k
ports per interface, so for example one other setting that you need to
add is
to remove the 2 minutes windows of TCP_WAIT. If you don't change these
parameters, mind you, you are going to have serious issues in
production, for example
not enough ports to bind new sockets.

Here's a set of parameters that you might want to change in your kernel
to make it faster and better for many TCP connections.

# some spoof protection
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1

# General gigabit tuning:
net.core.rmem_max = 8738000
net.core.wmem_max = 6553600
net.ipv4.tcp_rmem = 8192 873800 8738000
net.ipv4.tcp_wmem = 4096 655360 6553600

# VERY important to reuse ports in TCP_WAIT
net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_max_tw_buckets = 360000
net.core.netdev_max_backlog = 2500
vm.min_free_kbytes = 65536
vm.swappiness = 0

# Ports dedicated to clients from this server
net.ipv4.ip_local_port_range = 30000 65535

Furthermore you also might want to use ulimit to increase the number
of allowed
fds per process and other parameters as tony already suggested.

--
Valentino Volonghi aka Dialtone
Now running MacOS X 10.5
Home Page: http://www.twisted.it
http://www.adroll.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 194 bytes
Desc: This is a digitally signed message part
Url : http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20090410/1285cc8b/attachment.pgp

[Lawrence Kesteloot]

Post by Lawrence Kesteloot
I'd rather not hack the kernel.

That's not hacking the kernel... [...] Furthermore you
also might want to use ulimit to increase the number
of allowed fds per process and other parameters as
tony already suggested.

You're right, my knowledge of FD limits is apparently over a decade
out of date. Last time I tried this you had to change a #define and
rebuild everything.

This removes a large objection I had to RabbitMQ.

Thanks!

Lawrence

[Richard Heycock]

Post by Lawrence Kesteloot

Post by Lawrence Kesteloot
I'd rather not hack the kernel.

That's not hacking the kernel... [...] Furthermore you
also might want to use ulimit to increase the number
of allowed fds per process and other parameters as
tony already suggested.

You're right, my knowledge of FD limits is apparently over a decade
out of date. Last time I tried this you had to change a #define and
rebuild everything.

I think it might more than a decade ;-)

rgh

Post by Lawrence Kesteloot
This removes a large objection I had to RabbitMQ.
Thanks!
Lawrence

[Tony Garnock-Jones]

Post by Lawrence Kesteloot
multiplexer processes
that do nothing but multiplex 1000 TCP AMQP connections into a single
one? I don't know if the AMQP protocol makes this possible. It might
need to keep track of virtual connections within the single connection.

An interesting idea! One can easily imagine an AMQP 0-9-1 proxy that
accepts connections and multiplexes the opened channels into channels on
a single connection to the real RabbitMQ Server. Essentially an "AMQP
concentrator".

It ought even to be fairly easy to implement, using perhaps the
(internals of the) Java client library... anyone fancy having a go?

Tony