Recommendations for server process with thousands of listening ports?

[Johnny Luong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 06/22/2011 03:40 PM, Jordan Sissel wrote:
> On Wed, Jun 22, 2011 at 3:33 PM, Vassilis Bekiaris <v.bekiaris at saicon.gr>wrote:
> 
>> **
>> Hi!
>>
>> recently (on another unrelated thread) an issue came up with failure to
>> open sockets due to the limit of file descriptors available per user per
>> process (this was on Ubuntu linux IIRC), could this be your case? Check this
>> article by Trustin for further reading:
>>
>> http://gleamynode.net/articles/1557/
>>
>> Cheers,
>> Vassilis
>>
> 
> Turns out my issue was that Fedora ships with a default 'user process' limit
> of 1024 - which is roughly threads in java.
> 
> So that solves the crashing OOM/no-more-threads problem. My next question is
> this:
> 
> Why are so many threads created? My experience in other event-driven
> networking systems (eventmachine, libev, libevent, etc) is that only 1
> thread (or only a few) are used to process network IO; in Netty, using the
> Executors.newCachedThreadPool it seems to make 1 thread per server socket.
> If I switch to using newFixedThreadPool(N) only the first N sockets are
> usable.
> 
> I feel like I'm doing something wrong here; should I be expecting a 1:1
> thread:socket mapping? That seems odd.
> 
> -Jordan
> 
> 
> 
>>
>> On 23/06/2011 1:14 , Jordan Sissel wrote:
>>
>> Howdy :)
>>
>>  I have a service that currently listens on thousands of tcp and udp ports
>> (for different channels of data); due to various problems, I am rewriting it
>> in Java and decided to use Netty.
>>
>>  I've tried a few different ways of doing things -
>>
>>  First, one NioServerSocketChannelFactory (or NioDatagramChannelFactory)
>> per listening port. Problem is, after a few thousand server channels (via
>> ConnectionlessBootstrap and ServerBootstrap), I get OOM while creating
>> threads.
>>
>>  Second, one NioServerSocketChannelFactory shared across all TCP server
>> ports and one NioDatagramChannelFactory for all UDP server ports. Problem
>> here is that after about a few hundred open sockets with error
>> "org.jboss.netty.channel.ChannelException: Failed to bind to: /
>> 0.0.0.0:15725" - watching strace shows bind(2) successfully bind on this
>> port, for example, so the cause is not a bind specific failure. Trolling the
>> logs, I see about 911 successful TCP server sockets before tcp fails with
>> this message
>>
>>  I am using NioServerSocketChannelFactory with
>> Executors.newCachedThreadPool().
>>
>>  I can try to make some standalone code that reproduces this behavior for
>> further study, but if there's anything obvious I'm missing, please let me
>> know.
>>
>>  -Jordna
>>
>>
>> _______________________________________________
>> netty-users mailing listnetty-users at lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/netty-users
>>
>>
>> _______________________________________________
>> netty-users mailing list
>> netty-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
> 
> 
> 
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users

Hi,

If you take a look at the javadoc for NioServerSocketChannelFactory,
you'll see a section underneath about how it deals with the threading
which I think explains what you are seeing.

http://docs.jboss.org/netty/3.2/api/org/jboss/netty/channel/socket/nio/NioServerSocketChannelFactory.html

The threads that are used for doing the "work" are not the same ones as
the ones doing accept calls on each of the bound ports.

- -Johnny
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJOApRTAAoJENQfv1FjsMMvA8IP/1GtKJR4IJ1Idof4Vk73s1Jg
e9G9FGYDyWsASQw3Vy9q0wZewiLBnK1ACNzzCdRyj5Pp+RGJNauHyrTEbzlko5ls
yGVl9eyCsO/adQMVf8Cy9P/WwT+Zzf0nkcmg4b9FTRyxpeAO7xauM6TGLaHgwQtB
m/tvJjFik73CorKUykqow8L9bdRvQP06L4rrNvbLXrj0WsfxFp4BsjADu2avgQCA
pe5Pt+1bD6bi9mbnWS2Gaiibv0elTPjcTwQzA3dj5OZISwbFkTWqzPbGTDSFWzPl
uFW1mYjQ14lqCwSCXgx9ICHan15Tu03bOeJjZDzbQR312haLxlO8ffPCMc3E78eN
qbcFawgiaKfgky4GykGxxl1iqH/6uAp9iLqB4687eq057/e115y0rADHmlEdjP9e
zIoftM1I5CzbPHFZzGPqAllPMsK4dnmH0xjZnIO9SJqAmrTcV1E5VS9D1pDlNij+
DE1CeR9r+iBgLrAvignVpLNbDkMZPykvZD2KuCwyyS5a7PhvK0wRqOfH8J2UunNi
8okEyWkD0pMUwwv0nlX21qJ84aFTG7v9jm9OYQydOCr7ypqyKL1Oj0oZQi0gxTpT
lgD+U8YxtovogTz+puOvZjcAb/oZ3CrbvmsbgjRIuMu8N4O7jEd6JZlzM/Gcoqkq
+lR1vRsFmB+EjB3v5axq
=it3u
-----END PGP SIGNATURE-----