Strange Problem

[Trustin Lee (이희승)]

I still have no idea even after the IRC conversation with you.  It
might be more of a O/S or network configuration issue?

— Trustin Lee, http://gleamynode.net/

On Thu, Nov 26, 2009 at 8:45 PM, Alan Wolff <fear2tread at gmail.com> wrote:
> Hi
>
> I experienced a strange problem out of the blue last night with my
> netty-based server running on Debian Linux. The problem is not
> necessarily with netty, but any ideas would be appreciated:
>
> Most connections were not being accepted, even from localhost. Tcpdump
>  immediately showed SYN packet being received, but no ACK sent.
> LoggingHandler's logging showed nothing about the connection. If and
> when a connection was successfully accepted, the usual
> OPEN,BOUND,CONNECTED was shown in logs immediately.
> Here are the debugging steps I took:
>
> 1) Check if other TCP connections could be made to other app servers
> running on the host (yes)
> 2) Observe exception logs and stdout log for thing like max file open
> limits reached (nothing and setting is very high)
> 3) Observe /var/log/messages (nothing in log)
> 4) Observe TCP backlog queue (64 connections waiting in queue, sysctl
> settings were at about 1000)
> 5) Flush iptable rules (didn't help)
> 6) Observe netty's BOSS thread in Yourkit/jstack (nothing out of ordinary)
> 7) Try latest JDK (6u17) (didn't help)
> 8) Stop the server, try start it. (didn't help)
> 9) OS reboot (didn't help)
> 10) Block all new connections to server but keep current connections
> in place. (Problem gone when trying to open a connection to server
> from localhost. When allowing internet traffic again, problem appeared
> again)
>
> I would like to mention that there has commonly been 2x the amount of
> established connections before and load was normal and nothing out of
> the ordinary.
>
> >From my understanding, the chain of events for a connection to be accepted are:
>
> 1 SYN arrives at NIC-> 2 kernel checks if it can allow it-> 3 iptables
> checks if it can allow it-> 4 netty and JVM call accept()
>
> So I need to figure at which point the SYN packet is being dropped or ignored.
>
> The next time it happens (if it happens) I was thinking of unloading
> iptables's modules from kernel and trying IBM JVM.
>
> Does anyone have ideas on further debugging such as TCP tunables or
> other OS limitations?
>
> Thanks a lot!
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>