Hi Trustin,<div>I tried your suggested changes (3) and redeployed - unfortunately I dont get much time in this environment as it is actively being used. It turned out that for some reason part of the system didn't behave as expected so we had a malfunction rolled back the changes (and use channel future with timeout) - its very weird and only happens in this environment. Sorry I cant be much more help than that at the moment. When I first implemented a workaround I found that logging on the underlying socket (using setAccessible) - said the connection was notClosed, but notBound (i.e. it was open) and netty was reported it was, notBound, notConnected etc.</div>
<div><br></div><div>I will report more findings next week,</div><div>Best Regards,</div><div>Neil.<br><br><div class="gmail_quote">On Thu, May 27, 2010 at 2:21 PM, "Trustin Lee (이희승)" <span dir="ltr"><<a href="mailto:trustin@gmail.com">trustin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hi Neil,<br>
<br>
That doesn't sound right. It would be great if you try a couple things<br>
for me to track down the problem:<br>
<br>
1) After calling Channel.close() for the leaked connection, what do<br>
isOpen(), isBound(), and isConnected() return?<br>
<br>
2) Could you increase the connect future timeout and dump the heap while<br>
it waits for the connect future using jmap? Once you send me the heap<br>
dump, I can understand Netty's internal status more easily.<br>
<br>
3) Could you let me know if specifying "connectTimeoutMillis" option<br>
instead of using future timeout fixes the problem?<br>
<br>
bootstrap.setOption("connectTimeoutMillis", 10000);<br>
...<br>
bootstrap.connect().awaitUninterruptibly(); // note no argument<br>
<br>
I believe (3) will fix your problem, but I need (1) and (2) to figure<br>
out the exact steps to reproduce the problem and fix it fundamentally.<br>
<br>
If possible, it would be very nice if I can figure out if this is a bug<br>
or not and fix it as soon as possible so that more stable CR (or Final)<br>
version is released tomorrow.<br>
<br>
Thanks,<br>
Trustin<br>
<div><div></div><div class="h5"><br>
Neil Avery wrote:<br>
> All,<br>
> Trustin - firstly - thanks again for such an awesome network framework -<br>
> it has provided to be amazingly scalable for some of the loads we have<br>
> been putting through it; 4000 servers streaming GBs of data and its been<br>
> rock solid!<br>
><br>
> Platform Netty 3.2RC1, RedHat Linux, Java 1.6.0_18<br>
><br>
> Unfortunately we have finally hit a weird issue which only occurs on<br>
> this particular platform. Using the NioClientSocketChannelFactory to<br>
> create a client, after a period of use we start to experience a timeout<br>
> when trying to bind. The code is standard in that it follows the<br>
> tutorials and we have never seen this problem before. In this case there<br>
> is little load on the system and its only sending messages to 6 peers.<br>
><br>
> Now for the weird bit - there is no cause excepton() sent to the handler<br>
> (our nio listener) and - when printing<br>
> the channelFuture.getChannel().getLocalAddress() it prints out a valid<br>
> client address. Using lsof & netstat we can see the connection is<br>
> actually bound. As part of the failure, we attempt to free resources for<br>
> the channel and they also fail to release the bound connection...so we<br>
> leak connections. Cleanup is attempted using; closeFuture =<br>
> channelFuture.getChannel().close() and then wait 10seconds for the<br>
> closeFuture to complete.<br>
><br>
> So I guess I have 3 questions:<br>
> - are we handling the connection bind correctly? - it doesnt seem for<br>
> this case as it does connect the code doesnt think it has<br>
> - are we handling closeDown correctly? -<br>
> channelFuture.getChannel().close() doesnt seem to release these resources...<br>
> - is there a brute force way to access the underlying socketChannel and<br>
> attempt to release it?<br>
><br>
> Any help appreciated,<br>
> Regard Neil.<br>
><br>
> ClientBootstrap bootstrap = new ClientBootstrap(nettyClientfactory);<br>
> handler = new NettyClientHandler(uri);<br>
> bootstrap.getPipeline().addLast(HANDLER, handler);<br>
> bootstrap.setOption(TCP_NO_DELAY, true);<br>
> bootstrap.setOption(KEEP_ALIVE, true);<br>
><br>
> channelFuture = bootstrap.connect(new InetSocketAddress(uri.getHost(),<br>
> uri.getPort()));<br>
><br>
> try {<br>
> boolean success = channelFuture.await(10, TimeUnit.SECONDS);<br>
> if (!success) {<br>
> if (handler.getException() != null) {<br>
> throw new RuntimeException(String.format("%s\n Failed to establish<br>
> Connection within %d secs:%s", this.toString(),<br>
> connectionEstablishTimeout, uri), handler.getException()); } else {<br>
> throw new RuntimeException(String.format("%s\n Failed to establish<br>
> Connection within %d secs:%s", this.toString(),<br>
> connectionEstablishTimeout, uri)); }<br>
><br>
><br>
> --<br>
><br>
</div></div>> <a href="http://www.liquidlabs.co.uk" target="_blank">www.liquidlabs.co.uk</a> <<a href="http://www.liquidlabs.co.uk" target="_blank">http://www.liquidlabs.co.uk</a>><br>
><br>
><br>
> ------------------------------------------------------------------------<br>
><br>
> _______________________________________________<br>
> netty-users mailing list<br>
> <a href="mailto:netty-users@lists.jboss.org">netty-users@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/netty-users" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-users</a><br>
<font color="#888888"><br>
--<br>
what we call human nature in actuality is human habit<br>
<a href="http://gleamynode.net/" target="_blank">http://gleamynode.net/</a><br>
<br>
<br>
</font><br>_______________________________________________<br>
netty-users mailing list<br>
<a href="mailto:netty-users@lists.jboss.org">netty-users@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/netty-users" target="_blank">https://lists.jboss.org/mailman/listinfo/netty-users</a><br></blockquote></div><br><br clear="all"><br>-- <br><br><a href="http://www.liquidlabs.co.uk">www.liquidlabs.co.uk</a><br>
</div>