Shutting down the netty server takes long time when using SSLHandler.
Trustin Lee
tlee at redhat.com
Fri Apr 17 04:04:49 EDT 2009
You're welcome. Thanks for testing!
— Trustin Lee, http://gleamynode.net/
On Thu, Apr 16, 2009 at 5:53 PM, Virat Gohil <virat4lug at gmail.com> wrote:
> Hi Trustin,
>
> Thank you very much for working on this issue with such dedication :)
>
> I tested your fix using the SSLHandler.java revision number 1202. The
> fix seems to be working fine. I will keep testing it for some more
> rounds and see how it behaves.
>
> Thanks!!
>
> Virat
>
>
> On Wed, Apr 15, 2009 at 9:01 PM, Trustin Lee <tlee at redhat.com> wrote:
>> I found a workaround for the ClosedChannelException and checked the
>> fix in, too. It seems to work pretty well now. However, I'm not sure
>> if the new behavior will break existing non-Netty clients. We will
>> see..
>>
>> — Trustin Lee, http://gleamynode.net/
>>
>> On Thu, Apr 16, 2009 at 12:04 AM, Trustin Lee <tlee at redhat.com> wrote:
>>> Hi Virat,
>>>
>>> I've just checked in the workaround for this issue. SslHandler now
>>> closes the connection immediately after sending the closure_notify and
>>> swallows a connection reset by peer error if SSL session is closed. I
>>> made this change because otherwise SslHandler will be vulnerable to a
>>> DoS attack. An attacker would be able to prevent Netty from closing a
>>> connection forever just by sending a partial SSL message.
>>>
>>> I used preliminary regular expression matcher to swallow the
>>> exception, so it will not work for localized messages, but will work
>>> for most systems.
>>>
>>> Another exception that can be raised rarely is ClosedChannelException.
>>> It can be raised when SslHandler attempts to send a closure_notify
>>> after the connection is closed. It is impossible to swallow this
>>> exception unfortunately because ClosedChannelException doesn't convey
>>> any information that tells which write request caused such an
>>> exception. I could define a new exception that adds such an
>>> information but it will cost a lot of CPU time because the exception
>>> cannot be cached to avoid the cost of initializing stack trace
>>> anymore. Let me keep thinking about working around this issue.
>>>
>>> — Trustin Lee, http://gleamynode.net/
>>>
>>> On Wed, Apr 15, 2009 at 11:12 PM, Virat Gohil <virat4lug at gmail.com> wrote:
>>>> Hi Trustin,
>>>>
>>>> Thanks for the in depth explanation! :)
>>>>
>>>> I understand that changing the new behavior will cause "connection
>>>> reset by peer" on the client side, which is acceptable in our
>>>> scenario. Can I make a request for enhancement which will allow me to
>>>> control the behavior using some API(s) exposed by Netty? You can
>>>> provide it in some later release.
>>>>
>>>> Meanwhile, I will use the future.awaitUninterruptibly(miliseconds);
>>>> function as a workaround. This will lead to some connections closing
>>>> properly while others terminating.
>>>>
>>>> Thanks,
>>>>
>>>> Virat
>>>>
>>>> On Wed, Apr 15, 2009 at 7:30 PM, Trustin Lee <tlee at redhat.com> wrote:
>>>>> I realized that this issue can be resolved by closing the connection
>>>>> immediately after sending closure_notify (new behavior) instead of
>>>>> waiting for the client to respond with closure_notify (current
>>>>> behavior). This is a legal behavior according to RFC2246 (7.2.1.
>>>>> Closure Alerts):
>>>>>
>>>>> Each party is required to send a close_notify alert before closing
>>>>> the write side of the connection. It is required that the other party
>>>>> respond with a close_notify alert of its own and close down the
>>>>> connection immediately, discarding any pending writes. It is not
>>>>> required for the initiator of the close to wait for the responding
>>>>> close_notify alert before closing the read side of the connection.
>>>>>
>>>>> However, doing this can cause a 'connection reset by peer' error on
>>>>> the client side because the client will try to respond to the
>>>>> closure_notify while the server closes the connection.
>>>>>
>>>>> Actually, there's nothing we can do with 'connection reset by peer'
>>>>> error and it is safe to ignore the exception. The problem is that
>>>>> Netty cannot tell if a SocketException has been raised because of
>>>>> connection reset or not because simple string matcher will not work
>>>>> for localized messages. It there is a definite way to detect the
>>>>> connection reset error then Netty could swallow it, which makes
>>>>> perfect sense.
>>>>>
>>>>> — Trustin Lee, http://gleamynode.net/
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Apr 15, 2009 at 9:58 PM, Virat Gohil <virat4lug at gmail.com> wrote:
>>>>>> Hi Trustin,
>>>>>>
>>>>>> Thanks for the quick response!! :)
>>>>>>
>>>>>> I tried your suggestion on revision 1187, same result. I am willing to
>>>>>> test more suggestions :)
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Virat
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 15, 2009 at 6:04 PM, Trustin Lee <tlee at redhat.com> wrote:
>>>>>>> Please try Revision 1187 before making the modification I suggested.
>>>>>>> I've just checked in the potential fix for this issue.
>>>>>>>
>>>>>>> — Trustin Lee, http://gleamynode.net/
>>>>>>>
>>>>>>> On Wed, Apr 15, 2009 at 9:25 PM, Trustin Lee <tlee at redhat.com> wrote:
>>>>>>>> Hi Virat,
>>>>>>>>
>>>>>>>> On Wed, Apr 15, 2009 at 8:33 PM, Virat Gohil <virat.gohil at gmail.com> wrote:
>>>>>>>>> Hi All!
>>>>>>>>>
>>>>>>>>> I am facing a small problem shutting down my Netty based server with
>>>>>>>>> ~1200 connections.
>>>>>>>>>
>>>>>>>>> I am using the ChannelGroup as described in Getting Started guide,
>>>>>>>>> following is the code:
>>>>>>>>>
>>>>>>>>> public void stop()
>>>>>>>>> {
>>>>>>>>> if(timer!=null)
>>>>>>>>> {
>>>>>>>>> timer.stop();
>>>>>>>>> }
>>>>>>>>> if(g!=null && factory!=null)
>>>>>>>>> {
>>>>>>>>> ChannelGroupFuture future = g.close();
>>>>>>>>> future.awaitUninterruptibly();
>>>>>>>>> if(ch!=null)
>>>>>>>>> {
>>>>>>>>> ch.unbind();
>>>>>>>>> }
>>>>>>>>> try {
>>>>>>>>> bossExecutor.shutdownNow();
>>>>>>>>> workerExecutor.shutdownNow();
>>>>>>>>> workerExecutor.awaitTermination(3600, TimeUnit.SECONDS);
>>>>>>>>> bossExecutor.awaitTermination(3600, TimeUnit.SECONDS);
>>>>>>>>> } catch (InterruptedException e) {
>>>>>>>>> //print the exception
>>>>>>>>> }
>>>>>>>>> factory.releaseExternalResources();
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> The execution gets stuck at future.awaitUninterruptibly(); I tried
>>>>>>>>> debugging the issue and found the following:
>>>>>>>>>
>>>>>>>>> 1. when g.close() is called the channel group creates a hashtable and
>>>>>>>>> creates a new DefaultChannelGroupFuture, which becomes the registered
>>>>>>>>> listener on all these channels.
>>>>>>>>> 2. Whenever channel.close() is called, the DefaultChannelGroupFuture
>>>>>>>>> gets called and increments the succes/failure count.
>>>>>>>>> 3. if the success+failure count=number of channels in the group, then
>>>>>>>>> the operation is considered finished and the thread waiting on the
>>>>>>>>> defaultchannelgroupfuture is released.
>>>>>>>>>
>>>>>>>>> I observed in SSLHandler that Channels.close() is called only if the
>>>>>>>>> received frame was empty and the inbound was finished:
>>>>>>>>> SSLHandler.java:406 (decode())
>>>>>>>>> if (frame == null && engine.isInboundDone()) {
>>>>>>>>> synchronized (closeFutures) {
>>>>>>>>> for (;;) {
>>>>>>>>> ChannelFuture future = closeFutures.poll();
>>>>>>>>> if (future == null) {
>>>>>>>>> break;
>>>>>>>>> }
>>>>>>>>> Channels.close(ctx, future);
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>> Sometimes, either the frame is not null or inbound is not completed,
>>>>>>>>> this causes the SSLHandler to continue decoding. This leads in
>>>>>>>>> DefaultChannelGroupFuture.childListener.operationComplete() being
>>>>>>>>> called after a very long time.
>>>>>>>>
>>>>>>>> 1) What is the state of the actual connection? Is it closed or still connected?
>>>>>>>>
>>>>>>>> 2) What happens if you replace:
>>>>>>>>
>>>>>>>> if (frame == null && engine.isInboundDone()) {
>>>>>>>> with:
>>>>>>>>
>>>>>>>> if (frame == null && engine.isInboundDone() || !channel.isConnected()) {
>>>>>>>>
>>>>>>>> ?
>>>>>>>>
>>>>>>>>> What we would prefer to do, is to abandon the incomplete data in
>>>>>>>>> SSLHandler.decode() and close the channel immediately as soon as the
>>>>>>>>> server's shutdown method is called. Please let me know if I am missing
>>>>>>>>> something or if there is another way of achieving a faster shutdown.
>>>>>>>>
>>>>>>>> You are doing correctly and the ChannelGroupFuture should return
>>>>>>>> quickly. Thanks for reporting the problem!
>>>>>>>>
>>>>>>>> Trustin
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> netty-users mailing list
>>>>>>> netty-users at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> netty-users mailing list
>>>>>> netty-users at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> netty-users mailing list
>>>>> netty-users at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>>
>>>>
>>>> _______________________________________________
>>>> netty-users mailing list
>>>> netty-users at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/netty-users
>>>>
>>>
>>
>> _______________________________________________
>> netty-users mailing list
>> netty-users at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
>
> _______________________________________________
> netty-users mailing list
> netty-users at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/netty-users
>
More information about the netty-users
mailing list