From tsegismo at redhat.com  Wed Jun  1 12:13:25 2016
From: tsegismo at redhat.com (Thomas Segismont)
Date: Wed, 1 Jun 2016 18:13:25 +0200
Subject: [undertow-dev] Issue with Undertow Client and SSL
Message-ID: <CAHnC0xinKCGbBi91TBuUOodXQHR+yWJpu-eLvF-HvZzZjg3RMQ@mail.gmail.com>

Hi,

Openshift Origin uses Hawkular Metrics to store node and container data. In
this scenario, Hawkular Metrics calls the Kubernetes master server over
HTTPS to validate the client identity. This is implemented with Undertow
Client, as part of a ServletExtension (inside Wildfly 10) [1]. Works fine
in development and testing.

Now the Openshit team sees errors in the logs [2][3]. I couldn't reproduce
yet. Errors come in pair, first the "UT005001: An exception occurred
processing the request: java.lang.IllegalStateException: XNIO000017: Buffer
was already freed", and just after "XNIO001007: A channel event listener
threw an exception: java.lang.NullPointerException".

Does that ring a bell? I haven't been able to find a starting point  by
looking into the source the code.

Thanks

-- 
Thomas Segismont
JBoss ON Engineering Team

[1] https://git.io/vrNyP
[2] https://issues.jboss.org/browse/HWKMETRICS-408
[3] https://issues.jboss.org/secure/attachment/12405779/hawkular.log
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20160601/b2a740f0/attachment.html 

From tomaz.cerar at gmail.com  Wed Jun  1 14:35:47 2016
From: tomaz.cerar at gmail.com (=?UTF-8?B?VG9tYcW+IENlcmFy?=)
Date: Wed, 1 Jun 2016 20:35:47 +0200
Subject: [undertow-dev] Issue with Undertow Client and SSL
In-Reply-To: <CAHnC0xinKCGbBi91TBuUOodXQHR+yWJpu-eLvF-HvZzZjg3RMQ@mail.gmail.com>
References: <CAHnC0xinKCGbBi91TBuUOodXQHR+yWJpu-eLvF-HvZzZjg3RMQ@mail.gmail.com>
Message-ID: <CAMquZP6js1jKPFj1QW+cmHBd8J0ns8SrHiEWbLB-wN_Pd5PnKg@mail.gmail.com>

On top of my head, it sounds like a bug that was fixed long ago.

Are you sure that code that is throwing this is using latest undertow bits?

On Wed, Jun 1, 2016 at 6:13 PM, Thomas Segismont <tsegismo at redhat.com>
wrote:

> Hi,
>
> Openshift Origin uses Hawkular Metrics to store node and container data.
> In this scenario, Hawkular Metrics calls the Kubernetes master server over
> HTTPS to validate the client identity. This is implemented with Undertow
> Client, as part of a ServletExtension (inside Wildfly 10) [1]. Works fine
> in development and testing.
>
> Now the Openshit team sees errors in the logs [2][3]. I couldn't reproduce
> yet. Errors come in pair, first the "UT005001: An exception occurred
> processing the request: java.lang.IllegalStateException: XNIO000017: Buffer
> was already freed", and just after "XNIO001007: A channel event listener
> threw an exception: java.lang.NullPointerException".
>
> Does that ring a bell? I haven't been able to find a starting point  by
> looking into the source the code.
>
> Thanks
>
> --
> Thomas Segismont
> JBoss ON Engineering Team
>
> [1] https://git.io/vrNyP
> [2] https://issues.jboss.org/browse/HWKMETRICS-408
> [3] https://issues.jboss.org/secure/attachment/12405779/hawkular.log
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20160601/26beeb31/attachment.html 

From sdouglas at redhat.com  Wed Jun  1 16:55:51 2016
From: sdouglas at redhat.com (Stuart Douglas)
Date: Wed, 1 Jun 2016 22:55:51 +0200
Subject: [undertow-dev] Issue with Undertow Client and SSL
In-Reply-To: <CAHnC0xinKCGbBi91TBuUOodXQHR+yWJpu-eLvF-HvZzZjg3RMQ@mail.gmail.com>
References: <CAHnC0xinKCGbBi91TBuUOodXQHR+yWJpu-eLvF-HvZzZjg3RMQ@mail.gmail.com>
Message-ID: <CAD+L2cwH=_cD6VpeVpE3n-H6BgA6o49W7gLSYsbMnNmYWDR_jg@mail.gmail.com>

Try using io.undertow.protocols.ssl.UndertowXnioSsl instead of the
default XNIO one. I ended up writing a different SSL implementation
for Undertow, as Undertow and remoting had different requirements for
SSL. All the testing we do is done against the Undertow SSL
implementation.

Stuart

On Wed, Jun 1, 2016 at 6:13 PM, Thomas Segismont <tsegismo at redhat.com> wrote:
> Hi,
>
> Openshift Origin uses Hawkular Metrics to store node and container data. In
> this scenario, Hawkular Metrics calls the Kubernetes master server over
> HTTPS to validate the client identity. This is implemented with Undertow
> Client, as part of a ServletExtension (inside Wildfly 10) [1]. Works fine in
> development and testing.
>
> Now the Openshit team sees errors in the logs [2][3]. I couldn't reproduce
> yet. Errors come in pair, first the "UT005001: An exception occurred
> processing the request: java.lang.IllegalStateException: XNIO000017: Buffer
> was already freed", and just after "XNIO001007: A channel event listener
> threw an exception: java.lang.NullPointerException".
>
> Does that ring a bell? I haven't been able to find a starting point  by
> looking into the source the code.
>
> Thanks
>
> --
> Thomas Segismont
> JBoss ON Engineering Team
>
> [1] https://git.io/vrNyP
> [2] https://issues.jboss.org/browse/HWKMETRICS-408
> [3] https://issues.jboss.org/secure/attachment/12405779/hawkular.log
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev

From tsegismo at redhat.com  Thu Jun  2 04:26:18 2016
From: tsegismo at redhat.com (Thomas Segismont)
Date: Thu, 2 Jun 2016 10:26:18 +0200
Subject: [undertow-dev] Issue with Undertow Client and SSL
In-Reply-To: <CAMquZP6js1jKPFj1QW+cmHBd8J0ns8SrHiEWbLB-wN_Pd5PnKg@mail.gmail.com>
References: <CAHnC0xinKCGbBi91TBuUOodXQHR+yWJpu-eLvF-HvZzZjg3RMQ@mail.gmail.com>
	<CAMquZP6js1jKPFj1QW+cmHBd8J0ns8SrHiEWbLB-wN_Pd5PnKg@mail.gmail.com>
Message-ID: <CAHnC0xgf5E-2DqaS6ix28f5WKiDWKTViMdm+Qu66ALnyjwgYkg@mail.gmail.com>

We use the Undertow and XNIO jars which come with Wildfly 10. They are made
accessible to the application due to the presence of an Undertow extension.
At least, this is my understanding. I would like to avoid embedding later
versions of Undertow and XNIO inside the WAR.

2016-06-01 20:35 GMT+02:00 Toma? Cerar <tomaz.cerar at gmail.com>:

> On top of my head, it sounds like a bug that was fixed long ago.
>
> Are you sure that code that is throwing this is using latest undertow bits?
>
> On Wed, Jun 1, 2016 at 6:13 PM, Thomas Segismont <tsegismo at redhat.com>
> wrote:
>
>> Hi,
>>
>> Openshift Origin uses Hawkular Metrics to store node and container data.
>> In this scenario, Hawkular Metrics calls the Kubernetes master server over
>> HTTPS to validate the client identity. This is implemented with Undertow
>> Client, as part of a ServletExtension (inside Wildfly 10) [1]. Works fine
>> in development and testing.
>>
>> Now the Openshit team sees errors in the logs [2][3]. I couldn't
>> reproduce yet. Errors come in pair, first the "UT005001: An exception
>> occurred processing the request: java.lang.IllegalStateException:
>> XNIO000017: Buffer was already freed", and just after "XNIO001007: A
>> channel event listener threw an exception: java.lang.NullPointerException".
>>
>> Does that ring a bell? I haven't been able to find a starting point  by
>> looking into the source the code.
>>
>> Thanks
>>
>> --
>> Thomas Segismont
>> JBoss ON Engineering Team
>>
>> [1] https://git.io/vrNyP
>> [2] https://issues.jboss.org/browse/HWKMETRICS-408
>> [3] https://issues.jboss.org/secure/attachment/12405779/hawkular.log
>>
>> _______________________________________________
>> undertow-dev mailing list
>> undertow-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>
>
>


-- 
Thomas Segismont
JBoss ON Engineering Team
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20160602/f70ec681/attachment.html 

From tsegismo at redhat.com  Thu Jun  2 11:50:45 2016
From: tsegismo at redhat.com (Thomas Segismont)
Date: Thu, 2 Jun 2016 17:50:45 +0200
Subject: [undertow-dev] Issue with Undertow Client and SSL
In-Reply-To: <CAD+L2cwH=_cD6VpeVpE3n-H6BgA6o49W7gLSYsbMnNmYWDR_jg@mail.gmail.com>
References: <CAHnC0xinKCGbBi91TBuUOodXQHR+yWJpu-eLvF-HvZzZjg3RMQ@mail.gmail.com>
	<CAD+L2cwH=_cD6VpeVpE3n-H6BgA6o49W7gLSYsbMnNmYWDR_jg@mail.gmail.com>
Message-ID: <CAHnC0xg=QNH9wyTJyGBHPr9ukvH1Ndk3vmf8Zwnxes8w4UJPEQ@mail.gmail.com>

Switching to UndertowXnioSsl solved the issue

Thank you!

2016-06-01 22:55 GMT+02:00 Stuart Douglas <sdouglas at redhat.com>:

> Try using io.undertow.protocols.ssl.UndertowXnioSsl instead of the
> default XNIO one. I ended up writing a different SSL implementation
> for Undertow, as Undertow and remoting had different requirements for
> SSL. All the testing we do is done against the Undertow SSL
> implementation.
>
> Stuart
>
> On Wed, Jun 1, 2016 at 6:13 PM, Thomas Segismont <tsegismo at redhat.com>
> wrote:
> > Hi,
> >
> > Openshift Origin uses Hawkular Metrics to store node and container data.
> In
> > this scenario, Hawkular Metrics calls the Kubernetes master server over
> > HTTPS to validate the client identity. This is implemented with Undertow
> > Client, as part of a ServletExtension (inside Wildfly 10) [1]. Works
> fine in
> > development and testing.
> >
> > Now the Openshit team sees errors in the logs [2][3]. I couldn't
> reproduce
> > yet. Errors come in pair, first the "UT005001: An exception occurred
> > processing the request: java.lang.IllegalStateException: XNIO000017:
> Buffer
> > was already freed", and just after "XNIO001007: A channel event listener
> > threw an exception: java.lang.NullPointerException".
> >
> > Does that ring a bell? I haven't been able to find a starting point  by
> > looking into the source the code.
> >
> > Thanks
> >
> > --
> > Thomas Segismont
> > JBoss ON Engineering Team
> >
> > [1] https://git.io/vrNyP
> > [2] https://issues.jboss.org/browse/HWKMETRICS-408
> > [3] https://issues.jboss.org/secure/attachment/12405779/hawkular.log
> >
> > _______________________________________________
> > undertow-dev mailing list
> > undertow-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/undertow-dev
>



-- 
Thomas Segismont
JBoss ON Engineering Team
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20160602/51632205/attachment-0001.html 

From matt at matthicks.com  Fri Jun  3 11:01:22 2016
From: matt at matthicks.com (Hicks, Matt)
Date: Fri, 03 Jun 2016 15:01:22 +0000
Subject: [undertow-dev] Proxying WebSockets
In-Reply-To: <CAD+L2czV_CyKLo+5JGH3vr+0FnnzGLLcvCHVqFzKycQ37jmy4Q@mail.gmail.com>
References: <1464184472315-882e71d7-331eeff4-505beb4b@mixmax.com>
	<CAD+L2czV_CyKLo+5JGH3vr+0FnnzGLLcvCHVqFzKycQ37jmy4Q@mail.gmail.com>
Message-ID: <CAFeDf1_JR8ajWtDFTAnJizYDzMNqGK01krywFuZLvGaHut9qyQ@mail.gmail.com>

Hi Stuart, any updates on this?  Sorry to pester, I'm just hoping to use
Undertow for a project I'm working on, but it's contingent on being able to
proxy websockets.

On Wed, May 25, 2016 at 9:39 AM Stuart Douglas <sdouglas at redhat.com> wrote:

> I am traveling and in meetings all week, I will look into it next week.
>
> Stuart
>
> On Wed, May 25, 2016 at 3:54 PM, Hicks, Matt <matt at matthicks.com> wrote:
>
>> Stuart, is there anything else you need from me to look into this further?
>>
>> On Mon, May 23, 2016 at 11:29 AM Hicks, Matt <matt at matthicks.com> wrote:
>>
>>> Yes, I do. You can run the example code and it runs against
>>> hyperscala.org.  If you go to
>>> http://hyperscala.org/example/realtime/chat.html or
>>> http://localhost:8080/example/realtime/chat.html you can test the Chat
>>> example that uses WebSockets.  If you connect directly it works fine, if
>>> you connect via the Proxy it doesn't.
>>>
>>> On Mon, May 23, 2016, 11:17 AM Stuart Douglas <sdouglas at redhat.com>
>>> wrote:
>>>
>>>> Do you have a reproduce I can run (complete with JS etc so I can see
>>>> exactly what is going on)?
>>>>
>>>> Stuart
>>>>
>>>> On Mon, May 23, 2016 at 3:24 PM, Hicks, Matt <matt at matthicks.com>
>>>> wrote:
>>>>
>>>>> Stuart, I created a ticket with all the information I know to provide:
>>>>>
>>>>>
>>>>> [UNDERTOW-714] Broken WebSocket Proxying Support - JBoss Issue Tracker
>>>>> <https://issues.jboss.org/browse/UNDERTOW-714?_sscc=t>
>>>>> I have been attempting to write a test proxy server to proxy
>>>>> explicitly to another server.
>>>>> <https://issues.jboss.org/browse/UNDERTOW-714?_sscc=t>
>>>>> issues.jboss.org
>>>>> <https://issues.jboss.org/browse/UNDERTOW-714?_sscc=t> [image: Mixmax]
>>>>> <https://mixmax.com/r/rExsBBgirNC89QPSv>
>>>>>
>>>>>
>>>>> I'm not positive which side is closing the connection, but looking at
>>>>> the JavaScript console it's showing the connection is being closed almost
>>>>> immediately after any message is sent from the browser to the server.
>>>>>
>>>>> On Mon, May 23, 2016 at 2:48 AM Stuart Douglas <sdouglas at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> When you say it does not work exactly what behaviour are you seeing?
>>>>>> Does the connection get established? If it gets established and then fails
>>>>>> which side drops the connection?
>>>>>>
>>>>>> We have extensive tests for this in the test suite, as web socket
>>>>>> tests are run through the proxy server, we need more info to see why it is
>>>>>> not working for you.
>>>>>>
>>>>>> Stuart
>>>>>>
>>>>>> On Fri, May 20, 2016 at 1:18 AM, Hicks, Matt <matt at matthicks.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I did some additional testing with 2.0.0.Alpha1 and got the exact
>>>>>>> same results.  If there is some way to accomplish proper proxying with
>>>>>>> Undertow I'm not seeing it.
>>>>>>>
>>>>>>> Should I file a bug report?
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> undertow-dev mailing list
>>>>>>> undertow-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>>>>>>
>>>>>>
>>>>>>
>>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20160603/54335621/attachment.html 

From sdouglas at redhat.com  Fri Jun  3 16:57:54 2016
From: sdouglas at redhat.com (Stuart Douglas)
Date: Fri, 3 Jun 2016 22:57:54 +0200
Subject: [undertow-dev] Proxying WebSockets
In-Reply-To: <CAFeDf1_JR8ajWtDFTAnJizYDzMNqGK01krywFuZLvGaHut9qyQ@mail.gmail.com>
References: <1464184472315-882e71d7-331eeff4-505beb4b@mixmax.com>
	<CAD+L2czV_CyKLo+5JGH3vr+0FnnzGLLcvCHVqFzKycQ37jmy4Q@mail.gmail.com>
	<CAFeDf1_JR8ajWtDFTAnJizYDzMNqGK01krywFuZLvGaHut9qyQ@mail.gmail.com>
Message-ID: <CAD+L2cyasUmcZbAZqkG62U3od-2JagmEcCpHZmDvOEHRw22jsQ@mail.gmail.com>

I just tested this, the proxy is working fine, for some reason your site is
closing the connection (probably because the URL that is sent over the
connection does not match?).

If you open up Wireshark and look at what actually gets sent it looks like:

Browser text frame:
::json::{\"type\":\"init\",\"pageId\":\"f00f144d9ac5fd206e6304087148d345\",\"url\":\"
http://localhost:8080/example/realtime/chat.html\"}

Browser text frame:
::json::{\"id\":\"chatMessage\",\"type\":\"change\",\"target\":\"chatMessage\",\"value\":\"test3\"}

Server close frame:
Close: Internal Server (1011)
Reason: RuntimeException

I would suggest that the "url" pointing to localhost is breaking something
in your server side code, but the proxy is working 100% as expected.

Stuart



On Fri, Jun 3, 2016 at 5:01 PM, Hicks, Matt <matt at matthicks.com> wrote:

> Hi Stuart, any updates on this?  Sorry to pester, I'm just hoping to use
> Undertow for a project I'm working on, but it's contingent on being able to
> proxy websockets.
>
>
> On Wed, May 25, 2016 at 9:39 AM Stuart Douglas <sdouglas at redhat.com>
> wrote:
>
>> I am traveling and in meetings all week, I will look into it next week.
>>
>> Stuart
>>
>> On Wed, May 25, 2016 at 3:54 PM, Hicks, Matt <matt at matthicks.com> wrote:
>>
>>> Stuart, is there anything else you need from me to look into this
>>> further?
>>>
>>> On Mon, May 23, 2016 at 11:29 AM Hicks, Matt <matt at matthicks.com> wrote:
>>>
>>>> Yes, I do. You can run the example code and it runs against
>>>> hyperscala.org.  If you go to
>>>> http://hyperscala.org/example/realtime/chat.html or
>>>> http://localhost:8080/example/realtime/chat.html you can test the Chat
>>>> example that uses WebSockets.  If you connect directly it works fine, if
>>>> you connect via the Proxy it doesn't.
>>>>
>>>> On Mon, May 23, 2016, 11:17 AM Stuart Douglas <sdouglas at redhat.com>
>>>> wrote:
>>>>
>>>>> Do you have a reproduce I can run (complete with JS etc so I can see
>>>>> exactly what is going on)?
>>>>>
>>>>> Stuart
>>>>>
>>>>> On Mon, May 23, 2016 at 3:24 PM, Hicks, Matt <matt at matthicks.com>
>>>>> wrote:
>>>>>
>>>>>> Stuart, I created a ticket with all the information I know to provide:
>>>>>>
>>>>>>
>>>>>> [UNDERTOW-714] Broken WebSocket Proxying Support - JBoss Issue
>>>>>> Tracker <https://issues.jboss.org/browse/UNDERTOW-714?_sscc=t>
>>>>>> I have been attempting to write a test proxy server to proxy
>>>>>> explicitly to another server.
>>>>>> <https://issues.jboss.org/browse/UNDERTOW-714?_sscc=t>
>>>>>> issues.jboss.org
>>>>>> <https://issues.jboss.org/browse/UNDERTOW-714?_sscc=t> [image:
>>>>>> Mixmax] <https://mixmax.com/r/rExsBBgirNC89QPSv>
>>>>>>
>>>>>>
>>>>>> I'm not positive which side is closing the connection, but looking at
>>>>>> the JavaScript console it's showing the connection is being closed almost
>>>>>> immediately after any message is sent from the browser to the server.
>>>>>>
>>>>>> On Mon, May 23, 2016 at 2:48 AM Stuart Douglas <sdouglas at redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> When you say it does not work exactly what behaviour are you seeing?
>>>>>>> Does the connection get established? If it gets established and then fails
>>>>>>> which side drops the connection?
>>>>>>>
>>>>>>> We have extensive tests for this in the test suite, as web socket
>>>>>>> tests are run through the proxy server, we need more info to see why it is
>>>>>>> not working for you.
>>>>>>>
>>>>>>> Stuart
>>>>>>>
>>>>>>> On Fri, May 20, 2016 at 1:18 AM, Hicks, Matt <matt at matthicks.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I did some additional testing with 2.0.0.Alpha1 and got the exact
>>>>>>>> same results.  If there is some way to accomplish proper proxying with
>>>>>>>> Undertow I'm not seeing it.
>>>>>>>>
>>>>>>>> Should I file a bug report?
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> undertow-dev mailing list
>>>>>>>> undertow-dev at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20160603/9dc83abc/attachment-0001.html 

From sdouglas at redhat.com  Sun Jun 12 21:37:35 2016
From: sdouglas at redhat.com (Stuart Douglas)
Date: Mon, 13 Jun 2016 11:37:35 +1000
Subject: [undertow-dev] Undertow 1.4 release and HTTP/2
Message-ID: <CAD+L2czxjmMdktaxQq9TnQfXk5fUf5mmf_Aa-jpePq6g3Umz-g@mail.gmail.com>

Hi everyone,

Undertow 1.4 Final should be released very soon and will be part of
Wildfly 10. The main feature that we have added in this release is
direct support for ALPN on JDK8 without requiring modification of the
boot class path.

If anyone who is using HTTP/2 would like to try this out and provide
feedback that would be great. The current ALPN solution will continue
to be supported, however you will need to set
-Dio.undertow.disable-jdk8-alpn=true to disable the new
implementation.

JDK9 has ALPN support out of the box so this only applies to JDK8.

Stuart

From jared.wolinsky at mlb.com  Tue Jun 14 12:01:32 2016
From: jared.wolinsky at mlb.com (Wolinsky, Jared)
Date: Tue, 14 Jun 2016 16:01:32 +0000
Subject: [undertow-dev] How to handle exceptions thrown from worker threads
Message-ID: <7A82725263715943B0BE14F7CA24C448678C5BAE@BAM-MBX103.MLBAM.mlb.org>

Good morning,

We're using Undertow in a new web service at MLB, and I haven't been able to find a solution for handling errors that occur in requests that have been dispatched to worker threads.  Here is a link to my SO question: http://stackoverflow.com/questions/37753726/how-to-handle-exceptions-thrown-from-undertow-worker-threads.  Ignoring worker threads, my model for error handling is to have an outer handler that maintains a map of exception type -> {log level, http status code}.  Then I allow exceptions to propagate out from the actual API handlers, hitting the error handler which logs the error and responds to the client with the correct status code based on the exception type.  When exceptions are thrown from worker threads, however, they break out of the normal handler chain and never hit the error handler, resulting in a 500 response every time.  Here are options I've considered for handling this problem:


  1.  Inside my error handler, set a DefaultResponseListener (https://http2.undertow.io/documentation/core/error-handling.html) on the exchange that contains my error handling logic.  I hit a wall here after realizing the listener only has access to the Exchange, not to the exception that will be thrown.
  2.  Set an UncaughtExceptionHandler on each handler that's on a worker thread.  This code would be added to my base handler class, do a !exchange.isInIoThread() check, set the error handler, then call the handleRequest method.  The caveat here is that I can only do this for handlers that I have control over, not built-in handlers that dispatch to worker threads such as the AuthenticationCallHandler.
  3.  Use an AspectJ-based approach where I create an @Around advice that executes for all of my handlers that will wrap the execution in a try-catch, and take care of the error handling.


Are there best practices for handling this?


Thanks,

Jared Wolinsky

Senior Software Engineer, MLB Advanced Media



**********************************************************

MLB.com <http://mlb.com> - Where Baseball is Always On.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20160614/53e73c7c/attachment.html 

From bill at dartalley.com  Tue Jun 14 12:22:40 2016
From: bill at dartalley.com (Bill O'Neil)
Date: Tue, 14 Jun 2016 12:22:40 -0400
Subject: [undertow-dev] How to handle exceptions thrown from worker
	threads
In-Reply-To: <7A82725263715943B0BE14F7CA24C448678C5BAE@BAM-MBX103.MLBAM.mlb.org>
References: <7A82725263715943B0BE14F7CA24C448678C5BAE@BAM-MBX103.MLBAM.mlb.org>
Message-ID: <CANd+gFzx++hm9bcqF9cHhZ5m9Y1DJDM95D_9mm6sd61vm+9H-Q@mail.gmail.com>

I'm not sure if this is the recommended approach but I went with your
option 2 and it works well for me. Take a look at this PR for how I handled
it. https://github.com/undertow-io/undertow/pull/407

As Stuart mentioned you can used the WrappedHandler instead of my
implementation or create something similar to mine.

The caveat here is that I can only do this for handlers that I have control
> over, not built-in handlers that dispatch to worker threads such as the
> AuthenticationCallHandler.

As for this as long as you dispatch before AuthenticationCallHandler it
should not dispatch again. Any Handler that dispatches should first check
to see if the exchange has already been dispatched and short circuit if it
has. So this case should be covered.

see
https://github.com/undertow-io/undertow/blob/master/core/src/main/java/io/undertow/security/handlers/AuthenticationCallHandler.java#L45

On Tue, Jun 14, 2016 at 12:01 PM, Wolinsky, Jared <jared.wolinsky at mlb.com>
wrote:

> Good morning,
>
> We're using Undertow in a new web service at MLB, and I haven't been able
> to find a solution for handling errors that occur in requests that have
> been dispatched to worker threads.  Here is a link to my SO question:
> http://stackoverflow.com/questions/37753726/how-to-handle-exceptions-thrown-from-undertow-worker-threads.
> Ignoring worker threads, my model for error handling is to have an outer
> handler that maintains a map of exception type -> {log level, http status
> code}.  Then I allow exceptions to propagate out from the actual API
> handlers, hitting the error handler which logs the error and responds to
> the client with the correct status code based on the exception type.  When
> exceptions are thrown from worker threads, however, they break out of the
> normal handler chain and never hit the error handler, resulting in a 500
> response every time.  Here are options I've considered for handling this
> problem:
>
>
>    1. Inside my error handler, set a DefaultResponseListener (
>    https://http2.undertow.io/documentation/core/error-handling.html) on
>    the exchange that contains my error handling logic.  I hit a wall here
>    after realizing the listener only has access to the Exchange, not to the
>    exception that will be thrown.
>    2. Set an UncaughtExceptionHandler on each handler that's on a worker
>    thread.  This code would be added to my base handler class, do a
>    *!exchange.isInIoThread()* check, set the error handler, then call the
>    handleRequest method.  The caveat here is that I can only do this for
>    handlers that I have control over, not built-in handlers that dispatch to
>    worker threads such as the AuthenticationCallHandler.
>    3. Use an AspectJ-based approach where I create an @Around advice that
>    executes for all of my handlers that will wrap the execution in a
>    try-catch, and take care of the error handling.
>
>
> Are there best practices for handling this?
>
>
> Thanks,
>
> Jared Wolinsky
>
> Senior Software Engineer, MLB Advanced Media
>
>
> **********************************************************
>
> MLB.com <http://mlb.com> - Where Baseball is Always On.
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20160614/17a2085c/attachment.html 

From afroufeq at grancanaria.com  Wed Jun 15 07:47:44 2016
From: afroufeq at grancanaria.com (afroufeq)
Date: Wed, 15 Jun 2016 12:47:44 +0100
Subject: [undertow-dev] Rule in undertow-handlers.conf to redirect HTTP to
	HTTPS
Message-ID: <cae5b36e-5912-bf73-b8de-2b9fe533874a@grancanaria.com>

I have the same question that Victor in:

http://stackoverflow.com/questions/37791759/rule-in-undertow-handlers-conf-to-redirect-http-to-https

Does anyone know the answer?

Cheers,



From sdouglas at redhat.com  Wed Jun 15 22:25:34 2016
From: sdouglas at redhat.com (Stuart Douglas)
Date: Thu, 16 Jun 2016 12:25:34 +1000
Subject: [undertow-dev] Rule in undertow-handlers.conf to redirect HTTP
	to HTTPS
In-Reply-To: <cae5b36e-5912-bf73-b8de-2b9fe533874a@grancanaria.com>
References: <cae5b36e-5912-bf73-b8de-2b9fe533874a@grancanaria.com>
Message-ID: <CAD+L2cyL+XYeYtvAT6Hft2=ic1pQ1UFfinufc-gmSWBKH-qC4Q@mail.gmail.com>

Answered

On Wed, Jun 15, 2016 at 9:47 PM, afroufeq <afroufeq at grancanaria.com> wrote:
> I have the same question that Victor in:
>
> http://stackoverflow.com/questions/37791759/rule-in-undertow-handlers-conf-to-redirect-http-to-https
>
> Does anyone know the answer?
>
> Cheers,
>
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev

From sdouglas at redhat.com  Wed Jun 15 22:37:08 2016
From: sdouglas at redhat.com (Stuart Douglas)
Date: Thu, 16 Jun 2016 12:37:08 +1000
Subject: [undertow-dev] How to handle exceptions thrown from worker
	threads
In-Reply-To: <7A82725263715943B0BE14F7CA24C448678C5BAE@BAM-MBX103.MLBAM.mlb.org>
References: <7A82725263715943B0BE14F7CA24C448678C5BAE@BAM-MBX103.MLBAM.mlb.org>
Message-ID: <CAD+L2cz5jmSXFFLEeLW0dvU0=cBpqOH89yQ3Tw=T3p6gyEu=Vw@mail.gmail.com>

I think you are correct that we don't handle this case very well, the
basic error handing strategy has always been to just catch the error
in a handler, but when you don't have control over the dispatch site
this gets tricky.

We should definitely make the error available in
DefaultResponseListener, probably just as an attachment on the
exchange. Would this be enough to meet your requirements, or would
some kind of actual error handling interface be better?
DefaultResponseListener may not always be called if the exchange has
already been finished when the error occurs (i.e. the page has already
been sent to the user).

Stuart

On Wed, Jun 15, 2016 at 2:01 AM, Wolinsky, Jared <jared.wolinsky at mlb.com> wrote:
> Good morning,
>
> We're using Undertow in a new web service at MLB, and I haven't been able to
> find a solution for handling errors that occur in requests that have been
> dispatched to worker threads.  Here is a link to my SO question:
> http://stackoverflow.com/questions/37753726/how-to-handle-exceptions-thrown-from-undertow-worker-threads.
> Ignoring worker threads, my model for error handling is to have an outer
> handler that maintains a map of exception type -> {log level, http status
> code}.  Then I allow exceptions to propagate out from the actual API
> handlers, hitting the error handler which logs the error and responds to the
> client with the correct status code based on the exception type.  When
> exceptions are thrown from worker threads, however, they break out of the
> normal handler chain and never hit the error handler, resulting in a 500
> response every time.  Here are options I've considered for handling this
> problem:
>
> Inside my error handler, set a DefaultResponseListener
> (https://http2.undertow.io/documentation/core/error-handling.html) on the
> exchange that contains my error handling logic.  I hit a wall here after
> realizing the listener only has access to the Exchange, not to the exception
> that will be thrown.
> Set an UncaughtExceptionHandler on each handler that's on a worker thread.
> This code would be added to my base handler class, do a
> !exchange.isInIoThread() check, set the error handler, then call the
> handleRequest method.  The caveat here is that I can only do this for
> handlers that I have control over, not built-in handlers that dispatch to
> worker threads such as the AuthenticationCallHandler.
> Use an AspectJ-based approach where I create an @Around advice that executes
> for all of my handlers that will wrap the execution in a try-catch, and take
> care of the error handling.
>
>
> Are there best practices for handling this?
>
>
> Thanks,
>
> Jared Wolinsky
>
> Senior Software Engineer, MLB Advanced Media
>
>
>
> **********************************************************
>
> MLB.com - Where Baseball is Always On.
>
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev

From jared.wolinsky at mlb.com  Thu Jun 16 10:17:11 2016
From: jared.wolinsky at mlb.com (Wolinsky, Jared)
Date: Thu, 16 Jun 2016 14:17:11 +0000
Subject: [undertow-dev] How to handle exceptions thrown from worker
 threads
In-Reply-To: <CAD+L2cz5jmSXFFLEeLW0dvU0=cBpqOH89yQ3Tw=T3p6gyEu=Vw@mail.gmail.com>
References: <7A82725263715943B0BE14F7CA24C448678C5BAE@BAM-MBX103.MLBAM.mlb.org>,
	<CAD+L2cz5jmSXFFLEeLW0dvU0=cBpqOH89yQ3Tw=T3p6gyEu=Vw@mail.gmail.com>
Message-ID: <7A82725263715943B0BE14F7CA24C448678D8CFF@BAM-MBX103.MLBAM.mlb.org>

For me, adding the exception onto the exchange would be fine, along with a mention in the error handling section of the docs.  I think it's ok if the DefaultResponseListener isn't called if the exchange is ended because that is some handler explicitly taking care of the response and overriding the default case.  That said, I don't think an explicit error handling interface would be a bad way to go either.

I was actually able to work around this using Bill O'Neil's advice and dispatching the request every time in my ErrorHandler, which is the outer handler in my chain.  Then I try-catch the next.handleRequest call, and do my exception handling there.  

Thanks,
Jared
________________________________________
From: Stuart Douglas [sdouglas at redhat.com]
Sent: Wednesday, June 15, 2016 10:37 PM
To: Wolinsky, Jared
Cc: undertow-dev at lists.jboss.org
Subject: Re: [undertow-dev] How to handle exceptions thrown from worker threads

I think you are correct that we don't handle this case very well, the
basic error handing strategy has always been to just catch the error
in a handler, but when you don't have control over the dispatch site
this gets tricky.

We should definitely make the error available in
DefaultResponseListener, probably just as an attachment on the
exchange. Would this be enough to meet your requirements, or would
some kind of actual error handling interface be better?
DefaultResponseListener may not always be called if the exchange has
already been finished when the error occurs (i.e. the page has already
been sent to the user).

Stuart

On Wed, Jun 15, 2016 at 2:01 AM, Wolinsky, Jared <jared.wolinsky at mlb.com> wrote:
> Good morning,
>
> We're using Undertow in a new web service at MLB, and I haven't been able to
> find a solution for handling errors that occur in requests that have been
> dispatched to worker threads.  Here is a link to my SO question:
> http://stackoverflow.com/questions/37753726/how-to-handle-exceptions-thrown-from-undertow-worker-threads.
> Ignoring worker threads, my model for error handling is to have an outer
> handler that maintains a map of exception type -> {log level, http status
> code}.  Then I allow exceptions to propagate out from the actual API
> handlers, hitting the error handler which logs the error and responds to the
> client with the correct status code based on the exception type.  When
> exceptions are thrown from worker threads, however, they break out of the
> normal handler chain and never hit the error handler, resulting in a 500
> response every time.  Here are options I've considered for handling this
> problem:
>
> Inside my error handler, set a DefaultResponseListener
> (https://http2.undertow.io/documentation/core/error-handling.html) on the
> exchange that contains my error handling logic.  I hit a wall here after
> realizing the listener only has access to the Exchange, not to the exception
> that will be thrown.
> Set an UncaughtExceptionHandler on each handler that's on a worker thread.
> This code would be added to my base handler class, do a
> !exchange.isInIoThread() check, set the error handler, then call the
> handleRequest method.  The caveat here is that I can only do this for
> handlers that I have control over, not built-in handlers that dispatch to
> worker threads such as the AuthenticationCallHandler.
> Use an AspectJ-based approach where I create an @Around advice that executes
> for all of my handlers that will wrap the execution in a try-catch, and take
> care of the error handling.
>
>
> Are there best practices for handling this?
>
>
> Thanks,
>
> Jared Wolinsky
>
> Senior Software Engineer, MLB Advanced Media
>
>
>
> **********************************************************
>
> MLB.com - Where Baseball is Always On.
>
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev



**********************************************************

MLB.com <http://mlb.com> - Where Baseball is Always On.



From peter.royal at pobox.com  Thu Jun 16 13:58:30 2016
From: peter.royal at pobox.com (peter royal)
Date: Thu, 16 Jun 2016 12:58:30 -0500
Subject: [undertow-dev] Websocket connections and IO thread affinity
Message-ID: <1466099910.3550017.639859769.76A4CEB5@webmail.messagingengine.com>

(I believe the following is true... please correct me if not!)

I have an application which heavily utilizes web sockets. It is an
internal application which uses a small number of connections with
reasonable load on each.

When a new connection is received by Undertow, there is an
at-connection-time assignment of an XNIO IO Thread to the connection.
This is causing uneven load on my IO threads, due to chance. 

I'm increasing the number of IO threads as a temporary fix, but it might
be useful to be able to either migrate a long-lived connection to
another IO thread (harder) or do better load balancing amongst IO
threads. For the latter, if Undertow was able to provide a strategy for
picking a thread in NioXnioWorker.getIoThread(hashCode), it could try
and pick a thread that had fewer connections assigned to it.

Has anyone else run into this problem? Would a fix be accepted?

-pete

-- 
(peter.royal|osi)@pobox.com - http://fotap.org/~osi

From jason.greene at redhat.com  Thu Jun 16 14:41:50 2016
From: jason.greene at redhat.com (Jason T. Greene)
Date: Thu, 16 Jun 2016 14:41:50 -0400 (EDT)
Subject: [undertow-dev] Websocket connections and IO thread affinity
In-Reply-To: <1466099910.3550017.639859769.76A4CEB5@webmail.messagingengine.com>
References: <1466099910.3550017.639859769.76A4CEB5@webmail.messagingengine.com>
Message-ID: <9662EDD0-8415-450D-9C0E-0F483C74A97E@redhat.com>

We recently changed xnio to balance connections by default using a strategy similar to the new SO_REUSEPORT facility in the Linux kernel (3.3.3 or later). In the short future,  we will be switching to the native facility when accessible in the JDK NIO implementation. Older versions had a feature called balancing tokens that you could use to balance connections fairly, but it had to be especially configured.  


> On Jun 16, 2016, at 1:00 PM, peter royal <peter.royal at pobox.com> wrote:
> 
> (I believe the following is true... please correct me if not!)
> 
> I have an application which heavily utilizes web sockets. It is an
> internal application which uses a small number of connections with
> reasonable load on each.
> 
> When a new connection is received by Undertow, there is an
> at-connection-time assignment of an XNIO IO Thread to the connection.
> This is causing uneven load on my IO threads, due to chance. 
> 
> I'm increasing the number of IO threads as a temporary fix, but it might
> be useful to be able to either migrate a long-lived connection to
> another IO thread (harder) or do better load balancing amongst IO
> threads. For the latter, if Undertow was able to provide a strategy for
> picking a thread in NioXnioWorker.getIoThread(hashCode), it could try
> and pick a thread that had fewer connections assigned to it.
> 
> Has anyone else run into this problem? Would a fix be accepted?
> 
> -pete
> 
> -- 
> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev


From jason.greene at redhat.com  Thu Jun 16 14:49:22 2016
From: jason.greene at redhat.com (Jason T. Greene)
Date: Thu, 16 Jun 2016 14:49:22 -0400 (EDT)
Subject: [undertow-dev] Websocket connections and IO thread affinity
In-Reply-To: <9662EDD0-8415-450D-9C0E-0F483C74A97E@redhat.com>
References: <1466099910.3550017.639859769.76A4CEB5@webmail.messagingengine.com>
	<9662EDD0-8415-450D-9C0E-0F483C74A97E@redhat.com>
Message-ID: <45A13C5B-5A80-4FCD-9BCA-35B00EC013AF@redhat.com>

Sorry that got a bit truncated.  My next paragraph was going to ask which version you are using to see if this approach helps you. I think we'd be open to alternative balancing strategies.

> On Jun 16, 2016, at 1:41 PM, Jason T. Greene <jason.greene at redhat.com> wrote:
> 
> We recently changed xnio to balance connections by default using a strategy similar to the new SO_REUSEPORT facility in the Linux kernel (3.3.3 or later). In the short future,  we will be switching to the native facility when accessible in the JDK NIO implementation. Older versions had a feature called balancing tokens that you could use to balance connections fairly, but it had to be especially configured.  
> 
> 
>> On Jun 16, 2016, at 1:00 PM, peter royal <peter.royal at pobox.com> wrote:
>> 
>> (I believe the following is true... please correct me if not!)
>> 
>> I have an application which heavily utilizes web sockets. It is an
>> internal application which uses a small number of connections with
>> reasonable load on each.
>> 
>> When a new connection is received by Undertow, there is an
>> at-connection-time assignment of an XNIO IO Thread to the connection.
>> This is causing uneven load on my IO threads, due to chance. 
>> 
>> I'm increasing the number of IO threads as a temporary fix, but it might
>> be useful to be able to either migrate a long-lived connection to
>> another IO thread (harder) or do better load balancing amongst IO
>> threads. For the latter, if Undertow was able to provide a strategy for
>> picking a thread in NioXnioWorker.getIoThread(hashCode), it could try
>> and pick a thread that had fewer connections assigned to it.
>> 
>> Has anyone else run into this problem? Would a fix be accepted?
>> 
>> -pete
>> 
>> -- 
>> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
>> _______________________________________________
>> undertow-dev mailing list
>> undertow-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/undertow-dev


From peter.royal at pobox.com  Thu Jun 16 15:01:01 2016
From: peter.royal at pobox.com (peter royal)
Date: Thu, 16 Jun 2016 14:01:01 -0500
Subject: [undertow-dev] Websocket connections and IO thread affinity
In-Reply-To: <9662EDD0-8415-450D-9C0E-0F483C74A97E@redhat.com>
References: <1466099910.3550017.639859769.76A4CEB5@webmail.messagingengine.com>
	<9662EDD0-8415-450D-9C0E-0F483C74A97E@redhat.com>
Message-ID: <1466103661.3565919.639925417.0C3C1816@webmail.messagingengine.com>

Gotcha. I was digging through things and found the change where the new
strategy was introduced. With my current # of IO threads it is giving
un-even weighings:

thread, connections
0, 6
1, 5
2, 3
3, 2

I'm going to double my IO threads, but it will still be less than
optimal, but improved:

thread, connections
0, 2
1, 1
2, 1
3, 1
4, 4
5, 4
6, 2
7, 1

Random is only slightly better, eyeballing things.

I'm using Undertow 1.3.22 which uses XNIO 3.3.6.  Linux kernel 2.6.32
though.

Digging into my problem more, I would probably need to balance on more
than just connection count per IO thread, as some connections are busier
than others. 

Can you point me towards any references about the forthcoming access to
native facility? I'm curious as to how that will work

-pete

-- 
(peter.royal|osi)@pobox.com - http://fotap.org/~osi

On Thu, Jun 16, 2016, at 01:41 PM, Jason T. Greene wrote:
> We recently changed xnio to balance connections by default using a
> strategy similar to the new SO_REUSEPORT facility in the Linux kernel
> (3.3.3 or later). In the short future,  we will be switching to the
> native facility when accessible in the JDK NIO implementation. Older
> versions had a feature called balancing tokens that you could use to
> balance connections fairly, but it had to be especially configured.  
> 
> 
> > On Jun 16, 2016, at 1:00 PM, peter royal <peter.royal at pobox.com> wrote:
> > 
> > (I believe the following is true... please correct me if not!)
> > 
> > I have an application which heavily utilizes web sockets. It is an
> > internal application which uses a small number of connections with
> > reasonable load on each.
> > 
> > When a new connection is received by Undertow, there is an
> > at-connection-time assignment of an XNIO IO Thread to the connection.
> > This is causing uneven load on my IO threads, due to chance. 
> > 
> > I'm increasing the number of IO threads as a temporary fix, but it might
> > be useful to be able to either migrate a long-lived connection to
> > another IO thread (harder) or do better load balancing amongst IO
> > threads. For the latter, if Undertow was able to provide a strategy for
> > picking a thread in NioXnioWorker.getIoThread(hashCode), it could try
> > and pick a thread that had fewer connections assigned to it.
> > 
> > Has anyone else run into this problem? Would a fix be accepted?
> > 
> > -pete
> > 
> > -- 
> > (peter.royal|osi)@pobox.com - http://fotap.org/~osi
> > _______________________________________________
> > undertow-dev mailing list
> > undertow-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/undertow-dev

From jason.greene at redhat.com  Thu Jun 16 15:17:14 2016
From: jason.greene at redhat.com (Jason Greene)
Date: Thu, 16 Jun 2016 14:17:14 -0500
Subject: [undertow-dev] Websocket connections and IO thread affinity
In-Reply-To: <1466103661.3565919.639925417.0C3C1816@webmail.messagingengine.com>
References: <1466099910.3550017.639859769.76A4CEB5@webmail.messagingengine.com>
	<9662EDD0-8415-450D-9C0E-0F483C74A97E@redhat.com>
	<1466103661.3565919.639925417.0C3C1816@webmail.messagingengine.com>
Message-ID: <C0DA4CF3-6B5E-443D-8334-93F4E8016BD0@redhat.com>

The way our current approach works, which is the same approach as SO_REUSEPORT?s impl is that address:port are hashed to select the destination, this is mainly so we can transition with no real behavioral surprises. If you have some connections lasting significantly longer than others, then you will eventually go out of balance because the current allocation state isn?t a factor into the decision. It?s possible to do more advanced algorithms factoring in state, but once you do that you tie yourself to a single threaded acceptor (although thats currently the case with our emulated SO_REUSEPORT implementation). For many workloads this won?t matter though, as you need massive connection rates to hit the accept stability limits.

Maybe you want to play with modifying QueuedTcpNioServer to compare a few different algorithms? You could try balancing active connection count as one strategy, and perhaps thread user cpu time as another. For both approaches you probably want to have i/o threads individually updating a volatile statistic field as part of their standard work, and then the accept queuing thread scanning those values to select the best destination.

> On Jun 16, 2016, at 2:01 PM, peter royal <peter.royal at pobox.com> wrote:
> 
> Gotcha. I was digging through things and found the change where the new
> strategy was introduced. With my current # of IO threads it is giving
> un-even weighings:
> 
> thread, connections
> 0, 6
> 1, 5
> 2, 3
> 3, 2
> 
> I'm going to double my IO threads, but it will still be less than
> optimal, but improved:
> 
> thread, connections
> 0, 2
> 1, 1
> 2, 1
> 3, 1
> 4, 4
> 5, 4
> 6, 2
> 7, 1
> 
> Random is only slightly better, eyeballing things.
> 
> I'm using Undertow 1.3.22 which uses XNIO 3.3.6.  Linux kernel 2.6.32
> though.
> 
> Digging into my problem more, I would probably need to balance on more
> than just connection count per IO thread, as some connections are busier
> than others. 
> 
> Can you point me towards any references about the forthcoming access to
> native facility? I'm curious as to how that will work
> 
> -pete
> 
> -- 
> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
> 
> On Thu, Jun 16, 2016, at 01:41 PM, Jason T. Greene wrote:
>> We recently changed xnio to balance connections by default using a
>> strategy similar to the new SO_REUSEPORT facility in the Linux kernel
>> (3.3.3 or later). In the short future,  we will be switching to the
>> native facility when accessible in the JDK NIO implementation. Older
>> versions had a feature called balancing tokens that you could use to
>> balance connections fairly, but it had to be especially configured.  
>> 
>> 
>>> On Jun 16, 2016, at 1:00 PM, peter royal <peter.royal at pobox.com> wrote:
>>> 
>>> (I believe the following is true... please correct me if not!)
>>> 
>>> I have an application which heavily utilizes web sockets. It is an
>>> internal application which uses a small number of connections with
>>> reasonable load on each.
>>> 
>>> When a new connection is received by Undertow, there is an
>>> at-connection-time assignment of an XNIO IO Thread to the connection.
>>> This is causing uneven load on my IO threads, due to chance. 
>>> 
>>> I'm increasing the number of IO threads as a temporary fix, but it might
>>> be useful to be able to either migrate a long-lived connection to
>>> another IO thread (harder) or do better load balancing amongst IO
>>> threads. For the latter, if Undertow was able to provide a strategy for
>>> picking a thread in NioXnioWorker.getIoThread(hashCode), it could try
>>> and pick a thread that had fewer connections assigned to it.
>>> 
>>> Has anyone else run into this problem? Would a fix be accepted?
>>> 
>>> -pete
>>> 
>>> -- 
>>> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
>>> _______________________________________________
>>> undertow-dev mailing list
>>> undertow-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/undertow-dev

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat



From peter.royal at pobox.com  Thu Jun 16 15:36:15 2016
From: peter.royal at pobox.com (peter royal)
Date: Thu, 16 Jun 2016 14:36:15 -0500
Subject: [undertow-dev] Websocket connections and IO thread affinity
In-Reply-To: <C0DA4CF3-6B5E-443D-8334-93F4E8016BD0@redhat.com>
References: <1466099910.3550017.639859769.76A4CEB5@webmail.messagingengine.com>
	<9662EDD0-8415-450D-9C0E-0F483C74A97E@redhat.com>
	<1466103661.3565919.639925417.0C3C1816@webmail.messagingengine.com>
	<C0DA4CF3-6B5E-443D-8334-93F4E8016BD0@redhat.com>
Message-ID: <1466105775.3574369.639969105.11ADCB79@webmail.messagingengine.com>

Understood. 

I'm going to test with increased IO threads, and if that fixes things
I'm good. Using thread user CPU time might be a good metric, as looking
at that the imbalance is clear:

CPU: 2673514 ms
CPU: 31270 ms
CPU: 61962 ms
CPU: 7952561 ms

As I think through this more, optimal balancing requires pushing a lot
of application-specific info down low, because a given WS connection
might be high volume or not. it would be easier to migrate a connection
that is detected to be high volume to another IO thread, but that'd be a
hugely invasive change. The optimal strategy for me might just be to
have 1 thread per connection as the counts aren't very high.

Thanks for the help!

-- 
(peter.royal|osi)@pobox.com - http://fotap.org/~osi

On Thu, Jun 16, 2016, at 02:17 PM, Jason Greene wrote:
> The way our current approach works, which is the same approach as
> SO_REUSEPORT?s impl is that address:port are hashed to select the
> destination, this is mainly so we can transition with no real behavioral
> surprises. If you have some connections lasting significantly longer than
> others, then you will eventually go out of balance because the current
> allocation state isn?t a factor into the decision. It?s possible to do
> more advanced algorithms factoring in state, but once you do that you tie
> yourself to a single threaded acceptor (although thats currently the case
> with our emulated SO_REUSEPORT implementation). For many workloads this
> won?t matter though, as you need massive connection rates to hit the
> accept stability limits.
> 
> Maybe you want to play with modifying QueuedTcpNioServer to compare a few
> different algorithms? You could try balancing active connection count as
> one strategy, and perhaps thread user cpu time as another. For both
> approaches you probably want to have i/o threads individually updating a
> volatile statistic field as part of their standard work, and then the
> accept queuing thread scanning those values to select the best
> destination.
> 
> > On Jun 16, 2016, at 2:01 PM, peter royal <peter.royal at pobox.com> wrote:
> > 
> > Gotcha. I was digging through things and found the change where the new
> > strategy was introduced. With my current # of IO threads it is giving
> > un-even weighings:
> > 
> > thread, connections
> > 0, 6
> > 1, 5
> > 2, 3
> > 3, 2
> > 
> > I'm going to double my IO threads, but it will still be less than
> > optimal, but improved:
> > 
> > thread, connections
> > 0, 2
> > 1, 1
> > 2, 1
> > 3, 1
> > 4, 4
> > 5, 4
> > 6, 2
> > 7, 1
> > 
> > Random is only slightly better, eyeballing things.
> > 
> > I'm using Undertow 1.3.22 which uses XNIO 3.3.6.  Linux kernel 2.6.32
> > though.
> > 
> > Digging into my problem more, I would probably need to balance on more
> > than just connection count per IO thread, as some connections are busier
> > than others. 
> > 
> > Can you point me towards any references about the forthcoming access to
> > native facility? I'm curious as to how that will work
> > 
> > -pete
> > 
> > -- 
> > (peter.royal|osi)@pobox.com - http://fotap.org/~osi
> > 
> > On Thu, Jun 16, 2016, at 01:41 PM, Jason T. Greene wrote:
> >> We recently changed xnio to balance connections by default using a
> >> strategy similar to the new SO_REUSEPORT facility in the Linux kernel
> >> (3.3.3 or later). In the short future,  we will be switching to the
> >> native facility when accessible in the JDK NIO implementation. Older
> >> versions had a feature called balancing tokens that you could use to
> >> balance connections fairly, but it had to be especially configured.  
> >> 
> >> 
> >>> On Jun 16, 2016, at 1:00 PM, peter royal <peter.royal at pobox.com> wrote:
> >>> 
> >>> (I believe the following is true... please correct me if not!)
> >>> 
> >>> I have an application which heavily utilizes web sockets. It is an
> >>> internal application which uses a small number of connections with
> >>> reasonable load on each.
> >>> 
> >>> When a new connection is received by Undertow, there is an
> >>> at-connection-time assignment of an XNIO IO Thread to the connection.
> >>> This is causing uneven load on my IO threads, due to chance. 
> >>> 
> >>> I'm increasing the number of IO threads as a temporary fix, but it might
> >>> be useful to be able to either migrate a long-lived connection to
> >>> another IO thread (harder) or do better load balancing amongst IO
> >>> threads. For the latter, if Undertow was able to provide a strategy for
> >>> picking a thread in NioXnioWorker.getIoThread(hashCode), it could try
> >>> and pick a thread that had fewer connections assigned to it.
> >>> 
> >>> Has anyone else run into this problem? Would a fix be accepted?
> >>> 
> >>> -pete
> >>> 
> >>> -- 
> >>> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
> >>> _______________________________________________
> >>> undertow-dev mailing list
> >>> undertow-dev at lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/undertow-dev
> 
> --
> Jason T. Greene
> WildFly Lead / JBoss EAP Platform Architect
> JBoss, a division of Red Hat
> 


From jason.greene at redhat.com  Thu Jun 16 15:59:47 2016
From: jason.greene at redhat.com (Jason Greene)
Date: Thu, 16 Jun 2016 14:59:47 -0500
Subject: [undertow-dev] Websocket connections and IO thread affinity
In-Reply-To: <1466105775.3574369.639969105.11ADCB79@webmail.messagingengine.com>
References: <1466099910.3550017.639859769.76A4CEB5@webmail.messagingengine.com>
	<9662EDD0-8415-450D-9C0E-0F483C74A97E@redhat.com>
	<1466103661.3565919.639925417.0C3C1816@webmail.messagingengine.com>
	<C0DA4CF3-6B5E-443D-8334-93F4E8016BD0@redhat.com>
	<1466105775.3574369.639969105.11ADCB79@webmail.messagingengine.com>
Message-ID: <C1A4245C-2DCB-4948-B086-320D22EC6031@redhat.com>

Are you using WildFly or Undertow standalone? 

If you are using undertow standalone, you might want to try enabling dispatch to worker (this is the default on WildFly):
webSocketDeploymentInfo.setDispatchToWorkerThread(true)

If you have message handlers that use significant CPU time or introduce blocking (which a disparity like you see could potentially indicate), then it can negatively impact the I/O thread?s ability to handle connection events efficiently, and so dispatching to the worker pool allows for long running tasks to execute without interfering with other connections/activity. 

> On Jun 16, 2016, at 2:36 PM, peter royal <peter.royal at pobox.com> wrote:
> 
> Understood. 
> 
> I'm going to test with increased IO threads, and if that fixes things
> I'm good. Using thread user CPU time might be a good metric, as looking
> at that the imbalance is clear:
> 
> CPU: 2673514 ms
> CPU: 31270 ms
> CPU: 61962 ms
> CPU: 7952561 ms
> 
> As I think through this more, optimal balancing requires pushing a lot
> of application-specific info down low, because a given WS connection
> might be high volume or not. it would be easier to migrate a connection
> that is detected to be high volume to another IO thread, but that'd be a
> hugely invasive change. The optimal strategy for me might just be to
> have 1 thread per connection as the counts aren't very high.
> 
> Thanks for the help!
> 
> -- 
> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
> 
> On Thu, Jun 16, 2016, at 02:17 PM, Jason Greene wrote:
>> The way our current approach works, which is the same approach as
>> SO_REUSEPORT?s impl is that address:port are hashed to select the
>> destination, this is mainly so we can transition with no real behavioral
>> surprises. If you have some connections lasting significantly longer than
>> others, then you will eventually go out of balance because the current
>> allocation state isn?t a factor into the decision. It?s possible to do
>> more advanced algorithms factoring in state, but once you do that you tie
>> yourself to a single threaded acceptor (although thats currently the case
>> with our emulated SO_REUSEPORT implementation). For many workloads this
>> won?t matter though, as you need massive connection rates to hit the
>> accept stability limits.
>> 
>> Maybe you want to play with modifying QueuedTcpNioServer to compare a few
>> different algorithms? You could try balancing active connection count as
>> one strategy, and perhaps thread user cpu time as another. For both
>> approaches you probably want to have i/o threads individually updating a
>> volatile statistic field as part of their standard work, and then the
>> accept queuing thread scanning those values to select the best
>> destination.
>> 
>>> On Jun 16, 2016, at 2:01 PM, peter royal <peter.royal at pobox.com> wrote:
>>> 
>>> Gotcha. I was digging through things and found the change where the new
>>> strategy was introduced. With my current # of IO threads it is giving
>>> un-even weighings:
>>> 
>>> thread, connections
>>> 0, 6
>>> 1, 5
>>> 2, 3
>>> 3, 2
>>> 
>>> I'm going to double my IO threads, but it will still be less than
>>> optimal, but improved:
>>> 
>>> thread, connections
>>> 0, 2
>>> 1, 1
>>> 2, 1
>>> 3, 1
>>> 4, 4
>>> 5, 4
>>> 6, 2
>>> 7, 1
>>> 
>>> Random is only slightly better, eyeballing things.
>>> 
>>> I'm using Undertow 1.3.22 which uses XNIO 3.3.6.  Linux kernel 2.6.32
>>> though.
>>> 
>>> Digging into my problem more, I would probably need to balance on more
>>> than just connection count per IO thread, as some connections are busier
>>> than others. 
>>> 
>>> Can you point me towards any references about the forthcoming access to
>>> native facility? I'm curious as to how that will work
>>> 
>>> -pete
>>> 
>>> -- 
>>> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
>>> 
>>> On Thu, Jun 16, 2016, at 01:41 PM, Jason T. Greene wrote:
>>>> We recently changed xnio to balance connections by default using a
>>>> strategy similar to the new SO_REUSEPORT facility in the Linux kernel
>>>> (3.3.3 or later). In the short future,  we will be switching to the
>>>> native facility when accessible in the JDK NIO implementation. Older
>>>> versions had a feature called balancing tokens that you could use to
>>>> balance connections fairly, but it had to be especially configured.  
>>>> 
>>>> 
>>>>> On Jun 16, 2016, at 1:00 PM, peter royal <peter.royal at pobox.com> wrote:
>>>>> 
>>>>> (I believe the following is true... please correct me if not!)
>>>>> 
>>>>> I have an application which heavily utilizes web sockets. It is an
>>>>> internal application which uses a small number of connections with
>>>>> reasonable load on each.
>>>>> 
>>>>> When a new connection is received by Undertow, there is an
>>>>> at-connection-time assignment of an XNIO IO Thread to the connection.
>>>>> This is causing uneven load on my IO threads, due to chance. 
>>>>> 
>>>>> I'm increasing the number of IO threads as a temporary fix, but it might
>>>>> be useful to be able to either migrate a long-lived connection to
>>>>> another IO thread (harder) or do better load balancing amongst IO
>>>>> threads. For the latter, if Undertow was able to provide a strategy for
>>>>> picking a thread in NioXnioWorker.getIoThread(hashCode), it could try
>>>>> and pick a thread that had fewer connections assigned to it.
>>>>> 
>>>>> Has anyone else run into this problem? Would a fix be accepted?
>>>>> 
>>>>> -pete
>>>>> 
>>>>> -- 
>>>>> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
>>>>> _______________________________________________
>>>>> undertow-dev mailing list
>>>>> undertow-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>> 
>> --
>> Jason T. Greene
>> WildFly Lead / JBoss EAP Platform Architect
>> JBoss, a division of Red Hat
>> 

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat



From peter.royal at pobox.com  Thu Jun 16 16:15:54 2016
From: peter.royal at pobox.com (peter royal)
Date: Thu, 16 Jun 2016 15:15:54 -0500
Subject: [undertow-dev] Websocket connections and IO thread affinity
In-Reply-To: <C1A4245C-2DCB-4948-B086-320D22EC6031@redhat.com>
References: <1466099910.3550017.639859769.76A4CEB5@webmail.messagingengine.com>
	<9662EDD0-8415-450D-9C0E-0F483C74A97E@redhat.com>
	<1466103661.3565919.639925417.0C3C1816@webmail.messagingengine.com>
	<C0DA4CF3-6B5E-443D-8334-93F4E8016BD0@redhat.com>
	<1466105775.3574369.639969105.11ADCB79@webmail.messagingengine.com>
	<C1A4245C-2DCB-4948-B086-320D22EC6031@redhat.com>
Message-ID: <1466108154.3584990.640000809.5C53BB8C@webmail.messagingengine.com>

Undertow standalone. 

Thanks for the reminder about that - I went back and looked at things
and realized I was doing deserialization of incoming messages on the IO
thread prior to dispatching to the worker thread. The outbound path was
clean though. 

Thanks again!

-pete

-- 
(peter.royal|osi)@pobox.com - http://fotap.org/~osi

On Thu, Jun 16, 2016, at 02:59 PM, Jason Greene wrote:
> Are you using WildFly or Undertow standalone? 
> 
> If you are using undertow standalone, you might want to try enabling
> dispatch to worker (this is the default on WildFly):
> webSocketDeploymentInfo.setDispatchToWorkerThread(true)
> 
> If you have message handlers that use significant CPU time or introduce
> blocking (which a disparity like you see could potentially indicate),
> then it can negatively impact the I/O thread?s ability to handle
> connection events efficiently, and so dispatching to the worker pool
> allows for long running tasks to execute without interfering with other
> connections/activity. 
> 
> > On Jun 16, 2016, at 2:36 PM, peter royal <peter.royal at pobox.com> wrote:
> > 
> > Understood. 
> > 
> > I'm going to test with increased IO threads, and if that fixes things
> > I'm good. Using thread user CPU time might be a good metric, as looking
> > at that the imbalance is clear:
> > 
> > CPU: 2673514 ms
> > CPU: 31270 ms
> > CPU: 61962 ms
> > CPU: 7952561 ms
> > 
> > As I think through this more, optimal balancing requires pushing a lot
> > of application-specific info down low, because a given WS connection
> > might be high volume or not. it would be easier to migrate a connection
> > that is detected to be high volume to another IO thread, but that'd be a
> > hugely invasive change. The optimal strategy for me might just be to
> > have 1 thread per connection as the counts aren't very high.
> > 
> > Thanks for the help!
> > 
> > -- 
> > (peter.royal|osi)@pobox.com - http://fotap.org/~osi
> > 
> > On Thu, Jun 16, 2016, at 02:17 PM, Jason Greene wrote:
> >> The way our current approach works, which is the same approach as
> >> SO_REUSEPORT?s impl is that address:port are hashed to select the
> >> destination, this is mainly so we can transition with no real behavioral
> >> surprises. If you have some connections lasting significantly longer than
> >> others, then you will eventually go out of balance because the current
> >> allocation state isn?t a factor into the decision. It?s possible to do
> >> more advanced algorithms factoring in state, but once you do that you tie
> >> yourself to a single threaded acceptor (although thats currently the case
> >> with our emulated SO_REUSEPORT implementation). For many workloads this
> >> won?t matter though, as you need massive connection rates to hit the
> >> accept stability limits.
> >> 
> >> Maybe you want to play with modifying QueuedTcpNioServer to compare a few
> >> different algorithms? You could try balancing active connection count as
> >> one strategy, and perhaps thread user cpu time as another. For both
> >> approaches you probably want to have i/o threads individually updating a
> >> volatile statistic field as part of their standard work, and then the
> >> accept queuing thread scanning those values to select the best
> >> destination.
> >> 
> >>> On Jun 16, 2016, at 2:01 PM, peter royal <peter.royal at pobox.com> wrote:
> >>> 
> >>> Gotcha. I was digging through things and found the change where the new
> >>> strategy was introduced. With my current # of IO threads it is giving
> >>> un-even weighings:
> >>> 
> >>> thread, connections
> >>> 0, 6
> >>> 1, 5
> >>> 2, 3
> >>> 3, 2
> >>> 
> >>> I'm going to double my IO threads, but it will still be less than
> >>> optimal, but improved:
> >>> 
> >>> thread, connections
> >>> 0, 2
> >>> 1, 1
> >>> 2, 1
> >>> 3, 1
> >>> 4, 4
> >>> 5, 4
> >>> 6, 2
> >>> 7, 1
> >>> 
> >>> Random is only slightly better, eyeballing things.
> >>> 
> >>> I'm using Undertow 1.3.22 which uses XNIO 3.3.6.  Linux kernel 2.6.32
> >>> though.
> >>> 
> >>> Digging into my problem more, I would probably need to balance on more
> >>> than just connection count per IO thread, as some connections are busier
> >>> than others. 
> >>> 
> >>> Can you point me towards any references about the forthcoming access to
> >>> native facility? I'm curious as to how that will work
> >>> 
> >>> -pete
> >>> 
> >>> -- 
> >>> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
> >>> 
> >>> On Thu, Jun 16, 2016, at 01:41 PM, Jason T. Greene wrote:
> >>>> We recently changed xnio to balance connections by default using a
> >>>> strategy similar to the new SO_REUSEPORT facility in the Linux kernel
> >>>> (3.3.3 or later). In the short future,  we will be switching to the
> >>>> native facility when accessible in the JDK NIO implementation. Older
> >>>> versions had a feature called balancing tokens that you could use to
> >>>> balance connections fairly, but it had to be especially configured.  
> >>>> 
> >>>> 
> >>>>> On Jun 16, 2016, at 1:00 PM, peter royal <peter.royal at pobox.com> wrote:
> >>>>> 
> >>>>> (I believe the following is true... please correct me if not!)
> >>>>> 
> >>>>> I have an application which heavily utilizes web sockets. It is an
> >>>>> internal application which uses a small number of connections with
> >>>>> reasonable load on each.
> >>>>> 
> >>>>> When a new connection is received by Undertow, there is an
> >>>>> at-connection-time assignment of an XNIO IO Thread to the connection.
> >>>>> This is causing uneven load on my IO threads, due to chance. 
> >>>>> 
> >>>>> I'm increasing the number of IO threads as a temporary fix, but it might
> >>>>> be useful to be able to either migrate a long-lived connection to
> >>>>> another IO thread (harder) or do better load balancing amongst IO
> >>>>> threads. For the latter, if Undertow was able to provide a strategy for
> >>>>> picking a thread in NioXnioWorker.getIoThread(hashCode), it could try
> >>>>> and pick a thread that had fewer connections assigned to it.
> >>>>> 
> >>>>> Has anyone else run into this problem? Would a fix be accepted?
> >>>>> 
> >>>>> -pete
> >>>>> 
> >>>>> -- 
> >>>>> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
> >>>>> _______________________________________________
> >>>>> undertow-dev mailing list
> >>>>> undertow-dev at lists.jboss.org
> >>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
> >> 
> >> --
> >> Jason T. Greene
> >> WildFly Lead / JBoss EAP Platform Architect
> >> JBoss, a division of Red Hat
> >> 
> 
> --
> Jason T. Greene
> WildFly Lead / JBoss EAP Platform Architect
> JBoss, a division of Red Hat
> 


From sdouglas at redhat.com  Thu Jun 16 20:13:53 2016
From: sdouglas at redhat.com (Stuart Douglas)
Date: Fri, 17 Jun 2016 10:13:53 +1000
Subject: [undertow-dev] Websocket connections and IO thread affinity
In-Reply-To: <1466108154.3584990.640000809.5C53BB8C@webmail.messagingengine.com>
References: <1466099910.3550017.639859769.76A4CEB5@webmail.messagingengine.com>
	<9662EDD0-8415-450D-9C0E-0F483C74A97E@redhat.com>
	<1466103661.3565919.639925417.0C3C1816@webmail.messagingengine.com>
	<C0DA4CF3-6B5E-443D-8334-93F4E8016BD0@redhat.com>
	<1466105775.3574369.639969105.11ADCB79@webmail.messagingengine.com>
	<C1A4245C-2DCB-4948-B086-320D22EC6031@redhat.com>
	<1466108154.3584990.640000809.5C53BB8C@webmail.messagingengine.com>
Message-ID: <CAD+L2cxVvXuO+9Nt0K5rJLNMCiifnOgi6NNEEkxhEtnCaHCfEQ@mail.gmail.com>

Another approach we could potentially try is simply maintaining a
count of active connections per thread, and allocating new connections
to the one with the lowest number of open connections.

This will result in the connect itself being slightly slower, however
it guarantees the most even load distribution you can get without
somehow moving connections between threads (which is not really
something I see as being doable, any gains would probably be lost due
additional thread safety constructs being needed).

Stuart

On Fri, Jun 17, 2016 at 6:15 AM, peter royal <peter.royal at pobox.com> wrote:
> Undertow standalone.
>
> Thanks for the reminder about that - I went back and looked at things
> and realized I was doing deserialization of incoming messages on the IO
> thread prior to dispatching to the worker thread. The outbound path was
> clean though.
>
> Thanks again!
>
> -pete
>
> --
> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
>
> On Thu, Jun 16, 2016, at 02:59 PM, Jason Greene wrote:
>> Are you using WildFly or Undertow standalone?
>>
>> If you are using undertow standalone, you might want to try enabling
>> dispatch to worker (this is the default on WildFly):
>> webSocketDeploymentInfo.setDispatchToWorkerThread(true)
>>
>> If you have message handlers that use significant CPU time or introduce
>> blocking (which a disparity like you see could potentially indicate),
>> then it can negatively impact the I/O thread?s ability to handle
>> connection events efficiently, and so dispatching to the worker pool
>> allows for long running tasks to execute without interfering with other
>> connections/activity.
>>
>> > On Jun 16, 2016, at 2:36 PM, peter royal <peter.royal at pobox.com> wrote:
>> >
>> > Understood.
>> >
>> > I'm going to test with increased IO threads, and if that fixes things
>> > I'm good. Using thread user CPU time might be a good metric, as looking
>> > at that the imbalance is clear:
>> >
>> > CPU: 2673514 ms
>> > CPU: 31270 ms
>> > CPU: 61962 ms
>> > CPU: 7952561 ms
>> >
>> > As I think through this more, optimal balancing requires pushing a lot
>> > of application-specific info down low, because a given WS connection
>> > might be high volume or not. it would be easier to migrate a connection
>> > that is detected to be high volume to another IO thread, but that'd be a
>> > hugely invasive change. The optimal strategy for me might just be to
>> > have 1 thread per connection as the counts aren't very high.
>> >
>> > Thanks for the help!
>> >
>> > --
>> > (peter.royal|osi)@pobox.com - http://fotap.org/~osi
>> >
>> > On Thu, Jun 16, 2016, at 02:17 PM, Jason Greene wrote:
>> >> The way our current approach works, which is the same approach as
>> >> SO_REUSEPORT?s impl is that address:port are hashed to select the
>> >> destination, this is mainly so we can transition with no real behavioral
>> >> surprises. If you have some connections lasting significantly longer than
>> >> others, then you will eventually go out of balance because the current
>> >> allocation state isn?t a factor into the decision. It?s possible to do
>> >> more advanced algorithms factoring in state, but once you do that you tie
>> >> yourself to a single threaded acceptor (although thats currently the case
>> >> with our emulated SO_REUSEPORT implementation). For many workloads this
>> >> won?t matter though, as you need massive connection rates to hit the
>> >> accept stability limits.
>> >>
>> >> Maybe you want to play with modifying QueuedTcpNioServer to compare a few
>> >> different algorithms? You could try balancing active connection count as
>> >> one strategy, and perhaps thread user cpu time as another. For both
>> >> approaches you probably want to have i/o threads individually updating a
>> >> volatile statistic field as part of their standard work, and then the
>> >> accept queuing thread scanning those values to select the best
>> >> destination.
>> >>
>> >>> On Jun 16, 2016, at 2:01 PM, peter royal <peter.royal at pobox.com> wrote:
>> >>>
>> >>> Gotcha. I was digging through things and found the change where the new
>> >>> strategy was introduced. With my current # of IO threads it is giving
>> >>> un-even weighings:
>> >>>
>> >>> thread, connections
>> >>> 0, 6
>> >>> 1, 5
>> >>> 2, 3
>> >>> 3, 2
>> >>>
>> >>> I'm going to double my IO threads, but it will still be less than
>> >>> optimal, but improved:
>> >>>
>> >>> thread, connections
>> >>> 0, 2
>> >>> 1, 1
>> >>> 2, 1
>> >>> 3, 1
>> >>> 4, 4
>> >>> 5, 4
>> >>> 6, 2
>> >>> 7, 1
>> >>>
>> >>> Random is only slightly better, eyeballing things.
>> >>>
>> >>> I'm using Undertow 1.3.22 which uses XNIO 3.3.6.  Linux kernel 2.6.32
>> >>> though.
>> >>>
>> >>> Digging into my problem more, I would probably need to balance on more
>> >>> than just connection count per IO thread, as some connections are busier
>> >>> than others.
>> >>>
>> >>> Can you point me towards any references about the forthcoming access to
>> >>> native facility? I'm curious as to how that will work
>> >>>
>> >>> -pete
>> >>>
>> >>> --
>> >>> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
>> >>>
>> >>> On Thu, Jun 16, 2016, at 01:41 PM, Jason T. Greene wrote:
>> >>>> We recently changed xnio to balance connections by default using a
>> >>>> strategy similar to the new SO_REUSEPORT facility in the Linux kernel
>> >>>> (3.3.3 or later). In the short future,  we will be switching to the
>> >>>> native facility when accessible in the JDK NIO implementation. Older
>> >>>> versions had a feature called balancing tokens that you could use to
>> >>>> balance connections fairly, but it had to be especially configured.
>> >>>>
>> >>>>
>> >>>>> On Jun 16, 2016, at 1:00 PM, peter royal <peter.royal at pobox.com> wrote:
>> >>>>>
>> >>>>> (I believe the following is true... please correct me if not!)
>> >>>>>
>> >>>>> I have an application which heavily utilizes web sockets. It is an
>> >>>>> internal application which uses a small number of connections with
>> >>>>> reasonable load on each.
>> >>>>>
>> >>>>> When a new connection is received by Undertow, there is an
>> >>>>> at-connection-time assignment of an XNIO IO Thread to the connection.
>> >>>>> This is causing uneven load on my IO threads, due to chance.
>> >>>>>
>> >>>>> I'm increasing the number of IO threads as a temporary fix, but it might
>> >>>>> be useful to be able to either migrate a long-lived connection to
>> >>>>> another IO thread (harder) or do better load balancing amongst IO
>> >>>>> threads. For the latter, if Undertow was able to provide a strategy for
>> >>>>> picking a thread in NioXnioWorker.getIoThread(hashCode), it could try
>> >>>>> and pick a thread that had fewer connections assigned to it.
>> >>>>>
>> >>>>> Has anyone else run into this problem? Would a fix be accepted?
>> >>>>>
>> >>>>> -pete
>> >>>>>
>> >>>>> --
>> >>>>> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
>> >>>>> _______________________________________________
>> >>>>> undertow-dev mailing list
>> >>>>> undertow-dev at lists.jboss.org
>> >>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>> >>
>> >> --
>> >> Jason T. Greene
>> >> WildFly Lead / JBoss EAP Platform Architect
>> >> JBoss, a division of Red Hat
>> >>
>>
>> --
>> Jason T. Greene
>> WildFly Lead / JBoss EAP Platform Architect
>> JBoss, a division of Red Hat
>>
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev


From sdouglas at redhat.com  Thu Jun 16 20:45:35 2016
From: sdouglas at redhat.com (Stuart Douglas)
Date: Fri, 17 Jun 2016 10:45:35 +1000
Subject: [undertow-dev] How to handle exceptions thrown from worker
	threads
In-Reply-To: <7A82725263715943B0BE14F7CA24C448678D8CFF@BAM-MBX103.MLBAM.mlb.org>
References: <7A82725263715943B0BE14F7CA24C448678C5BAE@BAM-MBX103.MLBAM.mlb.org>
	<CAD+L2cz5jmSXFFLEeLW0dvU0=cBpqOH89yQ3Tw=T3p6gyEu=Vw@mail.gmail.com>
	<7A82725263715943B0BE14F7CA24C448678D8CFF@BAM-MBX103.MLBAM.mlb.org>
Message-ID: <CAD+L2cx5LQVm9gE8exmwX1BoqwmKZ70D3OzyBDUS5Po5+LfCBQ@mail.gmail.com>

I have added support for this, which will be available in Undertow
1.4.0 which should come out very shortly. The exception is will be
stored under the io.undertow.server.DefaultResponseListener#EXCEPTION
attachment key).

Stuart

On Fri, Jun 17, 2016 at 12:17 AM, Wolinsky, Jared
<jared.wolinsky at mlb.com> wrote:
> For me, adding the exception onto the exchange would be fine, along with a mention in the error handling section of the docs.  I think it's ok if the DefaultResponseListener isn't called if the exchange is ended because that is some handler explicitly taking care of the response and overriding the default case.  That said, I don't think an explicit error handling interface would be a bad way to go either.
>
> I was actually able to work around this using Bill O'Neil's advice and dispatching the request every time in my ErrorHandler, which is the outer handler in my chain.  Then I try-catch the next.handleRequest call, and do my exception handling there.
>
> Thanks,
> Jared
> ________________________________________
> From: Stuart Douglas [sdouglas at redhat.com]
> Sent: Wednesday, June 15, 2016 10:37 PM
> To: Wolinsky, Jared
> Cc: undertow-dev at lists.jboss.org
> Subject: Re: [undertow-dev] How to handle exceptions thrown from worker threads
>
> I think you are correct that we don't handle this case very well, the
> basic error handing strategy has always been to just catch the error
> in a handler, but when you don't have control over the dispatch site
> this gets tricky.
>
> We should definitely make the error available in
> DefaultResponseListener, probably just as an attachment on the
> exchange. Would this be enough to meet your requirements, or would
> some kind of actual error handling interface be better?
> DefaultResponseListener may not always be called if the exchange has
> already been finished when the error occurs (i.e. the page has already
> been sent to the user).
>
> Stuart
>
> On Wed, Jun 15, 2016 at 2:01 AM, Wolinsky, Jared <jared.wolinsky at mlb.com> wrote:
>> Good morning,
>>
>> We're using Undertow in a new web service at MLB, and I haven't been able to
>> find a solution for handling errors that occur in requests that have been
>> dispatched to worker threads.  Here is a link to my SO question:
>> http://stackoverflow.com/questions/37753726/how-to-handle-exceptions-thrown-from-undertow-worker-threads.
>> Ignoring worker threads, my model for error handling is to have an outer
>> handler that maintains a map of exception type -> {log level, http status
>> code}.  Then I allow exceptions to propagate out from the actual API
>> handlers, hitting the error handler which logs the error and responds to the
>> client with the correct status code based on the exception type.  When
>> exceptions are thrown from worker threads, however, they break out of the
>> normal handler chain and never hit the error handler, resulting in a 500
>> response every time.  Here are options I've considered for handling this
>> problem:
>>
>> Inside my error handler, set a DefaultResponseListener
>> (https://http2.undertow.io/documentation/core/error-handling.html) on the
>> exchange that contains my error handling logic.  I hit a wall here after
>> realizing the listener only has access to the Exchange, not to the exception
>> that will be thrown.
>> Set an UncaughtExceptionHandler on each handler that's on a worker thread.
>> This code would be added to my base handler class, do a
>> !exchange.isInIoThread() check, set the error handler, then call the
>> handleRequest method.  The caveat here is that I can only do this for
>> handlers that I have control over, not built-in handlers that dispatch to
>> worker threads such as the AuthenticationCallHandler.
>> Use an AspectJ-based approach where I create an @Around advice that executes
>> for all of my handlers that will wrap the execution in a try-catch, and take
>> care of the error handling.
>>
>>
>> Are there best practices for handling this?
>>
>>
>> Thanks,
>>
>> Jared Wolinsky
>>
>> Senior Software Engineer, MLB Advanced Media
>>
>>
>>
>> **********************************************************
>>
>> MLB.com - Where Baseball is Always On.
>>
>>
>> _______________________________________________
>> undertow-dev mailing list
>> undertow-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
>
>
> **********************************************************
>
> MLB.com <http://mlb.com> - Where Baseball is Always On.
>


From bill at dartalley.com  Thu Jun 16 20:48:27 2016
From: bill at dartalley.com (Bill O'Neil)
Date: Thu, 16 Jun 2016 20:48:27 -0400
Subject: [undertow-dev] How to handle exceptions thrown from worker
	threads
In-Reply-To: <CAD+L2cx5LQVm9gE8exmwX1BoqwmKZ70D3OzyBDUS5Po5+LfCBQ@mail.gmail.com>
References: <7A82725263715943B0BE14F7CA24C448678C5BAE@BAM-MBX103.MLBAM.mlb.org>
	<CAD+L2cz5jmSXFFLEeLW0dvU0=cBpqOH89yQ3Tw=T3p6gyEu=Vw@mail.gmail.com>
	<7A82725263715943B0BE14F7CA24C448678D8CFF@BAM-MBX103.MLBAM.mlb.org>
	<CAD+L2cx5LQVm9gE8exmwX1BoqwmKZ70D3OzyBDUS5Po5+LfCBQ@mail.gmail.com>
Message-ID: <CANd+gFwBJrFKQVBBgmQXRTxoQtfz6v6ujAopavFKKoJo662LCg@mail.gmail.com>

There is something similar here.
https://github.com/undertow-io/undertow/blob/master/core/src/main/java/io/undertow/server/handlers/ExceptionHandler.java#L17

Should they use the same Key or be kept separate?

On Thu, Jun 16, 2016 at 8:45 PM, Stuart Douglas <sdouglas at redhat.com> wrote:

> I have added support for this, which will be available in Undertow
> 1.4.0 which should come out very shortly. The exception is will be
> stored under the io.undertow.server.DefaultResponseListener#EXCEPTION
> attachment key).
>
> Stuart
>
> On Fri, Jun 17, 2016 at 12:17 AM, Wolinsky, Jared
> <jared.wolinsky at mlb.com> wrote:
> > For me, adding the exception onto the exchange would be fine, along with
> a mention in the error handling section of the docs.  I think it's ok if
> the DefaultResponseListener isn't called if the exchange is ended because
> that is some handler explicitly taking care of the response and overriding
> the default case.  That said, I don't think an explicit error handling
> interface would be a bad way to go either.
> >
> > I was actually able to work around this using Bill O'Neil's advice and
> dispatching the request every time in my ErrorHandler, which is the outer
> handler in my chain.  Then I try-catch the next.handleRequest call, and do
> my exception handling there.
> >
> > Thanks,
> > Jared
> > ________________________________________
> > From: Stuart Douglas [sdouglas at redhat.com]
> > Sent: Wednesday, June 15, 2016 10:37 PM
> > To: Wolinsky, Jared
> > Cc: undertow-dev at lists.jboss.org
> > Subject: Re: [undertow-dev] How to handle exceptions thrown from worker
> threads
> >
> > I think you are correct that we don't handle this case very well, the
> > basic error handing strategy has always been to just catch the error
> > in a handler, but when you don't have control over the dispatch site
> > this gets tricky.
> >
> > We should definitely make the error available in
> > DefaultResponseListener, probably just as an attachment on the
> > exchange. Would this be enough to meet your requirements, or would
> > some kind of actual error handling interface be better?
> > DefaultResponseListener may not always be called if the exchange has
> > already been finished when the error occurs (i.e. the page has already
> > been sent to the user).
> >
> > Stuart
> >
> > On Wed, Jun 15, 2016 at 2:01 AM, Wolinsky, Jared <jared.wolinsky at mlb.com>
> wrote:
> >> Good morning,
> >>
> >> We're using Undertow in a new web service at MLB, and I haven't been
> able to
> >> find a solution for handling errors that occur in requests that have
> been
> >> dispatched to worker threads.  Here is a link to my SO question:
> >>
> http://stackoverflow.com/questions/37753726/how-to-handle-exceptions-thrown-from-undertow-worker-threads
> .
> >> Ignoring worker threads, my model for error handling is to have an outer
> >> handler that maintains a map of exception type -> {log level, http
> status
> >> code}.  Then I allow exceptions to propagate out from the actual API
> >> handlers, hitting the error handler which logs the error and responds
> to the
> >> client with the correct status code based on the exception type.  When
> >> exceptions are thrown from worker threads, however, they break out of
> the
> >> normal handler chain and never hit the error handler, resulting in a 500
> >> response every time.  Here are options I've considered for handling this
> >> problem:
> >>
> >> Inside my error handler, set a DefaultResponseListener
> >> (https://http2.undertow.io/documentation/core/error-handling.html) on
> the
> >> exchange that contains my error handling logic.  I hit a wall here after
> >> realizing the listener only has access to the Exchange, not to the
> exception
> >> that will be thrown.
> >> Set an UncaughtExceptionHandler on each handler that's on a worker
> thread.
> >> This code would be added to my base handler class, do a
> >> !exchange.isInIoThread() check, set the error handler, then call the
> >> handleRequest method.  The caveat here is that I can only do this for
> >> handlers that I have control over, not built-in handlers that dispatch
> to
> >> worker threads such as the AuthenticationCallHandler.
> >> Use an AspectJ-based approach where I create an @Around advice that
> executes
> >> for all of my handlers that will wrap the execution in a try-catch, and
> take
> >> care of the error handling.
> >>
> >>
> >> Are there best practices for handling this?
> >>
> >>
> >> Thanks,
> >>
> >> Jared Wolinsky
> >>
> >> Senior Software Engineer, MLB Advanced Media
> >>
> >>
> >>
> >> **********************************************************
> >>
> >> MLB.com - Where Baseball is Always On.
> >>
> >>
> >> _______________________________________________
> >> undertow-dev mailing list
> >> undertow-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/undertow-dev
> >
> >
> >
> > **********************************************************
> >
> > MLB.com <http://mlb.com> - Where Baseball is Always On.
> >
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20160616/81e86781/attachment.html 

From bill at dartalley.com  Thu Jun 16 20:50:09 2016
From: bill at dartalley.com (Bill O'Neil)
Date: Thu, 16 Jun 2016 20:50:09 -0400
Subject: [undertow-dev] How to handle exceptions thrown from worker
	threads
In-Reply-To: <CANd+gFwBJrFKQVBBgmQXRTxoQtfz6v6ujAopavFKKoJo662LCg@mail.gmail.com>
References: <7A82725263715943B0BE14F7CA24C448678C5BAE@BAM-MBX103.MLBAM.mlb.org>
	<CAD+L2cz5jmSXFFLEeLW0dvU0=cBpqOH89yQ3Tw=T3p6gyEu=Vw@mail.gmail.com>
	<7A82725263715943B0BE14F7CA24C448678D8CFF@BAM-MBX103.MLBAM.mlb.org>
	<CAD+L2cx5LQVm9gE8exmwX1BoqwmKZ70D3OzyBDUS5Po5+LfCBQ@mail.gmail.com>
	<CANd+gFwBJrFKQVBBgmQXRTxoQtfz6v6ujAopavFKKoJo662LCg@mail.gmail.com>
Message-ID: <CANd+gFy9+=mrZwhoUxMQssQT3f0_ZinA21ydb3B84_SL-T7Y=Q@mail.gmail.com>

To be honest I don't remember why I added that there. It doesn't make sense
to only add the attachment when the exception is handled and ignore it when
it is not.

On Thu, Jun 16, 2016 at 8:48 PM, Bill O'Neil <bill at dartalley.com> wrote:

> There is something similar here.
> https://github.com/undertow-io/undertow/blob/master/core/src/main/java/io/undertow/server/handlers/ExceptionHandler.java#L17
>
> Should they use the same Key or be kept separate?
>
> On Thu, Jun 16, 2016 at 8:45 PM, Stuart Douglas <sdouglas at redhat.com>
> wrote:
>
>> I have added support for this, which will be available in Undertow
>> 1.4.0 which should come out very shortly. The exception is will be
>> stored under the io.undertow.server.DefaultResponseListener#EXCEPTION
>> attachment key).
>>
>> Stuart
>>
>> On Fri, Jun 17, 2016 at 12:17 AM, Wolinsky, Jared
>> <jared.wolinsky at mlb.com> wrote:
>> > For me, adding the exception onto the exchange would be fine, along
>> with a mention in the error handling section of the docs.  I think it's ok
>> if the DefaultResponseListener isn't called if the exchange is ended
>> because that is some handler explicitly taking care of the response and
>> overriding the default case.  That said, I don't think an explicit error
>> handling interface would be a bad way to go either.
>> >
>> > I was actually able to work around this using Bill O'Neil's advice and
>> dispatching the request every time in my ErrorHandler, which is the outer
>> handler in my chain.  Then I try-catch the next.handleRequest call, and do
>> my exception handling there.
>> >
>> > Thanks,
>> > Jared
>> > ________________________________________
>> > From: Stuart Douglas [sdouglas at redhat.com]
>> > Sent: Wednesday, June 15, 2016 10:37 PM
>> > To: Wolinsky, Jared
>> > Cc: undertow-dev at lists.jboss.org
>> > Subject: Re: [undertow-dev] How to handle exceptions thrown from worker
>> threads
>> >
>> > I think you are correct that we don't handle this case very well, the
>> > basic error handing strategy has always been to just catch the error
>> > in a handler, but when you don't have control over the dispatch site
>> > this gets tricky.
>> >
>> > We should definitely make the error available in
>> > DefaultResponseListener, probably just as an attachment on the
>> > exchange. Would this be enough to meet your requirements, or would
>> > some kind of actual error handling interface be better?
>> > DefaultResponseListener may not always be called if the exchange has
>> > already been finished when the error occurs (i.e. the page has already
>> > been sent to the user).
>> >
>> > Stuart
>> >
>> > On Wed, Jun 15, 2016 at 2:01 AM, Wolinsky, Jared <
>> jared.wolinsky at mlb.com> wrote:
>> >> Good morning,
>> >>
>> >> We're using Undertow in a new web service at MLB, and I haven't been
>> able to
>> >> find a solution for handling errors that occur in requests that have
>> been
>> >> dispatched to worker threads.  Here is a link to my SO question:
>> >>
>> http://stackoverflow.com/questions/37753726/how-to-handle-exceptions-thrown-from-undertow-worker-threads
>> .
>> >> Ignoring worker threads, my model for error handling is to have an
>> outer
>> >> handler that maintains a map of exception type -> {log level, http
>> status
>> >> code}.  Then I allow exceptions to propagate out from the actual API
>> >> handlers, hitting the error handler which logs the error and responds
>> to the
>> >> client with the correct status code based on the exception type.  When
>> >> exceptions are thrown from worker threads, however, they break out of
>> the
>> >> normal handler chain and never hit the error handler, resulting in a
>> 500
>> >> response every time.  Here are options I've considered for handling
>> this
>> >> problem:
>> >>
>> >> Inside my error handler, set a DefaultResponseListener
>> >> (https://http2.undertow.io/documentation/core/error-handling.html) on
>> the
>> >> exchange that contains my error handling logic.  I hit a wall here
>> after
>> >> realizing the listener only has access to the Exchange, not to the
>> exception
>> >> that will be thrown.
>> >> Set an UncaughtExceptionHandler on each handler that's on a worker
>> thread.
>> >> This code would be added to my base handler class, do a
>> >> !exchange.isInIoThread() check, set the error handler, then call the
>> >> handleRequest method.  The caveat here is that I can only do this for
>> >> handlers that I have control over, not built-in handlers that dispatch
>> to
>> >> worker threads such as the AuthenticationCallHandler.
>> >> Use an AspectJ-based approach where I create an @Around advice that
>> executes
>> >> for all of my handlers that will wrap the execution in a try-catch,
>> and take
>> >> care of the error handling.
>> >>
>> >>
>> >> Are there best practices for handling this?
>> >>
>> >>
>> >> Thanks,
>> >>
>> >> Jared Wolinsky
>> >>
>> >> Senior Software Engineer, MLB Advanced Media
>> >>
>> >>
>> >>
>> >> **********************************************************
>> >>
>> >> MLB.com - Where Baseball is Always On.
>> >>
>> >>
>> >> _______________________________________________
>> >> undertow-dev mailing list
>> >> undertow-dev at lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/undertow-dev
>> >
>> >
>> >
>> > **********************************************************
>> >
>> > MLB.com <http://mlb.com> - Where Baseball is Always On.
>> >
>>
>> _______________________________________________
>> undertow-dev mailing list
>> undertow-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20160616/2f22d23f/attachment-0001.html 

From frederick.michielssen at openanalytics.eu  Fri Jun 17 09:50:02 2016
From: frederick.michielssen at openanalytics.eu (Frederick Michielssen)
Date: Fri, 17 Jun 2016 15:50:02 +0200 (CEST)
Subject: [undertow-dev] Websocket proxy: monitor (in)activity
Message-ID: <608073034.290985.1466171402501.JavaMail.zimbra@openanalytics.eu>

Hi,

Our application uses Undertow to serve a Spring-boot application, as well as proxy websocket connections to a backend server.
Now we would like to monitor activity on the websocket, so that specific resources can be freed when the websocket has been inactive for a long time.
Is there an appropriate way we can detect this in Undertow? Maybe some timeout setting that can be used for this purpose?

-Frederick

From toby at tcrawley.org  Sun Jun 19 10:54:02 2016
From: toby at tcrawley.org (Toby Crawley)
Date: Sun, 19 Jun 2016 10:54:02 -0400
Subject: [undertow-dev] Fwd: File descriptor in bad state
In-Reply-To: <13c058c9-b079-4b82-8a01-4f3ed0c1ffda@googlegroups.com>
References: <13c058c9-b079-4b82-8a01-4f3ed0c1ffda@googlegroups.com>
Message-ID: <CAA3Huya11YtX2uTwD0kKBqCQ2Si2dYkQZ49MFCDg_irUcjYOoA@mail.gmail.com>

This came in from an Immutant user on Solaris - are there any known
undertow issues on Solaris? Is it a platform you even test on?

This is using Undertow 1.3.0.Beta9 - we're using 1.3.23.Final in
Immutant HEAD, so can have the user try that if you think it might
help.

- Toby


---------- Forwarded message ----------
From: Dusan Miloradovic <dusan.miloradovic at gmail.com>
Date: Sun, Jun 19, 2016 at 2:33 AM
Subject: File descriptor in bad state
To: Immutant <immutant at googlegroups.com>


Hi

I hope someone can  give me some directions about this issue I have .
I recently upgraded the production server from HTTP Kit to Immutant,
because effectively the server was becoming  unresponsive every week
or so. I was hoping that the issue was with the HTTP Kit itself, and
the move to Immutant would solve the issue.  With the HTTP Kit I
didn't get any error logs when it was  hanging, but with Immutant,
this is what I got:

Exception in thread "XNIO-1 I/O-4" java.lang.InternalError: File
descriptor in bad state
        at sun.nio.ch.EventPortWrapper.release(EventPortWrapper.java:235)
        at sun.nio.ch.EventPortSelectorImpl.implDereg(EventPortSelectorImpl.java:144)
        at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:150)
        at sun.nio.ch.EventPortSelectorImpl.doSelect(EventPortSelectorImpl.java:75)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:510)
Caused by: java.io.IOException: File descriptor in bad state
        at sun.nio.ch.SolarisEventPort.port_dissociate(Native Method)
        at sun.nio.ch.EventPortWrapper.release(EventPortWrapper.java:233)
        ... 6 more

I am running Immutant version 2.1.4 on Sun Solaris version SunOS 5.11
with Oracle JDK 1.7 and  clojure 1.8.0

Are there any Immutant/Undertow parameters I should be aware of with
regards to the problem?

Thanks

--
You received this message because you are subscribed to the Google
Groups "Immutant" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to immutant+unsubscribe at googlegroups.com.
To post to this group, send email to immutant at googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/immutant/13c058c9-b079-4b82-8a01-4f3ed0c1ffda%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

From sdouglas at redhat.com  Sun Jun 19 20:54:24 2016
From: sdouglas at redhat.com (Stuart Douglas)
Date: Mon, 20 Jun 2016 10:54:24 +1000
Subject: [undertow-dev] Fwd: File descriptor in bad state
In-Reply-To: <CAA3Huya11YtX2uTwD0kKBqCQ2Si2dYkQZ49MFCDg_irUcjYOoA@mail.gmail.com>
References: <13c058c9-b079-4b82-8a01-4f3ed0c1ffda@googlegroups.com>
	<CAA3Huya11YtX2uTwD0kKBqCQ2Si2dYkQZ49MFCDg_irUcjYOoA@mail.gmail.com>
Message-ID: <CAD+L2czz5OTfz1K9so_wQci3iqGY2p1qiRJ_RaG9iz9HYmy7TQ@mail.gmail.com>

I have never seen that error before, updating to the latest XNIO and
Undertow may help.

Does the error repeat constantly or is it one off?

Stuart

On Mon, Jun 20, 2016 at 12:54 AM, Toby Crawley <toby at tcrawley.org> wrote:
> This came in from an Immutant user on Solaris - are there any known
> undertow issues on Solaris? Is it a platform you even test on?
>
> This is using Undertow 1.3.0.Beta9 - we're using 1.3.23.Final in
> Immutant HEAD, so can have the user try that if you think it might
> help.
>
> - Toby
>
>
> ---------- Forwarded message ----------
> From: Dusan Miloradovic <dusan.miloradovic at gmail.com>
> Date: Sun, Jun 19, 2016 at 2:33 AM
> Subject: File descriptor in bad state
> To: Immutant <immutant at googlegroups.com>
>
>
> Hi
>
> I hope someone can  give me some directions about this issue I have .
> I recently upgraded the production server from HTTP Kit to Immutant,
> because effectively the server was becoming  unresponsive every week
> or so. I was hoping that the issue was with the HTTP Kit itself, and
> the move to Immutant would solve the issue.  With the HTTP Kit I
> didn't get any error logs when it was  hanging, but with Immutant,
> this is what I got:
>
> Exception in thread "XNIO-1 I/O-4" java.lang.InternalError: File
> descriptor in bad state
>         at sun.nio.ch.EventPortWrapper.release(EventPortWrapper.java:235)
>         at sun.nio.ch.EventPortSelectorImpl.implDereg(EventPortSelectorImpl.java:144)
>         at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:150)
>         at sun.nio.ch.EventPortSelectorImpl.doSelect(EventPortSelectorImpl.java:75)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>         at org.xnio.nio.WorkerThread.run(WorkerThread.java:510)
> Caused by: java.io.IOException: File descriptor in bad state
>         at sun.nio.ch.SolarisEventPort.port_dissociate(Native Method)
>         at sun.nio.ch.EventPortWrapper.release(EventPortWrapper.java:233)
>         ... 6 more
>
> I am running Immutant version 2.1.4 on Sun Solaris version SunOS 5.11
> with Oracle JDK 1.7 and  clojure 1.8.0
>
> Are there any Immutant/Undertow parameters I should be aware of with
> regards to the problem?
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google
> Groups "Immutant" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to immutant+unsubscribe at googlegroups.com.
> To post to this group, send email to immutant at googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/immutant/13c058c9-b079-4b82-8a01-4f3ed0c1ffda%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev

From sdouglas at redhat.com  Sun Jun 19 20:57:18 2016
From: sdouglas at redhat.com (Stuart Douglas)
Date: Mon, 20 Jun 2016 10:57:18 +1000
Subject: [undertow-dev] Websocket proxy: monitor (in)activity
In-Reply-To: <608073034.290985.1466171402501.JavaMail.zimbra@openanalytics.eu>
References: <608073034.290985.1466171402501.JavaMail.zimbra@openanalytics.eu>
Message-ID: <CAD+L2cz7_Ey5ju8=stMUthQhJtpX6FCnRj4h3g=rh9+anGcjLg@mail.gmail.com>

Are you using Wildfly or embedded Undertow?

Either way you basically want to set an idle timeout on the conduit.
Setting io.undertow.UndertowOptions#IDLE_TIMEOUT should do it, if you
are using the embedded builder you need to call setServerOptions with
this setting and your timeout, if you are using wildfly you need to
set in on the listener in standalone.xml.

Stuart

On Fri, Jun 17, 2016 at 11:50 PM, Frederick Michielssen
<frederick.michielssen at openanalytics.eu> wrote:
> Hi,
>
> Our application uses Undertow to serve a Spring-boot application, as well as proxy websocket connections to a backend server.
> Now we would like to monitor activity on the websocket, so that specific resources can be freed when the websocket has been inactive for a long time.
> Is there an appropriate way we can detect this in Undertow? Maybe some timeout setting that can be used for this purpose?
>
> -Frederick
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev

From jason.greene at redhat.com  Sun Jun 19 21:02:11 2016
From: jason.greene at redhat.com (Jason T. Greene)
Date: Sun, 19 Jun 2016 21:02:11 -0400 (EDT)
Subject: [undertow-dev] Fwd: File descriptor in bad state
In-Reply-To: <CAD+L2czz5OTfz1K9so_wQci3iqGY2p1qiRJ_RaG9iz9HYmy7TQ@mail.gmail.com>
References: <13c058c9-b079-4b82-8a01-4f3ed0c1ffda@googlegroups.com>
	<CAA3Huya11YtX2uTwD0kKBqCQ2Si2dYkQZ49MFCDg_irUcjYOoA@mail.gmail.com>
	<CAD+L2czz5OTfz1K9so_wQci3iqGY2p1qiRJ_RaG9iz9HYmy7TQ@mail.gmail.com>
Message-ID: <F99A745B-EB7D-4DB1-BCD9-5C6E2225911D@redhat.com>

You might want to update your JDK as well, the caused by looks like it's trying to remove a socket from the Solaris event set.

> On Jun 19, 2016, at 7:55 PM, Stuart Douglas <sdouglas at redhat.com> wrote:
> 
> I have never seen that error before, updating to the latest XNIO and
> Undertow may help.
> 
> Does the error repeat constantly or is it one off?
> 
> Stuart
> 
>> On Mon, Jun 20, 2016 at 12:54 AM, Toby Crawley <toby at tcrawley.org> wrote:
>> This came in from an Immutant user on Solaris - are there any known
>> undertow issues on Solaris? Is it a platform you even test on?
>> 
>> This is using Undertow 1.3.0.Beta9 - we're using 1.3.23.Final in
>> Immutant HEAD, so can have the user try that if you think it might
>> help.
>> 
>> - Toby
>> 
>> 
>> ---------- Forwarded message ----------
>> From: Dusan Miloradovic <dusan.miloradovic at gmail.com>
>> Date: Sun, Jun 19, 2016 at 2:33 AM
>> Subject: File descriptor in bad state
>> To: Immutant <immutant at googlegroups.com>
>> 
>> 
>> Hi
>> 
>> I hope someone can  give me some directions about this issue I have .
>> I recently upgraded the production server from HTTP Kit to Immutant,
>> because effectively the server was becoming  unresponsive every week
>> or so. I was hoping that the issue was with the HTTP Kit itself, and
>> the move to Immutant would solve the issue.  With the HTTP Kit I
>> didn't get any error logs when it was  hanging, but with Immutant,
>> this is what I got:
>> 
>> Exception in thread "XNIO-1 I/O-4" java.lang.InternalError: File
>> descriptor in bad state
>>        at sun.nio.ch.EventPortWrapper.release(EventPortWrapper.java:235)
>>        at sun.nio.ch.EventPortSelectorImpl.implDereg(EventPortSelectorImpl.java:144)
>>        at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:150)
>>        at sun.nio.ch.EventPortSelectorImpl.doSelect(EventPortSelectorImpl.java:75)
>>        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>        at org.xnio.nio.WorkerThread.run(WorkerThread.java:510)
>> Caused by: java.io.IOException: File descriptor in bad state
>>        at sun.nio.ch.SolarisEventPort.port_dissociate(Native Method)
>>        at sun.nio.ch.EventPortWrapper.release(EventPortWrapper.java:233)
>>        ... 6 more
>> 
>> I am running Immutant version 2.1.4 on Sun Solaris version SunOS 5.11
>> with Oracle JDK 1.7 and  clojure 1.8.0
>> 
>> Are there any Immutant/Undertow parameters I should be aware of with
>> regards to the problem?
>> 
>> Thanks
>> 
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Immutant" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to immutant+unsubscribe at googlegroups.com.
>> To post to this group, send email to immutant at googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/immutant/13c058c9-b079-4b82-8a01-4f3ed0c1ffda%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>> _______________________________________________
>> undertow-dev mailing list
>> undertow-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/undertow-dev
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev


From toby at tcrawley.org  Mon Jun 20 10:03:07 2016
From: toby at tcrawley.org (Toby Crawley)
Date: Mon, 20 Jun 2016 10:03:07 -0400
Subject: [undertow-dev] Fwd: File descriptor in bad state
In-Reply-To: <F99A745B-EB7D-4DB1-BCD9-5C6E2225911D@redhat.com>
References: <13c058c9-b079-4b82-8a01-4f3ed0c1ffda@googlegroups.com>
	<CAA3Huya11YtX2uTwD0kKBqCQ2Si2dYkQZ49MFCDg_irUcjYOoA@mail.gmail.com>
	<CAD+L2czz5OTfz1K9so_wQci3iqGY2p1qiRJ_RaG9iz9HYmy7TQ@mail.gmail.com>
	<F99A745B-EB7D-4DB1-BCD9-5C6E2225911D@redhat.com>
Message-ID: <CAA3HuyZSAsCN=dMeyMr_Yct7bWTE+zNSwJv3HtfLpOFowoU4YA@mail.gmail.com>

Thanks Jason, Stuart. I'll pass those suggestions along and let you
know if they don't help.

- Toby

On Sun, Jun 19, 2016 at 9:02 PM, Jason T. Greene
<jason.greene at redhat.com> wrote:
> You might want to update your JDK as well, the caused by looks like it's trying to remove a socket from the Solaris event set.
>
>> On Jun 19, 2016, at 7:55 PM, Stuart Douglas <sdouglas at redhat.com> wrote:
>>
>> I have never seen that error before, updating to the latest XNIO and
>> Undertow may help.
>>
>> Does the error repeat constantly or is it one off?
>>
>> Stuart
>>
>>> On Mon, Jun 20, 2016 at 12:54 AM, Toby Crawley <toby at tcrawley.org> wrote:
>>> This came in from an Immutant user on Solaris - are there any known
>>> undertow issues on Solaris? Is it a platform you even test on?
>>>
>>> This is using Undertow 1.3.0.Beta9 - we're using 1.3.23.Final in
>>> Immutant HEAD, so can have the user try that if you think it might
>>> help.
>>>
>>> - Toby
>>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Dusan Miloradovic <dusan.miloradovic at gmail.com>
>>> Date: Sun, Jun 19, 2016 at 2:33 AM
>>> Subject: File descriptor in bad state
>>> To: Immutant <immutant at googlegroups.com>
>>>
>>>
>>> Hi
>>>
>>> I hope someone can  give me some directions about this issue I have .
>>> I recently upgraded the production server from HTTP Kit to Immutant,
>>> because effectively the server was becoming  unresponsive every week
>>> or so. I was hoping that the issue was with the HTTP Kit itself, and
>>> the move to Immutant would solve the issue.  With the HTTP Kit I
>>> didn't get any error logs when it was  hanging, but with Immutant,
>>> this is what I got:
>>>
>>> Exception in thread "XNIO-1 I/O-4" java.lang.InternalError: File
>>> descriptor in bad state
>>>        at sun.nio.ch.EventPortWrapper.release(EventPortWrapper.java:235)
>>>        at sun.nio.ch.EventPortSelectorImpl.implDereg(EventPortSelectorImpl.java:144)
>>>        at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:150)
>>>        at sun.nio.ch.EventPortSelectorImpl.doSelect(EventPortSelectorImpl.java:75)
>>>        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>>>        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>>>        at org.xnio.nio.WorkerThread.run(WorkerThread.java:510)
>>> Caused by: java.io.IOException: File descriptor in bad state
>>>        at sun.nio.ch.SolarisEventPort.port_dissociate(Native Method)
>>>        at sun.nio.ch.EventPortWrapper.release(EventPortWrapper.java:233)
>>>        ... 6 more
>>>
>>> I am running Immutant version 2.1.4 on Sun Solaris version SunOS 5.11
>>> with Oracle JDK 1.7 and  clojure 1.8.0
>>>
>>> Are there any Immutant/Undertow parameters I should be aware of with
>>> regards to the problem?
>>>
>>> Thanks
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Immutant" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to immutant+unsubscribe at googlegroups.com.
>>> To post to this group, send email to immutant at googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/immutant/13c058c9-b079-4b82-8a01-4f3ed0c1ffda%40googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>> _______________________________________________
>>> undertow-dev mailing list
>>> undertow-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>> _______________________________________________
>> undertow-dev mailing list
>> undertow-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/undertow-dev

From peter.royal at pobox.com  Tue Jun 21 15:51:24 2016
From: peter.royal at pobox.com (peter royal)
Date: Tue, 21 Jun 2016 14:51:24 -0500
Subject: [undertow-dev] Websocket connections and IO thread affinity
In-Reply-To: <CAD+L2cxVvXuO+9Nt0K5rJLNMCiifnOgi6NNEEkxhEtnCaHCfEQ@mail.gmail.com>
References: <1466099910.3550017.639859769.76A4CEB5@webmail.messagingengine.com>
	<9662EDD0-8415-450D-9C0E-0F483C74A97E@redhat.com>
	<1466103661.3565919.639925417.0C3C1816@webmail.messagingengine.com>
	<C0DA4CF3-6B5E-443D-8334-93F4E8016BD0@redhat.com>
	<1466105775.3574369.639969105.11ADCB79@webmail.messagingengine.com>
	<C1A4245C-2DCB-4948-B086-320D22EC6031@redhat.com>
	<1466108154.3584990.640000809.5C53BB8C@webmail.messagingengine.com>
	<CAD+L2cxVvXuO+9Nt0K5rJLNMCiifnOgi6NNEEkxhEtnCaHCfEQ@mail.gmail.com>
Message-ID: <1466538684.242359.644485193.592397D9@webmail.messagingengine.com>

Adding additional IO threads was sufficient to alleviate my problem for
now. 

While investigating I also realized I was deserializing inbound
websocket messages on the IO thread, and moving that to a worker thread
has also reduced the amount of non-IO work being doing on the IO
threads.

Thanks all,

-pete

-- 
(peter.royal|osi)@pobox.com - http://fotap.org/~osi

On Thu, Jun 16, 2016, at 07:13 PM, Stuart Douglas wrote:
> Another approach we could potentially try is simply maintaining a
> count of active connections per thread, and allocating new connections
> to the one with the lowest number of open connections.
> 
> This will result in the connect itself being slightly slower, however
> it guarantees the most even load distribution you can get without
> somehow moving connections between threads (which is not really
> something I see as being doable, any gains would probably be lost due
> additional thread safety constructs being needed).
> 
> Stuart
> 
> On Fri, Jun 17, 2016 at 6:15 AM, peter royal <peter.royal at pobox.com>
> wrote:
> > Undertow standalone.
> >
> > Thanks for the reminder about that - I went back and looked at things
> > and realized I was doing deserialization of incoming messages on the IO
> > thread prior to dispatching to the worker thread. The outbound path was
> > clean though.
> >
> > Thanks again!
> >
> > -pete
> >
> > --
> > (peter.royal|osi)@pobox.com - http://fotap.org/~osi
> >
> > On Thu, Jun 16, 2016, at 02:59 PM, Jason Greene wrote:
> >> Are you using WildFly or Undertow standalone?
> >>
> >> If you are using undertow standalone, you might want to try enabling
> >> dispatch to worker (this is the default on WildFly):
> >> webSocketDeploymentInfo.setDispatchToWorkerThread(true)
> >>
> >> If you have message handlers that use significant CPU time or introduce
> >> blocking (which a disparity like you see could potentially indicate),
> >> then it can negatively impact the I/O thread?s ability to handle
> >> connection events efficiently, and so dispatching to the worker pool
> >> allows for long running tasks to execute without interfering with other
> >> connections/activity.
> >>
> >> > On Jun 16, 2016, at 2:36 PM, peter royal <peter.royal at pobox.com> wrote:
> >> >
> >> > Understood.
> >> >
> >> > I'm going to test with increased IO threads, and if that fixes things
> >> > I'm good. Using thread user CPU time might be a good metric, as looking
> >> > at that the imbalance is clear:
> >> >
> >> > CPU: 2673514 ms
> >> > CPU: 31270 ms
> >> > CPU: 61962 ms
> >> > CPU: 7952561 ms
> >> >
> >> > As I think through this more, optimal balancing requires pushing a lot
> >> > of application-specific info down low, because a given WS connection
> >> > might be high volume or not. it would be easier to migrate a connection
> >> > that is detected to be high volume to another IO thread, but that'd be a
> >> > hugely invasive change. The optimal strategy for me might just be to
> >> > have 1 thread per connection as the counts aren't very high.
> >> >
> >> > Thanks for the help!
> >> >
> >> > --
> >> > (peter.royal|osi)@pobox.com - http://fotap.org/~osi
> >> >
> >> > On Thu, Jun 16, 2016, at 02:17 PM, Jason Greene wrote:
> >> >> The way our current approach works, which is the same approach as
> >> >> SO_REUSEPORT?s impl is that address:port are hashed to select the
> >> >> destination, this is mainly so we can transition with no real behavioral
> >> >> surprises. If you have some connections lasting significantly longer than
> >> >> others, then you will eventually go out of balance because the current
> >> >> allocation state isn?t a factor into the decision. It?s possible to do
> >> >> more advanced algorithms factoring in state, but once you do that you tie
> >> >> yourself to a single threaded acceptor (although thats currently the case
> >> >> with our emulated SO_REUSEPORT implementation). For many workloads this
> >> >> won?t matter though, as you need massive connection rates to hit the
> >> >> accept stability limits.
> >> >>
> >> >> Maybe you want to play with modifying QueuedTcpNioServer to compare a few
> >> >> different algorithms? You could try balancing active connection count as
> >> >> one strategy, and perhaps thread user cpu time as another. For both
> >> >> approaches you probably want to have i/o threads individually updating a
> >> >> volatile statistic field as part of their standard work, and then the
> >> >> accept queuing thread scanning those values to select the best
> >> >> destination.
> >> >>
> >> >>> On Jun 16, 2016, at 2:01 PM, peter royal <peter.royal at pobox.com> wrote:
> >> >>>
> >> >>> Gotcha. I was digging through things and found the change where the new
> >> >>> strategy was introduced. With my current # of IO threads it is giving
> >> >>> un-even weighings:
> >> >>>
> >> >>> thread, connections
> >> >>> 0, 6
> >> >>> 1, 5
> >> >>> 2, 3
> >> >>> 3, 2
> >> >>>
> >> >>> I'm going to double my IO threads, but it will still be less than
> >> >>> optimal, but improved:
> >> >>>
> >> >>> thread, connections
> >> >>> 0, 2
> >> >>> 1, 1
> >> >>> 2, 1
> >> >>> 3, 1
> >> >>> 4, 4
> >> >>> 5, 4
> >> >>> 6, 2
> >> >>> 7, 1
> >> >>>
> >> >>> Random is only slightly better, eyeballing things.
> >> >>>
> >> >>> I'm using Undertow 1.3.22 which uses XNIO 3.3.6.  Linux kernel 2.6.32
> >> >>> though.
> >> >>>
> >> >>> Digging into my problem more, I would probably need to balance on more
> >> >>> than just connection count per IO thread, as some connections are busier
> >> >>> than others.
> >> >>>
> >> >>> Can you point me towards any references about the forthcoming access to
> >> >>> native facility? I'm curious as to how that will work
> >> >>>
> >> >>> -pete
> >> >>>
> >> >>> --
> >> >>> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
> >> >>>
> >> >>> On Thu, Jun 16, 2016, at 01:41 PM, Jason T. Greene wrote:
> >> >>>> We recently changed xnio to balance connections by default using a
> >> >>>> strategy similar to the new SO_REUSEPORT facility in the Linux kernel
> >> >>>> (3.3.3 or later). In the short future,  we will be switching to the
> >> >>>> native facility when accessible in the JDK NIO implementation. Older
> >> >>>> versions had a feature called balancing tokens that you could use to
> >> >>>> balance connections fairly, but it had to be especially configured.
> >> >>>>
> >> >>>>
> >> >>>>> On Jun 16, 2016, at 1:00 PM, peter royal <peter.royal at pobox.com> wrote:
> >> >>>>>
> >> >>>>> (I believe the following is true... please correct me if not!)
> >> >>>>>
> >> >>>>> I have an application which heavily utilizes web sockets. It is an
> >> >>>>> internal application which uses a small number of connections with
> >> >>>>> reasonable load on each.
> >> >>>>>
> >> >>>>> When a new connection is received by Undertow, there is an
> >> >>>>> at-connection-time assignment of an XNIO IO Thread to the connection.
> >> >>>>> This is causing uneven load on my IO threads, due to chance.
> >> >>>>>
> >> >>>>> I'm increasing the number of IO threads as a temporary fix, but it might
> >> >>>>> be useful to be able to either migrate a long-lived connection to
> >> >>>>> another IO thread (harder) or do better load balancing amongst IO
> >> >>>>> threads. For the latter, if Undertow was able to provide a strategy for
> >> >>>>> picking a thread in NioXnioWorker.getIoThread(hashCode), it could try
> >> >>>>> and pick a thread that had fewer connections assigned to it.
> >> >>>>>
> >> >>>>> Has anyone else run into this problem? Would a fix be accepted?
> >> >>>>>
> >> >>>>> -pete
> >> >>>>>
> >> >>>>> --
> >> >>>>> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
> >> >>>>> _______________________________________________
> >> >>>>> undertow-dev mailing list
> >> >>>>> undertow-dev at lists.jboss.org
> >> >>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
> >> >>
> >> >> --
> >> >> Jason T. Greene
> >> >> WildFly Lead / JBoss EAP Platform Architect
> >> >> JBoss, a division of Red Hat
> >> >>
> >>
> >> --
> >> Jason T. Greene
> >> WildFly Lead / JBoss EAP Platform Architect
> >> JBoss, a division of Red Hat
> >>
> >
> > _______________________________________________
> > undertow-dev mailing list
> > undertow-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/undertow-dev


From thomas.walas at 1und1.de  Wed Jun 22 10:35:09 2016
From: thomas.walas at 1und1.de (Thomas Walas)
Date: Wed, 22 Jun 2016 14:35:09 +0000
Subject: [undertow-dev] Issue with SOAP application deployment with https
Message-ID: <10a0e924bf6c47f38cb2b57a2f861b8e@KAPPEX007-MBX.webde.local>

Hi all,

this is my first to post undertow-dev, so I hope I'm at the right location.

My problem is deploying a SOAP application on a ssl secured Wildfly 10.

The webservice subsystem complains about a non-present wsdl port it can run HTTP/1.1 on. It used to get this information from the web subsystem, but with Undertow this functionality seems to have been removed.

The result is a failing SOAP deployment, a work-around would be to set <wsdl-secure-port/> in the webservice subsystem.

Can you provide a fix to reenable the forwarding of the secured http port to the webservice subsystem?

Thanks in advance!

Thomas Walas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20160622/0771f9db/attachment.html 

From sdouglas at redhat.com  Wed Jun 22 17:47:58 2016
From: sdouglas at redhat.com (Stuart Douglas)
Date: Thu, 23 Jun 2016 07:47:58 +1000
Subject: [undertow-dev] Issue with SOAP application deployment with https
In-Reply-To: <10a0e924bf6c47f38cb2b57a2f861b8e@KAPPEX007-MBX.webde.local>
References: <10a0e924bf6c47f38cb2b57a2f861b8e@KAPPEX007-MBX.webde.local>
Message-ID: <CAD+L2cyEDPjz6PmZpOiqarHRUvWrfqOVkPnZF=+CgCeFBwyVzA@mail.gmail.com>

This sounds more like a wildfly web services config issue, have you
tried posting on the Wildfly forum?

Stuart

On Thu, Jun 23, 2016 at 12:35 AM, Thomas Walas <thomas.walas at 1und1.de> wrote:
> Hi all,
>
>
>
> this is my first to post undertow-dev, so I hope I?m at the right location.
>
>
>
> My problem is deploying a SOAP application on a ssl secured Wildfly 10.
>
>
>
> The webservice subsystem complains about a non-present wsdl port it can run
> HTTP/1.1 on. It used to get this information from the web subsystem, but
> with Undertow this functionality seems to have been removed.
>
>
>
> The result is a failing SOAP deployment, a work-around would be to set
> <wsdl-secure-port/> in the webservice subsystem.
>
>
>
> Can you provide a fix to reenable the forwarding of the secured http port to
> the webservice subsystem?
>
>
>
> Thanks in advance!
>
>
>
> Thomas Walas
>
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev


From frederick.michielssen at openanalytics.eu  Thu Jun 23 05:52:32 2016
From: frederick.michielssen at openanalytics.eu (Frederick Michielssen)
Date: Thu, 23 Jun 2016 11:52:32 +0200 (CEST)
Subject: [undertow-dev] Websocket proxy: monitor (in)activity
In-Reply-To: <CAD+L2cz7_Ey5ju8=stMUthQhJtpX6FCnRj4h3g=rh9+anGcjLg@mail.gmail.com>
References: <608073034.290985.1466171402501.JavaMail.zimbra@openanalytics.eu>
	<CAD+L2cz7_Ey5ju8=stMUthQhJtpX6FCnRj4h3g=rh9+anGcjLg@mail.gmail.com>
Message-ID: <367947099.692530.1466675552289.JavaMail.zimbra@openanalytics.eu>

Hi Stuart,

Thanks for the pointer.
This appears to work, but what I would like to do next, is to perform additional actions when the websocket closes.
At first glance, I would use DeploymentInfo.addSessionListener, but these sessions will expire (and call sessionDestroyed) regardless of websocket activity, right?

- Frederick

----- Original Message -----
From: "Stuart Douglas" <sdouglas at redhat.com>
To: "Frederick Michielssen" <frederick.michielssen at openanalytics.eu>
Cc: "undertow-dev" <undertow-dev at lists.jboss.org>
Sent: Monday, June 20, 2016 2:57:18 AM
Subject: Re: [undertow-dev] Websocket proxy: monitor (in)activity

Are you using Wildfly or embedded Undertow?

Either way you basically want to set an idle timeout on the conduit.
Setting io.undertow.UndertowOptions#IDLE_TIMEOUT should do it, if you
are using the embedded builder you need to call setServerOptions with
this setting and your timeout, if you are using wildfly you need to
set in on the listener in standalone.xml.

Stuart

On Fri, Jun 17, 2016 at 11:50 PM, Frederick Michielssen
<frederick.michielssen at openanalytics.eu> wrote:
> Hi,
>
> Our application uses Undertow to serve a Spring-boot application, as well as proxy websocket connections to a backend server.
> Now we would like to monitor activity on the websocket, so that specific resources can be freed when the websocket has been inactive for a long time.
> Is there an appropriate way we can detect this in Undertow? Maybe some timeout setting that can be used for this purpose?
>
> -Frederick
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev

From sdouglas at redhat.com  Thu Jun 23 06:27:36 2016
From: sdouglas at redhat.com (Stuart Douglas)
Date: Thu, 23 Jun 2016 20:27:36 +1000
Subject: [undertow-dev] Websocket proxy: monitor (in)activity
In-Reply-To: <367947099.692530.1466675552289.JavaMail.zimbra@openanalytics.eu>
References: <608073034.290985.1466171402501.JavaMail.zimbra@openanalytics.eu>
	<CAD+L2cz7_Ey5ju8=stMUthQhJtpX6FCnRj4h3g=rh9+anGcjLg@mail.gmail.com>
	<367947099.692530.1466675552289.JavaMail.zimbra@openanalytics.eu>
Message-ID: <CAD+L2cypB+SiZFVVXLwgoOQyM=i+SdJqCUD+Vw7Rm2cKx3JSUw@mail.gmail.com>

Are you talking about the proxy or backend nodes?

The proxy basically knows nothing about websockets, it treats the
upgrade as a TCP stream, and can't do any special handling. You could
still use the undertow native API to register a close listener though,
but it will be invoked every time the connection closes.

The session listener is not related, basically the HTTP session is not
related to the websocket session.

Stuart


On Thu, Jun 23, 2016 at 7:52 PM, Frederick Michielssen
<frederick.michielssen at openanalytics.eu> wrote:
> Hi Stuart,
>
> Thanks for the pointer.
> This appears to work, but what I would like to do next, is to perform additional actions when the websocket closes.
> At first glance, I would use DeploymentInfo.addSessionListener, but these sessions will expire (and call sessionDestroyed) regardless of websocket activity, right?
>
> - Frederick
>
> ----- Original Message -----
> From: "Stuart Douglas" <sdouglas at redhat.com>
> To: "Frederick Michielssen" <frederick.michielssen at openanalytics.eu>
> Cc: "undertow-dev" <undertow-dev at lists.jboss.org>
> Sent: Monday, June 20, 2016 2:57:18 AM
> Subject: Re: [undertow-dev] Websocket proxy: monitor (in)activity
>
> Are you using Wildfly or embedded Undertow?
>
> Either way you basically want to set an idle timeout on the conduit.
> Setting io.undertow.UndertowOptions#IDLE_TIMEOUT should do it, if you
> are using the embedded builder you need to call setServerOptions with
> this setting and your timeout, if you are using wildfly you need to
> set in on the listener in standalone.xml.
>
> Stuart
>
> On Fri, Jun 17, 2016 at 11:50 PM, Frederick Michielssen
> <frederick.michielssen at openanalytics.eu> wrote:
>> Hi,
>>
>> Our application uses Undertow to serve a Spring-boot application, as well as proxy websocket connections to a backend server.
>> Now we would like to monitor activity on the websocket, so that specific resources can be freed when the websocket has been inactive for a long time.
>> Is there an appropriate way we can detect this in Undertow? Maybe some timeout setting that can be used for this purpose?
>>
>> -Frederick
>> _______________________________________________
>> undertow-dev mailing list
>> undertow-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/undertow-dev

From ping.li at firemon.com  Wed Jun 29 16:16:24 2016
From: ping.li at firemon.com (PingShan Li)
Date: Wed, 29 Jun 2016 20:16:24 +0000
Subject: [undertow-dev] permessage-deflate extension is not supported,
 getInstalledExtensions() returns empty set in
 ServerWebSocketContainer.java
Message-ID: <CY1PR10MB03140FBB3D67F9F1F2FE9398FA230@CY1PR10MB0314.namprd10.prod.outlook.com>

--- Problem:
client's request for compression is ignored.

The client sends the request for compression:

Sec-WebSocket-Extensions: permessage-deflate; client_no_context_takeover; client_max_window_bits\r\n


Webserver receives the client request,


org.springframework.web.socket.server.support.AbstractHandshakeHandler.java:

doHandshake function: try to see if requested extension is supported

String subProtocol = selectProtocol(headers.getSecWebSocketProtocol(), wsHandler);
List<WebSocketExtension> requested = headers.getSecWebSocketExtensions();
List<WebSocketExtension> supported = this.requestUpgradeStrategy.getSupportedExtensions(request);
List<WebSocketExtension> extensions = filterRequestedExtensions(request, requested, supported);

getSupportedExtensions calls into the following function, which always returns empty set for installed extensions.

io.undertow.websockets.jsr.ServerWebSocketContainer.java:

    public Set<Extension> getInstalledExtensions() {
        return Collections.emptySet();
    }

My question is:
how to tell spring framework that the permessage-deflate extension is supported by undertow?

Environement:
Undertow 1.3.22 Final is used by springframework:
+- org.springframework.boot:spring-boot-starter-undertow:jar:1.3.5.RELEASE:compile
[INFO] |  |  +- io.undertow:undertow-core:jar:1.3.22.Final:compile
[INFO] |  |  +- io.undertow:undertow-servlet:jar:1.3.22.Final:compile
[INFO] |  |  +- io.undertow:undertow-websockets-jsr:jar:1.3.22.Final:compile



Thanks




-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20160629/a077dcc5/attachment-0001.html 

From sdouglas at redhat.com  Wed Jun 29 20:00:31 2016
From: sdouglas at redhat.com (Stuart Douglas)
Date: Thu, 30 Jun 2016 10:00:31 +1000
Subject: [undertow-dev] permessage-deflate extension is not supported,
 getInstalledExtensions() returns empty set in
 ServerWebSocketContainer.java
In-Reply-To: <CY1PR10MB03140FBB3D67F9F1F2FE9398FA230@CY1PR10MB0314.namprd10.prod.outlook.com>
References: <CY1PR10MB03140FBB3D67F9F1F2FE9398FA230@CY1PR10MB0314.namprd10.prod.outlook.com>
Message-ID: <CAD+L2cwukkqWtjWEDAm_o7+9U+OznNsCiCYZUSig654e5VB2Dg@mail.gmail.com>

You need to invoke
io.undertow.websockets.jsr.WebSocketDeploymentInfo#addExtension with
an instance of io.undertow.websockets.extensions.PerMessageDeflateHandshake


Stuart

On Thu, Jun 30, 2016 at 6:16 AM, PingShan Li <ping.li at firemon.com> wrote:
> --- Problem:
> client's request for compression is ignored.
>
> The client sends the request for compression:
>
> Sec-WebSocket-Extensions: permessage-deflate; client_no_context_takeover;
> client_max_window_bits\r\n
>
>
> Webserver receives the client request,
>
>
> org.springframework.web.socket.server.support.AbstractHandshakeHandler.java:
>
> doHandshake function: try to see if requested extension is supported
>
> String subProtocol = selectProtocol(headers.getSecWebSocketProtocol(),
> wsHandler);
> List<WebSocketExtension> requested = headers.getSecWebSocketExtensions();
> List<WebSocketExtension> supported =
> this.requestUpgradeStrategy.getSupportedExtensions(request);
> List<WebSocketExtension> extensions = filterRequestedExtensions(request,
> requested, supported);
>
> getSupportedExtensions calls into the following function, which always
> returns empty set for installed extensions.
>
> io.undertow.websockets.jsr.ServerWebSocketContainer.java:
>
>     public Set<Extension> getInstalledExtensions() {
>         return Collections.emptySet();
>     }
>
> My question is:
> how to tell spring framework that the permessage-deflate extension is
> supported by undertow?
>
> Environement:
> Undertow 1.3.22 Final is used by springframework:
> +-
> org.springframework.boot:spring-boot-starter-undertow:jar:1.3.5.RELEASE:compile
> [INFO] |  |  +- io.undertow:undertow-core:jar:1.3.22.Final:compile
> [INFO] |  |  +- io.undertow:undertow-servlet:jar:1.3.22.Final:compile
> [INFO] |  |  +- io.undertow:undertow-websockets-jsr:jar:1.3.22.Final:compile
>
>
> Thanks
>
>
>
>
>
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev