[undertow-dev] Websocket connections and IO thread affinity

[Stuart Douglas]

Another approach we could potentially try is simply maintaining a
count of active connections per thread, and allocating new connections
to the one with the lowest number of open connections.

This will result in the connect itself being slightly slower, however
it guarantees the most even load distribution you can get without
somehow moving connections between threads (which is not really
something I see as being doable, any gains would probably be lost due
additional thread safety constructs being needed).

Stuart

On Fri, Jun 17, 2016 at 6:15 AM, peter royal <peter.royal at pobox.com> wrote:
> Undertow standalone.
>
> Thanks for the reminder about that - I went back and looked at things
> and realized I was doing deserialization of incoming messages on the IO
> thread prior to dispatching to the worker thread. The outbound path was
> clean though.
>
> Thanks again!
>
> -pete
>
> --
> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
>
> On Thu, Jun 16, 2016, at 02:59 PM, Jason Greene wrote:
>> Are you using WildFly or Undertow standalone?
>>
>> If you are using undertow standalone, you might want to try enabling
>> dispatch to worker (this is the default on WildFly):
>> webSocketDeploymentInfo.setDispatchToWorkerThread(true)
>>
>> If you have message handlers that use significant CPU time or introduce
>> blocking (which a disparity like you see could potentially indicate),
>> then it can negatively impact the I/O thread’s ability to handle
>> connection events efficiently, and so dispatching to the worker pool
>> allows for long running tasks to execute without interfering with other
>> connections/activity.
>>
>> > On Jun 16, 2016, at 2:36 PM, peter royal <peter.royal at pobox.com> wrote:
>> >
>> > Understood.
>> >
>> > I'm going to test with increased IO threads, and if that fixes things
>> > I'm good. Using thread user CPU time might be a good metric, as looking
>> > at that the imbalance is clear:
>> >
>> > CPU: 2673514 ms
>> > CPU: 31270 ms
>> > CPU: 61962 ms
>> > CPU: 7952561 ms
>> >
>> > As I think through this more, optimal balancing requires pushing a lot
>> > of application-specific info down low, because a given WS connection
>> > might be high volume or not. it would be easier to migrate a connection
>> > that is detected to be high volume to another IO thread, but that'd be a
>> > hugely invasive change. The optimal strategy for me might just be to
>> > have 1 thread per connection as the counts aren't very high.
>> >
>> > Thanks for the help!
>> >
>> > --
>> > (peter.royal|osi)@pobox.com - http://fotap.org/~osi
>> >
>> > On Thu, Jun 16, 2016, at 02:17 PM, Jason Greene wrote:
>> >> The way our current approach works, which is the same approach as
>> >> SO_REUSEPORT’s impl is that address:port are hashed to select the
>> >> destination, this is mainly so we can transition with no real behavioral
>> >> surprises. If you have some connections lasting significantly longer than
>> >> others, then you will eventually go out of balance because the current
>> >> allocation state isn’t a factor into the decision. It’s possible to do
>> >> more advanced algorithms factoring in state, but once you do that you tie
>> >> yourself to a single threaded acceptor (although thats currently the case
>> >> with our emulated SO_REUSEPORT implementation). For many workloads this
>> >> won’t matter though, as you need massive connection rates to hit the
>> >> accept stability limits.
>> >>
>> >> Maybe you want to play with modifying QueuedTcpNioServer to compare a few
>> >> different algorithms? You could try balancing active connection count as
>> >> one strategy, and perhaps thread user cpu time as another. For both
>> >> approaches you probably want to have i/o threads individually updating a
>> >> volatile statistic field as part of their standard work, and then the
>> >> accept queuing thread scanning those values to select the best
>> >> destination.
>> >>
>> >>> On Jun 16, 2016, at 2:01 PM, peter royal <peter.royal at pobox.com> wrote:
>> >>>
>> >>> Gotcha. I was digging through things and found the change where the new
>> >>> strategy was introduced. With my current # of IO threads it is giving
>> >>> un-even weighings:
>> >>>
>> >>> thread, connections
>> >>> 0, 6
>> >>> 1, 5
>> >>> 2, 3
>> >>> 3, 2
>> >>>
>> >>> I'm going to double my IO threads, but it will still be less than
>> >>> optimal, but improved:
>> >>>
>> >>> thread, connections
>> >>> 0, 2
>> >>> 1, 1
>> >>> 2, 1
>> >>> 3, 1
>> >>> 4, 4
>> >>> 5, 4
>> >>> 6, 2
>> >>> 7, 1
>> >>>
>> >>> Random is only slightly better, eyeballing things.
>> >>>
>> >>> I'm using Undertow 1.3.22 which uses XNIO 3.3.6.  Linux kernel 2.6.32
>> >>> though.
>> >>>
>> >>> Digging into my problem more, I would probably need to balance on more
>> >>> than just connection count per IO thread, as some connections are busier
>> >>> than others.
>> >>>
>> >>> Can you point me towards any references about the forthcoming access to
>> >>> native facility? I'm curious as to how that will work
>> >>>
>> >>> -pete
>> >>>
>> >>> --
>> >>> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
>> >>>
>> >>> On Thu, Jun 16, 2016, at 01:41 PM, Jason T. Greene wrote:
>> >>>> We recently changed xnio to balance connections by default using a
>> >>>> strategy similar to the new SO_REUSEPORT facility in the Linux kernel
>> >>>> (3.3.3 or later). In the short future,  we will be switching to the
>> >>>> native facility when accessible in the JDK NIO implementation. Older
>> >>>> versions had a feature called balancing tokens that you could use to
>> >>>> balance connections fairly, but it had to be especially configured.
>> >>>>
>> >>>>
>> >>>>> On Jun 16, 2016, at 1:00 PM, peter royal <peter.royal at pobox.com> wrote:
>> >>>>>
>> >>>>> (I believe the following is true... please correct me if not!)
>> >>>>>
>> >>>>> I have an application which heavily utilizes web sockets. It is an
>> >>>>> internal application which uses a small number of connections with
>> >>>>> reasonable load on each.
>> >>>>>
>> >>>>> When a new connection is received by Undertow, there is an
>> >>>>> at-connection-time assignment of an XNIO IO Thread to the connection.
>> >>>>> This is causing uneven load on my IO threads, due to chance.
>> >>>>>
>> >>>>> I'm increasing the number of IO threads as a temporary fix, but it might
>> >>>>> be useful to be able to either migrate a long-lived connection to
>> >>>>> another IO thread (harder) or do better load balancing amongst IO
>> >>>>> threads. For the latter, if Undertow was able to provide a strategy for
>> >>>>> picking a thread in NioXnioWorker.getIoThread(hashCode), it could try
>> >>>>> and pick a thread that had fewer connections assigned to it.
>> >>>>>
>> >>>>> Has anyone else run into this problem? Would a fix be accepted?
>> >>>>>
>> >>>>> -pete
>> >>>>>
>> >>>>> --
>> >>>>> (peter.royal|osi)@pobox.com - http://fotap.org/~osi
>> >>>>> _______________________________________________
>> >>>>> undertow-dev mailing list
>> >>>>> undertow-dev at lists.jboss.org
>> >>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>> >>
>> >> --
>> >> Jason T. Greene
>> >> WildFly Lead / JBoss EAP Platform Architect
>> >> JBoss, a division of Red Hat
>> >>
>>
>> --
>> Jason T. Greene
>> WildFly Lead / JBoss EAP Platform Architect
>> JBoss, a division of Red Hat
>>
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev