Re: [undertow-dev] Websocket connections and IO thread affinity

Undertow standalone. Thanks for the reminder about that - I went back and looked at things and realized I was doing deserialization of incoming messages on the IO thread prior to dispatching to the worker thread. The outbound path was clean though. Thanks again! -pete -- (peter.royal|osi)(a)pobox.com - http://fotap.org/~osi On Thu, Jun 16, 2016, at 02:59 PM, Jason Greene wrote: > Are you using WildFly or Undertow standalone? > > If you are using undertow standalone, you might want to try enabling > dispatch to worker (this is the default on WildFly): > webSocketDeploymentInfo.setDispatchToWorkerThread(true) > > If you have message handlers that use significant CPU time or introduce > blocking (which a disparity like you see could potentially indicate), > then it can negatively impact the I/O thread’s ability to handle > connection events efficiently, and so dispatching to the worker pool > allows for long running tasks to execute without interfering with other > connections/activity. > > > On Jun 16, 2016, at 2:36 PM, peter royal <peter.royal(a)pobox.com&gt; wrote: > > > > Understood. > > > > I'm going to test with increased IO threads, and if that fixes things > > I'm good. Using thread user CPU time might be a good metric, as looking > > at that the imbalance is clear: > > > > CPU: 2673514 ms > > CPU: 31270 ms > > CPU: 61962 ms > > CPU: 7952561 ms > > > > As I think through this more, optimal balancing requires pushing a lot > > of application-specific info down low, because a given WS connection > > might be high volume or not. it would be easier to migrate a connection > > that is detected to be high volume to another IO thread, but that'd be a > > hugely invasive change. The optimal strategy for me might just be to > > have 1 thread per connection as the counts aren't very high. > > > > Thanks for the help! > > > > -- > > (peter.royal|osi)(a)pobox.com - http://fotap.org/~osi > > > > On Thu, Jun 16, 2016, at 02:17 PM, Jason Greene wrote: > >> The way our current approach works, which is the same approach as > >> SO_REUSEPORT’s impl is that address:port are hashed to select the > >> destination, this is mainly so we can transition with no real behavioral > >> surprises. If you have some connections lasting significantly longer than > >> others, then you will eventually go out of balance because the current > >> allocation state isn’t a factor into the decision. It’s possible to do > >> more advanced algorithms factoring in state, but once you do that you tie > >> yourself to a single threaded acceptor (although thats currently the case > >> with our emulated SO_REUSEPORT implementation). For many workloads this > >> won’t matter though, as you need massive connection rates to hit the > >> accept stability limits. > >> > >> Maybe you want to play with modifying QueuedTcpNioServer to compare a few > >> different algorithms? You could try balancing active connection count as > >> one strategy, and perhaps thread user cpu time as another. For both > >> approaches you probably want to have i/o threads individually updating a > >> volatile statistic field as part of their standard work, and then the > >> accept queuing thread scanning those values to select the best > >> destination. > >> > >>> On Jun 16, 2016, at 2:01 PM, peter royal <peter.royal(a)pobox.com&gt; wrote: > >>> > >>> Gotcha. I was digging through things and found the change where the new > >>> strategy was introduced. With my current # of IO threads it is giving > >>> un-even weighings: > >>> > >>> thread, connections > >>> 0, 6 > >>> 1, 5 > >>> 2, 3 > >>> 3, 2 > >>> > >>> I'm going to double my IO threads, but it will still be less than > >>> optimal, but improved: > >>> > >>> thread, connections > >>> 0, 2 > >>> 1, 1 > >>> 2, 1 > >>> 3, 1 > >>> 4, 4 > >>> 5, 4 > >>> 6, 2 > >>> 7, 1 > >>> > >>> Random is only slightly better, eyeballing things. > >>> > >>> I'm using Undertow 1.3.22 which uses XNIO 3.3.6. Linux kernel 2.6.32 > >>> though. > >>> > >>> Digging into my problem more, I would probably need to balance on more > >>> than just connection count per IO thread, as some connections are busier > >>> than others. > >>> > >>> Can you point me towards any references about the forthcoming access to > >>> native facility? I'm curious as to how that will work > >>> > >>> -pete > >>> > >>> -- > >>> (peter.royal|osi)(a)pobox.com - http://fotap.org/~osi > >>> > >>> On Thu, Jun 16, 2016, at 01:41 PM, Jason T. Greene wrote: > >>>> We recently changed xnio to balance connections by default using a > >>>> strategy similar to the new SO_REUSEPORT facility in the Linux kernel > >>>> (3.3.3 or later). In the short future, we will be switching to the > >>>> native facility when accessible in the JDK NIO implementation. Older > >>>> versions had a feature called balancing tokens that you could use to > >>>> balance connections fairly, but it had to be especially configured. > >>>> > >>>> > >>>>> On Jun 16, 2016, at 1:00 PM, peter royal <peter.royal(a)pobox.com&gt; wrote: > >>>>> > >>>>> (I believe the following is true... please correct me if not!) > >>>>> > >>>>> I have an application which heavily utilizes web sockets. It is an > >>>>> internal application which uses a small number of connections with > >>>>> reasonable load on each. > >>>>> > >>>>> When a new connection is received by Undertow, there is an > >>>>> at-connection-time assignment of an XNIO IO Thread to the connection. > >>>>> This is causing uneven load on my IO threads, due to chance. > >>>>> > >>>>> I'm increasing the number of IO threads as a temporary fix, but it might > >>>>> be useful to be able to either migrate a long-lived connection to > >>>>> another IO thread (harder) or do better load balancing amongst IO > >>>>> threads. For the latter, if Undertow was able to provide a strategy for > >>>>> picking a thread in NioXnioWorker.getIoThread(hashCode), it could try > >>>>> and pick a thread that had fewer connections assigned to it. > >>>>> > >>>>> Has anyone else run into this problem? Would a fix be accepted? > >>>>> > >>>>> -pete > >>>>> > >>>>> -- > >>>>> (peter.royal|osi)(a)pobox.com - http://fotap.org/~osi > >>>>> _______________________________________________ > >>>>> undertow-dev mailing list > >>>>> undertow-dev(a)lists.jboss.org > >>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev > >> > >> -- > >> Jason T. Greene > >> WildFly Lead / JBoss EAP Platform Architect > >> JBoss, a division of Red Hat > >> > > -- > Jason T. Greene > WildFly Lead / JBoss EAP Platform Architect > JBoss, a division of Red Hat > _______________________________________________ undertow-dev mailing list undertow-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/undertow-dev