2:17 p.m.
The way our current approach works, which is the same approach as SO_REUSEPORT’s impl is
that address:port are hashed to select the destination, this is mainly so we can
transition with no real behavioral surprises. If you have some connections lasting
significantly longer than others, then you will eventually go out of balance because the
current allocation state isn’t a factor into the decision. It’s possible to do more
advanced algorithms factoring in state, but once you do that you tie yourself to a single
threaded acceptor (although thats currently the case with our emulated SO_REUSEPORT
implementation). For many workloads this won’t matter though, as you need massive
connection rates to hit the accept stability limits.
Maybe you want to play with modifying QueuedTcpNioServer to compare a few different
algorithms? You could try balancing active connection count as one strategy, and perhaps
thread user cpu time as another. For both approaches you probably want to have i/o threads
individually updating a volatile statistic field as part of their standard work, and then
the accept queuing thread scanning those values to select the best destination.
On Jun 16, 2016, at 2:01 PM, peter royal
<peter.royal(a)pobox.com> wrote:
Gotcha. I was digging through things and found the change where the new
strategy was introduced. With my current # of IO threads it is giving
un-even weighings:
thread, connections
0, 6
1, 5
2, 3
3, 2
I'm going to double my IO threads, but it will still be less than
optimal, but improved:
thread, connections
0, 2
1, 1
2, 1
3, 1
4, 4
5, 4
6, 2
7, 1
Random is only slightly better, eyeballing things.
I'm using Undertow 1.3.22 which uses XNIO 3.3.6. Linux kernel 2.6.32
though.
Digging into my problem more, I would probably need to balance on more
than just connection count per IO thread, as some connections are busier
than others.
Can you point me towards any references about the forthcoming access to
native facility? I'm curious as to how that will work
-pete
--
(peter.royal|osi)(a)pobox.com - http://fotap.org/~osi
On Thu, Jun 16, 2016, at 01:41 PM, Jason T. Greene wrote:
> We recently changed xnio to balance connections by default using a
> strategy similar to the new SO_REUSEPORT facility in the Linux kernel
> (3.3.3 or later). In the short future, we will be switching to the
> native facility when accessible in the JDK NIO implementation. Older
> versions had a feature called balancing tokens that you could use to
> balance connections fairly, but it had to be especially configured.
>
>
>> On Jun 16, 2016, at 1:00 PM, peter royal <peter.royal(a)pobox.com> wrote:
>>
>> (I believe the following is true... please correct me if not!)
>>
>> I have an application which heavily utilizes web sockets. It is an
>> internal application which uses a small number of connections with
>> reasonable load on each.
>>
>> When a new connection is received by Undertow, there is an
>> at-connection-time assignment of an XNIO IO Thread to the connection.
>> This is causing uneven load on my IO threads, due to chance.
>>
>> I'm increasing the number of IO threads as a temporary fix, but it might
>> be useful to be able to either migrate a long-lived connection to
>> another IO thread (harder) or do better load balancing amongst IO
>> threads. For the latter, if Undertow was able to provide a strategy for
>> picking a thread in NioXnioWorker.getIoThread(hashCode), it could try
>> and pick a thread that had fewer connections assigned to it.
>>
>> Has anyone else run into this problem? Would a fix be accepted?
>>
>> -pete
>>
>> --
>> (peter.royal|osi)(a)pobox.com - http://fotap.org/~osi
>> _______________________________________________
>> undertow-dev mailing list
>> undertow-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/undertow-dev
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat