[undertow-dev] Unable to concurrently use all available IO Threads under load on Red Hat
Stuart Douglas
sdouglas at redhat.com
Thu Jul 26 20:13:57 EDT 2018
They are all even numbers :-(
This does not play well with our hash if C1 is also even:
(((C1 * 23) + P) * 23 + C2) % 8
If C1 is even the C1 * 23 is even. This means ((C1 * 23) + P) * 23 is even.
Depending on the value of C2 this means the result is always even or always
odd, so with an evenly divisible number of threads you are only ever going
to allocate to half of them.
The good news is this should be easily fixed by using an odd number of IO
threads, but we probably should revisit this.
Stuart
On Fri, Jul 27, 2018 at 4:34 AM R. Matt Barnett <barnett at rice.edu> wrote:
> Backlog setting is 1000.
>
> Is this what you are interested in from netstat? This was for ab with a
> -c of 50.
>
>
> [barnett at apigateway_test ~]$ java -jar
> undertow-test-0.1.0-jar-with-dependencies.jar &
> [1] 7329
> [barnett at apigateway_test ~]$ Jul 26, 2018 1:30:22 PM org.xnio.Xnio
> <clinit>
> INFO: XNIO version 3.3.8.Final
> Jul 26, 2018 1:30:23 PM org.xnio.nio.NioXnio <clinit>
> INFO: XNIO NIO Implementation Version 3.3.8.Final
>
>
> Server started on port 8080
> 1
> 2
> 3
> 4
> [barnett at apigateway_test ~]$ netstat -t | grep apigateway_loadge | grep
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51580
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51614
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51622
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51626
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51612
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51578
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51636
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51616
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51582
> ESTABLISHED
> tcp6 0 0 apigateway_tes:webcache apigateway_loadge:51556
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51588
> ESTABLISHED
> tcp6 0 0 apigateway_tes:webcache apigateway_loadge:51558
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51586
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51648
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51632
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51652
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51654
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51574
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51640
> ESTABLISHED
> tcp6 0 0 apigateway_tes:webcache apigateway_loadge:51564
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51590
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51610
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51594
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51592
> ESTABLISHED
> tcp6 0 0 apigateway_tes:webcache apigateway_loadge:51568
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51620
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51598
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51600
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51584
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51630
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51596
> ESTABLISHED
> tcp6 0 0 apigateway_tes:webcache apigateway_loadge:51566
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51650
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51656
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51624
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51662
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51642
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51604
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51608
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51634
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51658
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51628
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51660
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51572
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51606
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51602
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51638
> ESTABLISHED
> tcp6 0 0 apigateway_tes:webcache apigateway_loadge:51570
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51618
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51646
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51644
> ESTABLISHED
> tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51576
> ESTABLISHED
>
> On 7/25/2018 9:23 PM, Jason Greene wrote:
>
> Could you post a netstat output so we can see what port numbers your host
> is picking?
>
> Also is your backlog setting low by chance?
>
> On Jul 25, 2018, at 6:24 PM, Stuart Douglas <sdouglas at redhat.com> wrote:
>
> The mapping is done by a hash of the remote IP+port. It sounds like maybe
> this machine is allocating ports in a way that does not map well to our
> hash.
>
> Because the remote IP is the same it is really only the port that comes
> into effect. The algorithm is
> in org.xnio.nio.QueuedNioTcpServer#handleReady and in this case would
> simplify down to:
>
> (((C1 * 23) + P) * 23 + C2) % 8
>
> Where C1 is a hash of the remote IP, and C2 is a hash of the local IP+port
> combo.
>
> Stuart
>
> On Thu, Jul 26, 2018 at 3:52 AM R. Matt Barnett <barnett at rice.edu> wrote:
>
>> I did. I set the concurrency level of ab to 128. I still see only 4
>> overlaps:
>>
>> $ java -jar undertow-test-0.1.0-jar-with-dependencies.jar &
>>
>> Server started on port 8080
>> 1
>> 2
>> 3
>> 4
>>
>> $ netstat -t | grep apigateway_loadge | grep -c ESTABLISHED
>> 126
>>
>>
>> What is the algorithm for mapping connections to IO threads? As a new
>> Undertow user I had assumed round robin, but it sounds like this is not the
>> case.
>>
>>
>> -- Matt
>>
>> On 7/25/2018 11:49 AM, Bill O'Neil wrote:
>>
>> Did you try setting the concurrency level much higher than 8 like I
>> suggested earlier? You are probably having multiple connections assigned to
>> the same IO threads.
>>
>> On Wed, Jul 25, 2018 at 12:26 PM, R. Matt Barnett <barnett at rice.edu>
>> wrote:
>>
>>> Corrected test to resolve test/set race.
>>>
>>>
>>> https://gist.github.com/rmbarnett-rice/1179c4ad1d3344bb247c8b8daed3e4fa
>>>
>>>
>>> I've also discovered this morning that I *can* see 1-8 printed on Red
>>> Hat when I generate load using ab from Windows, but only 1-4 when
>>> running ab on Red Hat (both locally and from a remote server). I'm
>>> wondering if perhaps there is some sort of connection reuse shenanigans
>>> going on. My assumption of the use of the -c 8 parameter was "make 8
>>> sockets" but maybe not. I'll dig in and report back.
>>>
>>>
>>> -- Matt
>>>
>>>
>>> On 7/24/2018 6:56 PM, R. Matt Barnett wrote:
>>> > Hello,
>>> >
>>> > I'm experiencing an Undertow performance issue I fail to understand. I
>>> > am able to reproduce the issue with the code linked bellow. The problem
>>> > is that on Red Hat (and not Windows) I'm unable to concurrently process
>>> > more than 4 overlapping requests even with 8 configured IO Threads.
>>> > For example, if I run the following program (1 file, 55 lines):
>>> >
>>> >
>>> https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5
>>> >
>>> > ... on Red Hat and then send requests to the server using Apache
>>> > Benchmark...
>>> >
>>> > > ab -n 1000 -c 8 localhost:8080/
>>> >
>>> > I see the following output from the Undertow process:
>>> >
>>> > Server started on port 8080
>>> >
>>> > 1
>>> > 2
>>> > 3
>>> > 4
>>> >
>>> > I believe this demonstrates that only 4 requests are ever processed in
>>> > parallel. I would expect 8. In fact, when I run the same experiment
>>> on
>>> > Windows I see the expected output of
>>> >
>>> > Server started on port 8080
>>> > 1
>>> > 2
>>> > 3
>>> > 4
>>> > 5
>>> > 6
>>> > 7
>>> > 8
>>> >
>>> > Any thoughts as to what might explain this behavior?
>>> >
>>> > Best,
>>> >
>>> > Matt
>>> >
>>> > _______________________________________________
>>> > undertow-dev mailing list
>>> > undertow-dev at lists.jboss.org
>>> > https://lists.jboss.org/mailman/listinfo/undertow-dev
>>>
>>> _______________________________________________
>>> undertow-dev mailing list
>>> undertow-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>>
>>
>>
>> _______________________________________________
>> undertow-dev mailing list
>> undertow-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20180727/d69fb8c0/attachment-0001.html
More information about the undertow-dev
mailing list