[undertow-dev] Unable to concurrently use all available IO Threads under load on Red Hat
R. Matt Barnett
barnett at rice.edu
Mon Aug 13 16:49:25 EDT 2018
Did you all ever open a ticket on this? If so could you link me so I can
follow along?
Thanks,
Matt
On 7/26/2018 9:11 PM, Jason Greene wrote:
> Looks like we need to tweak the hash:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=07f4c90062f8fc7c8c26f8f95324cbe8fa3145a5
>
>
>
>
> On Jul 26, 2018, at 7:13 PM, Stuart Douglas <sdouglas at redhat.com
> <mailto:sdouglas at redhat.com>> wrote:
>
>> They are all even numbers :-(
>>
>> This does not play well with our hash if C1 is also even:
>>
>> (((C1 * 23) + P) * 23 + C2) % 8
>>
>> If C1 is even the C1 * 23 is even. This means ((C1 * 23) + P) * 23 is
>> even. Depending on the value of C2 this means the result is always
>> even or always odd, so with an evenly divisible number of threads you
>> are only ever going to allocate to half of them.
>>
>> The good news is this should be easily fixed by using an odd number
>> of IO threads, but we probably should revisit this.
>>
>> Stuart
>>
>> On Fri, Jul 27, 2018 at 4:34 AM R. Matt Barnett <barnett at rice.edu
>> <mailto:barnett at rice.edu>> wrote:
>>
>> Backlog setting is 1000.
>>
>> Is this what you are interested in from netstat? This was for ab
>> with a -c of 50.
>>
>>
>> [barnett at apigateway_test ~]$ java -jar
>> undertow-test-0.1.0-jar-with-dependencies.jar &
>> [1] 7329
>> [barnett at apigateway_test ~]$ Jul 26, 2018 1:30:22 PM
>> org.xnio.Xnio <clinit>
>> INFO: XNIO version 3.3.8.Final
>> Jul 26, 2018 1:30:23 PM org.xnio.nio.NioXnio <clinit>
>> INFO: XNIO NIO Implementation Version 3.3.8.Final
>>
>>
>> Server started on port 8080
>> 1
>> 2
>> 3
>> 4
>> [barnett at apigateway_test ~]$ netstat -t | grep apigateway_loadge
>> | grep ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51580 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51614 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51622 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51626 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51612 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51578 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51636 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51616 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51582 ESTABLISHED
>> tcp6 0 0 apigateway_tes:webcache
>> apigateway_loadge:51556 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51588 ESTABLISHED
>> tcp6 0 0 apigateway_tes:webcache
>> apigateway_loadge:51558 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51586 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51648 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51632 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51652 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51654 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51574 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51640 ESTABLISHED
>> tcp6 0 0 apigateway_tes:webcache
>> apigateway_loadge:51564 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51590 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51610 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51594 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51592 ESTABLISHED
>> tcp6 0 0 apigateway_tes:webcache
>> apigateway_loadge:51568 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51620 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51598 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51600 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51584 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51630 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51596 ESTABLISHED
>> tcp6 0 0 apigateway_tes:webcache
>> apigateway_loadge:51566 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51650 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51656 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51624 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51662 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51642 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51604 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51608 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51634 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51658 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51628 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51660 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51572 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51606 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51602 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51638 ESTABLISHED
>> tcp6 0 0 apigateway_tes:webcache
>> apigateway_loadge:51570 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51618 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51646 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51644 ESTABLISHED
>> tcp6 97 0 apigateway_tes:webcache
>> apigateway_loadge:51576 ESTABLISHED
>>
>>
>> On 7/25/2018 9:23 PM, Jason Greene wrote:
>>> Could you post a netstat output so we can see what port numbers
>>> your host is picking?
>>>
>>> Also is your backlog setting low by chance?
>>>
>>> On Jul 25, 2018, at 6:24 PM, Stuart Douglas <sdouglas at redhat.com
>>> <mailto:sdouglas at redhat.com>> wrote:
>>>
>>>> The mapping is done by a hash of the remote IP+port. It sounds
>>>> like maybe this machine is allocating ports in a way that does
>>>> not map well to our hash.
>>>>
>>>> Because the remote IP is the same it is really only the port
>>>> that comes into effect. The algorithm is
>>>> in org.xnio.nio.QueuedNioTcpServer#handleReady and in this case
>>>> would simplify down to:
>>>>
>>>> (((C1 * 23) + P) * 23 + C2) % 8
>>>>
>>>> Where C1 is a hash of the remote IP, and C2 is a hash of the
>>>> local IP+port combo.
>>>>
>>>> Stuart
>>>>
>>>> On Thu, Jul 26, 2018 at 3:52 AM R. Matt Barnett
>>>> <barnett at rice.edu <mailto:barnett at rice.edu>> wrote:
>>>>
>>>> I did. I set the concurrency level of ab to 128. I still
>>>> see only 4 overlaps:
>>>>
>>>> $ java -jar undertow-test-0.1.0-jar-with-dependencies.jar &
>>>>
>>>> Server started on port 8080
>>>> 1
>>>> 2
>>>> 3
>>>> 4
>>>>
>>>> $ netstat -t | grep apigateway_loadge | grep -c ESTABLISHED
>>>> 126
>>>>
>>>>
>>>> What is the algorithm for mapping connections to IO
>>>> threads? As a new Undertow user I had assumed round robin,
>>>> but it sounds like this is not the case.
>>>>
>>>>
>>>> -- Matt
>>>>
>>>>
>>>> On 7/25/2018 11:49 AM, Bill O'Neil wrote:
>>>>> Did you try setting the concurrency level much higher than
>>>>> 8 like I suggested earlier? You are probably having
>>>>> multiple connections assigned to the same IO threads.
>>>>>
>>>>> On Wed, Jul 25, 2018 at 12:26 PM, R. Matt Barnett
>>>>> <barnett at rice.edu <mailto:barnett at rice.edu>> wrote:
>>>>>
>>>>> Corrected test to resolve test/set race.
>>>>>
>>>>>
>>>>> https://gist.github.com/rmbarnett-rice/1179c4ad1d3344bb247c8b8daed3e4fa
>>>>>
>>>>>
>>>>> I've also discovered this morning that I *can* see 1-8
>>>>> printed on Red
>>>>> Hat when I generate load using ab from Windows, but
>>>>> only 1-4 when
>>>>> running ab on Red Hat (both locally and from a remote
>>>>> server). I'm
>>>>> wondering if perhaps there is some sort of connection
>>>>> reuse shenanigans
>>>>> going on. My assumption of the use of the -c 8
>>>>> parameter was "make 8
>>>>> sockets" but maybe not. I'll dig in and report back.
>>>>>
>>>>>
>>>>> -- Matt
>>>>>
>>>>>
>>>>> On 7/24/2018 6:56 PM, R. Matt Barnett wrote:
>>>>> > Hello,
>>>>> >
>>>>> > I'm experiencing an Undertow performance issue I
>>>>> fail to understand. I
>>>>> > am able to reproduce the issue with the code linked
>>>>> bellow. The problem
>>>>> > is that on Red Hat (and not Windows) I'm unable to
>>>>> concurrently process
>>>>> > more than 4 overlapping requests even with 8
>>>>> configured IO Threads.
>>>>> > For example, if I run the following program (1 file,
>>>>> 55 lines):
>>>>> >
>>>>> >
>>>>> https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5
>>>>> >
>>>>> > ... on Red Hat and then send requests to the server
>>>>> using Apache
>>>>> > Benchmark...
>>>>> >
>>>>> > > ab -n 1000 -c 8 localhost:8080/
>>>>> >
>>>>> > I see the following output from the Undertow process:
>>>>> >
>>>>> > Server started on port 8080
>>>>> >
>>>>> > 1
>>>>> > 2
>>>>> > 3
>>>>> > 4
>>>>> >
>>>>> > I believe this demonstrates that only 4 requests are
>>>>> ever processed in
>>>>> > parallel. I would expect 8. In fact, when I run
>>>>> the same experiment on
>>>>> > Windows I see the expected output of
>>>>> >
>>>>> > Server started on port 8080
>>>>> > 1
>>>>> > 2
>>>>> > 3
>>>>> > 4
>>>>> > 5
>>>>> > 6
>>>>> > 7
>>>>> > 8
>>>>> >
>>>>> > Any thoughts as to what might explain this behavior?
>>>>> >
>>>>> > Best,
>>>>> >
>>>>> > Matt
>>>>> >
>>>>> > _______________________________________________
>>>>> > undertow-dev mailing list
>>>>> > undertow-dev at lists.jboss.org
>>>>> <mailto:undertow-dev at lists.jboss.org>
>>>>> > https://lists.jboss.org/mailman/listinfo/undertow-dev
>>>>>
>>>>> _______________________________________________
>>>>> undertow-dev mailing list
>>>>> undertow-dev at lists.jboss.org
>>>>> <mailto:undertow-dev at lists.jboss.org>
>>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> undertow-dev mailing list
>>>> undertow-dev at lists.jboss.org
>>>> <mailto:undertow-dev at lists.jboss.org>
>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>>>
>>>> _______________________________________________
>>>> undertow-dev mailing list
>>>> undertow-dev at lists.jboss.org <mailto:undertow-dev at lists.jboss.org>
>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20180813/5a9bacf1/attachment-0001.html
More information about the undertow-dev
mailing list