Re: [undertow-dev] Unable to concurrently use all available IO Threads under load on Red Hat

Thursday, 26 July 2018

Backlog setting is 1000.

Is this what you are interested in from netstat?  This was for ab with a 
-c of 50.

[barnett@apigateway_test ~]$ java -jar 
undertow-test-0.1.0-jar-with-dependencies.jar &
[1] 7329
[barnett@apigateway_test ~]$ Jul 26, 2018 1:30:22 PM org.xnio.Xnio <clinit>
INFO: XNIO version 3.3.8.Final
Jul 26, 2018 1:30:23 PM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.3.8.Final

Server started on port 8080
1
2
3
4
[barnett@apigateway_test ~]$ netstat -t | grep apigateway_loadge | grep 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51580 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51614 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51622 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51626 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51612 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51578 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51636 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51616 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51582 
ESTABLISHED
tcp6       0      0 apigateway_tes:webcache apigateway_loadge:51556 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51588 
ESTABLISHED
tcp6       0      0 apigateway_tes:webcache apigateway_loadge:51558 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51586 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51648 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51632 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51652 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51654 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51574 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51640 
ESTABLISHED
tcp6       0      0 apigateway_tes:webcache apigateway_loadge:51564 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51590 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51610 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51594 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51592 
ESTABLISHED
tcp6       0      0 apigateway_tes:webcache apigateway_loadge:51568 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51620 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51598 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51600 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51584 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51630 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51596 
ESTABLISHED
tcp6       0      0 apigateway_tes:webcache apigateway_loadge:51566 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51650 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51656 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51624 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51662 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51642 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51604 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51608 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51634 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51658 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51628 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51660 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51572 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51606 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51602 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51638 
ESTABLISHED
tcp6       0      0 apigateway_tes:webcache apigateway_loadge:51570 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51618 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51646 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51644 
ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51576 
ESTABLISHED

On 7/25/2018 9:23 PM, Jason Greene wrote:
...
 Could you post a netstat output so we can see what port numbers your

 host is picking?

 Also is your backlog setting low by chance?

 On Jul 25, 2018, at 6:24 PM, Stuart Douglas <sdouglas(a)redhat.com 
 <mailto:sdouglas@redhat.com>> wrote:

> The mapping is done by a hash of the remote IP+port. It sounds like 
> maybe this machine is allocating ports in a way that does not map 
> well to our hash.
>
> Because the remote IP is the same it is really only the port that 
> comes into effect. The algorithm is 
> in org.xnio.nio.QueuedNioTcpServer#handleReady and in this case would 
> simplify down to:
>
> (((C1 * 23) + P) * 23 + C2) % 8
>
> Where C1 is a hash of the remote IP, and C2 is a hash of the local 
> IP+port combo.
>
> Stuart
>
> On Thu, Jul 26, 2018 at 3:52 AM R. Matt Barnett <barnett(a)rice.edu 
> <mailto:barnett@rice.edu>> wrote:
>
>     I did. I set the concurrency level of ab to 128. I still see only
>     4 overlaps:
>
>         $ java -jar undertow-test-0.1.0-jar-with-dependencies.jar &
>
>         Server started on port 8080
>         1
>         2
>         3
>         4
>
>         $ netstat -t | grep apigateway_loadge | grep -c ESTABLISHED
>         126
>
>
>     What is the algorithm for mapping connections to IO threads?  As
>     a new Undertow user I had assumed round robin, but it sounds like
>     this is not the case.
>
>
>     -- Matt
>
>
>     On 7/25/2018 11:49 AM, Bill O'Neil wrote:
>>     Did you try setting the concurrency level much higher than 8
>>     like I suggested earlier? You are probably having multiple
>>     connections assigned to the same IO threads.
>>
>>     On Wed, Jul 25, 2018 at 12:26 PM, R. Matt Barnett
>>     <barnett(a)rice.edu <mailto:barnett@rice.edu>> wrote:
>>
>>         Corrected test to resolve test/set race.
>>
>>
>>         https://gist.github.com/rmbarnett-rice/1179c4ad1d3344bb247c8b8daed3e4fa
>>
>>
>>         I've also discovered this morning that I *can* see 1-8
>>         printed on Red
>>         Hat when I generate load using ab from Windows, but only 1-4
>>         when
>>         running ab on Red Hat (both locally and from a remote
>>         server).  I'm
>>         wondering if perhaps there is some sort of connection reuse
>>         shenanigans
>>         going on.  My assumption of the use of the -c 8 parameter
>>         was "make 8
>>         sockets" but maybe not.  I'll dig in and report back.
>>
>>
>>         -- Matt
>>
>>
>>         On 7/24/2018 6:56 PM, R. Matt Barnett wrote:
>>         > Hello,
>>         >
>>         > I'm experiencing an Undertow performance issue I fail to
>>         understand.  I
>>         > am able to reproduce the issue with the code linked
>>         bellow. The problem
>>         > is that on Red Hat (and not Windows) I'm unable to
>>         concurrently process
>>         > more than 4 overlapping requests even with 8 configured IO
>>         Threads.
>>         > For example, if I run the following program (1 file, 55
>>         lines):
>>         >
>>         >
>>         https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5
>>         >
>>         > ... on Red Hat and then send requests to the server using
>>         Apache
>>         > Benchmark...
>>         >
>>         >       > ab -n 1000 -c 8 localhost:8080/
>>         >
>>         > I see the following output from the Undertow process:
>>         >
>>         >       Server started on port 8080
>>         >
>>         >       1
>>         >       2
>>         >       3
>>         >       4
>>         >
>>         > I believe this demonstrates that only 4 requests are ever
>>         processed in
>>         > parallel.  I would expect 8.  In fact, when I run the same
>>         experiment on
>>         > Windows I see the expected output of
>>         >
>>         >       Server started on port 8080
>>         >       1
>>         >       2
>>         >       3
>>         >       4
>>         >       5
>>         >       6
>>         >       7
>>         >       8
>>         >
>>         > Any thoughts as to what might explain this behavior?
>>         >
>>         > Best,
>>         >
>>         > Matt
>>         >
>>         > _______________________________________________
>>         > undertow-dev mailing list
>>         > undertow-dev(a)lists.jboss.org
>>         <mailto:undertow-dev@lists.jboss.org>
>>         > https://lists.jboss.org/mailman/listinfo/undertow-dev
>>
>>         _______________________________________________
>>         undertow-dev mailing list
>>         undertow-dev(a)lists.jboss.org
>>         <mailto:undertow-dev@lists.jboss.org>
>>         https://lists.jboss.org/mailman/listinfo/undertow-dev
>>
>>
>
>     _______________________________________________
>     undertow-dev mailing list
>     undertow-dev(a)lists.jboss.org <mailto:undertow-dev@lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/undertow-dev
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev(a)lists.jboss.org <mailto:undertow-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/undertow-dev 

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

Re: [undertow-dev] Unable to concurrently use all available IO Threads under load on Red Hat