I have created https://issues.jboss.org/browse/XNIO-328. 

Stuart

On Tue, Aug 14, 2018 at 6:49 AM R. Matt Barnett <barnett@rice.edu> wrote:

Did you all ever open a ticket on this? If so could you link me so I can follow along?


Thanks,

Matt


On 7/26/2018 9:11 PM, Jason Greene wrote:
Looks like we need to tweak the hash:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=07f4c90062f8fc7c8c26f8f95324cbe8fa3145a5



On Jul 26, 2018, at 7:13 PM, Stuart Douglas <sdouglas@redhat.com> wrote:

They are all even numbers :-(

This does not play well with our hash if C1 is also even:

(((C1 * 23) + P) * 23 + C2) % 8

If C1 is even the C1 * 23 is even. This means ((C1 * 23) + P) * 23 is even. Depending on the value of C2 this means the result is always even or always odd, so with an evenly divisible number of threads you are only ever going to allocate to half of them.

The good news is this should be easily fixed by using an odd number of IO threads, but we probably should revisit this.

Stuart

On Fri, Jul 27, 2018 at 4:34 AM R. Matt Barnett <barnett@rice.edu> wrote:

Backlog setting is 1000.

Is this what you are interested in from netstat?  This was for ab with a -c of 50.


[barnett@apigateway_test ~]$ java -jar undertow-test-0.1.0-jar-with-dependencies.jar &
[1] 7329
[barnett@apigateway_test ~]$ Jul 26, 2018 1:30:22 PM org.xnio.Xnio <clinit>
INFO: XNIO version 3.3.8.Final
Jul 26, 2018 1:30:23 PM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.3.8.Final


Server started on port 8080
1
2
3
4
[barnett@apigateway_test ~]$ netstat -t | grep apigateway_loadge | grep ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51580 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51614 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51622 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51626 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51612 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51578 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51636 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51616 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51582 ESTABLISHED
tcp6       0      0 apigateway_tes:webcache apigateway_loadge:51556 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51588 ESTABLISHED
tcp6       0      0 apigateway_tes:webcache apigateway_loadge:51558 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51586 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51648 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51632 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51652 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51654 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51574 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51640 ESTABLISHED
tcp6       0      0 apigateway_tes:webcache apigateway_loadge:51564 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51590 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51610 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51594 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51592 ESTABLISHED
tcp6       0      0 apigateway_tes:webcache apigateway_loadge:51568 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51620 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51598 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51600 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51584 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51630 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51596 ESTABLISHED
tcp6       0      0 apigateway_tes:webcache apigateway_loadge:51566 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51650 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51656 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51624 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51662 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51642 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51604 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51608 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51634 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51658 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51628 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51660 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51572 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51606 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51602 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51638 ESTABLISHED
tcp6       0      0 apigateway_tes:webcache apigateway_loadge:51570 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51618 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51646 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51644 ESTABLISHED
tcp6      97      0 apigateway_tes:webcache apigateway_loadge:51576 ESTABLISHED


On 7/25/2018 9:23 PM, Jason Greene wrote:
Could you post a netstat output so we can see what port numbers your host is picking?

Also is your backlog setting low by chance? 

On Jul 25, 2018, at 6:24 PM, Stuart Douglas <sdouglas@redhat.com> wrote:

The mapping is done by a hash of the remote IP+port. It sounds like maybe this machine is allocating ports in a way that does not map well to our hash. 

Because the remote IP is the same it is really only the port that comes into effect. The algorithm is in org.xnio.nio.QueuedNioTcpServer#handleReady and in this case would simplify down to:

(((C1 * 23) + P) * 23 + C2) % 8

Where C1 is a hash of the remote IP, and C2 is a hash of the local IP+port combo. 

Stuart

On Thu, Jul 26, 2018 at 3:52 AM R. Matt Barnett <barnett@rice.edu> wrote:

I did. I set the concurrency level of ab to 128. I still see only 4 overlaps:

$ java -jar undertow-test-0.1.0-jar-with-dependencies.jar &

Server started on port 8080
1
2
3
4

$ netstat -t | grep apigateway_loadge | grep -c ESTABLISHED
126


What is the algorithm for mapping connections to IO threads?  As a new Undertow user I had assumed round robin, but it sounds like this is not the case.


-- Matt


On 7/25/2018 11:49 AM, Bill O'Neil wrote:
Did you try setting the concurrency level much higher than 8 like I suggested earlier? You are probably having multiple connections assigned to the same IO threads.

On Wed, Jul 25, 2018 at 12:26 PM, R. Matt Barnett <barnett@rice.edu> wrote:
Corrected test to resolve test/set race.


https://gist.github.com/rmbarnett-rice/1179c4ad1d3344bb247c8b8daed3e4fa


I've also discovered this morning that I *can* see 1-8 printed on Red
Hat when I generate load using ab from Windows, but only 1-4 when
running ab on Red Hat (both locally and from a remote server).  I'm
wondering if perhaps there is some sort of connection reuse shenanigans
going on.  My assumption of the use of the -c 8 parameter was "make 8
sockets" but maybe not.  I'll dig in and report back.


-- Matt


On 7/24/2018 6:56 PM, R. Matt Barnett wrote:
> Hello,
>
> I'm experiencing an Undertow performance issue I fail to understand.  I
> am able to reproduce the issue with the code linked bellow. The problem
> is that on Red Hat (and not Windows) I'm unable to concurrently process
> more than 4 overlapping requests even with 8 configured IO Threads.
> For example, if I run the following program (1 file, 55 lines):
>
> https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5
>
> ... on Red Hat and then send requests to the server using Apache
> Benchmark...
>
>       > ab -n 1000 -c 8 localhost:8080/
>
> I see the following output from the Undertow process:
>
>       Server started on port 8080
>
>       1
>       2
>       3
>       4
>
> I believe this demonstrates that only 4 requests are ever processed in
> parallel.  I would expect 8.  In fact, when I run the same experiment on
> Windows I see the expected output of
>
>       Server started on port 8080
>       1
>       2
>       3
>       4
>       5
>       6
>       7
>       8
>
> Any thoughts as to what might explain this behavior?
>
> Best,
>
> Matt
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev

_______________________________________________
undertow-dev mailing list
undertow-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev


_______________________________________________
undertow-dev mailing list
undertow-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
_______________________________________________
undertow-dev mailing list
undertow-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev