Hello, I'm experiencing an Undertow performance issue I fail to understand. I am able to reproduce the issue with the code linked bellow. The problem is that on Red Hat (and not Windows) I'm unable to concurrently process more than 4 overlapping requests even with 8 configured IO Threads. For example, if I run the following program (1 file, 55 lines): https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5 ... on Red Hat and then send requests to the server using Apache Benchmark... > ab -n 1000 -c 8 localhost:8080/ I see the following output from the Undertow process: Server started on port 8080 1 2 3 4 I believe this demonstrates that only 4 requests are ever processed in parallel. I would expect 8. In fact, when I run the same experiment on Windows I see the expected output of Server started on port 8080 1 2 3 4 5 6 7 8 Any thoughts as to what might explain this behavior? Best, Matt _______________________________________________ undertow-dev mailing list undertow-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/undertow-dev

My test is a little bogus anyway because I realized on the drive home I did an invalid test/set on max seen. But I think the conclusion still stands because we only see 4 printfs. I'm not very experienced with nio, but the way I assumed Undertow worked, at a high level, was as follows: 1.) Each incoming socket connection generated a channel. 2.) Each channel created by step 1.) was associated with a singleton selector. 3.) All IO threads polled the singleton selector waiting for requests to process. Sort of a multi-producer/multi-consumer with a singleton queue model. Is this not the case? Is it the case that only one thread can poll from a selector? -- Matt Quoting Stuart Douglas <sdouglas(a)redhat.com&gt;: > There is no guarantee that connections will be evenly distributed between > IO threads. Once a client has connected the connection is tied to that IO > thread, so it may be that you are just ending up with 2 connections on 4 > threads. > > Stuart > > > > > On Wed, Jul 25, 2018 at 10:02 AM R. Matt Barnett <barnett(a)rice.edu&gt; wrote: > >> Hello, >> >> I'm experiencing an Undertow performance issue I fail to understand. I >> am able to reproduce the issue with the code linked bellow. The problem >> is that on Red Hat (and not Windows) I'm unable to concurrently process >> more than 4 overlapping requests even with 8 configured IO Threads. >> For example, if I run the following program (1 file, 55 lines): >> >> https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5 >> >> ... on Red Hat and then send requests to the server using Apache >> Benchmark... >> >> > ab -n 1000 -c 8 localhost:8080/ >> >> I see the following output from the Undertow process: >> >> Server started on port 8080 >> >> 1 >> 2 >> 3 >> 4 >> >> I believe this demonstrates that only 4 requests are ever processed in >> parallel. I would expect 8. In fact, when I run the same experiment on >> Windows I see the expected output of >> >> Server started on port 8080 >> 1 >> 2 >> 3 >> 4 >> 5 >> 6 >> 7 >> 8 >> >> Any thoughts as to what might explain this behavior? >> >> Best, >> >> Matt >> >> _______________________________________________ >> undertow-dev mailing list >> undertow-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/undertow-dev _______________________________________________ undertow-dev mailing list undertow-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/undertow-dev

Hello, I'm experiencing an Undertow performance issue I fail to understand.  I am able to reproduce the issue with the code linked bellow. The problem is that on Red Hat (and not Windows) I'm unable to concurrently process more than 4 overlapping requests even with 8 configured IO Threads. For example, if I run the following program (1 file, 55 lines): https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5 ... on Red Hat and then send requests to the server using Apache Benchmark...     > ab -n 1000 -c 8 localhost:8080/ I see the following output from the Undertow process:     Server started on port 8080     1     2     3     4 I believe this demonstrates that only 4 requests are ever processed in parallel.  I would expect 8.  In fact, when I run the same experiment on Windows I see the expected output of     Server started on port 8080     1     2     3     4     5     6     7     8 Any thoughts as to what might explain this behavior? Best, Matt _______________________________________________ undertow-dev mailing list undertow-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/undertow-dev

Did you try setting the concurrency level much higher than 8 like I suggested earlier? You are probably having multiple connections assigned to the same IO threads. On Wed, Jul 25, 2018 at 12:26 PM, R. Matt Barnett <barnett(a)rice.edu <mailto:barnett@rice.edu>> wrote: Corrected test to resolve test/set race. https://gist.github.com/rmbarnett-rice/1179c4ad1d3344bb247c8b8daed3e4fa <https://gist.github.com/rmbarnett-rice/1179c4ad1d3344bb247c8b8daed3e4fa> I've also discovered this morning that I *can* see 1-8 printed on Red Hat when I generate load using ab from Windows, but only 1-4 when running ab on Red Hat (both locally and from a remote server).  I'm wondering if perhaps there is some sort of connection reuse shenanigans going on.  My assumption of the use of the -c 8 parameter was "make 8 sockets" but maybe not.  I'll dig in and report back. -- Matt On 7/24/2018 6:56 PM, R. Matt Barnett wrote: > Hello, > > I'm experiencing an Undertow performance issue I fail to understand.  I > am able to reproduce the issue with the code linked bellow. The problem > is that on Red Hat (and not Windows) I'm unable to concurrently process > more than 4 overlapping requests even with 8 configured IO Threads. > For example, if I run the following program (1 file, 55 lines): > > https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5 <https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5> > > ... on Red Hat and then send requests to the server using Apache > Benchmark... > >       > ab -n 1000 -c 8 localhost:8080/ > > I see the following output from the Undertow process: > >       Server started on port 8080 > >       1 >       2 >       3 >       4 > > I believe this demonstrates that only 4 requests are ever processed in > parallel.  I would expect 8.  In fact, when I run the same experiment on > Windows I see the expected output of > >       Server started on port 8080 >       1 >       2 >       3 >       4 >       5 >       6 >       7 >       8 > > Any thoughts as to what might explain this behavior? > > Best, > > Matt > > _______________________________________________ > undertow-dev mailing list > undertow-dev(a)lists.jboss.org <mailto:undertow-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/undertow-dev <https://lists.jboss.org/mailman/listinfo/undertow-dev> _______________________________________________ undertow-dev mailing list undertow-dev(a)lists.jboss.org <mailto:undertow-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/undertow-dev <https://lists.jboss.org/mailman/listinfo/undertow-dev>

I did. I set the concurrency level of ab to 128. I still see only 4 overlaps: $ java -jar undertow-test-0.1.0-jar-with-dependencies.jar & Server started on port 8080 1 2 3 4 $ netstat -t | grep apigateway_loadge | grep -c ESTABLISHED 126 What is the algorithm for mapping connections to IO threads? As a new Undertow user I had assumed round robin, but it sounds like this is not the case. -- Matt On 7/25/2018 11:49 AM, Bill O'Neil wrote: Did you try setting the concurrency level much higher than 8 like I suggested earlier? You are probably having multiple connections assigned to the same IO threads. On Wed, Jul 25, 2018 at 12:26 PM, R. Matt Barnett <barnett(a)rice.edu&gt; wrote: > Corrected test to resolve test/set race. > > > https://gist.github.com/rmbarnett-rice/1179c4ad1d3344bb247c8b8daed3e4fa > > > I've also discovered this morning that I *can* see 1-8 printed on Red > Hat when I generate load using ab from Windows, but only 1-4 when > running ab on Red Hat (both locally and from a remote server). I'm > wondering if perhaps there is some sort of connection reuse shenanigans > going on. My assumption of the use of the -c 8 parameter was "make 8 > sockets" but maybe not. I'll dig in and report back. > > > -- Matt > > > On 7/24/2018 6:56 PM, R. Matt Barnett wrote: > > Hello, > > > > I'm experiencing an Undertow performance issue I fail to understand. I > > am able to reproduce the issue with the code linked bellow. The problem > > is that on Red Hat (and not Windows) I'm unable to concurrently process > > more than 4 overlapping requests even with 8 configured IO Threads. > > For example, if I run the following program (1 file, 55 lines): > > > > https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5 > > > > ... on Red Hat and then send requests to the server using Apache > > Benchmark... > > > > > ab -n 1000 -c 8 localhost:8080/ > > > > I see the following output from the Undertow process: > > > > Server started on port 8080 > > > > 1 > > 2 > > 3 > > 4 > > > > I believe this demonstrates that only 4 requests are ever processed in > > parallel. I would expect 8. In fact, when I run the same experiment on > > Windows I see the expected output of > > > > Server started on port 8080 > > 1 > > 2 > > 3 > > 4 > > 5 > > 6 > > 7 > > 8 > > > > Any thoughts as to what might explain this behavior? > > > > Best, > > > > Matt > > > > _______________________________________________ > > undertow-dev mailing list > > undertow-dev(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/undertow-dev > > _______________________________________________ > undertow-dev mailing list > undertow-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/undertow-dev > _______________________________________________ undertow-dev mailing list undertow-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/undertow-dev

Backlog setting is 1000. Is this what you are interested in from netstat? This was for ab with a -c of 50. [barnett@apigateway_test ~]$ java -jar undertow-test-0.1.0-jar-with-dependencies.jar & [1] 7329 [barnett@apigateway_test ~]$ Jul 26, 2018 1:30:22 PM org.xnio.Xnio <clinit> INFO: XNIO version 3.3.8.Final Jul 26, 2018 1:30:23 PM org.xnio.nio.NioXnio <clinit> INFO: XNIO NIO Implementation Version 3.3.8.Final Server started on port 8080 1 2 3 4 [barnett@apigateway_test ~]$ netstat -t | grep apigateway_loadge | grep ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51580 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51614 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51622 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51626 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51612 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51578 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51636 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51616 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51582 ESTABLISHED tcp6 0 0 apigateway_tes:webcache apigateway_loadge:51556 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51588 ESTABLISHED tcp6 0 0 apigateway_tes:webcache apigateway_loadge:51558 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51586 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51648 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51632 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51652 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51654 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51574 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51640 ESTABLISHED tcp6 0 0 apigateway_tes:webcache apigateway_loadge:51564 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51590 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51610 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51594 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51592 ESTABLISHED tcp6 0 0 apigateway_tes:webcache apigateway_loadge:51568 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51620 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51598 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51600 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51584 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51630 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51596 ESTABLISHED tcp6 0 0 apigateway_tes:webcache apigateway_loadge:51566 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51650 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51656 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51624 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51662 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51642 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51604 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51608 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51634 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51658 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51628 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51660 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51572 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51606 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51602 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51638 ESTABLISHED tcp6 0 0 apigateway_tes:webcache apigateway_loadge:51570 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51618 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51646 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51644 ESTABLISHED tcp6 97 0 apigateway_tes:webcache apigateway_loadge:51576 ESTABLISHED On 7/25/2018 9:23 PM, Jason Greene wrote: Could you post a netstat output so we can see what port numbers your host is picking? Also is your backlog setting low by chance? On Jul 25, 2018, at 6:24 PM, Stuart Douglas <sdouglas(a)redhat.com&gt; wrote: The mapping is done by a hash of the remote IP+port. It sounds like maybe this machine is allocating ports in a way that does not map well to our hash. Because the remote IP is the same it is really only the port that comes into effect. The algorithm is in org.xnio.nio.QueuedNioTcpServer#handleReady and in this case would simplify down to: (((C1 * 23) + P) * 23 + C2) % 8 Where C1 is a hash of the remote IP, and C2 is a hash of the local IP+port combo. Stuart On Thu, Jul 26, 2018 at 3:52 AM R. Matt Barnett <barnett(a)rice.edu&gt; wrote: > I did. I set the concurrency level of ab to 128. I still see only 4 > overlaps: > > $ java -jar undertow-test-0.1.0-jar-with-dependencies.jar & > > Server started on port 8080 > 1 > 2 > 3 > 4 > > $ netstat -t | grep apigateway_loadge | grep -c ESTABLISHED > 126 > > > What is the algorithm for mapping connections to IO threads? As a new > Undertow user I had assumed round robin, but it sounds like this is not the > case. > > > -- Matt > > On 7/25/2018 11:49 AM, Bill O'Neil wrote: > > Did you try setting the concurrency level much higher than 8 like I > suggested earlier? You are probably having multiple connections assigned to > the same IO threads. > > On Wed, Jul 25, 2018 at 12:26 PM, R. Matt Barnett <barnett(a)rice.edu&gt; > wrote: > >> Corrected test to resolve test/set race. >> >> >> https://gist.github.com/rmbarnett-rice/1179c4ad1d3344bb247c8b8daed3e4fa >> >> >> I've also discovered this morning that I *can* see 1-8 printed on Red >> Hat when I generate load using ab from Windows, but only 1-4 when >> running ab on Red Hat (both locally and from a remote server). I'm >> wondering if perhaps there is some sort of connection reuse shenanigans >> going on. My assumption of the use of the -c 8 parameter was "make 8 >> sockets" but maybe not. I'll dig in and report back. >> >> >> -- Matt >> >> >> On 7/24/2018 6:56 PM, R. Matt Barnett wrote: >> > Hello, >> > >> > I'm experiencing an Undertow performance issue I fail to understand. I >> > am able to reproduce the issue with the code linked bellow. The problem >> > is that on Red Hat (and not Windows) I'm unable to concurrently process >> > more than 4 overlapping requests even with 8 configured IO Threads. >> > For example, if I run the following program (1 file, 55 lines): >> > >> > >> https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5 >> > >> > ... on Red Hat and then send requests to the server using Apache >> > Benchmark... >> > >> > > ab -n 1000 -c 8 localhost:8080/ >> > >> > I see the following output from the Undertow process: >> > >> > Server started on port 8080 >> > >> > 1 >> > 2 >> > 3 >> > 4 >> > >> > I believe this demonstrates that only 4 requests are ever processed in >> > parallel. I would expect 8. In fact, when I run the same experiment >> on >> > Windows I see the expected output of >> > >> > Server started on port 8080 >> > 1 >> > 2 >> > 3 >> > 4 >> > 5 >> > 6 >> > 7 >> > 8 >> > >> > Any thoughts as to what might explain this behavior? >> > >> > Best, >> > >> > Matt >> > >> > _______________________________________________ >> > undertow-dev mailing list >> > undertow-dev(a)lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/undertow-dev >> >> _______________________________________________ >> undertow-dev mailing list >> undertow-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/undertow-dev >> > > > _______________________________________________ > undertow-dev mailing list > undertow-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/undertow-dev _______________________________________________ undertow-dev mailing list undertow-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/undertow-dev

Looks like we need to tweak the hash: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit... On Jul 26, 2018, at 7:13 PM, Stuart Douglas <sdouglas(a)redhat.com <mailto:sdouglas@redhat.com>> wrote: > They are all even numbers :-( > > This does not play well with our hash if C1 is also even: > > (((C1 * 23) + P) * 23 + C2) % 8 > > If C1 is even the C1 * 23 is even. This means ((C1 * 23) + P) * 23 is > even. Depending on the value of C2 this means the result is always > even or always odd, so with an evenly divisible number of threads you > are only ever going to allocate to half of them. > > The good news is this should be easily fixed by using an odd number > of IO threads, but we probably should revisit this. > > Stuart > > On Fri, Jul 27, 2018 at 4:34 AM R. Matt Barnett <barnett(a)rice.edu > <mailto:barnett@rice.edu>> wrote: > > Backlog setting is 1000. > > Is this what you are interested in from netstat? This was for ab > with a -c of 50. > > > [barnett@apigateway_test ~]$ java -jar > undertow-test-0.1.0-jar-with-dependencies.jar & > [1] 7329 > [barnett@apigateway_test ~]$ Jul 26, 2018 1:30:22 PM > org.xnio.Xnio <clinit> > INFO: XNIO version 3.3.8.Final > Jul 26, 2018 1:30:23 PM org.xnio.nio.NioXnio <clinit> > INFO: XNIO NIO Implementation Version 3.3.8.Final > > > Server started on port 8080 > 1 > 2 > 3 > 4 > [barnett@apigateway_test ~]$ netstat -t | grep apigateway_loadge > | grep ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51580 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51614 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51622 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51626 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51612 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51578 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51636 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51616 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51582 ESTABLISHED > tcp6       0      0 apigateway_tes:webcache > apigateway_loadge:51556 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51588 ESTABLISHED > tcp6       0      0 apigateway_tes:webcache > apigateway_loadge:51558 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51586 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51648 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51632 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51652 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51654 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51574 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51640 ESTABLISHED > tcp6       0      0 apigateway_tes:webcache > apigateway_loadge:51564 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51590 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51610 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51594 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51592 ESTABLISHED > tcp6       0      0 apigateway_tes:webcache > apigateway_loadge:51568 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51620 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51598 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51600 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51584 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51630 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51596 ESTABLISHED > tcp6       0      0 apigateway_tes:webcache > apigateway_loadge:51566 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51650 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51656 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51624 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51662 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51642 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51604 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51608 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51634 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51658 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51628 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51660 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51572 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51606 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51602 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51638 ESTABLISHED > tcp6       0      0 apigateway_tes:webcache > apigateway_loadge:51570 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51618 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51646 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51644 ESTABLISHED > tcp6      97      0 apigateway_tes:webcache > apigateway_loadge:51576 ESTABLISHED > > > On 7/25/2018 9:23 PM, Jason Greene wrote: >> Could you post a netstat output so we can see what port numbers >> your host is picking? >> >> Also is your backlog setting low by chance? >> >> On Jul 25, 2018, at 6:24 PM, Stuart Douglas <sdouglas(a)redhat.com >> <mailto:sdouglas@redhat.com>> wrote: >> >>> The mapping is done by a hash of the remote IP+port. It sounds >>> like maybe this machine is allocating ports in a way that does >>> not map well to our hash. >>> >>> Because the remote IP is the same it is really only the port >>> that comes into effect. The algorithm is >>> in org.xnio.nio.QueuedNioTcpServer#handleReady and in this case >>> would simplify down to: >>> >>> (((C1 * 23) + P) * 23 + C2) % 8 >>> >>> Where C1 is a hash of the remote IP, and C2 is a hash of the >>> local IP+port combo. >>> >>> Stuart >>> >>> On Thu, Jul 26, 2018 at 3:52 AM R. Matt Barnett >>> <barnett(a)rice.edu <mailto:barnett@rice.edu>> wrote: >>> >>> I did. I set the concurrency level of ab to 128. I still >>> see only 4 overlaps: >>> >>> $ java -jar undertow-test-0.1.0-jar-with-dependencies.jar & >>> >>> Server started on port 8080 >>> 1 >>> 2 >>> 3 >>> 4 >>> >>> $ netstat -t | grep apigateway_loadge | grep -c ESTABLISHED >>> 126 >>> >>> >>> What is the algorithm for mapping connections to IO >>> threads?  As a new Undertow user I had assumed round robin, >>> but it sounds like this is not the case. >>> >>> >>> -- Matt >>> >>> >>> On 7/25/2018 11:49 AM, Bill O'Neil wrote: >>>> Did you try setting the concurrency level much higher than >>>> 8 like I suggested earlier? You are probably having >>>> multiple connections assigned to the same IO threads. >>>> >>>> On Wed, Jul 25, 2018 at 12:26 PM, R. Matt Barnett >>>> <barnett(a)rice.edu <mailto:barnett@rice.edu>> wrote: >>>> >>>> Corrected test to resolve test/set race. >>>> >>>> >>>> https://gist.github.com/rmbarnett-rice/1179c4ad1d3344bb247c8b8daed3e4fa >>>> >>>> >>>> I've also discovered this morning that I *can* see 1-8 >>>> printed on Red >>>> Hat when I generate load using ab from Windows, but >>>> only 1-4 when >>>> running ab on Red Hat (both locally and from a remote >>>> server).  I'm >>>> wondering if perhaps there is some sort of connection >>>> reuse shenanigans >>>> going on.  My assumption of the use of the -c 8 >>>> parameter was "make 8 >>>> sockets" but maybe not. I'll dig in and report back. >>>> >>>> >>>> -- Matt >>>> >>>> >>>> On 7/24/2018 6:56 PM, R. Matt Barnett wrote: >>>> > Hello, >>>> > >>>> > I'm experiencing an Undertow performance issue I >>>> fail to understand.  I >>>> > am able to reproduce the issue with the code linked >>>> bellow. The problem >>>> > is that on Red Hat (and not Windows) I'm unable to >>>> concurrently process >>>> > more than 4 overlapping requests even with 8 >>>> configured IO Threads. >>>> > For example, if I run the following program (1 file, >>>> 55 lines): >>>> > >>>> > >>>> https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5 >>>> > >>>> > ... on Red Hat and then send requests to the server >>>> using Apache >>>> > Benchmark... >>>> > >>>> >       > ab -n 1000 -c 8 localhost:8080/ >>>> > >>>> > I see the following output from the Undertow process: >>>> > >>>> >       Server started on port 8080 >>>> > >>>> >       1 >>>> >       2 >>>> >       3 >>>> >       4 >>>> > >>>> > I believe this demonstrates that only 4 requests are >>>> ever processed in >>>> > parallel.  I would expect 8.  In fact, when I run >>>> the same experiment on >>>> > Windows I see the expected output of >>>> > >>>> >       Server started on port 8080 >>>> >       1 >>>> >       2 >>>> >       3 >>>> >       4 >>>> >       5 >>>> >       6 >>>> >       7 >>>> >       8 >>>> > >>>> > Any thoughts as to what might explain this behavior? >>>> > >>>> > Best, >>>> > >>>> > Matt >>>> > >>>> > _______________________________________________ >>>> > undertow-dev mailing list >>>> > undertow-dev(a)lists.jboss.org >>>> <mailto:undertow-dev@lists.jboss.org> >>>> > https://lists.jboss.org/mailman/listinfo/undertow-dev >>>> >>>> _______________________________________________ >>>> undertow-dev mailing list >>>> undertow-dev(a)lists.jboss.org >>>> <mailto:undertow-dev@lists.jboss.org> >>>> https://lists.jboss.org/mailman/listinfo/undertow-dev >>>> >>>> >>> >>> _______________________________________________ >>> undertow-dev mailing list >>> undertow-dev(a)lists.jboss.org >>> <mailto:undertow-dev@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/undertow-dev >>> >>> _______________________________________________ >>> undertow-dev mailing list >>> undertow-dev(a)lists.jboss.org <mailto:undertow-dev@lists.jboss.org> >>> https://lists.jboss.org/mailman/listinfo/undertow-dev >

I have created https://issues.jboss.org/browse/XNIO-328. Stuart On Tue, Aug 14, 2018 at 6:49 AM R. Matt Barnett <barnett(a)rice.edu <mailto:barnett@rice.edu>> wrote: Did you all ever open a ticket on this? If so could you link me so I can follow along? Thanks, Matt On 7/26/2018 9:11 PM, Jason Greene wrote: > Looks like we need to tweak the hash: > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit... > > > > > On Jul 26, 2018, at 7:13 PM, Stuart Douglas <sdouglas(a)redhat.com > <mailto:sdouglas@redhat.com>> wrote: > >> They are all even numbers :-( >> >> This does not play well with our hash if C1 is also even: >> >> (((C1 * 23) + P) * 23 + C2) % 8 >> >> If C1 is even the C1 * 23 is even. This means ((C1 * 23) + P) * >> 23 is even. Depending on the value of C2 this means the result >> is always even or always odd, so with an evenly divisible number >> of threads you are only ever going to allocate to half of them. >> >> The good news is this should be easily fixed by using an odd >> number of IO threads, but we probably should revisit this. >> >> Stuart >> >> On Fri, Jul 27, 2018 at 4:34 AM R. Matt Barnett >> <barnett(a)rice.edu <mailto:barnett@rice.edu>> wrote: >> >> Backlog setting is 1000. >> >> Is this what you are interested in from netstat?  This was >> for ab with a -c of 50. >> >> >> [barnett@apigateway_test ~]$ java -jar >> undertow-test-0.1.0-jar-with-dependencies.jar & >> [1] 7329 >> [barnett@apigateway_test ~]$ Jul 26, 2018 1:30:22 PM >> org.xnio.Xnio <clinit> >> INFO: XNIO version 3.3.8.Final >> Jul 26, 2018 1:30:23 PM org.xnio.nio.NioXnio <clinit> >> INFO: XNIO NIO Implementation Version 3.3.8.Final >> >> >> Server started on port 8080 >> 1 >> 2 >> 3 >> 4 >> [barnett@apigateway_test ~]$ netstat -t | grep >> apigateway_loadge | grep ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51580 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51614 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51622 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51626 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51612 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51578 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51636 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51616 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51582 ESTABLISHED >> tcp6       0      0 apigateway_tes:webcache >> apigateway_loadge:51556 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51588 ESTABLISHED >> tcp6       0      0 apigateway_tes:webcache >> apigateway_loadge:51558 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51586 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51648 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51632 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51652 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51654 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51574 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51640 ESTABLISHED >> tcp6       0      0 apigateway_tes:webcache >> apigateway_loadge:51564 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51590 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51610 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51594 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51592 ESTABLISHED >> tcp6       0      0 apigateway_tes:webcache >> apigateway_loadge:51568 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51620 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51598 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51600 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51584 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51630 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51596 ESTABLISHED >> tcp6       0      0 apigateway_tes:webcache >> apigateway_loadge:51566 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51650 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51656 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51624 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51662 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51642 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51604 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51608 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51634 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51658 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51628 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51660 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51572 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51606 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51602 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51638 ESTABLISHED >> tcp6       0      0 apigateway_tes:webcache >> apigateway_loadge:51570 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51618 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51646 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51644 ESTABLISHED >> tcp6      97      0 apigateway_tes:webcache >> apigateway_loadge:51576 ESTABLISHED >> >> >> On 7/25/2018 9:23 PM, Jason Greene wrote: >>> Could you post a netstat output so we can see what port >>> numbers your host is picking? >>> >>> Also is your backlog setting low by chance? >>> >>> On Jul 25, 2018, at 6:24 PM, Stuart Douglas >>> <sdouglas(a)redhat.com <mailto:sdouglas@redhat.com>> wrote: >>> >>>> The mapping is done by a hash of the remote IP+port. It >>>> sounds like maybe this machine is allocating ports in a >>>> way that does not map well to our hash. >>>> >>>> Because the remote IP is the same it is really only the >>>> port that comes into effect. The algorithm is >>>> in org.xnio.nio.QueuedNioTcpServer#handleReady and in this >>>> case would simplify down to: >>>> >>>> (((C1 * 23) + P) * 23 + C2) % 8 >>>> >>>> Where C1 is a hash of the remote IP, and C2 is a hash of >>>> the local IP+port combo. >>>> >>>> Stuart >>>> >>>> On Thu, Jul 26, 2018 at 3:52 AM R. Matt Barnett >>>> <barnett(a)rice.edu <mailto:barnett@rice.edu>> wrote: >>>> >>>> I did. I set the concurrency level of ab to 128. I >>>> still see only 4 overlaps: >>>> >>>> $ java -jar >>>> undertow-test-0.1.0-jar-with-dependencies.jar & >>>> >>>> Server started on port 8080 >>>> 1 >>>> 2 >>>> 3 >>>> 4 >>>> >>>> $ netstat -t | grep apigateway_loadge | grep -c >>>> ESTABLISHED >>>> 126 >>>> >>>> >>>> What is the algorithm for mapping connections to IO >>>> threads?  As a new Undertow user I had assumed round >>>> robin, but it sounds like this is not the case. >>>> >>>> >>>> -- Matt >>>> >>>> >>>> On 7/25/2018 11:49 AM, Bill O'Neil wrote: >>>>> Did you try setting the concurrency level much higher >>>>> than 8 like I suggested earlier? You are probably >>>>> having multiple connections assigned to the same IO >>>>> threads. >>>>> >>>>> On Wed, Jul 25, 2018 at 12:26 PM, R. Matt Barnett >>>>> <barnett(a)rice.edu <mailto:barnett@rice.edu>> wrote: >>>>> >>>>> Corrected test to resolve test/set race. >>>>> >>>>> >>>>> https://gist.github.com/rmbarnett-rice/1179c4ad1d3344bb247c8b8daed3e4fa >>>>> >>>>> >>>>> I've also discovered this morning that I *can* >>>>> see 1-8 printed on Red >>>>> Hat when I generate load using ab from Windows, >>>>> but only 1-4 when >>>>> running ab on Red Hat (both locally and from a >>>>> remote server).  I'm >>>>> wondering if perhaps there is some sort of >>>>> connection reuse shenanigans >>>>> going on.  My assumption of the use of the -c 8 >>>>> parameter was "make 8 >>>>> sockets" but maybe not.  I'll dig in and report back. >>>>> >>>>> >>>>> -- Matt >>>>> >>>>> >>>>> On 7/24/2018 6:56 PM, R. Matt Barnett wrote: >>>>> > Hello, >>>>> > >>>>> > I'm experiencing an Undertow performance issue >>>>> I fail to understand.  I >>>>> > am able to reproduce the issue with the code >>>>> linked bellow. The problem >>>>> > is that on Red Hat (and not Windows) I'm unable >>>>> to concurrently process >>>>> > more than 4 overlapping requests even with 8 >>>>> configured IO Threads. >>>>> > For example, if I run the following program (1 >>>>> file, 55 lines): >>>>> > >>>>> > >>>>> https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5 >>>>> > >>>>> > ... on Red Hat and then send requests to the >>>>> server using Apache >>>>> > Benchmark... >>>>> > >>>>> >       > ab -n 1000 -c 8 localhost:8080/ >>>>> > >>>>> > I see the following output from the Undertow >>>>> process: >>>>> > >>>>> > Server started on port 8080 >>>>> > >>>>> >       1 >>>>> >       2 >>>>> >       3 >>>>> >       4 >>>>> > >>>>> > I believe this demonstrates that only 4 >>>>> requests are ever processed in >>>>> > parallel. I would expect 8.  In fact, when I >>>>> run the same experiment on >>>>> > Windows I see the expected output of >>>>> > >>>>> > Server started on port 8080 >>>>> >       1 >>>>> >       2 >>>>> >       3 >>>>> >       4 >>>>> >       5 >>>>> >       6 >>>>> >       7 >>>>> >       8 >>>>> > >>>>> > Any thoughts as to what might explain this >>>>> behavior? >>>>> > >>>>> > Best, >>>>> > >>>>> > Matt >>>>> > >>>>> > _______________________________________________ >>>>> > undertow-dev mailing list >>>>> > undertow-dev(a)lists.jboss.org >>>>> <mailto:undertow-dev@lists.jboss.org> >>>>> > >>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev >>>>> >>>>> _______________________________________________ >>>>> undertow-dev mailing list >>>>> undertow-dev(a)lists.jboss.org >>>>> <mailto:undertow-dev@lists.jboss.org> >>>>> https://lists.jboss.org/mailman/listinfo/undertow-dev >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> undertow-dev mailing list >>>> undertow-dev(a)lists.jboss.org >>>> <mailto:undertow-dev@lists.jboss.org> >>>> https://lists.jboss.org/mailman/listinfo/undertow-dev >>>> >>>> _______________________________________________ >>>> undertow-dev mailing list >>>> undertow-dev(a)lists.jboss.org >>>> <mailto:undertow-dev@lists.jboss.org> >>>> https://lists.jboss.org/mailman/listinfo/undertow-dev >>

I've been able to observe 1...8 on Red Hat by adding the following statements to my handler (and setting the worker thread pool size to 8): @Overridepublic void handleRequest(HttpServerExchange httpServerExchange) throws Exception { if (httpServerExchange.isInIoThread()) { httpServerExchange.dispatch(this); return; } ... } I have a few questions about this technique though: 1.) How are dispatch actions mapped onto worker threads? New connections were not mapped to available idle IO threads, so is it possible dispatches also won't be mapped to available idle worker threads but instead queued for currently busy threads?

2.) The Undertow documentation states that HttpServerExchange is not thread-safe. However the documentation states that dispatch(...) has happens-before semantics with respect to the worker thread accessing httpServerExchange. That would seem to make it ok to read from httpServerExchange in the worker thread. Assuming that an IO thread will be responsible for writing the http response back to the client, what steps do I need to take in the body of handleRequest to ensure that my writes to httpServerExchange in the worker thread are observed by the IO thread responsible for transmitting the response to the client? Is invoking httpServerExchange.endExchange(); in the worker thread as the final statement sufficient?

-- Matt On 7/25/2018 11:26 AM, R. Matt Barnett wrote: Corrected test to resolve test/set race. https://gist.github.com/rmbarnett-rice/1179c4ad1d3344bb247c8b8daed3e4fa I've also discovered this morning that I *can* see 1-8 printed on Red Hat when I generate load using ab from Windows, but only 1-4 when running ab on Red Hat (both locally and from a remote server). I'm wondering if perhaps there is some sort of connection reuse shenanigans going on. My assumption of the use of the -c 8 parameter was "make 8 sockets" but maybe not. I'll dig in and report back. -- Matt On 7/24/2018 6:56 PM, R. Matt Barnett wrote: Hello, I'm experiencing an Undertow performance issue I fail to understand. I am able to reproduce the issue with the code linked bellow. The problem is that on Red Hat (and not Windows) I'm unable to concurrently process more than 4 overlapping requests even with 8 configured IO Threads. For example, if I run the following program (1 file, 55 lines): https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5 ... on Red Hat and then send requests to the server using Apache Benchmark... > ab -n 1000 -c 8 localhost:8080/ I see the following output from the Undertow process: Server started on port 8080 1 2 3 4 I believe this demonstrates that only 4 requests are ever processed in parallel. I would expect 8. In fact, when I run the same experiment on Windows I see the expected output of Server started on port 8080 1 2 3 4 5 6 7 8 Any thoughts as to what might explain this behavior? Best, Matt _______________________________________________ undertow-dev mailing listundertow-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/undertow-dev _______________________________________________ undertow-dev mailing listundertow-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/undertow-dev _______________________________________________ undertow-dev mailing list undertow-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/undertow-dev

So just to be clear, am I correct in my understanding that is it safe to invoke any method on the Sender instance returned by getResponseSender() from a Worker Thread without any extra threading considerations? Can you confirm this is true as well for the HeaderMap instance returned by getResponseHeaders() and also for invocations of setStatusCode(...) ? Thanks. On 7/25/2018 6:31 PM, Stuart Douglas wrote: On Thu, Jul 26, 2018 at 5:27 AM R. Matt Barnett <barnett(a)rice.edu&gt; wrote: > I've been able to observe 1...8 on Red Hat by adding the following > statements to my handler (and setting the worker thread pool size to 8): > > > @Overridepublic void handleRequest(HttpServerExchange httpServerExchange) throws Exception > { > if (httpServerExchange.isInIoThread()) { > httpServerExchange.dispatch(this); > return; > } > ... > } > > I have a few questions about this technique though: > > 1.) How are dispatch actions mapped onto worker threads? New connections were not mapped to available idle IO threads, so is it possible dispatches also won't be mapped to available idle worker threads but instead queued for currently > busy threads? > > IO threads are tied to the connection. Once a connection has been accepted only that IO thread will be used to service it. This avoids contention from having a larger number of IO threads waiting on a single selector. The worker thread pool is basically just a normal executor, that will run tasks in a FIFO manner. > 2.) The Undertow documentation states that HttpServerExchange is not thread-safe. However the documentation states that dispatch(...) has happens-before semantics with respect to the worker thread accessing httpServerExchange. > That would seem to make it ok to read from httpServerExchange in the worker thread. Assuming that an IO thread will be responsible for writing the http response back to the client, what steps do I need to take in the body > of handleRequest to ensure that my writes to httpServerExchange in the worker thread are observed by the IO thread responsible for transmitting the response to the client? Is invoking httpServerExchange.endExchange(); in the > worker thread as the final statement sufficient? > > Not all writes are done from the IO thread. For instance if you use blocking IO and are using a Stream then the writes are done from the worker. If you use the Sender to perform async IO then the initial write is done from the original thread, and the IO thread is only involved if the response is too larger to just write out immediately. In this case though the Sender will take care of the thread safety aspects, as the underlying SelectionKey will not have its interest ops set until after the current stack has returned. Basically if you call dispatch(), or perform an action that requires async IO nothing happens immediately, it just sets a flag in the HttpServerExchange. Once the call stack returns (i.e. the current thread is done) one of three things will happen: - If dispatch was called the dispatch task will be run in an executor - If async IO was required the underlying SelectionKey will have its interest ops modified, so the IO thread can perform the IO - If neither of the above happened then the exchange is ended. Stuart > > -- Matt > > On 7/25/2018 11:26 AM, R. Matt Barnett wrote: > > Corrected test to resolve test/set race. > > https://gist.github.com/rmbarnett-rice/1179c4ad1d3344bb247c8b8daed3e4fa > > > I've also discovered this morning that I *can* see 1-8 printed on Red > Hat when I generate load using ab from Windows, but only 1-4 when > running ab on Red Hat (both locally and from a remote server). I'm > wondering if perhaps there is some sort of connection reuse shenanigans > going on. My assumption of the use of the -c 8 parameter was "make 8 > sockets" but maybe not. I'll dig in and report back. > > > -- Matt > > > On 7/24/2018 6:56 PM, R. Matt Barnett wrote: > > Hello, > > I'm experiencing an Undertow performance issue I fail to understand. I > am able to reproduce the issue with the code linked bellow. The problem > is that on Red Hat (and not Windows) I'm unable to concurrently process > more than 4 overlapping requests even with 8 configured IO Threads. > For example, if I run the following program (1 file, 55 lines): > https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5 > > ... on Red Hat and then send requests to the server using Apache > Benchmark... > > > ab -n 1000 -c 8 localhost:8080/ > > I see the following output from the Undertow process: > > Server started on port 8080 > > 1 > 2 > 3 > 4 > > I believe this demonstrates that only 4 requests are ever processed in > parallel. I would expect 8. In fact, when I run the same experiment on > Windows I see the expected output of > > Server started on port 8080 > 1 > 2 > 3 > 4 > 5 > 6 > 7 > 8 > > Any thoughts as to what might explain this behavior? > > Best, > > Matt > > _______________________________________________ > undertow-dev mailing listundertow-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/undertow-dev > > _______________________________________________ > undertow-dev mailing listundertow-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/undertow-dev > > > _______________________________________________ > undertow-dev mailing list > undertow-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/undertow-dev