[undertow-dev] Unable to concurrently use all available IO Threads under load on Red Hat

[Carter Kozak]

When I run the test case locally I immediately see 1-8 printed.
Perhaps there aren't many cores available on your test machine?

Also note that writing the response is handled asynchronously, so it's
likely not a very large percentage of the total request handling time
spent inside your HttpHandler implementation.

-ck

On Tue, Jul 24, 2018 at 8:59 PM, Bill O'Neil <bill at dartalley.com> wrote:
> I believe one of the selector implementations buckets the connections to IO
> threads based on the port the socket is open on. This means with bad luck
> you could have 4 connections waiting on a single IO thread while other IO
> threads are idle especially if the connections are keepalive. I would try
> boosting the concurrency level quite a bit and see if you notice more IO
> threads being used.
>
>
> On Tue, Jul 24, 2018 at 8:56 PM, R. Matt Barnett <barnett at rice.edu> wrote:
>>
>> My test is a little bogus anyway because I realized on the drive home
>> I did an invalid test/set on max seen.  But I think the conclusion
>> still stands because we only see 4 printfs.
>>
>> I'm not very experienced with nio, but the way I assumed Undertow
>> worked, at a high level, was as follows:
>>
>> 1.) Each incoming socket connection generated a channel.
>> 2.) Each channel created by step 1.) was associated with a singleton
>> selector.
>> 3.) All IO threads polled the singleton selector waiting for requests
>> to process.
>>
>> Sort of a multi-producer/multi-consumer with a singleton queue model.
>>
>> Is this not the case? Is it the case that only one thread can poll
>> from a selector?
>>
>> -- Matt
>>
>> Quoting Stuart Douglas <sdouglas at redhat.com>:
>>
>> > There is no guarantee that connections will be evenly distributed
>> > between
>> > IO threads. Once a client has connected the connection is tied to that
>> > IO
>> > thread, so it may be that you are just ending up with 2 connections on 4
>> > threads.
>> >
>> > Stuart
>> >
>> >
>> >
>> >
>> > On Wed, Jul 25, 2018 at 10:02 AM R. Matt Barnett <barnett at rice.edu>
>> > wrote:
>> >
>> >> Hello,
>> >>
>> >> I'm experiencing an Undertow performance issue I fail to understand.  I
>> >> am able to reproduce the issue with the code linked bellow. The problem
>> >> is that on Red Hat (and not Windows) I'm unable to concurrently process
>> >> more than 4 overlapping requests even with 8 configured IO Threads.
>> >> For example, if I run the following program (1 file, 55 lines):
>> >>
>> >> https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5
>> >>
>> >> ... on Red Hat and then send requests to the server using Apache
>> >> Benchmark...
>> >>
>> >>      > ab -n 1000 -c 8 localhost:8080/
>> >>
>> >> I see the following output from the Undertow process:
>> >>
>> >>      Server started on port 8080
>> >>
>> >>      1
>> >>      2
>> >>      3
>> >>      4
>> >>
>> >> I believe this demonstrates that only 4 requests are ever processed in
>> >> parallel.  I would expect 8.  In fact, when I run the same experiment
>> >> on
>> >> Windows I see the expected output of
>> >>
>> >>      Server started on port 8080
>> >>      1
>> >>      2
>> >>      3
>> >>      4
>> >>      5
>> >>      6
>> >>      7
>> >>      8
>> >>
>> >> Any thoughts as to what might explain this behavior?
>> >>
>> >> Best,
>> >>
>> >> Matt
>> >>
>> >> _______________________________________________
>> >> undertow-dev mailing list
>> >> undertow-dev at lists.jboss.org
>> >> https://lists.jboss.org/mailman/listinfo/undertow-dev
>>
>>
>>
>> _______________________________________________
>> undertow-dev mailing list
>> undertow-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
>
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev