[undertow-dev] Unable to concurrently use all available IO Threads under load on Red Hat

[Bill O'Neil]

I believe one of the selector implementations buckets the connections to IO
threads based on the port the socket is open on. This means with bad luck
you could have 4 connections waiting on a single IO thread while other IO
threads are idle especially if the connections are keepalive. I would try
boosting the concurrency level quite a bit and see if you notice more IO
threads being used.


On Tue, Jul 24, 2018 at 8:56 PM, R. Matt Barnett <barnett at rice.edu> wrote:

> My test is a little bogus anyway because I realized on the drive home
> I did an invalid test/set on max seen.  But I think the conclusion
> still stands because we only see 4 printfs.
>
> I'm not very experienced with nio, but the way I assumed Undertow
> worked, at a high level, was as follows:
>
> 1.) Each incoming socket connection generated a channel.
> 2.) Each channel created by step 1.) was associated with a singleton
> selector.
> 3.) All IO threads polled the singleton selector waiting for requests
> to process.
>
> Sort of a multi-producer/multi-consumer with a singleton queue model.
>
> Is this not the case? Is it the case that only one thread can poll
> from a selector?
>
> -- Matt
>
> Quoting Stuart Douglas <sdouglas at redhat.com>:
>
> > There is no guarantee that connections will be evenly distributed between
> > IO threads. Once a client has connected the connection is tied to that IO
> > thread, so it may be that you are just ending up with 2 connections on 4
> > threads.
> >
> > Stuart
> >
> >
> >
> >
> > On Wed, Jul 25, 2018 at 10:02 AM R. Matt Barnett <barnett at rice.edu>
> wrote:
> >
> >> Hello,
> >>
> >> I'm experiencing an Undertow performance issue I fail to understand.  I
> >> am able to reproduce the issue with the code linked bellow. The problem
> >> is that on Red Hat (and not Windows) I'm unable to concurrently process
> >> more than 4 overlapping requests even with 8 configured IO Threads.
> >> For example, if I run the following program (1 file, 55 lines):
> >>
> >> https://gist.github.com/rmbarnett-rice/668db6b4e9f8f8da7093a3659b6ae2b5
> >>
> >> ... on Red Hat and then send requests to the server using Apache
> >> Benchmark...
> >>
> >>      > ab -n 1000 -c 8 localhost:8080/
> >>
> >> I see the following output from the Undertow process:
> >>
> >>      Server started on port 8080
> >>
> >>      1
> >>      2
> >>      3
> >>      4
> >>
> >> I believe this demonstrates that only 4 requests are ever processed in
> >> parallel.  I would expect 8.  In fact, when I run the same experiment on
> >> Windows I see the expected output of
> >>
> >>      Server started on port 8080
> >>      1
> >>      2
> >>      3
> >>      4
> >>      5
> >>      6
> >>      7
> >>      8
> >>
> >> Any thoughts as to what might explain this behavior?
> >>
> >> Best,
> >>
> >> Matt
> >>
> >> _______________________________________________
> >> undertow-dev mailing list
> >> undertow-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
>
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20180724/d6f3c4d6/attachment.html