[undertow-dev] ProxyHandler returns 503 under load

[Stuart Douglas]

If there is only 20 clients then in theory this should not happen, however
to some extent this is inherently racey. What can happen is that there is a
slight delay between the client receiving the full response and the
connection being returned to the pool (there is nothing that can be done
about this, it is impossible to atomically send the last content and return
the thread in one action).

I would try calling setMaxQueueSize and add a small queue (say 20) which
should resolve the issue. If not then this is definitely a bug.

Stuart

On Thu, Feb 14, 2019 at 1:53 AM Carter Kozak <ckozak at ckozak.net> wrote:

> When the connections-per-thread limit is reached, requests result in 503
> responses. Is the problem resolved by increasing the connections-per-thread
> value to 50 or 100?
>
> On Wed, Feb 13, 2019, at 04:42, Jocke Eriksson wrote:
>
> Hello,
>
> We are trying to build an API gateway using undertow as a servlet
> extension, this has been working great until we started our performance
> tests.
> Our setup is two gateways balancing traffic towards 10+ servers and we are
> using JBoss EAP 7.0 We have observed that some of our requests fail with
> 503 response. When we get a 503 response from a request, we have also
> observed that it never reaches the proxy target.
> I made a small test on my machine to see if I could get the same error, to
> my surprise this was very easy. First I tried with the following code.
>
> public class ApiGatewayServletExtension implements ServletExtension {
>
>     @Override
>     public void handleDeployment(DeploymentInfo deploymentInfo,
> ServletContext servletContext) {
>         LOG.info("Deploying {}...",
> MethodHandles.lookup().lookupClass().getSimpleName());
>         deploymentInfo.addInitialHandlerChainWrapper(handler -> {
>             try {
>                 LoadBalancingProxyClient loadBalancer = new
> LoadBalancingProxyClient()
>                         .addHost(new URI("http://localhost:8282"))
>                         .setConnectionsPerThread(20);
>                 return new ProxyHandler(loadBalancer, 30000, handler);
>             } catch (URISyntaxException ex) {
>                 throw new IllegalStateException(ex);
>             }
>         });
>     }
> }
>
> I used JMeter with 20 threads making a simple rest GET request and after
> 30 seconds or so the test stopped with a 503 response.
>
> Then I created a small java application using undertow 2.0.17.Final like
> this.
>
>     public static void main(final String[] args) {
>
>         try {
>             LoadBalancingProxyClient loadBalancer = new
> LoadBalancingProxyClient()
>                     .addHost(new URI("http://localhost:8282"))
>                     .setConnectionsPerThread(20);
>             ProxyHandler proxyHandler = new ProxyHandler(loadBalancer,
> 30000, ResponseCodeHandler.HANDLE_404);
>
>             Undertow server = Undertow.builder()
>                     .addHttpListener(8080, "localhost")
>                     .setHandler(proxyHandler).build();
>
>             server.start();
>
>         } catch (Exception ex) {
>             System.out.println(ex.getMessage());
>         }
>     }
>
> Getting the same result. I have also tried just loading the home page of
> one of our servers and still the same result.
>
> Any help would be very much appreciated. I have not created a bug because
> I'm pretty sure we are just missing something here.
>
> Regards Joakim Eriksson.
>
>
>
>
>
>
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
>
>
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20190214/51c3f29c/attachment.html