[undertow-dev] ProxyHandler returns 503 under load

[Jocke Eriksson]

We made it work better by changing the worker configuration in JBoss like this.

<worker name="default-undertow" io-threads="256" task-max-threads="2048" />

<http-listener name="default" socket-binding="http" redirect-socket="https" worker="default-undertow" />

This looks like it could be a bit too much, but we can finetune that later.

We do not use mod_cluster, do you recommend us to setConnectionsPerThread on the proxy client?

Thanks for the help, Joakim.

________________________________
Från: undertow-dev-bounces at lists.jboss.org <undertow-dev-bounces at lists.jboss.org> för Carter Kozak <ckozak at ckozak.net>
Skickat: den 13 februari 2019 14:16
Till: undertow-dev at lists.jboss.org
Ämne: Re: [undertow-dev] ProxyHandler returns 503 under load

When the connections-per-thread limit is reached, requests result in 503 responses. Is the problem resolved by increasing the connections-per-thread value to 50 or 100?

On Wed, Feb 13, 2019, at 04:42, Jocke Eriksson wrote:
Hello,

We are trying to build an API gateway using undertow as a servlet extension, this has been working great until we started our performance tests.
Our setup is two gateways balancing traffic towards 10+ servers and we are using JBoss EAP 7.0 We have observed that some of our requests fail with 503 response. When we get a 503 response from a request, we have also observed that it never reaches the proxy target.
I made a small test on my machine to see if I could get the same error, to my surprise this was very easy. First I tried with the following code.

public class ApiGatewayServletExtension implements ServletExtension {

    @Override
    public void handleDeployment(DeploymentInfo deploymentInfo, ServletContext servletContext) {
        LOG.info("Deploying {}...", MethodHandles.lookup().lookupClass().getSimpleName());
        deploymentInfo.addInitialHandlerChainWrapper(handler -> {
            try {
                LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
                        .addHost(new URI("http://localhost:8282"))
                        .setConnectionsPerThread(20);
                return new ProxyHandler(loadBalancer, 30000, handler);
            } catch (URISyntaxException ex) {
                throw new IllegalStateException(ex);
            }
        });
    }
}

I used JMeter with 20 threads making a simple rest GET request and after 30 seconds or so the test stopped with a 503 response.

Then I created a small java application using undertow 2.0.17.Final like this.

    public static void main(final String[] args) {

        try {
            LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
                    .addHost(new URI("http://localhost:8282"))
                    .setConnectionsPerThread(20);
            ProxyHandler proxyHandler = new ProxyHandler(loadBalancer, 30000, ResponseCodeHandler.HANDLE_404);

            Undertow server = Undertow.builder()
                    .addHttpListener(8080, "localhost")
                    .setHandler(proxyHandler).build();

            server.start();

        } catch (Exception ex) {
            System.out.println(ex.getMessage());
        }
    }

Getting the same result. I have also tried just loading the home page of one of our servers and still the same result.

Any help would be very much appreciated. I have not created a bug because I'm pretty sure we are just missing something here.

Regards Joakim Eriksson.







_______________________________________________
undertow-dev mailing list
undertow-dev at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20190214/d92df2d4/attachment-0001.html