[undertow-dev] ProxyHandler returns 503 under load

[Jocke Eriksson]

I will try this and get back to you after our tests.

Ragrads Joakim.
________________________________
Fillrån: undertow-dev-bounces at lists.jboss.org <undertow-dev-bounces at lists.jboss.org> för Stuart Douglas <sdouglas at redhat.com>
Skickat: den 14 februari 2019 04:11
Till: Carter Kozak
Kopia: Undertow Developers
Ämne: Re: [undertow-dev] ProxyHandler returns 503 under load

If there is only 20 clients then in theory this should not happen, however to some extent this is inherently racey. What can happen is that there is a slight delay between the client receiving the full response and the connection being returned to the pool (there is nothing that can be done about this, it is impossible to atomically send the last content and return the thread in one action).

I would try calling setMaxQueueSize and add a small queue (say 20) which should resolve the issue. If not then this is definitely a bug.

Stuart

On Thu, Feb 14, 2019 at 1:53 AM Carter Kozak <ckozak at ckozak.net<mailto:ckozak at ckozak.net>> wrote:
When the connections-per-thread limit is reached, requests result in 503 responses. Is the problem resolved by increasing the connections-per-thread value to 50 or 100?

On Wed, Feb 13, 2019, at 04:42, Jocke Eriksson wrote:
Hello,

We are trying to build an API gateway using undertow as a servlet extension, this has been working great until we started our performance tests.
Our setup is two gateways balancing traffic towards 10+ servers and we are using JBoss EAP 7.0 We have observed that some of our requests fail with 503 response. When we get a 503 response from a request, we have also observed that it never reaches the proxy target.
I made a small test on my machine to see if I could get the same error, to my surprise this was very easy. First I tried with the following code.

public class ApiGatewayServletExtension implements ServletExtension {

    @Override
    public void handleDeployment(DeploymentInfo deploymentInfo, ServletContext servletContext) {
        LOG.info("Deploying {}...", MethodHandles.lookup().lookupClass().getSimpleName());
        deploymentInfo.addInitialHandlerChainWrapper(handler -> {
            try {
                LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
                        .addHost(new URI("http://localhost:8282"))
                        .setConnectionsPerThread(20);
                return new ProxyHandler(loadBalancer, 30000, handler);
            } catch (URISyntaxException ex) {
                throw new IllegalStateException(ex);
            }
        });
    }
}

I used JMeter with 20 threads making a simple rest GET request and after 30 seconds or so the test stopped with a 503 response.

Then I created a small java application using undertow 2.0.17.Final like this.

    public static void main(final String[] args) {

        try {
            LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
                    .addHost(new URI("http://localhost:8282"))
                    .setConnectionsPerThread(20);
            ProxyHandler proxyHandler = new ProxyHandler(loadBalancer, 30000, ResponseCodeHandler.HANDLE_404);

            Undertow server = Undertow.builder()
                    .addHttpListener(8080, "localhost")
                    .setHandler(proxyHandler).build();

            server.start();

        } catch (Exception ex) {
            System.out.println(ex.getMessage());
        }
    }

Getting the same result. I have also tried just loading the home page of one of our servers and still the same result.

Any help would be very much appreciated. I have not created a bug because I'm pretty sure we are just missing something here.

Regards Joakim Eriksson.







_______________________________________________
undertow-dev mailing list
undertow-dev at lists.jboss.org<mailto:undertow-dev at lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/undertow-dev

_______________________________________________
undertow-dev mailing list
undertow-dev at lists.jboss.org<mailto:undertow-dev at lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/undertow-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/undertow-dev/attachments/20190214/f36b798e/attachment.html