4:39 a.m.
Hello,
We are trying to build an API gateway using undertow as a servlet extension, this has been
working great until we started our performance tests.
Our setup is two gateways balancing traffic towards 10+ servers and we are using JBoss EAP
7.0 We have observed that some of our requests fail with 503 response. When we get a 503
response from a request, we have also observed that it never reaches the proxy target.
I made a small test on my machine to see if I could get the same error, to my surprise
this was very easy. First I tried with the following code.
public class ApiGatewayServletExtension implements ServletExtension {
@Override
public void handleDeployment(DeploymentInfo deploymentInfo, ServletContext
servletContext) {
LOG.info("Deploying {}...",
MethodHandles.lookup().lookupClass().getSimpleName());
deploymentInfo.addInitialHandlerChainWrapper(handler -> {
try {
LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
.addHost(new URI("http://localhost:8282"))
.setConnectionsPerThread(20);
return new ProxyHandler(loadBalancer, 30000, handler);
} catch (URISyntaxException ex) {
throw new IllegalStateException(ex);
}
});
}
}
I used JMeter with 20 threads making a simple rest GET request and after 30 seconds or so
the test stopped with a 503 response.
Then I created a small java application using undertow 2.0.17.Final like this.
public static void main(final String[] args) {
try {
LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
.addHost(new URI("http://localhost:8282"))
.setConnectionsPerThread(20);
ProxyHandler proxyHandler = new ProxyHandler(loadBalancer, 30000,
ResponseCodeHandler.HANDLE_404);
Undertow server = Undertow.builder()
.addHttpListener(8080, "localhost")
.setHandler(proxyHandler).build();
server.start();
} catch (Exception ex) {
System.out.println(ex.getMessage());
}
}
Getting the same result. I have also tried just loading the home page of one of our
servers and still the same result.
Any help would be very much appreciated. I have not created a bug because I'm pretty
sure we are just missing something here.
Regards Joakim Eriksson.
8:16 a.m.
When the connections-per-thread limit is reached, requests result in 503 responses. Is the
problem resolved by increasing the connections-per-thread value to 50 or 100?
On Wed, Feb 13, 2019, at 04:42, Jocke Eriksson wrote:
Hello,
We are trying to build an API gateway using undertow as a servlet extension, this has
been working great until we started our performance tests.
Our setup is two gateways balancing traffic towards 10+ servers and we are using JBoss
EAP 7.0 We have observed that some of our requests fail with 503 response. When we get a
503 response from a request, we have also observed that it never reaches the proxy
target.
I made a small test on my machine to see if I could get the same error, to my surprise
this was very easy. First I tried with the following code.
public class ApiGatewayServletExtension implements ServletExtension {
@Override
public void handleDeployment(DeploymentInfo deploymentInfo, ServletContext
servletContext) {
LOG.info("Deploying {}...",
MethodHandles.lookup().lookupClass().getSimpleName());
deploymentInfo.addInitialHandlerChainWrapper(handler -> {
try {
LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
.addHost(new URI("http://localhost:8282"))
.setConnectionsPerThread(20);
return new ProxyHandler(loadBalancer, 30000, handler);
} catch (URISyntaxException ex) {
throw new IllegalStateException(ex);
}
});
}
}
I used JMeter with 20 threads making a simple rest GET request and after 30 seconds or so
the test stopped with a 503 response.
Then I created a small java application using undertow 2.0.17.Final like this.
public static void main(final String[] args) {
try {
LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
.addHost(new URI("http://localhost:8282"))
.setConnectionsPerThread(20);
ProxyHandler proxyHandler = new ProxyHandler(loadBalancer, 30000,
ResponseCodeHandler.HANDLE_404);
Undertow server = Undertow.builder()
.addHttpListener(8080, "localhost")
.setHandler(proxyHandler).build();
server.start();
} catch (Exception ex) {
System.out.println(ex.getMessage());
}
}
Getting the same result. I have also tried just loading the home page of one of our
servers and still the same result.
Any help would be very much appreciated. I have not created a bug because I'm pretty
sure we are just missing something here.
Regards Joakim Eriksson.
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
10:11 p.m.
If there is only 20 clients then in theory this should not happen, however
to some extent this is inherently racey. What can happen is that there is a
slight delay between the client receiving the full response and the
connection being returned to the pool (there is nothing that can be done
about this, it is impossible to atomically send the last content and return
the thread in one action).
I would try calling setMaxQueueSize and add a small queue (say 20) which
should resolve the issue. If not then this is definitely a bug.
Stuart
On Thu, Feb 14, 2019 at 1:53 AM Carter Kozak <ckozak(a)ckozak.net> wrote:
When the connections-per-thread limit is reached, requests result in
503
responses. Is the problem resolved by increasing the connections-per-thread
value to 50 or 100?
On Wed, Feb 13, 2019, at 04:42, Jocke Eriksson wrote:
Hello,
We are trying to build an API gateway using undertow as a servlet
extension, this has been working great until we started our performance
tests.
Our setup is two gateways balancing traffic towards 10+ servers and we are
using JBoss EAP 7.0 We have observed that some of our requests fail with
503 response. When we get a 503 response from a request, we have also
observed that it never reaches the proxy target.
I made a small test on my machine to see if I could get the same error, to
my surprise this was very easy. First I tried with the following code.
public class ApiGatewayServletExtension implements ServletExtension {
@Override
public void handleDeployment(DeploymentInfo deploymentInfo,
ServletContext servletContext) {
LOG.info("Deploying {}...",
MethodHandles.lookup().lookupClass().getSimpleName());
deploymentInfo.addInitialHandlerChainWrapper(handler -> {
try {
LoadBalancingProxyClient loadBalancer = new
LoadBalancingProxyClient()
.addHost(new URI("http://localhost:8282"))
.setConnectionsPerThread(20);
return new ProxyHandler(loadBalancer, 30000, handler);
} catch (URISyntaxException ex) {
throw new IllegalStateException(ex);
}
});
}
}
I used JMeter with 20 threads making a simple rest GET request and after
30 seconds or so the test stopped with a 503 response.
Then I created a small java application using undertow 2.0.17.Final like
this.
public static void main(final String[] args) {
try {
LoadBalancingProxyClient loadBalancer = new
LoadBalancingProxyClient()
.addHost(new URI("http://localhost:8282"))
.setConnectionsPerThread(20);
ProxyHandler proxyHandler = new ProxyHandler(loadBalancer,
30000, ResponseCodeHandler.HANDLE_404);
Undertow server = Undertow.builder()
.addHttpListener(8080, "localhost")
.setHandler(proxyHandler).build();
server.start();
} catch (Exception ex) {
System.out.println(ex.getMessage());
}
}
Getting the same result. I have also tried just loading the home page of
one of our servers and still the same result.
Any help would be very much appreciated. I have not created a bug because
I'm pretty sure we are just missing something here.
Regards Joakim Eriksson.
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
1:59 a.m.
I will try this and get back to you after our tests.
Ragrads Joakim.
________________________________
Fillrån: undertow-dev-bounces(a)lists.jboss.org <undertow-dev-bounces(a)lists.jboss.org>
för Stuart Douglas <sdouglas(a)redhat.com>
Skickat: den 14 februari 2019 04:11
Till: Carter Kozak
Kopia: Undertow Developers
Ämne: Re: [undertow-dev] ProxyHandler returns 503 under load
If there is only 20 clients then in theory this should not happen, however to some extent
this is inherently racey. What can happen is that there is a slight delay between the
client receiving the full response and the connection being returned to the pool (there is
nothing that can be done about this, it is impossible to atomically send the last content
and return the thread in one action).
I would try calling setMaxQueueSize and add a small queue (say 20) which should resolve
the issue. If not then this is definitely a bug.
Stuart
On Thu, Feb 14, 2019 at 1:53 AM Carter Kozak
<ckozak@ckozak.net<mailto:ckozak@ckozak.net>> wrote:
When the connections-per-thread limit is reached, requests result in 503 responses. Is the
problem resolved by increasing the connections-per-thread value to 50 or 100?
On Wed, Feb 13, 2019, at 04:42, Jocke Eriksson wrote:
Hello,
We are trying to build an API gateway using undertow as a servlet extension, this has been
working great until we started our performance tests.
Our setup is two gateways balancing traffic towards 10+ servers and we are using JBoss EAP
7.0 We have observed that some of our requests fail with 503 response. When we get a 503
response from a request, we have also observed that it never reaches the proxy target.
I made a small test on my machine to see if I could get the same error, to my surprise
this was very easy. First I tried with the following code.
public class ApiGatewayServletExtension implements ServletExtension {
@Override
public void handleDeployment(DeploymentInfo deploymentInfo, ServletContext
servletContext) {
LOG.info("Deploying {}...",
MethodHandles.lookup().lookupClass().getSimpleName());
deploymentInfo.addInitialHandlerChainWrapper(handler -> {
try {
LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
.addHost(new URI("http://localhost:8282"))
.setConnectionsPerThread(20);
return new ProxyHandler(loadBalancer, 30000, handler);
} catch (URISyntaxException ex) {
throw new IllegalStateException(ex);
}
});
}
}
I used JMeter with 20 threads making a simple rest GET request and after 30 seconds or so
the test stopped with a 503 response.
Then I created a small java application using undertow 2.0.17.Final like this.
public static void main(final String[] args) {
try {
LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
.addHost(new URI("http://localhost:8282"))
.setConnectionsPerThread(20);
ProxyHandler proxyHandler = new ProxyHandler(loadBalancer, 30000,
ResponseCodeHandler.HANDLE_404);
Undertow server = Undertow.builder()
.addHttpListener(8080, "localhost")
.setHandler(proxyHandler).build();
server.start();
} catch (Exception ex) {
System.out.println(ex.getMessage());
}
}
Getting the same result. I have also tried just loading the home page of one of our
servers and still the same result.
Any help would be very much appreciated. I have not created a bug because I'm pretty
sure we are just missing something here.
Regards Joakim Eriksson.
_______________________________________________
undertow-dev mailing list
undertow-dev@lists.jboss.org<mailto:undertow-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/undertow-dev
_______________________________________________
undertow-dev mailing list
undertow-dev@lists.jboss.org<mailto:undertow-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/undertow-dev
5:52 a.m.
After adding more threads and using setMaxQueueSize, we are down to a handful of 503
responses. We will continue with some tunings
to see if we can get down to zero.
Thanks for the help Joakim.
________________________________
Från: undertow-dev-bounces(a)lists.jboss.org <undertow-dev-bounces(a)lists.jboss.org>
för Stuart Douglas <sdouglas(a)redhat.com>
Skickat: den 14 februari 2019 04:11
Till: Carter Kozak
Kopia: Undertow Developers
Ämne: Re: [undertow-dev] ProxyHandler returns 503 under load
If there is only 20 clients then in theory this should not happen, however to some extent
this is inherently racey. What can happen is that there is a slight delay between the
client receiving the full response and the connection being returned to the pool (there is
nothing that can be done about this, it is impossible to atomically send the last content
and return the thread in one action).
I would try calling setMaxQueueSize and add a small queue (say 20) which should resolve
the issue. If not then this is definitely a bug.
Stuart
On Thu, Feb 14, 2019 at 1:53 AM Carter Kozak
<ckozak@ckozak.net<mailto:ckozak@ckozak.net>> wrote:
When the connections-per-thread limit is reached, requests result in 503 responses. Is the
problem resolved by increasing the connections-per-thread value to 50 or 100?
On Wed, Feb 13, 2019, at 04:42, Jocke Eriksson wrote:
Hello,
We are trying to build an API gateway using undertow as a servlet extension, this has been
working great until we started our performance tests.
Our setup is two gateways balancing traffic towards 10+ servers and we are using JBoss EAP
7.0 We have observed that some of our requests fail with 503 response. When we get a 503
response from a request, we have also observed that it never reaches the proxy target.
I made a small test on my machine to see if I could get the same error, to my surprise
this was very easy. First I tried with the following code.
public class ApiGatewayServletExtension implements ServletExtension {
@Override
public void handleDeployment(DeploymentInfo deploymentInfo, ServletContext
servletContext) {
LOG.info("Deploying {}...",
MethodHandles.lookup().lookupClass().getSimpleName());
deploymentInfo.addInitialHandlerChainWrapper(handler -> {
try {
LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
.addHost(new URI("http://localhost:8282"))
.setConnectionsPerThread(20);
return new ProxyHandler(loadBalancer, 30000, handler);
} catch (URISyntaxException ex) {
throw new IllegalStateException(ex);
}
});
}
}
I used JMeter with 20 threads making a simple rest GET request and after 30 seconds or so
the test stopped with a 503 response.
Then I created a small java application using undertow 2.0.17.Final like this.
public static void main(final String[] args) {
try {
LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
.addHost(new URI("http://localhost:8282"))
.setConnectionsPerThread(20);
ProxyHandler proxyHandler = new ProxyHandler(loadBalancer, 30000,
ResponseCodeHandler.HANDLE_404);
Undertow server = Undertow.builder()
.addHttpListener(8080, "localhost")
.setHandler(proxyHandler).build();
server.start();
} catch (Exception ex) {
System.out.println(ex.getMessage());
}
}
Getting the same result. I have also tried just loading the home page of one of our
servers and still the same result.
Any help would be very much appreciated. I have not created a bug because I'm pretty
sure we are just missing something here.
Regards Joakim Eriksson.
_______________________________________________
undertow-dev mailing list
undertow-dev@lists.jboss.org<mailto:undertow-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/undertow-dev
_______________________________________________
undertow-dev mailing list
undertow-dev@lists.jboss.org<mailto:undertow-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/undertow-dev
1:58 a.m.
We made it work better by changing the worker configuration in JBoss like this.
<worker name="default-undertow" io-threads="256"
task-max-threads="2048" />
<http-listener name="default" socket-binding="http"
redirect-socket="https" worker="default-undertow" />
This looks like it could be a bit too much, but we can finetune that later.
We do not use mod_cluster, do you recommend us to setConnectionsPerThread on the proxy
client?
Thanks for the help, Joakim.
________________________________
Från: undertow-dev-bounces(a)lists.jboss.org <undertow-dev-bounces(a)lists.jboss.org>
för Carter Kozak <ckozak(a)ckozak.net>
Skickat: den 13 februari 2019 14:16
Till: undertow-dev(a)lists.jboss.org
Ämne: Re: [undertow-dev] ProxyHandler returns 503 under load
When the connections-per-thread limit is reached, requests result in 503 responses. Is the
problem resolved by increasing the connections-per-thread value to 50 or 100?
On Wed, Feb 13, 2019, at 04:42, Jocke Eriksson wrote:
Hello,
We are trying to build an API gateway using undertow as a servlet extension, this has been
working great until we started our performance tests.
Our setup is two gateways balancing traffic towards 10+ servers and we are using JBoss EAP
7.0 We have observed that some of our requests fail with 503 response. When we get a 503
response from a request, we have also observed that it never reaches the proxy target.
I made a small test on my machine to see if I could get the same error, to my surprise
this was very easy. First I tried with the following code.
public class ApiGatewayServletExtension implements ServletExtension {
@Override
public void handleDeployment(DeploymentInfo deploymentInfo, ServletContext
servletContext) {
LOG.info("Deploying {}...",
MethodHandles.lookup().lookupClass().getSimpleName());
deploymentInfo.addInitialHandlerChainWrapper(handler -> {
try {
LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
.addHost(new URI("http://localhost:8282"))
.setConnectionsPerThread(20);
return new ProxyHandler(loadBalancer, 30000, handler);
} catch (URISyntaxException ex) {
throw new IllegalStateException(ex);
}
});
}
}
I used JMeter with 20 threads making a simple rest GET request and after 30 seconds or so
the test stopped with a 503 response.
Then I created a small java application using undertow 2.0.17.Final like this.
public static void main(final String[] args) {
try {
LoadBalancingProxyClient loadBalancer = new LoadBalancingProxyClient()
.addHost(new URI("http://localhost:8282"))
.setConnectionsPerThread(20);
ProxyHandler proxyHandler = new ProxyHandler(loadBalancer, 30000,
ResponseCodeHandler.HANDLE_404);
Undertow server = Undertow.builder()
.addHttpListener(8080, "localhost")
.setHandler(proxyHandler).build();
server.start();
} catch (Exception ex) {
System.out.println(ex.getMessage());
}
}
Getting the same result. I have also tried just loading the home page of one of our
servers and still the same result.
Any help would be very much appreciated. I have not created a bug because I'm pretty
sure we are just missing something here.
Regards Joakim Eriksson.
_______________________________________________
undertow-dev mailing list
undertow-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/undertow-dev
1895
days inactive
1897
days old
5 comments
3 participants
participants (3)
-
Carter Kozak -
Jocke Eriksson -
Stuart Douglas