[jbossts-issues] [JBoss JIRA] (JBTM-3092) Performance TxnTest is processed with ServiceUnavailableException

Ondra Chaloupka (Jira) issues at jboss.org
Fri Jan 11 07:05:00 EST 2019 

     [ https://issues.jboss.org/browse/JBTM-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ondra Chaloupka updated JBTM-3092:
----------------------------------
    Description: 
The performance test `org.jboss.narayana.rts.TxnTest` runs with errors `javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable`. It seems the Undertow is not accepting requests for some reason. 

At first I thought it could be an issue of the Undertow that fully stopped the endpoint from some reason. But on generating the Undertow logs (attached) it kind of seems there are some request dropped and some are processed. Maybe there is only need to tune Undertow about dropping requests.

But it seems that even some requests are dropped the test passes.

The log contains errors
{code}
javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.handleErrorStatus(ClientInvocation.java:191)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.extractResult(ClientInvocation.java:154)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:444)
	at org.jboss.narayana.rts.TxnHelper.sendRequest(TxnHelper.java:87)
	at org.jboss.narayana.rts.TxnTest.testTxn(TxnTest.java:66)
	at org.jboss.narayana.rts.generated.TxnTest_testTxn_jmhTest.testTxn_thrpt_jmhStub(TxnTest_testTxn_jmhTest.java:143)
	at org.jboss.narayana.rts.generated.TxnTest_testTxn_jmhTest.testTxn_Throughput(TxnTest_testTxn_jmhTest.java:83)
	at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

javax.ws.rs.ProcessingException: Unable to invoke request
	at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:287)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:407)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.post(ClientInvocationBuilder.java:195)
	at org.jboss.narayana.rts.TxnHelper.beginTxn(TxnHelper.java:50)
	at org.jboss.narayana.rts.TxnTest.testTxn(TxnTest.java:59)
	at org.jboss.narayana.rts.generated.TxnTest_testTxn_jmhTest.testTxn_Throughput(TxnTest_testTxn_jmhTest.java:78)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8090 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MinimalClientExec.execute(MinimalClientExec.java:183)
	at org.apache.http.impl.client.MinimalHttpClient.doExecute(MinimalHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:84)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:283)
	... 16 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	... 22 more
{code}

  was:
The performance test `org.jboss.narayana.rts.TxnTest` runs with errors `javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable`. It seems the Undertow is not accepting requests for some reason. 

At first I thought it could be an issue of the Undertow that fully stopped the endpoint from some reason. But on generating the Undertow logs (attached) it kind of seems there are some request dropped and some are processed. Maybe there is only need to tune Undertow about dropping requests.

But it seems that even some requests are dropped the test passes.

The log contains errors
```
javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.handleErrorStatus(ClientInvocation.java:191)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.extractResult(ClientInvocation.java:154)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:444)
	at org.jboss.narayana.rts.TxnHelper.sendRequest(TxnHelper.java:87)
	at org.jboss.narayana.rts.TxnTest.testTxn(TxnTest.java:66)
	at org.jboss.narayana.rts.generated.TxnTest_testTxn_jmhTest.testTxn_thrpt_jmhStub(TxnTest_testTxn_jmhTest.java:143)
	at org.jboss.narayana.rts.generated.TxnTest_testTxn_jmhTest.testTxn_Throughput(TxnTest_testTxn_jmhTest.java:83)
	at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

javax.ws.rs.ProcessingException: Unable to invoke request
	at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:287)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:407)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.post(ClientInvocationBuilder.java:195)
	at org.jboss.narayana.rts.TxnHelper.beginTxn(TxnHelper.java:50)
	at org.jboss.narayana.rts.TxnTest.testTxn(TxnTest.java:59)
	at org.jboss.narayana.rts.generated.TxnTest_testTxn_jmhTest.testTxn_Throughput(TxnTest_testTxn_jmhTest.java:78)
	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8090 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MinimalClientExec.execute(MinimalClientExec.java:183)
	at org.apache.http.impl.client.MinimalHttpClient.doExecute(MinimalHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:84)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:283)
	... 16 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
	... 22 more
```



> Performance TxnTest is processed with ServiceUnavailableException 
> ------------------------------------------------------------------
>
>                 Key: JBTM-3092
>                 URL: https://issues.jboss.org/browse/JBTM-3092
>             Project: JBoss Transaction Manager
>          Issue Type: Bug
>          Components: Performance Testing
>            Reporter: Ondra Chaloupka
>            Assignee: Michael Musgrove
>            Priority: Major
>         Attachments: perf-failure-rts.log, undertow-log.zip
>
>
> The performance test `org.jboss.narayana.rts.TxnTest` runs with errors `javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable`. It seems the Undertow is not accepting requests for some reason. 
> At first I thought it could be an issue of the Undertow that fully stopped the endpoint from some reason. But on generating the Undertow logs (attached) it kind of seems there are some request dropped and some are processed. Maybe there is only need to tune Undertow about dropping requests.
> But it seems that even some requests are dropped the test passes.
> The log contains errors
> {code}
> javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
> 	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.handleErrorStatus(ClientInvocation.java:191)
> 	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.extractResult(ClientInvocation.java:154)
> 	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:444)
> 	at org.jboss.narayana.rts.TxnHelper.sendRequest(TxnHelper.java:87)
> 	at org.jboss.narayana.rts.TxnTest.testTxn(TxnTest.java:66)
> 	at org.jboss.narayana.rts.generated.TxnTest_testTxn_jmhTest.testTxn_thrpt_jmhStub(TxnTest_testTxn_jmhTest.java:143)
> 	at org.jboss.narayana.rts.generated.TxnTest_testTxn_jmhTest.testTxn_Throughput(TxnTest_testTxn_jmhTest.java:83)
> 	at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
> 	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:748)
> javax.ws.rs.ProcessingException: Unable to invoke request
> 	at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:287)
> 	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:407)
> 	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.post(ClientInvocationBuilder.java:195)
> 	at org.jboss.narayana.rts.TxnHelper.beginTxn(TxnHelper.java:50)
> 	at org.jboss.narayana.rts.TxnTest.testTxn(TxnTest.java:59)
> 	at org.jboss.narayana.rts.generated.TxnTest_testTxn_jmhTest.testTxn_Throughput(TxnTest_testTxn_jmhTest.java:78)
> 	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453)
> 	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8090 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
> 	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
> 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
> 	at org.apache.http.impl.execchain.MinimalClientExec.execute(MinimalClientExec.java:183)
> 	at org.apache.http.impl.client.MinimalHttpClient.doExecute(MinimalHttpClient.java:107)
> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:84)
> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
> 	at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:283)
> 	... 16 more
> Caused by: java.net.ConnectException: Connection refused (Connection refused)
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	at java.net.Socket.connect(Socket.java:589)
> 	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
> 	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
> 	... 22 more
> {code}



--
This message was sent by Atlassian Jira
(v7.12.1#712002)

More information about the jbossts-issues mailing list