[JBoss JIRA] (RTGOV-674) ElasticsearchRESTServer proxy blocks on Error
by Gary Brown (JIRA)
[ https://issues.jboss.org/browse/RTGOV-674?page=com.atlassian.jira.plugin.... ]
Gary Brown resolved RTGOV-674.
------------------------------
Fix Version/s: 2.2.0.Alpha3
Resolution: Done
Also applied to product branch.
Thanks.
> ElasticsearchRESTServer proxy blocks on Error
> ---------------------------------------------
>
> Key: RTGOV-674
> URL: https://issues.jboss.org/browse/RTGOV-674
> Project: RTGov (Run Time Governance)
> Issue Type: Bug
> Components: User Interface
> Affects Versions: 2.0.0.Final
> Reporter: Sascha Dirbach
> Assignee: Gary Brown
> Fix For: 2.2.0.Alpha3
>
>
> Hi,
> it looks like the InputStream of the proxyResponse is not closed correctly when there is an exception in org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(HttpResponse, HttpServletResponse)
> This results in a blocking behaviour, as the internally used HttpClient uses a ConnectionPool with the size of 2 connections per host, which will not be released in the above case.
> We experienced that with a SocketClosedException like this:
> ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe
> at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403)
> at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450)
> at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
> at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301)
> at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104)
> at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137)
> at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61)
> After two of these Exceptions the elasticsearch proxy did not react anymore until a server restart.
> A stacktrace of the running server revealed, that the threads are blocked in the connection pool:
> sun.misc.Unsafe.park()
> at java.util.concurrent.locks.LockSupport.park()
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> at org.apache.http.pool.PoolEntryFuture.await()
> at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking()
> at org.apache.http.pool.AbstractConnPool.access$000AbstractConnPool()
> at org.apache.http.pool.AbstractConnPool$2.getPoolEntry()
> at org.apache.http.pool.AbstractConnPool$2.getPoolEntry()
> at org.apache.http.pool.PoolEntryFuture.get()
> at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection()
> at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection()
> at org.apache.http.impl.client.DefaultRequestDirector.execute()
> at org.apache.http.impl.client.AbstractHttpClient.execute()
> at org.apache.http.impl.client.AbstractHttpClient.execute()
> at org.overlord.rtgov.elasticsearch.rest.ElasticsearchHttpClient.process()
> at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service()
> at javax.servlet.http.HttpServlet.service)
> Sascha
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 10 months
- 1
- 0
[JBoss JIRA] (RTGOV-674) ElasticsearchRESTServer proxy blocks on Error
by Sascha Dirbach (JIRA)
[ https://issues.jboss.org/browse/RTGOV-674?page=com.atlassian.jira.plugin.... ]
Sascha Dirbach updated RTGOV-674:
---------------------------------
Description:
Hi,
it looks like the InputStream of the proxyResponse is not closed correctly when there is an exception in org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(HttpResponse, HttpServletResponse)
This results in a blocking behaviour, as the internally used HttpClient uses a ConnectionPool with the size of 2 connections per host, which will not be released in the above case.
We experienced that with a SocketClosedException like this:
ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301)
at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104)
at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137)
at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61)
After two of these Exceptions the elasticsearch proxy did not react anymore until a server restart.
A stacktrace of the running server revealed, that the threads are blocked in the connection pool:
sun.misc.Unsafe.park()
at java.util.concurrent.locks.LockSupport.park()
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
at org.apache.http.pool.PoolEntryFuture.await()
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking()
at org.apache.http.pool.AbstractConnPool.access$000AbstractConnPool()
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry()
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry()
at org.apache.http.pool.PoolEntryFuture.get()
at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection()
at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection()
at org.apache.http.impl.client.DefaultRequestDirector.execute()
at org.apache.http.impl.client.AbstractHttpClient.execute()
at org.apache.http.impl.client.AbstractHttpClient.execute()
at org.overlord.rtgov.elasticsearch.rest.ElasticsearchHttpClient.process()
at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service()
at javax.servlet.http.HttpServlet.service)
Sascha
was:
Hi,
it looks like the InputStream of the proxyResponse is not closed correctly when there is an exception in org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(HttpResponse, HttpServletResponse)
This results in a blocking behaviour, as the internally used HttpClient uses a ConnectionPool with the size of 2 connections per host, which will not be released in the above case.
We experienced that with a SocketClosedException like this:
ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301)
at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104)
at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137)
at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61)
After two of these Exceptions the elasticsearch proxy did not react anymore until a server restart.
A stacktrace of the running server revealed, that the threads are blocked in the connection pool:
ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301)
at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104)
at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137)
at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61)
Sascha
> ElasticsearchRESTServer proxy blocks on Error
> ---------------------------------------------
>
> Key: RTGOV-674
> URL: https://issues.jboss.org/browse/RTGOV-674
> Project: RTGov (Run Time Governance)
> Issue Type: Bug
> Components: User Interface
> Affects Versions: 2.0.0.Final
> Reporter: Sascha Dirbach
> Assignee: Gary Brown
>
> Hi,
> it looks like the InputStream of the proxyResponse is not closed correctly when there is an exception in org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(HttpResponse, HttpServletResponse)
> This results in a blocking behaviour, as the internally used HttpClient uses a ConnectionPool with the size of 2 connections per host, which will not be released in the above case.
> We experienced that with a SocketClosedException like this:
> ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe
> at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403)
> at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450)
> at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
> at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301)
> at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104)
> at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137)
> at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61)
> After two of these Exceptions the elasticsearch proxy did not react anymore until a server restart.
> A stacktrace of the running server revealed, that the threads are blocked in the connection pool:
> sun.misc.Unsafe.park()
> at java.util.concurrent.locks.LockSupport.park()
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
> at org.apache.http.pool.PoolEntryFuture.await()
> at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking()
> at org.apache.http.pool.AbstractConnPool.access$000AbstractConnPool()
> at org.apache.http.pool.AbstractConnPool$2.getPoolEntry()
> at org.apache.http.pool.AbstractConnPool$2.getPoolEntry()
> at org.apache.http.pool.PoolEntryFuture.get()
> at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection()
> at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection()
> at org.apache.http.impl.client.DefaultRequestDirector.execute()
> at org.apache.http.impl.client.AbstractHttpClient.execute()
> at org.apache.http.impl.client.AbstractHttpClient.execute()
> at org.overlord.rtgov.elasticsearch.rest.ElasticsearchHttpClient.process()
> at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service()
> at javax.servlet.http.HttpServlet.service)
> Sascha
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 10 months
- 1
- 0
[JBoss JIRA] (RTGOV-674) ElasticsearchRESTServer proxy blocks on Error
by Sascha Dirbach (JIRA)
[ https://issues.jboss.org/browse/RTGOV-674?page=com.atlassian.jira.plugin.... ]
Sascha Dirbach updated RTGOV-674:
---------------------------------
Git Pull Request: https://github.com/Governance/rtgov/pull/338
> ElasticsearchRESTServer proxy blocks on Error
> ---------------------------------------------
>
> Key: RTGOV-674
> URL: https://issues.jboss.org/browse/RTGOV-674
> Project: RTGov (Run Time Governance)
> Issue Type: Bug
> Components: User Interface
> Affects Versions: 2.0.0.Final
> Reporter: Sascha Dirbach
> Assignee: Gary Brown
>
> Hi,
> it looks like the InputStream of the proxyResponse is not closed correctly when there is an exception in org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(HttpResponse, HttpServletResponse)
> This results in a blocking behaviour, as the internally used HttpClient uses a ConnectionPool with the size of 2 connections per host, which will not be released in the above case.
> We experienced that with a SocketClosedException like this:
> ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe
> at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403)
> at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450)
> at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
> at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301)
> at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104)
> at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137)
> at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61)
> After two of these Exceptions the elasticsearch proxy did not react anymore until a server restart.
> A stacktrace of the running server revealed, that the threads are blocked in the connection pool:
> ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe
> at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403)
> at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450)
> at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
> at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301)
> at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104)
> at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137)
> at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61)
> Sascha
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 10 months
- 1
- 0
[JBoss JIRA] (RTGOV-674) ElasticsearchRESTServer proxy blocks on Error
by Sascha Dirbach (JIRA)
Sascha Dirbach created RTGOV-674:
------------------------------------
Summary: ElasticsearchRESTServer proxy blocks on Error
Key: RTGOV-674
URL: https://issues.jboss.org/browse/RTGOV-674
Project: RTGov (Run Time Governance)
Issue Type: Bug
Components: User Interface
Affects Versions: 2.0.0.Final
Reporter: Sascha Dirbach
Assignee: Gary Brown
Hi,
it looks like the InputStream of the proxyResponse is not closed correctly when there is an exception in org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(HttpResponse, HttpServletResponse)
This results in a blocking behaviour, as the internally used HttpClient uses a ConnectionPool with the size of 2 connections per host, which will not be released in the above case.
We experienced that with a SocketClosedException like this:
ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301)
at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104)
at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137)
at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61)
After two of these Exceptions the elasticsearch proxy did not react anymore until a server restart.
A stacktrace of the running server revealed, that the threads are blocked in the connection pool:
ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301)
at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104)
at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137)
at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61)
Sascha
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 10 months
- 1
- 0
- ← Newer
- 1
- Older →