[overlord-issues] [JBoss JIRA] (RTGOV-674) ElasticsearchRESTServer proxy blocks on Error

ElasticsearchRESTServer proxy blocks on Error --------------------------------------------- Key: RTGOV-674 URL: https://issues.jboss.org/browse/RTGOV-674 Project: RTGov (Run Time Governance) Issue Type: Bug Components: User Interface Affects Versions: 2.0.0.Final Reporter: Sascha Dirbach Assignee: Gary Brown Fix For: 2.2.0.Alpha3 Hi, it looks like the InputStream of the proxyResponse is not closed correctly when there is an exception in org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(HttpResponse, HttpServletResponse) This results in a blocking behaviour, as the internally used HttpClient uses a ConnectionPool with the size of 2 connections per host, which will not be released in the above case. We experienced that with a SocketClosedException like this: ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301) at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104) at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137) at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61) After two of these Exceptions the elasticsearch proxy did not react anymore until a server restart. A stacktrace of the running server revealed, that the threads are blocked in the connection pool: sun.misc.Unsafe.park() at java.util.concurrent.locks.LockSupport.park() at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() at org.apache.http.pool.PoolEntryFuture.await() at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking() at org.apache.http.pool.AbstractConnPool.access$000AbstractConnPool() at org.apache.http.pool.AbstractConnPool$2.getPoolEntry() at org.apache.http.pool.AbstractConnPool$2.getPoolEntry() at org.apache.http.pool.PoolEntryFuture.get() at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection() at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection() at org.apache.http.impl.client.DefaultRequestDirector.execute() at org.apache.http.impl.client.AbstractHttpClient.execute() at org.apache.http.impl.client.AbstractHttpClient.execute() at org.overlord.rtgov.elasticsearch.rest.ElasticsearchHttpClient.process() at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service() at javax.servlet.http.HttpServlet.service) Sascha