From issues at jboss.org Tue Feb 9 08:39:00 2016 From: issues at jboss.org (Sascha Dirbach (JIRA)) Date: Tue, 9 Feb 2016 08:39:00 -0500 (EST) Subject: [overlord-issues] [JBoss JIRA] (RTGOV-674) ElasticsearchRESTServer proxy blocks on Error In-Reply-To: References: Message-ID: Sascha Dirbach created RTGOV-674: ------------------------------------ Summary: ElasticsearchRESTServer proxy blocks on Error Key: RTGOV-674 URL: https://issues.jboss.org/browse/RTGOV-674 Project: RTGov (Run Time Governance) Issue Type: Bug Components: User Interface Affects Versions: 2.0.0.Final Reporter: Sascha Dirbach Assignee: Gary Brown Hi, it looks like the InputStream of the proxyResponse is not closed correctly when there is an exception in org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(HttpResponse, HttpServletResponse) This results in a blocking behaviour, as the internally used HttpClient uses a ConnectionPool with the size of 2 connections per host, which will not be released in the above case. We experienced that with a SocketClosedException like this: ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301) at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104) at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137) at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61) After two of these Exceptions the elasticsearch proxy did not react anymore until a server restart. A stacktrace of the running server revealed, that the threads are blocked in the connection pool: ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301) at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104) at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137) at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61) Sascha -- This message was sent by Atlassian JIRA (v6.4.11#64026) From issues at jboss.org Tue Feb 9 08:45:00 2016 From: issues at jboss.org (Sascha Dirbach (JIRA)) Date: Tue, 9 Feb 2016 08:45:00 -0500 (EST) Subject: [overlord-issues] [JBoss JIRA] (RTGOV-674) ElasticsearchRESTServer proxy blocks on Error In-Reply-To: References: Message-ID: [ https://issues.jboss.org/browse/RTGOV-674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sascha Dirbach updated RTGOV-674: --------------------------------- Git Pull Request: https://github.com/Governance/rtgov/pull/338 > ElasticsearchRESTServer proxy blocks on Error > --------------------------------------------- > > Key: RTGOV-674 > URL: https://issues.jboss.org/browse/RTGOV-674 > Project: RTGov (Run Time Governance) > Issue Type: Bug > Components: User Interface > Affects Versions: 2.0.0.Final > Reporter: Sascha Dirbach > Assignee: Gary Brown > > Hi, > it looks like the InputStream of the proxyResponse is not closed correctly when there is an exception in org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(HttpResponse, HttpServletResponse) > This results in a blocking behaviour, as the internally used HttpClient uses a ConnectionPool with the size of 2 connections per host, which will not be released in the above case. > We experienced that with a SocketClosedException like this: > ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe > at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) > at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) > at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342) > at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301) > at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104) > at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137) > at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61) > After two of these Exceptions the elasticsearch proxy did not react anymore until a server restart. > A stacktrace of the running server revealed, that the threads are blocked in the connection pool: > ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe > at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) > at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) > at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342) > at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301) > at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104) > at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137) > at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61) > Sascha -- This message was sent by Atlassian JIRA (v6.4.11#64026) From issues at jboss.org Fri Feb 12 04:18:00 2016 From: issues at jboss.org (Sascha Dirbach (JIRA)) Date: Fri, 12 Feb 2016 04:18:00 -0500 (EST) Subject: [overlord-issues] [JBoss JIRA] (RTGOV-674) ElasticsearchRESTServer proxy blocks on Error In-Reply-To: References: Message-ID: [ https://issues.jboss.org/browse/RTGOV-674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sascha Dirbach updated RTGOV-674: --------------------------------- Description: Hi, it looks like the InputStream of the proxyResponse is not closed correctly when there is an exception in org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(HttpResponse, HttpServletResponse) This results in a blocking behaviour, as the internally used HttpClient uses a ConnectionPool with the size of 2 connections per host, which will not be released in the above case. We experienced that with a SocketClosedException like this: ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301) at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104) at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137) at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61) After two of these Exceptions the elasticsearch proxy did not react anymore until a server restart. A stacktrace of the running server revealed, that the threads are blocked in the connection pool: sun.misc.Unsafe.park() at java.util.concurrent.locks.LockSupport.park() at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() at org.apache.http.pool.PoolEntryFuture.await() at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking() at org.apache.http.pool.AbstractConnPool.access$000AbstractConnPool() at org.apache.http.pool.AbstractConnPool$2.getPoolEntry() at org.apache.http.pool.AbstractConnPool$2.getPoolEntry() at org.apache.http.pool.PoolEntryFuture.get() at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection() at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection() at org.apache.http.impl.client.DefaultRequestDirector.execute() at org.apache.http.impl.client.AbstractHttpClient.execute() at org.apache.http.impl.client.AbstractHttpClient.execute() at org.overlord.rtgov.elasticsearch.rest.ElasticsearchHttpClient.process() at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service() at javax.servlet.http.HttpServlet.service) Sascha was: Hi, it looks like the InputStream of the proxyResponse is not closed correctly when there is an exception in org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(HttpResponse, HttpServletResponse) This results in a blocking behaviour, as the internally used HttpClient uses a ConnectionPool with the size of 2 connections per host, which will not be released in the above case. We experienced that with a SocketClosedException like this: ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301) at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104) at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137) at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61) After two of these Exceptions the elasticsearch proxy did not react anymore until a server restart. A stacktrace of the running server revealed, that the threads are blocked in the connection pool: ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342) at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301) at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104) at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137) at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61) Sascha > ElasticsearchRESTServer proxy blocks on Error > --------------------------------------------- > > Key: RTGOV-674 > URL: https://issues.jboss.org/browse/RTGOV-674 > Project: RTGov (Run Time Governance) > Issue Type: Bug > Components: User Interface > Affects Versions: 2.0.0.Final > Reporter: Sascha Dirbach > Assignee: Gary Brown > > Hi, > it looks like the InputStream of the proxyResponse is not closed correctly when there is an exception in org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(HttpResponse, HttpServletResponse) > This results in a blocking behaviour, as the internally used HttpClient uses a ConnectionPool with the size of 2 connections per host, which will not be released in the above case. > We experienced that with a SocketClosedException like this: > ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe > at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) > at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) > at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342) > at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301) > at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104) > at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137) > at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61) > After two of these Exceptions the elasticsearch proxy did not react anymore until a server restart. > A stacktrace of the running server revealed, that the threads are blocked in the connection pool: > sun.misc.Unsafe.park() > at java.util.concurrent.locks.LockSupport.park() > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() > at org.apache.http.pool.PoolEntryFuture.await() > at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking() > at org.apache.http.pool.AbstractConnPool.access$000AbstractConnPool() > at org.apache.http.pool.AbstractConnPool$2.getPoolEntry() > at org.apache.http.pool.AbstractConnPool$2.getPoolEntry() > at org.apache.http.pool.PoolEntryFuture.get() > at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection() > at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection() > at org.apache.http.impl.client.DefaultRequestDirector.execute() > at org.apache.http.impl.client.AbstractHttpClient.execute() > at org.apache.http.impl.client.AbstractHttpClient.execute() > at org.overlord.rtgov.elasticsearch.rest.ElasticsearchHttpClient.process() > at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service() > at javax.servlet.http.HttpServlet.service) > Sascha -- This message was sent by Atlassian JIRA (v6.4.11#64026) From issues at jboss.org Fri Feb 12 11:38:00 2016 From: issues at jboss.org (Gary Brown (JIRA)) Date: Fri, 12 Feb 2016 11:38:00 -0500 (EST) Subject: [overlord-issues] [JBoss JIRA] (RTGOV-674) ElasticsearchRESTServer proxy blocks on Error In-Reply-To: References: Message-ID: [ https://issues.jboss.org/browse/RTGOV-674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Brown resolved RTGOV-674. ------------------------------ Fix Version/s: 2.2.0.Alpha3 Resolution: Done Also applied to product branch. Thanks. > ElasticsearchRESTServer proxy blocks on Error > --------------------------------------------- > > Key: RTGOV-674 > URL: https://issues.jboss.org/browse/RTGOV-674 > Project: RTGov (Run Time Governance) > Issue Type: Bug > Components: User Interface > Affects Versions: 2.0.0.Final > Reporter: Sascha Dirbach > Assignee: Gary Brown > Fix For: 2.2.0.Alpha3 > > > Hi, > it looks like the InputStream of the proxyResponse is not closed correctly when there is an exception in org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(HttpResponse, HttpServletResponse) > This results in a blocking behaviour, as the internally used HttpClient uses a ConnectionPool with the size of 2 connections per host, which will not be released in the above case. > We experienced that with a SocketClosedException like this: > ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/rtgov-ui]] (http-/0.0.0.0:9181-5) ElasticsearchRESTServer: null: ClientAbortException: java.net.SocketException: Broken pipe > at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) > at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) > at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342) > at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:301) > at org.apache.catalina.connector.CoyoteOutputStream.close(CoyoteOutputStream.java:104) > at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.copyResponseContent(ElasticsearchRESTServer.java:137) > at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service(ElasticsearchRESTServer.java:61) > After two of these Exceptions the elasticsearch proxy did not react anymore until a server restart. > A stacktrace of the running server revealed, that the threads are blocked in the connection pool: > sun.misc.Unsafe.park() > at java.util.concurrent.locks.LockSupport.park() > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() > at org.apache.http.pool.PoolEntryFuture.await() > at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking() > at org.apache.http.pool.AbstractConnPool.access$000AbstractConnPool() > at org.apache.http.pool.AbstractConnPool$2.getPoolEntry() > at org.apache.http.pool.AbstractConnPool$2.getPoolEntry() > at org.apache.http.pool.PoolEntryFuture.get() > at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection() > at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection() > at org.apache.http.impl.client.DefaultRequestDirector.execute() > at org.apache.http.impl.client.AbstractHttpClient.execute() > at org.apache.http.impl.client.AbstractHttpClient.execute() > at org.overlord.rtgov.elasticsearch.rest.ElasticsearchHttpClient.process() > at org.overlord.rtgov.elasticsearch.rest.ElasticsearchRESTServer.service() > at javax.servlet.http.HttpServlet.service) > Sascha -- This message was sent by Atlassian JIRA (v6.4.11#64026)