[jboss-jira] [JBoss JIRA] (JBJCA-1353) Parallelize flush of pools

[Flavia Rainone (JIRA)]

    [ https://issues.jboss.org/browse/JBJCA-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439300#comment-13439300 ] 

Flavia Rainone commented on JBJCA-1353:
---------------------------------------

I am not sure if there is an issue involved in parallelizing, but I'm investigating and seeing what can be done. [~maeste], please let me know if you see any problem that will prevent us from doing this optimization.

> Parallelize flush of pools
> --------------------------
>
>                 Key: JBJCA-1353
>                 URL: https://issues.jboss.org/browse/JBJCA-1353
>             Project: IronJacamar
>          Issue Type: Enhancement
>            Reporter: Jeff Mesnil
>            Assignee: Flavia Rainone
>
> This is related to WFLY-8492 where WildFly server does not shutdown in a timely fashion. There are several tasks to address WFLY-8492 and one of them is related to IronJacamar.
> In the thread dumps done when before we kill -9 the server because it did not shutdown in the expected time, we have a thread:
> {code}
> "ServerService Thread Pool -- 82" #322 prio=5 os_prio=31 tid=0x00007fcad286b800 nid=0x13007 waiting on condition [0x000070000c462000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	- parking to wait for  <0x0000000791e1cf20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
> 	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:375)
> 	- locked <0x0000000791e1cf60> (a java.lang.Object)
> 	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:315)
> 	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sessionStop(ActiveMQSessionContext.java:343)
> 	at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:662)
> 	at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:651)
> 	at org.apache.activemq.artemis.jms.client.ActiveMQSession.stop(ActiveMQSession.java:1023)
> 	at org.apache.activemq.artemis.jms.client.ActiveMQConnection.stop(ActiveMQConnection.java:321)
> 	- locked <0x0000000791e1c360> (a org.apache.activemq.artemis.jms.client.ActiveMQXAConnection)
> 	at org.apache.activemq.artemis.ra.ActiveMQRAManagedConnection.destroyHandles(ActiveMQRAManagedConnection.java:229)
> 	at org.apache.activemq.artemis.ra.ActiveMQRAManagedConnection.destroy(ActiveMQRAManagedConnection.java:268)
> 	at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.doDestroy(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1369)
> 	at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.flush(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:882)
> 	at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.shutdown(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1065)
> 	at org.jboss.jca.core.connectionmanager.pool.AbstractPool.shutdown(AbstractPool.java:930)
> 	- locked <0x0000000790f758d0> (a org.jboss.jca.core.connectionmanager.pool.strategy.PoolByCri)
> 	at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.shutdown(AbstractConnectionManager.java:286)
> 	- locked <0x0000000790f75868> (a org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl)
> 	at org.jboss.as.connector.services.resourceadapters.deployment.AbstractResourceAdapterDeploymentService.unregisterAll(AbstractResourceAdapterDeploymentService.java:199)
> 	at org.jboss.as.connector.services.resourceadapters.deployment.AbstractResourceAdapterDeploymentService$3.run(AbstractResourceAdapterDeploymentService.java:353)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> 	at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> {code}
> When the server is asked to shutdown, IJ will flush its pools. In turn, this calls Artemis RA managed connection destroy() which will stop any underlying JMS connection.
> The test is using a remote JMS that has been stopped and thus the underlying JMS connection will be blocked for some time.
> I had a look at SemaphoreConcurrentLinkedDequeManagedConnectionPool#flush[1] and it is destroying the pool's connections sequentially.
> This implies that the time to flush the pool can be potentially up to n * t where n is the number of connections managed by the pool and t is the timeout that each connections used when it is stopped.
> In the test run, n = 20, t = 30s so we have to wait 10 minutes to the the pool completely flushed.
> Would it be possible to flush the pool concurrently, so that the time to flush the pools would be close to t?
> Something like:
> {code}
> destroy.parallelStream().forEach(clw -> doDestroy(clw));
> {code}
> With such a code, the time to flush the pool would be close to t.
> In our test run, that would reduce the time to wait from 10 minutes to circa 30s.
> Would there be any issue with parallelizing the flush of the pool in such way?
> [1] https://github.com/ironjacamar/ironjacamar/blob/1.4/core/src/main/java/org/jboss/jca/core/connectionmanager/pool/mcp/SemaphoreConcurrentLinkedDequeManagedConnectionPool.java#L872-L884



--
This message was sent by Atlassian JIRA
(v7.2.3#72005)