Parallelize flush of pools -------------------------- Key: JBJCA-1353 URL: https://issues.jboss.org/browse/JBJCA-1353 Project: IronJacamar Issue Type: Enhancement Reporter: Jeff Mesnil Assignee: Flavia Rainone This is related to WFLY-8492 where WildFly server does not shutdown in a timely fashion. There are several tasks to address WFLY-8492 and one of them is related to IronJacamar. In the thread dumps done when before we kill -9 the server because it did not shutdown in the expected time, we have a thread: {code} "ServerService Thread Pool -- 82" #322 prio=5 os_prio=31 tid=0x00007fcad286b800 nid=0x13007 waiting on condition [0x000070000c462000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000791e1cf20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:375) - locked <0x0000000791e1cf60> (a java.lang.Object) at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:315) at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sessionStop(ActiveMQSessionContext.java:343) at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:662) at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.stop(ClientSessionImpl.java:651) at org.apache.activemq.artemis.jms.client.ActiveMQSession.stop(ActiveMQSession.java:1023) at org.apache.activemq.artemis.jms.client.ActiveMQConnection.stop(ActiveMQConnection.java:321) - locked <0x0000000791e1c360> (a org.apache.activemq.artemis.jms.client.ActiveMQXAConnection) at org.apache.activemq.artemis.ra.ActiveMQRAManagedConnection.destroyHandles(ActiveMQRAManagedConnection.java:229) at org.apache.activemq.artemis.ra.ActiveMQRAManagedConnection.destroy(ActiveMQRAManagedConnection.java:268) at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.doDestroy(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1369) at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.flush(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:882) at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.shutdown(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1065) at org.jboss.jca.core.connectionmanager.pool.AbstractPool.shutdown(AbstractPool.java:930) - locked <0x0000000790f758d0> (a org.jboss.jca.core.connectionmanager.pool.strategy.PoolByCri) at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.shutdown(AbstractConnectionManager.java:286) - locked <0x0000000790f75868> (a org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl) at org.jboss.as.connector.services.resourceadapters.deployment.AbstractResourceAdapterDeploymentService.unregisterAll(AbstractResourceAdapterDeploymentService.java:199) at org.jboss.as.connector.services.resourceadapters.deployment.AbstractResourceAdapterDeploymentService$3.run(AbstractResourceAdapterDeploymentService.java:353) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320) {code} When the server is asked to shutdown, IJ will flush its pools. In turn, this calls Artemis RA managed connection destroy() which will stop any underlying JMS connection. The test is using a remote JMS that has been stopped and thus the underlying JMS connection will be blocked for some time. I had a look at SemaphoreConcurrentLinkedDequeManagedConnectionPool#flush[1] and it is destroying the pool's connections sequentially. This implies that the time to flush the pool can be potentially up to n * t where n is the number of connections managed by the pool and t is the timeout that each connections used when it is stopped. In the test run, n = 20, t = 30s so we have to wait 10 minutes to the the pool completely flushed. Would it be possible to flush the pool concurrently, so that the time to flush the pools would be close to t? Something like: {code} destroy.parallelStream().forEach(clw -> doDestroy(clw)); {code} With such a code, the time to flush the pool would be close to t. In our test run, that would reduce the time to wait from 10 minutes to circa 30s. Would there be any issue with parallelizing the flush of the pool in such way? [1] https://github.com/ironjacamar/ironjacamar/blob/1.4/core/src/main/java/or...