[jboss-dev] Change in locking behavior in MainDeployer [WAS: Port conflict in tx manager - cluster-udp-SYNC-0
The reason the cluster-udp-SYNC-0 server is failing to start is because
a server (cluster-udp-1) from the previous test set is not shutting down
because threads are deadlocking. See attached thread dump.
What's changed here is what used to be a read lock acquisition in
MainDeploymentImpl.process() has changed into a write lock acquisition
in a jboss-deployers snapshot deployed a couple days ago. r84308 [1]
The root of the problem is thread AsynchKeyChangeHandler is
asynchronously responding to a notification of an undeploy on the other
node by 1) synchronizing on an HASingletonImpl and 2) calling into
MainDeployerImpl.addDeployment(). The addDeployment call needs to
acquire a read lock on a ReentrantReadWriteLock.
Meanwhile the test driver has moved on and is trying to undeploy the
HASingletonImpl via an RMI call. The RMI call has called
MainDeployerImpl.process() and acquired a write lock on the
ReentrantReadWriteLock. It then needs to synchronize on the HASingletonImpl.
Deadlock.
Does this lock need to be a write lock?
[1]
http://fisheye.jboss.org/browse/JBossAS/projects/jboss-deployers/trunk/de...
Brian Stansberry wrote:
The servers from the previous set of tests are failing to shut down;
I'll take a look why:
[server:stop] Shutting down server: cluster-udp-0
[server:stop] shutdownTimeout will be=45
[server:stop] Writing server thread dump to
/mnt/hudson_workspace/workspace/JBoss-AS-5.0.x-testSuite-sun15-sun16/JBossAS_5_0/build/output/jboss-5.0.1.GA/server/cluster-udp-0/log/threadDump.log
[server:stop] Failed to shutdown server "cluster-udp-0" before timeout.
Destroying the process.
[server:stop] Unable to shutdown server properly:
org.jboss.jbossas.servermanager.ServerShutdownException: Failed to
shutdown server before timeout.Process was destroyed.
[server:stop] Shutting down server: cluster-udp-1
[server:stop] shutdownTimeout will be=45
[server:stop] Writing server thread dump to
/mnt/hudson_workspace/workspace/JBoss-AS-5.0.x-testSuite-sun15-sun16/JBossAS_5_0/build/output/jboss-5.0.1.GA/server/cluster-udp-1/log/threadDump.log
[server:stop] Failed to shutdown server "cluster-udp-1" before timeout.
Destroying the process.
[server:stop] Unable to shutdown server properly:
org.jboss.jbossas.servermanager.ServerShutdownException: Failed to
shutdown server before timeout.Process was destroyed.configuration
(SYNC, no BR)
Dimitris Andreadis wrote:
> For some reason the cluster-udp-SYNC-0 test config in the
> JBoss-AS-5.0.x-testSuite-sun15-sun16 run shows a port conflict (don't
> appear up in the other runs thought, eg. testSuite-sun16):
>
>
http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-test...
>
>
>
http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-test...
>
>
> Some server failed to start.
> 06:01:19,991 INFO [TransactionManagerService] JBossTS Transaction
> Service (JTA version) - JBoss Inc.
> 06:01:19,992 INFO [TransactionManagerService] Setting up property
> manager MBean and JMX layer
> 06:01:20,314 WARN [arjLoggerI18N]
> [com.arjuna.ats.arjuna.recovery.TransactionStatusManager_14] - Failed
> to create server socket on address 10.16.93.89 and port: 4,713
> 06:01:20,316 ERROR [AbstractKernelController] Error installing to
> Create: name=TransactionManager state=Configured
> com.arjuna.ats.arjuna.exceptions.FatalError:
> [com.arjuna.ats.arjuna.recovery.TransactionStatusManager_9] - Could
> not get unique port.
> at
>
com.arjuna.ats.arjuna.recovery.TransactionStatusManager.start(TransactionStatusManager.java:185)
>
> at
>
com.arjuna.ats.arjuna.recovery.TransactionStatusManager.<init>(TransactionStatusManager.java:72)
>
> at
> com.arjuna.ats.arjuna.coordinator.TxControl.<clinit>(TxControl.java:355)
> at
>
com.arjuna.ats.jbossatx.jta.TransactionManagerService.create(TransactionManagerService.java:178)
>
>
--
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry(a)redhat.com
Attachments:
- threadDump-1.html (text/html — 244.9 KB)