[jboss-dev] Change in locking behavior in MainDeployer [WAS: Port conflict in tx manager - cluster-udp-SYNC-0

[Brian Stansberry]

The reason the cluster-udp-SYNC-0 server is failing to start is because 
a server (cluster-udp-1) from the previous test set is not shutting down 
because threads are deadlocking. See attached thread dump.

What's changed here is what used to be a read lock acquisition in 
MainDeploymentImpl.process() has changed into a write lock acquisition 
in a jboss-deployers snapshot deployed a couple days ago. r84308 [1]

The root of the problem is thread AsynchKeyChangeHandler is 
asynchronously responding to a notification of an undeploy on the other 
node by 1) synchronizing on an HASingletonImpl and 2) calling into 
MainDeployerImpl.addDeployment(). The addDeployment call needs to 
acquire a read lock on a ReentrantReadWriteLock.

Meanwhile the test driver has moved on and is trying to undeploy the 
HASingletonImpl via an RMI call. The RMI call has called 
MainDeployerImpl.process() and acquired a write lock on the 
ReentrantReadWriteLock. It then needs to synchronize on the HASingletonImpl.

Deadlock.

Does this lock need to be a write lock?

[1] 
http://fisheye.jboss.org/browse/JBossAS/projects/jboss-deployers/trunk/deployers-impl/src/main/java/org/jboss/deployers/plugins/main/MainDeployerImpl.java?r1=82090&r2=84308

Brian Stansberry wrote:
> The servers from the previous set of tests are failing to shut down; 
> I'll take a look why:
> 
> [server:stop] Shutting down server: cluster-udp-0
> [server:stop] shutdownTimeout will be=45
> [server:stop] Writing server thread dump to 
> /mnt/hudson_workspace/workspace/JBoss-AS-5.0.x-testSuite-sun15-sun16/JBossAS_5_0/build/output/jboss-5.0.1.GA/server/cluster-udp-0/log/threadDump.log 
> 
> [server:stop] Failed to shutdown server "cluster-udp-0" before timeout. 
> Destroying the process.
> [server:stop] Unable to shutdown server properly: 
> org.jboss.jbossas.servermanager.ServerShutdownException: Failed to 
> shutdown server before timeout.Process was destroyed.
> [server:stop] Shutting down server: cluster-udp-1
> [server:stop] shutdownTimeout will be=45
> [server:stop] Writing server thread dump to 
> /mnt/hudson_workspace/workspace/JBoss-AS-5.0.x-testSuite-sun15-sun16/JBossAS_5_0/build/output/jboss-5.0.1.GA/server/cluster-udp-1/log/threadDump.log 
> 
> [server:stop] Failed to shutdown server "cluster-udp-1" before timeout. 
> Destroying the process.
> [server:stop] Unable to shutdown server properly: 
> org.jboss.jbossas.servermanager.ServerShutdownException: Failed to 
> shutdown server before timeout.Process was destroyed.configuration 
> (SYNC, no BR)
> 
> Dimitris Andreadis wrote:
>> For some reason the cluster-udp-SYNC-0 test config in the 
>> JBoss-AS-5.0.x-testSuite-sun15-sun16 run shows a port conflict (don't 
>> appear up in the other runs thought, eg. testSuite-sun16):
>>
>> http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-testSuite-sun15-sun16/ 
>>
>>
>> http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-testSuite-sun15-sun16/lastBuild/artifact/JBossAS_5_0/build/output/jboss-5.0.1.GA/server/cluster-udp-SYNC-0/log/output.log 
>>
>>
>> Some server failed to start.
>> 06:01:19,991 INFO  [TransactionManagerService] JBossTS Transaction 
>> Service (JTA version) - JBoss Inc.
>> 06:01:19,992 INFO  [TransactionManagerService] Setting up property 
>> manager MBean and JMX layer
>> 06:01:20,314 WARN  [arjLoggerI18N] 
>> [com.arjuna.ats.arjuna.recovery.TransactionStatusManager_14] - Failed 
>> to create server socket on address 10.16.93.89 and port: 4,713
>> 06:01:20,316 ERROR [AbstractKernelController] Error installing to 
>> Create: name=TransactionManager state=Configured
>> com.arjuna.ats.arjuna.exceptions.FatalError: 
>> [com.arjuna.ats.arjuna.recovery.TransactionStatusManager_9] - Could 
>> not get unique port.
>>     at 
>> com.arjuna.ats.arjuna.recovery.TransactionStatusManager.start(TransactionStatusManager.java:185) 
>>
>>     at 
>> com.arjuna.ats.arjuna.recovery.TransactionStatusManager.<init>(TransactionStatusManager.java:72) 
>>
>>     at 
>> com.arjuna.ats.arjuna.coordinator.TxControl.<clinit>(TxControl.java:355)
>>     at 
>> com.arjuna.ats.jbossatx.jta.TransactionManagerService.create(TransactionManagerService.java:178) 
>>
>>
> 
> 


-- 
Brian Stansberry
Lead, AS Clustering
JBoss, a division of Red Hat
brian.stansberry at redhat.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-development/attachments/20090219/0e983b22/attachment.html