[jboss-jira] [JBoss JIRA] Commented: (JBREM-553) deadlock when disconnecting
Tom Elrod (JIRA)
jira-events at jboss.com
Sun Jul 23 16:02:11 EDT 2006
[ http://jira.jboss.com/jira/browse/JBREM-553?page=comments#action_12339918 ]
Tom Elrod commented on JBREM-553:
----------------------------------
Based on the thread dump (thanks for providing this btw), it looks like when the client disconnects, the client invoker goes to clear all the connection pools (MicroSocketClientInvoker.clearPool(ServerAddress). As it iterates through the Map of connection pools, it will try to synchronize on each connection pool it pulls out (line 529), where is where is having to wait for the lock during the disconnect (which is already held by the ConnectionValidator thread).
The ConnectionValidator is making simple call on server using client invoker (more on where this is used in a min). During this process, the MicroSocketClientInvoker has to get a connection from the connection pool (MicroSocketClientInvoker.getConnection()) and will synchronize the connection pool while doing this (line 649). This will only lock the pool while it can either get a connection from the pool or create a new connection. If all the connections within the pool are being used, will leave the synchronized block and retry (after sleeping 1 second).
So appears that the thread from the ConnectionValidator has the lock on the connection pool trying to get/create a new connection while the disconnect thread is trying to clean out the connection pools. This in of itself it weird because the lock on the pool when trying to get/create connection should not be very long, so should be releases after short period of time for the pool cleanup.
Now for how the ConnectionValidator comes into the picture in the first plase. There are only two ways in which a ConnectionValidator will be used. The first is if add a connection listener to the client (e.g. Client.addConnectionListener(ConnectionListener listener, int pingPeriod)). This will be used when user wants to know if server died while client has been idle. The other place is in a detector, where has not received a detection message from a server and will use the ConnectionValidator to make an active ping call on the server to check if it is still alive (e.g. AbstractDetector.checkInvokerServer(Detection detection, ClassLoader cl)).
I think in this particular case, is the detector that is using the ConnectionValidator.
> deadlock when disconnecting
> ---------------------------
>
> Key: JBREM-553
> URL: http://jira.jboss.com/jira/browse/JBREM-553
> Project: JBoss Remoting
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Affects Versions: 2.0.0.Beta2 (Boon)
> Reporter: John Mazzitelli
> Assigned To: Tom Elrod
> Fix For: 2.0.0.CR1 (Boon)
>
>
> I am seeing a deadlock in my test suite. It happens everytime. I haven't finished looking at all the data, but I'm submitting this JIRA now because I only started seeing this after building a new remoting jar from HEAD and installing it. I do not believe any changes that I did caused this (I haven't changed much if any code in my own stuff - though I will test this by switching out to an older remoting jar to see what happens).
> Here's the two threads that deadlocked on java.util.LinkedList<E> (id=2271:
> Thread [Thread-3562] (Suspended)
> owns: java.io.BufferedInputStream (id=2299)
> owns: java.util.LinkedList<E> (id=2271)
> java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) line: not available [native method]
> java.net.SocketInputStream.read(byte[], int, int) line: 129
> java.io.BufferedInputStream.fill() line: 218
> java.io.BufferedInputStream.read1(byte[], int, int) line: 256
> java.io.BufferedInputStream.read(byte[], int, int) line: 313
> java.io.ObjectInputStream$PeekInputStream.read(byte[], int, int) line: 2217
> java.io.ObjectInputStream$PeekInputStream.readFully(byte[], int, int) line: 2230
> java.io.ObjectInputStream$BlockDataInputStream.readShort() line: 2698
> org.jboss.remoting.loading.ObjectInputStreamWithClassLoader(java.io.ObjectInputStream).readStreamHeader() line: 750
> org.jboss.remoting.loading.ObjectInputStreamWithClassLoader(java.io.ObjectInputStream).<init>(java.io.InputStream) line: 268
> org.jboss.remoting.loading.ObjectInputStreamWithClassLoader.<init>(java.io.InputStream, java.lang.ClassLoader) line: 73
> org.jboss.remoting.serialization.impl.java.JavaSerializationManager.createInput(java.io.InputStream, java.lang.ClassLoader) line: 52
> org.jboss.remoting.transport.socket.ClientSocketWrapper.createInputStream(java.lang.String, java.net.Socket) line: 83
> org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(java.net.Socket, java.util.Map) line: 76
> org.jboss.remoting.transport.socket.ClientSocketWrapper.<init>(java.net.Socket, java.util.Map, java.lang.Integer) line: 54
> sun.reflect.GeneratedConstructorAccessor31.newInstance(java.lang.Object[]) line: not available
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) line: 27
> java.lang.reflect.Constructor<T>.newInstance(java.lang.Object...) line: 494
> org.jboss.remoting.transport.socket.SocketClientInvoker.createClientSocket(java.net.Socket, int, java.util.Map) line: 158
> org.jboss.remoting.transport.socket.SocketClientInvoker(org.jboss.remoting.transport.socket.MicroSocketClientInvoker).getConnection() line: 666
> org.jboss.remoting.transport.socket.SocketClientInvoker(org.jboss.remoting.transport.socket.MicroSocketClientInvoker).transport(java.lang.String, java.lang.Object, java.util.Map, org.jboss.remoting.marshal.Marshaller, org.jboss.remoting.marshal.UnMarshaller) line: 292
> org.jboss.remoting.transport.socket.SocketClientInvoker(org.jboss.remoting.MicroRemoteClientInvoker).invoke(org.jboss.remoting.InvocationRequest) line: 116
> org.jboss.remoting.ConnectionValidator$2.run() line: 170
> java.lang.Thread.run() line: 595
> Thread [main] (Suspended)
> owns: java.util.HashMap<K,V> (id=2272)
> owns: org.jboss.remoting.transport.sslsocket.SSLSocketClientInvoker (id=2273)
> owns: java.lang.Object (id=2274)
> owns: java.lang.Object (id=2275)
> owns: boolean[] (id=2276)
> waited by: Thread [JON Agent Main] (Running)
> waited by: Thread [JON Agent Main] (Running)
> waiting for: java.util.LinkedList<E> (id=2271)
> owned by: Thread [Thread-3562] (Running)
> org.jboss.remoting.transport.socket.MicroSocketClientInvoker.clearPool(org.jboss.remoting.transport.socket.ServerAddress) line: 509
> org.jboss.remoting.transport.socket.MicroSocketClientInvoker.clearPools() line: 541
> org.jboss.remoting.transport.sslsocket.SSLSocketClientInvoker(org.jboss.remoting.transport.socket.MicroSocketClientInvoker).handleDisconnect() line: 253
> org.jboss.remoting.transport.sslsocket.SSLSocketClientInvoker(org.jboss.remoting.MicroRemoteClientInvoker).disconnect() line: 287
> org.jboss.remoting.InvokerRegistry.destroyClientInvoker(org.jboss.remoting.InvokerLocator, java.util.Map) line: 225
> org.jboss.remoting.Client.disconnect() line: 536
> org.jboss.on.communications.command.client.JBossRemotingRemoteCommunicator.disconnect() line: 406
> org.jboss.on.communications.command.client.ClientCommandSender.stopSending(boolean) line: 725
> org.jboss.on.agent.AgentMain.shutdown() line: 436
> org.jboss.on.agent.AgentCommTest.tearDown() line: 146
> sun.reflect.GeneratedMethodAccessor26.invoke(java.lang.Object, java.lang.Object[]) line: not available
> sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) line: 25
> java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 585
> org.testng.internal.MethodHelper.invokeMethod(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) line: 552
> org.testng.internal.Invoker.invokeConfigurationMethod(java.lang.Object[], org.testng.ITestNGMethod, java.lang.Object[], boolean, org.testng.ITestResult) line: 318
> org.testng.internal.Invoker.invokeConfigurations(org.testng.IClass, org.testng.ITestNGMethod[], org.testng.xml.XmlSuite, java.util.Map<java.lang.String,java.lang.String>, java.lang.Object) line: 152
> org.testng.internal.Invoker.invokeMethod(java.lang.Object[], org.testng.ITestNGMethod, java.lang.Object[], org.testng.xml.XmlSuite, java.util.Map<java.lang.String,java.lang.String>, org.testng.ITestClass, org.testng.ITestNGMethod[], org.testng.ITestNGMethod[], org.testng.internal.ConfigurationGroupMethods) line: 483
> org.testng.internal.Invoker.invokeTestMethods(org.testng.ITestNGMethod, org.testng.xml.XmlSuite, java.util.Map<java.lang.String,java.lang.String>, org.testng.ITestNGMethod[], int, org.testng.internal.ConfigurationGroupMethods) line: 778
> org.testng.internal.TestMethodWorker.run() line: 105
> org.testng.TestRunner.privateRun(org.testng.xml.XmlTest) line: 682
> org.testng.TestRunner.run() line: 566
> org.testng.SuiteRunner.privateRun() line: 220
> org.testng.SuiteRunner.run() line: 146
> org.testng.TestNG.createAndRunSuiteRunners(org.testng.xml.XmlSuite) line: 713
> org.testng.TestNG.runSuitesLocally() line: 676
> org.apache.maven.surefire.testng.TestNGExecutor.executeTestNG(org.apache.maven.surefire.suite.SurefireTestSuite, java.lang.String, org.testng.xml.XmlSuite, org.apache.maven.surefire.report.ReporterManager) line: 64
> org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(org.apache.maven.surefire.report.ReporterManager, java.lang.ClassLoader) line: 75
> org.apache.maven.surefire.Surefire.run(java.util.List, java.util.List, java.lang.ClassLoader, java.lang.ClassLoader) line: 129
> sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) line: not available [native method]
> sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) line: 39
> sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) line: 25
> java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 585
> org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess() line: 225
> org.apache.maven.surefire.booter.SurefireBooter.main(java.lang.String[]) line: 747
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list