3:02 p.m.
[ http://jira.jboss.com/jira/browse/JBREM-553?page=comments#action_12339918 ]
Tom Elrod commented on JBREM-553:
----------------------------------
Based on the thread dump (thanks for providing this btw), it looks like when the client
disconnects, the client invoker goes to clear all the connection pools
(MicroSocketClientInvoker.clearPool(ServerAddress). As it iterates through the Map of
connection pools, it will try to synchronize on each connection pool it pulls out (line
529), where is where is having to wait for the lock during the disconnect (which is
already held by the ConnectionValidator thread).
The ConnectionValidator is making simple call on server using client invoker (more on
where this is used in a min). During this process, the MicroSocketClientInvoker has to
get a connection from the connection pool (MicroSocketClientInvoker.getConnection()) and
will synchronize the connection pool while doing this (line 649). This will only lock the
pool while it can either get a connection from the pool or create a new connection. If
all the connections within the pool are being used, will leave the synchronized block and
retry (after sleeping 1 second).
So appears that the thread from the ConnectionValidator has the lock on the connection
pool trying to get/create a new connection while the disconnect thread is trying to clean
out the connection pools. This in of itself it weird because the lock on the pool when
trying to get/create connection should not be very long, so should be releases after short
period of time for the pool cleanup.
Now for how the ConnectionValidator comes into the picture in the first plase. There are
only two ways in which a ConnectionValidator will be used. The first is if add a
connection listener to the client (e.g. Client.addConnectionListener(ConnectionListener
listener, int pingPeriod)). This will be used when user wants to know if server died
while client has been idle. The other place is in a detector, where has not received a
detection message from a server and will use the ConnectionValidator to make an active
ping call on the server to check if it is still alive (e.g.
AbstractDetector.checkInvokerServer(Detection detection, ClassLoader cl)).
I think in this particular case, is the detector that is using the ConnectionValidator.
deadlock when disconnecting
---------------------------
Key: JBREM-553
URL: http://jira.jboss.com/jira/browse/JBREM-553
Project: JBoss Remoting
Issue Type: Bug
Security Level: Public(Everyone can see)
Affects Versions: 2.0.0.Beta2 (Boon)
Reporter: John Mazzitelli
Assigned To: Tom Elrod
Fix For: 2.0.0.CR1 (Boon)
I am seeing a deadlock in my test suite. It happens everytime. I haven't finished
looking at all the data, but I'm submitting this JIRA now because I only started
seeing this after building a new remoting jar from HEAD and installing it. I do not
believe any changes that I did caused this (I haven't changed much if any code in my
own stuff - though I will test this by switching out to an older remoting jar to see what
happens).
Here's the two threads that deadlocked on java.util.LinkedList<E> (id=2271:
Thread [Thread-3562] (Suspended)
owns: java.io.BufferedInputStream (id=2299)
owns: java.util.LinkedList<E> (id=2271)
java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int)
line: not available [native method]
java.net.SocketInputStream.read(byte[], int, int) line: 129
java.io.BufferedInputStream.fill() line: 218
java.io.BufferedInputStream.read1(byte[], int, int) line: 256
java.io.BufferedInputStream.read(byte[], int, int) line: 313
java.io.ObjectInputStream$PeekInputStream.read(byte[], int, int) line: 2217
java.io.ObjectInputStream$PeekInputStream.readFully(byte[], int, int) line: 2230
java.io.ObjectInputStream$BlockDataInputStream.readShort() line: 2698
org.jboss.remoting.loading.ObjectInputStreamWithClassLoader(java.io.ObjectInputStream).readStreamHeader()
line: 750
org.jboss.remoting.loading.ObjectInputStreamWithClassLoader(java.io.ObjectInputStream).<init>(java.io.InputStream)
line: 268
org.jboss.remoting.loading.ObjectInputStreamWithClassLoader.<init>(java.io.InputStream,
java.lang.ClassLoader) line: 73
org.jboss.remoting.serialization.impl.java.JavaSerializationManager.createInput(java.io.InputStream,
java.lang.ClassLoader) line: 52
org.jboss.remoting.transport.socket.ClientSocketWrapper.createInputStream(java.lang.String,
java.net.Socket) line: 83
org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(java.net.Socket,
java.util.Map) line: 76
org.jboss.remoting.transport.socket.ClientSocketWrapper.<init>(java.net.Socket,
java.util.Map, java.lang.Integer) line: 54
sun.reflect.GeneratedConstructorAccessor31.newInstance(java.lang.Object[]) line: not
available
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) line: 27
java.lang.reflect.Constructor<T>.newInstance(java.lang.Object...) line: 494
org.jboss.remoting.transport.socket.SocketClientInvoker.createClientSocket(java.net.Socket,
int, java.util.Map) line: 158
org.jboss.remoting.transport.socket.SocketClientInvoker(org.jboss.remoting.transport.socket.MicroSocketClientInvoker).getConnection()
line: 666
org.jboss.remoting.transport.socket.SocketClientInvoker(org.jboss.remoting.transport.socket.MicroSocketClientInvoker).transport(java.lang.String,
java.lang.Object, java.util.Map, org.jboss.remoting.marshal.Marshaller,
org.jboss.remoting.marshal.UnMarshaller) line: 292
org.jboss.remoting.transport.socket.SocketClientInvoker(org.jboss.remoting.MicroRemoteClientInvoker).invoke(org.jboss.remoting.InvocationRequest)
line: 116
org.jboss.remoting.ConnectionValidator$2.run() line: 170
java.lang.Thread.run() line: 595
Thread [main] (Suspended)
owns: java.util.HashMap<K,V> (id=2272)
owns: org.jboss.remoting.transport.sslsocket.SSLSocketClientInvoker (id=2273)
owns: java.lang.Object (id=2274)
owns: java.lang.Object (id=2275)
owns: boolean[] (id=2276)
waited by: Thread [JON Agent Main] (Running)
waited by: Thread [JON Agent Main] (Running)
waiting for: java.util.LinkedList<E> (id=2271)
owned by: Thread [Thread-3562] (Running)
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.clearPool(org.jboss.remoting.transport.socket.ServerAddress)
line: 509
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.clearPools() line: 541
org.jboss.remoting.transport.sslsocket.SSLSocketClientInvoker(org.jboss.remoting.transport.socket.MicroSocketClientInvoker).handleDisconnect()
line: 253
org.jboss.remoting.transport.sslsocket.SSLSocketClientInvoker(org.jboss.remoting.MicroRemoteClientInvoker).disconnect()
line: 287
org.jboss.remoting.InvokerRegistry.destroyClientInvoker(org.jboss.remoting.InvokerLocator,
java.util.Map) line: 225
org.jboss.remoting.Client.disconnect() line: 536
org.jboss.on.communications.command.client.JBossRemotingRemoteCommunicator.disconnect()
line: 406
org.jboss.on.communications.command.client.ClientCommandSender.stopSending(boolean)
line: 725
org.jboss.on.agent.AgentMain.shutdown() line: 436
org.jboss.on.agent.AgentCommTest.tearDown() line: 146
sun.reflect.GeneratedMethodAccessor26.invoke(java.lang.Object, java.lang.Object[]) line:
not available
sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[])
line: 25
java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 585
org.testng.internal.MethodHelper.invokeMethod(java.lang.reflect.Method,
java.lang.Object, java.lang.Object[]) line: 552
org.testng.internal.Invoker.invokeConfigurationMethod(java.lang.Object[],
org.testng.ITestNGMethod, java.lang.Object[], boolean, org.testng.ITestResult) line: 318
org.testng.internal.Invoker.invokeConfigurations(org.testng.IClass,
org.testng.ITestNGMethod[], org.testng.xml.XmlSuite,
java.util.Map<java.lang.String,java.lang.String>, java.lang.Object) line: 152
org.testng.internal.Invoker.invokeMethod(java.lang.Object[], org.testng.ITestNGMethod,
java.lang.Object[], org.testng.xml.XmlSuite,
java.util.Map<java.lang.String,java.lang.String>, org.testng.ITestClass,
org.testng.ITestNGMethod[], org.testng.ITestNGMethod[],
org.testng.internal.ConfigurationGroupMethods) line: 483
org.testng.internal.Invoker.invokeTestMethods(org.testng.ITestNGMethod,
org.testng.xml.XmlSuite, java.util.Map<java.lang.String,java.lang.String>,
org.testng.ITestNGMethod[], int, org.testng.internal.ConfigurationGroupMethods) line:
778
org.testng.internal.TestMethodWorker.run() line: 105
org.testng.TestRunner.privateRun(org.testng.xml.XmlTest) line: 682
org.testng.TestRunner.run() line: 566
org.testng.SuiteRunner.privateRun() line: 220
org.testng.SuiteRunner.run() line: 146
org.testng.TestNG.createAndRunSuiteRunners(org.testng.xml.XmlSuite) line: 713
org.testng.TestNG.runSuitesLocally() line: 676
org.apache.maven.surefire.testng.TestNGExecutor.executeTestNG(org.apache.maven.surefire.suite.SurefireTestSuite,
java.lang.String, org.testng.xml.XmlSuite,
org.apache.maven.surefire.report.ReporterManager) line: 64
org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(org.apache.maven.surefire.report.ReporterManager,
java.lang.ClassLoader) line: 75
org.apache.maven.surefire.Surefire.run(java.util.List, java.util.List,
java.lang.ClassLoader, java.lang.ClassLoader) line: 129
sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object,
java.lang.Object[]) line: not available [native method]
sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) line:
39
sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[])
line: 25
java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 585
org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess() line: 225
org.apache.maven.surefire.booter.SurefireBooter.main(java.lang.String[]) line: 747
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira