[Design of Messaging on JBoss (Messaging/JBoss)] - Re: Another DeadLock, something around InVM
by clebert.suconic＠jboss.com
Just had another look at the original thread dump:
I can also see the waiting to shutdown the executor there:
| [junit] "main" prio=1 tid=0x08deb950 nid=0x76c8 waiting on condition [0xffc33000..0xffc33d58]
| [junit] at sun.misc.Unsafe.park(Native Method)
| [junit] at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
| [junit] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
| [junit] at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1028)
| [junit] at org.jboss.messaging.integration.transports.netty.NettyAcceptor.stop(NettyAcceptor.java:354)
| [junit] - locked <0xb2718638> (a org.jboss.messaging.integration.transports.netty.NettyAcceptor)
| [junit] at org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl.stop(RemotingServiceImpl.java:213)
| [junit] - locked <0xb2714c30> (a org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl)
| [junit] at org.jboss.messaging.core.server.impl.MessagingServiceImpl.stop(MessagingServiceImpl.java:67)
| [junit] at org.jboss.messaging.tests.integration.remoting.PingTest.tearDown(PingTest.java:102)
| [junit] at junit.framework.TestCase.runBare(TestCase.java:136)
| [junit] at junit.framework.TestResult$1.protect(TestResult.java:106)
| [junit] at junit.framework.TestResult.runProtected(TestResult.java:124)
| [junit] at junit.framework.TestResult.run(TestResult.java:109)
| [junit] at junit.framework.TestCase.run(TestCase.java:120)
| [junit] at junit.framework.TestSuite.runTest(TestSuite.java:230)
| [junit] at junit.framework.TestSuite.run(TestSuite.java:225)
| [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:421)
| [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:912)
| [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743)
|
So, this is the same condition replicated by PingStressTest.
And I also confirmed the test passed without the changes from r5990.
View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4215069#4215069
Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4215069
17 years, 1 month
[Design of Messaging on JBoss (Messaging/JBoss)] - Re: Another DeadLock, something around InVM
by clebert.suconic＠jboss.com
I believe (unless I"m missing something here) this is being caused by the recent changes on shutting down the executors:
I have found three threads waiting the same lock (00007fc50a611708):
On the thread dump (1):
- "Thread-1" waiting to the lock, before a call to failover, while the executor is being used.
- Another thread named "LocalThread i = 10" is holding the lock and waiting to shutdown the executor.
I would consider this a dead lock, even though the VM didn't report it so as it didn't explicity used to lock objects.
I am adding PingStressTest into stressTest (which is the test i previously reported here with a few modifications). The test is currently disabled. Change getNumberOfIterations to something like 10 or 100 to replicate this hang.
The test passed for me after reverting r5990.
I will run it a few more times and report the results later.
___________________________________________________________________________
*1 - Thread dump
| "LocalThread i = 10" prio=10 tid=0x00007fc50035c000 nid=0x6484 waiting on condition [0x0000000040849000..0x0000000040849c00]
| java.lang.Thread.State: TIMED_WAITING (parking)
| at sun.misc.Unsafe.park(Native Method)
| - parking to wait for <0x00007fc50a5eebe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
| at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
| at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
| at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1244)
| at org.jboss.messaging.integration.transports.netty.NettyConnector.close(NettyConnector.java:363)
| - locked <0x00007fc50a6114d0> (a org.jboss.messaging.integration.transports.netty.NettyConnector)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.checkCloseConnections(ConnectionManagerImpl.java:761)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:852)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:387)
| - locked <0x00007fc50a611708> (a java.lang.Object)
| - locked <0x00007fc50a6116f8> (a java.lang.Object)
| at org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1313)
| at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:761)
| at org.jboss.messaging.tests.stress.remote.PingStressTest$1LocalThread.run(PingStressTest.java:248)
|
|
| "Thread-1 (group:jbm-pinger-threads-896472140)" daemon prio=10 tid=0x00007fc500385c00 nid=0x6463 waiting for monitor entry [0x0000000045ece000..0x0000000045ecec00]
| java.lang.Thread.State: BLOCKED (on object monitor)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failover(ConnectionManagerImpl.java:493)
| - waiting to lock <0x00007fc50a611708> (a java.lang.Object)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionFailed(ConnectionManagerImpl.java:411)
| at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callListeners(RemotingConnectionImpl.java:530)
| at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:421)
| at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$Pinger.run(RemotingConnectionImpl.java:1537)
| - locked <0x00007fc50a614508> (a org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$Pinger)
| at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
| at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
| at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
| at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
| at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
| at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
| at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
| at java.lang.Thread.run(Thread.java:619)
|
|
|
| "LocalThread i = 10" prio=10 tid=0x00007fc50035c000 nid=0x6484 waiting on condition [0x0000000040849000..0x0000000040849c00]
| java.lang.Thread.State: TIMED_WAITING (parking)
| at sun.misc.Unsafe.park(Native Method)
| - parking to wait for <0x00007fc50a5eebe8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
| at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
| at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
| at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1244)
| at org.jboss.messaging.integration.transports.netty.NettyConnector.close(NettyConnector.java:363)
| - locked <0x00007fc50a6114d0> (a org.jboss.messaging.integration.transports.netty.NettyConnector)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.checkCloseConnections(ConnectionManagerImpl.java:761)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.returnConnection(ConnectionManagerImpl.java:852)
| at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.removeSession(ConnectionManagerImpl.java:387)
| - locked <0x00007fc50a611708> (a java.lang.Object)
| - locked <0x00007fc50a6116f8> (a java.lang.Object)
| at org.jboss.messaging.core.client.impl.ClientSessionImpl.doCleanup(ClientSessionImpl.java:1313)
| at org.jboss.messaging.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:761)
| at org.jboss.messaging.tests.stress.remote.PingStressTest$1LocalThread.run(PingStressTest.java:248)
|
|
View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4215066#4215066
Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4215066
17 years, 1 month
[Design of Security on JBoss] - Password masking at the MC level
by anil.saldhana＠jboss.com
I want to just place some of my thoughts here for future reference.
Scott had broached the topic of some aspects to encrypt/decrypt bean properties which is on our TODO list.
I also want to handle the issue of password masking (or any other sensitive bean properties) using XML encryption. It would be simple to encrypt the secret key using PKI but it is not very simple to deal with the keystore stuff (keystore password etc). Assuming that we know how to do key management properly (such as encrypting the tomcat keystore password for example which sets the JSSE keystore properties or other means of dealing with keystore), we can certainly use XML encryption to encrypt xml elements of MC bean definitions.
A writeup from me recently is here:
http://jboss.org/community/docs/DOC-13383
Apache XML Security is currently packaged in JBAS. It is pretty straight forward to write an utility to encrypt/decrypt xml elements (either based on the one I started in JBID or the one Jason wrote for WSS support in JBossWS).
I have not given too much thought on how to do it at the MC level yet but will in the next few days. We do have the concept of SubjectFactory in AS5 which is being injected into JCA connection factories. Maybe a specific implementation of SubjectFactory that can handle xml encryption may be good.
Just some of my thoughts.
View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4215052#4215052
Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4215052
17 years, 1 month
[Design the new POJO MicroContainer] - Re: InterruptedException not being cleared?
by scott.stark＠jboss.org
It looks like somewhere along this stack trace is where the interrupt is being set. I'm not sure why.
| 13:30:16,400 INFO [RuntimeDiscoveryExecutor] Running runtime discovery scan rooted at platform...
| 13:30:26,199 ERROR [ExceptionUtil] Topic[/topic/TestTopic] getNonDurableSubscriptionsCount
| java.lang.InterruptedException
| at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:157)
| at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.getQueuesForCondition(MessagingPostOffice.java:514)
| at org.jboss.jms.server.destination.ManagedTopic.getSubscriptionsCount(ManagedTopic.java:308)
| at org.jboss.jms.server.destination.ManagedTopic.getNonDurableSubscriptionsCount(ManagedTopic.java:104)
| at org.jboss.jms.server.destination.TopicService.getNonDurableSubscriptionsCount(TopicService.java:305)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.mx.interceptor.AttributeDispatcher.invoke(AttributeDispatcher.java:99)
| at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
| at org.jboss.mx.interceptor.ModelMBeanAttributeInterceptor.invoke(ModelMBeanAttributeInterceptor.java:197)
| at org.jboss.mx.interceptor.PersistenceInterceptor.invoke(PersistenceInterceptor.java:76)
| at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
| at org.jboss.mx.server.AbstractMBeanInvoker.getAttribute(AbstractMBeanInvoker.java:362)
| at org.jboss.mx.server.MBeanServerImpl.getAttribute(MBeanServerImpl.java:565)
| at org.jboss.system.deployers.managed.ServiceMetaDataICF.getValue(ServiceMetaDataICF.java:166)
| at org.jboss.system.deployers.managed.ServiceMetaDataICF.getValue(ServiceMetaDataICF.java:54)
| at org.jboss.managed.plugins.factory.AbstractManagedObjectPopulator.populateValues(AbstractManagedObjectPopulator.java:204)
| at org.jboss.managed.plugins.factory.AbstractManagedObjectPopulator.populateManagedObject(AbstractManagedObjectPopulator.java:130)
| at org.jboss.managed.plugins.factory.AbstractManagedObjectFactory.initManagedObject(AbstractManagedObjectFactory.java:367)
| at org.jboss.managed.plugins.factory.AbstractInstanceClassFactory.getManagedObjectValue(AbstractInstanceClassFactory.java:315)
| at org.jboss.managed.plugins.factory.AbstractInstanceClassFactory.getManagedObjectArray(AbstractInstanceClassFactory.java:331)
| at org.jboss.managed.plugins.factory.AbstractInstanceClassFactory.getValue(AbstractInstanceClassFactory.java:252)
| at org.jboss.managed.plugins.factory.AbstractManagedObjectPopulator.populateValues(AbstractManagedObjectPopulator.java:204)
| at org.jboss.managed.plugins.factory.AbstractManagedObjectPopulator.populateManagedObject(AbstractManagedObjectPopulator.java:130)
| at org.jboss.managed.plugins.factory.AbstractManagedObjectFactory.initManagedObject(AbstractManagedObjectFactory.java:367)
| at org.jboss.deployers.spi.deployer.helpers.DefaultManagedObjectCreator.build(DefaultManagedObjectCreator.java:81)
| at org.jboss.deployers.plugins.deployers.DeployersImpl.getManagedObjects(DeployersImpl.java:463)
| at org.jboss.deployers.plugins.main.MainDeployerImpl.getManagedObjects(MainDeployerImpl.java:928)
| at org.jboss.deployers.plugins.main.MainDeployerImpl.getManagedDeployment(MainDeployerImpl.java:905)
| at org.jboss.profileservice.management.ManagementViewImpl.getManagedDeployment(ManagementViewImpl.java:1303)
| at org.jboss.profileservice.management.ManagementViewImpl.loadProfile(ManagementViewImpl.java:215)
| at sun.reflect.GeneratedMethodAccessor317.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:121)
| at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
| at org.jboss.profileservice.remoting.ProfileServiceInvocationHandler.invoke(ProfileServiceInvocationHandler.java:99)
| at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:908)
| at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:106)
| at org.jboss.remoting.Client.invoke(Client.java:1708)
| at org.jboss.remoting.Client.invoke(Client.java:612)
| at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
| at org.jboss.aspects.remoting.MergeMetaDataInterceptor.invoke(MergeMetaDataInterceptor.java:74)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
| at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
| at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
| at AOPProxy$1.loadProfile(AOPProxy$1.java)
| at org.rhq.plugins.jbossas5.factory.ProfileServiceFactory.refreshCurrentProfileView(ProfileServiceFactory.java:113)
| at org.rhq.plugins.jbossas5.ManagedDeploymentComponent.getManagedDeployment(ManagedDeploymentComponent.java:335)
| at org.rhq.plugins.jbossas5.ManagedDeploymentComponent.getAvailability(ManagedDeploymentComponent.java:133)
| at sun.reflect.GeneratedMethodAccessor330.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| at java.lang.reflect.Method.invoke(Method.java:585)
| at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:466)
| at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
| at java.util.concurrent.FutureTask.run(FutureTask.java:123)
| at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
| at java.lang.Thread.run(Thread.java:613)
|
View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4215045#4215045
Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4215045
17 years, 1 month