[infinispan-issues] [JBoss JIRA] (ISPN-2153) Distributed ejb cache container on AS7 - Server hangs when invoking an SFSB

[Mircea Markus (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705572#comment-12705572 ] 

Mircea Markus commented on ISPN-2153:
-------------------------------------

+1 for all Dan's points.
@Paul - looking a bit at the AS code, a possible cause for this is the following:
- you create an KeyAffinityFactory with a single thread executor (Executors.newSingleThreadExecutor())
- multiple KAS services seem to be created by the same KeyAffinityFactory
- that means that the single thread might be shared between multiple KAS. So it might be waiting in one of the other instances and it would never get CPU to generate keys for the instance where the leave lock takes place.

I think we can determine whether this is the cause of the problem by identifying the thread single executor thread in the thread dump and determining the number of instances that are built by the same KeyAffinityService.       
 

                
> Distributed ejb cache container on AS7 - Server hangs when invoking an SFSB
> ---------------------------------------------------------------------------
>
>                 Key: ISPN-2153
>                 URL: https://issues.jboss.org/browse/ISPN-2153
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Distributed Cache
>    Affects Versions: 5.1.5.FINAL
>         Environment: JBoss AS 7.2.0.Alpha1-SNAPSHOT, EAP6
>            Reporter: Heinz Wilming
>            Assignee: Manik Surtani
>            Priority: Critical
>             Fix For: 5.2.0.ALPHA2, 5.1.6.CR1
>
>
> I'm working on scalable ha-clusters and I have a domain-mode cluster in mind which uses Infinispan-distribution to be scalable. So I started to make the cache-containers use distributed-caches instead of replicated ones.
> It starts up well and seems like everything is ok. But when I try to access a stateful-session-bean the server hangs. It only hangs when I try to use a distributed cache for the container ejb. If I use a replicated cache for ejb but still use distributed caches for cluster and web everything works fine. The problem does appear independent from the number of cluster nodes.
> By hanging I mean that the server won't complete the request. It will start to shutdown on CTRL+C but will not finish to. The only way to kill the server is kill -9. Attached to this post there is a file containing the stdout of the server with a thread-dump (kill -3) appended at the point where I requested a SFSB and nothing happened.
> Threaddump:
> [Server:server-one] "ajp-/192.168.122.165:8009-2" daemon prio=10 tid=0x08708800 nid=0x2583 runnable [0xb0882000]
> [Server:server-one]    java.lang.Thread.State: RUNNABLE
> [Server:server-one]     at java.util.HashMap.containsKey(HashMap.java:352)
> [Server:server-one]     at java.util.HashSet.contains(HashSet.java:201)
> [Server:server-one]     at org.infinispan.affinity.KeyAffinityServiceImpl.isNodeInConsistentHash(KeyAffinityServiceImpl.java:351)
> [Server:server-one]     at org.infinispan.affinity.KeyAffinityServiceImpl.getKeyForAddress(KeyAffinityServiceImpl.java:145)
> [Server:server-one]     at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.createIdentifier(InfinispanBackingCacheEntryStore.java:123)
> [Server:server-one]     at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.createIdentifier(InfinispanBackingCacheEntryStore.java:71)
> [Server:server-one]     at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.createIdentifier(SerializationGroupMemberContainer.java:83)
> [Server:server-one]     at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.createIdentifier(SerializationGroupMemberContainer.java:51)
> [Server:server-one]     at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.createIdentifier(PassivatingBackingCacheImpl.java:96)
> [Server:server-one]     at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.createIdentifier(PassivatingBackingCacheImpl.java:60)
> [Server:server-one]     at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.createIdentifier(AbstractCache.java:48)
> [Server:server-one]     at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.createIdentifier(AbstractCache.java:39)
> [Server:server-one]     at org.jboss.as.ejb3.component.stateful.StatefulSessionComponentInstance.<init>(StatefulSessionComponentInstance.java:70)
> [Server:server-one]     at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.instantiateComponentInstance(StatefulSessionComponent.java:259)
> [Server:server-one]     at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:150)
> [Server:server-one]     at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.constructComponentInstance(StatefulSessionComponent.java:145)
> [Server:server-one]     at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.constructComponentInstance(StatefulSessionComponent.java:76)
> [Server:server-one]     at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85)
> [Server:server-one]     at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.createInstance(StatefulSessionComponent.java:135)
> [Server:server-one]     at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.createInstance(StatefulSessionComponent.java:76)
> [Server:server-one]     at org.jboss.as.ejb3.cache.TransactionAwareObjectFactory.createInstance(TransactionAwareObjectFactory.java:53)
> [Server:server-one]     at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.create(PassivatingBackingCacheImpl.java:121)
> [Server:server-one]     at org.jboss.as.ejb3.cache.impl.GroupAwareCache.create(GroupAwareCache.java:67)
> [Server:server-one]     at org.jboss.as.ejb3.cache.impl.GroupAwareCache.create(GroupAwareCache.java:41)
> [Server:server-one]     at org.jboss.as.ejb3.component.stateful.StatefulSessionComponent.createSession(StatefulSessionComponent.java:241)
> [Server:server-one]     at org.jboss.as.weld.ejb.StatefulSessionObjectReferenceImpl.<init>(StatefulSessionObjectReferenceImpl.java:72)
> [Server:server-one]     at org.jboss.as.weld.services.bootstrap.WeldEjbServices.resolveEjb(WeldEjbServices.java:60)
> [Server:server-one]     at org.jboss.weld.bean.SessionBean.createReference(SessionBean.java:412)
> [Server:server-one]     at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.<init>(EnterpriseBeanProxyMethodHandler.java:69)
> [Server:server-one]     at org.jboss.weld.bean.SessionBean.create(SessionBean.java:297)
>  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira