[infinispan-issues] [JBoss JIRA] (ISPN-4213) clustered Wildfly unexpected nodes SuspectException

Hielke Hoeve (JIRA) issues at jboss.org
Wed Apr 16 06:38:34 EDT 2014 

Hielke Hoeve created ISPN-4213:
----------------------------------

             Summary: clustered Wildfly unexpected nodes SuspectException
                 Key: ISPN-4213
                 URL: https://issues.jboss.org/browse/ISPN-4213
             Project: Infinispan
          Issue Type: Bug
          Components: Core
    Affects Versions: 6.0.1.Final
         Environment: ubuntu 13.04, jboss wildfly 8.0.1, infinispan 6.0.1.Final, jgroups 3.4.3.Final
            Reporter: Hielke Hoeve
            Assignee: Dan Berindei


Our current production system has a cluster of 2 nodes using Wildfly 8.0.1. We have used the "ha" profile and "ha" socket-binding-group as a baseline to create the profile for the clusters. The cluster has a REST application deployed. This application uses 2 infinispan cache containers (hibernate and application).  In total these views have 5 local caches, 3 invalidation caches and 7 replicated caches. The cache containers use udp as transport. Our hibernate container is the default config as provided by the "ha" profile.

Last thursday and yesterday the one of the nodes started throwing SuspectException during a hibernate flush. The applications had only a few active users.

Node 1 says:
-----------------
2014-04-15 17:40:53,380 ERROR [org.jboss.as.ejb3.invocation] (default task-22) JBAS014134: EJB Invocation failed on component ConversatieResourceRESTService 
for method public javax.ws.rs.core.Response ...ConversatieResourceRESTService.setStatus(...event.Con
versatieStatusObject): javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
...
...
        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
        at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1306) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
        at org.infinispan.CacheImpl.putInternal(CacheImpl.java:878) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
        at org.infinispan.CacheImpl.put(CacheImpl.java:870) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
        at org.infinispan.DecoratedCache.put(DecoratedCache.java:401) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
        at org.infinispan.AbstractDelegatingCache.put(AbstractDelegatingCache.java:276) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
        at org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.update(TransactionalAccessDelegate.java:192) [hibernate-infinispan-4.3.4.Final.j
ar:4.3.4.Final]
        at org.hibernate.cache.infinispan.entity.TransactionalAccess.update(TransactionalAccess.java:89) [hibernate-infinispan-4.3.4.Final.jar:4.3.4.Final]
        at org.hibernate.action.internal.EntityUpdateAction.cacheUpdate(EntityUpdateAction.java:234) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
        at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:209) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:461) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:347) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
        at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:350) [hibernate-core-4.3.4.Final.j
ar:4.3.4.Final]
        at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
        at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1222) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
        at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:425) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
        at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallba
ckCoordinatorNonTrackingImpl.java:110) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
        ... 117 more
Caused by: SuspectedException
        at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:406)
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:353) [infinispan-core-6.0.1.F
inal.jar:6.0.1.Final]
        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167) [infinispan-core-6.0.1
.Final.jar:6.0.1.Final]
        ... 161 more
2014-04-15 17:40:53,707 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-7,shared=udp) ISPN000094: Received new cluster view: [node1:rest1-even/hibernate-even|2] (1) [node1:rest1-even/hibernate-even]
2014-04-15 17:40:53,707 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-14,shared=udp) ISPN000094: Received new cluster view: [node1:rest1-even/application-even|2] (1) [node1:rest1-even/application-even]

Node 2 says:
-----------------
2014-04-15 17:40:17,934 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-19,shared=udp) ISPN000094: Received new cluster view: [node2:rest2-even/hibernate-even|2] (1) [node2:rest2-even/hibernate-even]
2014-04-15 17:40:17,934 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-17,shared=udp) ISPN000094: Received new cluster view: [node2:rest2-even/application-even|2] (1) [node2:rest2-even/application-even]
...
...
2014-04-15 17:40:50,737 WARN  [org.jgroups.protocols.UDP] (Timer-2,shared=udp) JGRP000032: null: no physical address for node1:rest1-even/hibernate-even, dropping message
2014-04-15 17:40:52,543 WARN  [org.jgroups.protocols.UDP] (Timer-5,shared=udp) JGRP000032: null: no physical address for node1:rest1-even/application-even, dropping message
(repeats 100 times)
...

Is there any way I can get more logging than the WARNs above? Does anyone have pointers how or when this SuspectException is thrown?


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

More information about the infinispan-issues mailing list