[infinispan-issues] [JBoss JIRA] (ISPN-4213) clustered Wildfly unexpected nodes SuspectException

Wed Apr 16 11:56:34 EDT 2014

    [ https://issues.jboss.org/browse/ISPN-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12962321#comment-12962321 ] 

Hielke Hoeve commented on ISPN-4213:
------------------------------------

Restarting the suspected node after waiting more than 2 hours does not cause the nodes to rediscover each other. Nor do they rediscover each other when  restarting each node consecutively. They give no warning, error of info.

> clustered Wildfly unexpected nodes SuspectException
> ---------------------------------------------------
>
>                 Key: ISPN-4213
>                 URL: https://issues.jboss.org/browse/ISPN-4213
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 6.0.1.Final
>         Environment: ubuntu 13.04, jboss wildfly 8.0.1, infinispan 6.0.1.Final, jgroups 3.4.3.Final
>            Reporter: Hielke Hoeve
>            Assignee: Dan Berindei
>
> Our current production system has a cluster of 2 nodes using Wildfly 8.0.1. We have used the "ha" profile and "ha" socket-binding-group as a baseline to create the profile for the clusters. The cluster has a REST application deployed. This application uses 2 infinispan cache containers (hibernate and application).  In total these views have 5 local caches, 3 invalidation caches and 7 replicated caches. The cache containers use udp as transport. Our hibernate container is the default config as provided by the "ha" profile.
> Last thursday and yesterday the one of the nodes started throwing SuspectException during a hibernate flush. The applications had only a few active users.
> Node 1 says:
> -----------------
> 2014-04-15 17:40:53,380 ERROR [org.jboss.as.ejb3.invocation] (default task-22) JBAS014134: EJB Invocation failed on component ConversatieResourceRESTService 
> for method public javax.ws.rs.core.Response ...ConversatieResourceRESTService.setStatus(...event.Con
> versatieStatusObject): javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
> ...
> ...
>         at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
>         at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
>         at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
>         at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1306) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
>         at org.infinispan.CacheImpl.putInternal(CacheImpl.java:878) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
>         at org.infinispan.CacheImpl.put(CacheImpl.java:870) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
>         at org.infinispan.DecoratedCache.put(DecoratedCache.java:401) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
>         at org.infinispan.AbstractDelegatingCache.put(AbstractDelegatingCache.java:276) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
>         at org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.update(TransactionalAccessDelegate.java:192) [hibernate-infinispan-4.3.4.Final.j
> ar:4.3.4.Final]
>         at org.hibernate.cache.infinispan.entity.TransactionalAccess.update(TransactionalAccess.java:89) [hibernate-infinispan-4.3.4.Final.jar:4.3.4.Final]
>         at org.hibernate.action.internal.EntityUpdateAction.cacheUpdate(EntityUpdateAction.java:234) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
>         at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:209) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
>         at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:461) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
>         at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:347) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
>         at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:350) [hibernate-core-4.3.4.Final.j
> ar:4.3.4.Final]
>         at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
>         at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1222) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
>         at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:425) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
>         at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallba
> ckCoordinatorNonTrackingImpl.java:110) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
>         ... 117 more
> Caused by: SuspectedException
>         at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:406)
>         at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:353) [infinispan-core-6.0.1.F
> inal.jar:6.0.1.Final]
>         at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167) [infinispan-core-6.0.1
> .Final.jar:6.0.1.Final]
>         ... 161 more
> 2014-04-15 17:40:53,707 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-7,shared=udp) ISPN000094: Received new cluster view: [node1:rest1-even/hibernate-even|2] (1) [node1:rest1-even/hibernate-even]
> 2014-04-15 17:40:53,707 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-14,shared=udp) ISPN000094: Received new cluster view: [node1:rest1-even/application-even|2] (1) [node1:rest1-even/application-even]
> Node 2 says:
> -----------------
> 2014-04-15 17:40:17,934 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-19,shared=udp) ISPN000094: Received new cluster view: [node2:rest2-even/hibernate-even|2] (1) [node2:rest2-even/hibernate-even]
> 2014-04-15 17:40:17,934 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-17,shared=udp) ISPN000094: Received new cluster view: [node2:rest2-even/application-even|2] (1) [node2:rest2-even/application-even]
> ...
> ...
> 2014-04-15 17:40:50,737 WARN  [org.jgroups.protocols.UDP] (Timer-2,shared=udp) JGRP000032: null: no physical address for node1:rest1-even/hibernate-even, dropping message
> 2014-04-15 17:40:52,543 WARN  [org.jgroups.protocols.UDP] (Timer-5,shared=udp) JGRP000032: null: no physical address for node1:rest1-even/application-even, dropping message
> (repeats 100 times)
> ...
> Is there any way I can get more logging than the WARNs above? Does anyone have pointers how or when this SuspectException is thrown?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira