[infinispan-issues] [JBoss JIRA] (ISPN-2982) CLONE - State transfer does not end because some segments are erroneously reported as unreceived

[Dan Berindei (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766540#comment-12766540 ] 

Dan Berindei commented on ISPN-2982:
------------------------------------

The "Problems unmarshalling remote command from byte buffer" warning might be related, didn't it have an exception and a stack trace as well? The command that cannot be deserialized might be the {{ClusterTopologyControlCommand(CH_UPDATE)}} that tells the joiner that it has joined successfully.

Note, however, that this exception ({{CacheException: Initial state transfer timed out for cache LuceneIndexesMetadata on PersistentStateTransferQueryDistributedIndexTest-NodeC-6067}}) is a catch-all exception that appears on the node that is trying to join the cluster.

In order for the node to join successfully, all the nodes have to report that they finished receiving state to the coordinator (even if they didn't request any state), and the coordinator then has to tell the joiner that the state transfer is completed. If any of the state transfer steps fails, on any node, the joiner will probably report a state transfer timeout. So in general it's very important to know what the other nodes were doing, in order to find the root problem.
                
> CLONE - State transfer does not end because some segments are erroneously reported as unreceived
> ------------------------------------------------------------------------------------------------
>
>                 Key: ISPN-2982
>                 URL: https://issues.jboss.org/browse/ISPN-2982
>             Project: Infinispan
>          Issue Type: Bug
>          Components: State transfer
>    Affects Versions: 5.2.1.Final
>            Reporter: Ittay Dror
>            Assignee: Adrian Nistor
>            Priority: Critical
>         Attachments: jgroups.xml
>
>
> Hard to reproduce. I lost the last log where this was visible but still have a stack trace:
> org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
>         at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
>         at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:879)
>         at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:650)
>         at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:639)
>         at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:542)
>         at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:197)
>         at org.infinispan.CacheImpl.start(CacheImpl.java:517)
>         at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:689)
>         at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:652)
>         at org.infinispan.manager.DefaultCacheManager.access$100(DefaultCacheManager.java:126)
>         at org.infinispan.manager.DefaultCacheManager$1.run(DefaultCacheManager.java:574)
> Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache LuceneIndexesMetadata on PersistentStateTransferQueryDistributedIndexTest-NodeC-6067
>         at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:199)
>         at sun.reflect.GeneratedMethodAccessor139.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
>         ... 10 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira