[infinispan-issues] [JBoss JIRA] (ISPN-9351) nodes did not join to cluster because of timeoutException

[M S (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-9351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13603500#comment-13603500 ] 

M S commented on ISPN-9351:
---------------------------

One of recommendation was to have following setup in config file: <FRAG3 frag_size="1200"/>, but the issue is still reproducible on coordinator, after restarting
all except the coordinator. Attaching logs ... [^Rebooting14ControllrsOf15InCloud.zip] 

> nodes did not join to cluster because of timeoutException
> ---------------------------------------------------------
>
>                 Key: ISPN-9351
>                 URL: https://issues.jboss.org/browse/ISPN-9351
>             Project: Infinispan
>          Issue Type: Bug
>            Reporter: Robert Cernak
>         Attachments: 15nodesTryingToJoin1ClusterHowever5DidNotJoin .zip, Rebooting14ControllrsOf15InCloud.zip
>
>
> I was trying to connect 15nodes to 1 cluster, however nodes did not join.
> In logs, all nodes mostly had 2 kinds of exceptions, all caused by TimeoutException:
> 1:
> {noformat}
> 2018-07-03 07:35:31,670 DEBUG  [Camel (camel-1) thread #0 - seda://systemInitializer] (LocalTopologyManagerImpl.java:169) - Error sending join request for cache org.infinispan.CONFIG to coordinator
> org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 2 from 0125f36a-2860-4107-810a-d087224c9105-21637
> following by next exception 1 second after...
> 2018-07-03 07:35:31,968  INFO  [Camel (camel-1) thread #0 - seda://systemInitializer] (JGroupsTransport.java:702) - ISPN000080: Disconnecting JGroups channel cloud11-15
> 2018-07-03 07:35:32,262 DEBUG  [Camel (camel-1) thread #0 - seda://systemInitializer] (DefaultCacheManager.java:709) - Stopping cache manager cloud11-15 on null
> 2018-07-03 07:35:32,273  WARN  [Camel (camel-1) thread #0 - seda://systemInitializer] (DefaultCacheManager.java:736) - ISPN000189: While stopping a cache or cache manager, one of its components failed to stop
> java.util.concurrent.CompletionException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
> 	at java.util.concurrent.CompletableFuture.reportJoin(Unknown Source) ~[?:1.8.0_131]
> 	at java.util.concurrent.CompletableFuture.join(Unknown Source) ~[?:1.8.0_131]
> 	at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:688) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:734) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:711) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 2 from 0125f36a-2860-4107-810a-d087224c9105-21637
> {noformat}
> 2:
> {noformat}
> 2018-07-03 09:12:55,788  WARN  [Camel (camel-1) thread #2 - seda://northboundProvider] (TransactionImpl.java:429) - ISPN000927: exception while committing
> javax.transaction.xa.XAException: null
> 	at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:180) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.transaction.xa.XaTransactionTable.rollback(XaTransactionTable.java:137) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.transaction.xa.TransactionXaAdapter.rollback(TransactionXaAdapter.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:424) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.commons.tx.TransactionImpl.rollbackResources(TransactionImpl.java:477) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:332) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:132) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.commons.tx.TransactionManagerImpl.rollback(TransactionManagerImpl.java:80) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.cache.impl.CacheImpl.tryRollback(CacheImpl.java:1801) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.cache.impl.CacheImpl.executeCommandWithInjectedTx(CacheImpl.java:1731) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1707) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1370) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:655) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:544) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> 	at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:674) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
> .....
> 	Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 1043 from 0075f36a-2860-4107-810a-d087224c9105-32070
> {noformat}
> in attached zip including 
> -infinispan logs from all nodes 
> -cluster config
> -jgroups health status csv files from nodes(comma separated, time in csv is 2hours before time in logs)



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)