[infinispan-issues] [JBoss JIRA] (ISPN-9351) nodes did not join to cluster because of timeoutException

[Robert Cernak (JIRA)]

Robert Cernak created ISPN-9351:
-----------------------------------

             Summary: nodes did not join to cluster because of timeoutException
                 Key: ISPN-9351
                 URL: https://issues.jboss.org/browse/ISPN-9351
             Project: Infinispan
          Issue Type: Bug
            Reporter: Robert Cernak


I was trying to connect 15nodes to 1 cluster, however nodes did not join.
In logs, all nodes mostly had 2 kinds of exceptions, all caused by TimeoutException:

1:
{noformat}
2018-07-03 07:35:31,670 DEBUG  [Camel (camel-1) thread #0 - seda://systemInitializer] (LocalTopologyManagerImpl.java:169) - Error sending join request for cache org.infinispan.CONFIG to coordinator
org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 2 from 0125f36a-2860-4107-810a-d087224c9105-21637

following by next exception 1 second after...
2018-07-03 07:35:31,968  INFO  [Camel (camel-1) thread #0 - seda://systemInitializer] (JGroupsTransport.java:702) - ISPN000080: Disconnecting JGroups channel cloud11-15
2018-07-03 07:35:32,262 DEBUG  [Camel (camel-1) thread #0 - seda://systemInitializer] (DefaultCacheManager.java:709) - Stopping cache manager cloud11-15 on null
2018-07-03 07:35:32,273  WARN  [Camel (camel-1) thread #0 - seda://systemInitializer] (DefaultCacheManager.java:736) - ISPN000189: While stopping a cache or cache manager, one of its components failed to stop
java.util.concurrent.CompletionException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
	at java.util.concurrent.CompletableFuture.reportJoin(Unknown Source) ~[?:1.8.0_131]
	at java.util.concurrent.CompletableFuture.join(Unknown Source) ~[?:1.8.0_131]
	at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:688) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:734) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:711) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 2 from 0125f36a-2860-4107-810a-d087224c9105-21637
{noformat}


2:
{noformat}
2018-07-03 09:12:55,788  WARN  [Camel (camel-1) thread #2 - seda://northboundProvider] (TransactionImpl.java:429) - ISPN000927: exception while committing
javax.transaction.xa.XAException: null
	at org.infinispan.transaction.impl.TransactionCoordinator.rollback(TransactionCoordinator.java:180) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.transaction.xa.XaTransactionTable.rollback(XaTransactionTable.java:137) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.transaction.xa.TransactionXaAdapter.rollback(TransactionXaAdapter.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:424) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.tx.TransactionImpl.rollbackResources(TransactionImpl.java:477) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:332) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.tx.TransactionImpl.rollback(TransactionImpl.java:132) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.tx.TransactionManagerImpl.rollback(TransactionManagerImpl.java:80) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.CacheImpl.tryRollback(CacheImpl.java:1801) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.CacheImpl.executeCommandWithInjectedTx(CacheImpl.java:1731) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1707) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1370) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:655) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:544) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:358) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:674) [infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
.....
	Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 1043 from 0075f36a-2860-4107-810a-d087224c9105-32070
{noformat}

in attached zip including 
-infinispan logs from all nodes 
-cluster config
-jgroups health status csv files from nodes(comma separated, time in csv is 2hours before time in logs)



--
This message was sent by Atlassian JIRA
(v7.5.0#75005)