[infinispan-dev] state transfer timed out, where to configure?

[Andreas Kruthoff]

I've added my vote.

I've increased the chunk size-from 512 entries (default) to 10'000
entries, as from the xsd: "The number of cache entries to batch in each
transfer." I thought that an increased chunk-size helps to populate the
cache faster? Or is it the opposite? One entry is very small, just a
Long for key and value.


On 02/09/2015 02:31 PM, Dan Berindei wrote:
> I'm not 100% sure, but this sounds like you're hitting ISPN-5141 [1].
> Please add your vote, it might make it into 7.2.0.
>
> Lowering the state transfer chunkSize may also help, especially if your
> values are large. The chunkSize is the number of entries in a single
> chunk, you should aim at making the average chunk <= 500KB.
>
> Cheers
> Dan
>
> [1] https://issues.jboss.org/browse/ISPN-5141
>
>
> On Mon, Feb 9, 2015 at 11:48 AM, Andreas Kruthoff
> <andreas.kruthoff at nexustelecom.com
> <mailto:andreas.kruthoff at nexustelecom.com>> wrote:
>
>     Thanks. The cache is a distributed cache with 3 nodes and replication
>     factor 2. It contains ~40 million entries. As soon as a node has loaded
>     all entries from disk, new entries are added to this node with high
>     frequency (several thousand/s).
>
>     The other nodes might not be up yet. It takes 'forever' to bring them up
>     then...?
>
>     On 02/03/2015 12:32 PM, Dan Berindei wrote:
>     > Hi Andreas
>     >
>     > Yes, that's the one. 1200s sounds like a huge timeout, though, how big
>     > is your cache?
>     >
>     > Cheers
>     > Dan
>     >
>     >
>     > On Thu, Jan 29, 2015 at 7:25 PM, Andreas Kruthoff
>     > <andreas.kruthoff at nexustelecom.com
>     <mailto:andreas.kruthoff at nexustelecom.com>
>      > <mailto:andreas.kruthoff at nexustelecom.com
>     <mailto:andreas.kruthoff at nexustelecom.com>>> wrote:
>      >
>      >     Hi dev
>      >
>      >
>      >     I'm running into the following exception on the 3rd node out
>     of 2.
>      >     Distributed cluster, file store with a few millions of entries.
>      >
>      >     The 3rd node times out during startup, I think. "Initial
>     state transfer
>      >     timed out". How can I configure/increase the timeout in my
>      >     infinispan.xml?
>      >
>      >     Is it <state-transfer timeout="1200000"/> within <distributed
>     cache/> ?
>      >
>      >
>      >     thanks for help
>      >
>      >     -andreas
>      >
>      >
>      >     Exception in thread "main"
>     org.infinispan.commons.CacheException: Unable
>      >     to invoke method public void
>      >
>       org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete()
>      >     th
>      >     rows java.lang.InterruptedException on object of type
>      >     StateTransferManagerImpl
>      >               at
>      >
>       org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:170)
>      >               at
>      >
>       org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
>      >               at
>      >
>       org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
>      >               at
>      >
>       org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
>      >               at
>      >
>       org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
>      >               at
>      >
>       org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)
>      >               at
>      >     org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:813)
>      >               at
>      >
>       org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:584)
>      >               at
>      >
>       org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:539)
>      >               at
>      >
>       org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:416)
>      >               at
>      >     ch.nexustelecom.lbd.engine.ImeiCache.init(ImeiCache.java:49)
>      >               at
>      >
>       ch.nexustelecom.dexclient.engine.DefaultDexClientEngine.init(DefaultDexClientEngine.java:120)
>      >               at
>      >
>       ch.nexustelecom.dexclient.DexClient.initClient(DexClient.java:169)
>      >               at
>      >
>       ch.nexustelecom.dexclient.tool.DexClientManager.startup(DexClientManager.java:196)
>      >               at
>      >
>       ch.nexustelecom.dexclient.tool.DexClientManager.main(DexClientManager.java:83)
>      >     Caused by: org.infinispan.commons.CacheException: Initial
>     state transfer
>      >     timed out for cache infinicache-lbd-imei on m4sxhpsrm672-11986
>      >               at
>      >
>       org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:216)
>      >               at
>     sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>      >               at
>      >
>       sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>      >               at
>      >
>       sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>      >               at java.lang.reflect.Method.invoke(Method.java:606)
>      >               at
>      >
>       org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
>      >               ... 14 more
>      >
>      >     This email and any attachment may contain confidential
>     information
>      >     which is intended for use only by the addressee(s) named
>     above. If
>      >     you received this email by mistake, please notify the sender
>      >     immediately, and delete the email from your system. You are
>      >     prohibited from copying, disseminating or otherwise using the
>     email
>      >     or any attachment.
>      >
>      >     _______________________________________________
>      >     infinispan-dev mailing list
>      > infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>
>     <mailto:infinispan-dev at lists.jboss.org
>     <mailto:infinispan-dev at lists.jboss.org>>
>      > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>      >
>      >
>
>     This email and any attachment may contain confidential information
>     which is intended for use only by the addressee(s) named above. If
>     you received this email by mistake, please notify the sender
>     immediately, and delete the email from your system. You are
>     prohibited from copying, disseminating or otherwise using the email
>     or any attachment.
>
>     _______________________________________________
>     infinispan-dev mailing list
>     infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>

This email and any attachment may contain confidential information which is intended for use only by the addressee(s) named above. If you received this email by mistake, please notify the sender immediately, and delete the email from your system. You are prohibited from copying, disseminating or otherwise using the email or any attachment.