[infinispan-dev] AsyncCacheWriter is dead

[Dan Berindei]

Hi Andreas

Have you tried to see much worse your performance is without write-behind?
I wouldn't expect write-behind to help a lot with SingleFileStore
(file-store): since it never calls fsync, writes only go to the OS
write-behind buffers before returning.

Regarding the error per se, I'm afraid the error message doesn't say much.
You should enable DEBUG logging for org.infinispan.persistence and see if
the async writer threads log any "Failed to process async modifications"
messages and create a bug in JIRA. It might also help to get a thread dump
after you've seen the error.

Cheers
Dan



On Fri, Nov 21, 2014 at 12:28 PM, Andreas Kruthoff <
andreas.kruthoff at nexustelecom.com> wrote:

> Hi dev
>
>
> I'm running infinispan-7.0.1 (will soon upgrade to 7.0.2).
>
> I've configured 2 distributed caches, both are similar.
>
> Example cache1:
> ...
> <distributed-cache name="infinicache-lbd-imei" mode="SYNC"
> async-marshalling="false" owners="2"
>      l1-lifespan="600000" l1-cleanup-interval="60000" statistics="false">
>    <eviction strategy="LIRS" max-entries="20000000"/>
>    <expiration max-idle="86400000" interval="30000"/>
>    <persistence passivation="true">
>      <file-store
>          path="/data1/infini-lbd"
>          preload="true" shared="true" purge="false">
>        <write-behind/>
>      </file-store>
>    </persistence>
> </distributed-cache>
> ...
>
> Today, I've seen that one of the two caches suddenly doesn't exist
> anymore somehow (JMX access to numberOfEntries returns nothing?!).
>
> After looking into the application logfile, I've found this:
>
> 2014-11-19 13:45:47,310 ERROR
> [AsyncStoreCoordinator-infinicache-lbd-imei] AsyncCacheWriter.java:267
> ISPN000055: Unexpected error in AsyncStoreCoordinator thread.
> AsyncCacheWriter is dead!
> org.infinispan.util.concurrent.TimeoutException: ISPN000233: Waiting on
> work threads latch failed:
> java.util.concurrent.CountDownLatch at 6e30928d[Count = 1]
>          at
>
> org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreCoordinator.workerThreadsAwait(AsyncCacheWriter.java:297)
> ~[infinispan-embedded.jar:7.0.1.Final]
>          at
>
> org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreCoordinator.run(AsyncCacheWriter.java:254)
> ~[infinispan-embedded.jar:7.0.1.Final]
>          at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_72]
>
>
>
> No error before nor after this entry. It looks like it's related to
> <write-behind/> to me, so I'll remove that setting.
>
> But I'd like to write asynchronously for best performance.
>
> Any help would be appreciated!
>
> -andreas
>
> This email and any attachment may contain confidential information which
> is intended for use only by the addressee(s) named above. If you received
> this email by mistake, please notify the sender immediately, and delete the
> email from your system. You are prohibited from copying, disseminating or
> otherwise using the email or any attachment.
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20141124/8962f2fb/attachment-0001.html