Hi Andreas

Have you tried to see much worse your performance is without write-behind? I wouldn't expect write-behind to help a lot with SingleFileStore (file-store): since it never calls fsync, writes only go to the OS write-behind buffers before returning.

Regarding the error per se, I'm afraid the error message doesn't say much. You should enable DEBUG logging for org.infinispan.persistence and see if the async writer threads log any "Failed to process async modifications" messages and create a bug in JIRA. It might also help to get a thread dump after you've seen the error.

Cheers
Dan



On Fri, Nov 21, 2014 at 12:28 PM, Andreas Kruthoff <andreas.kruthoff@nexustelecom.com> wrote:
Hi dev


I'm running infinispan-7.0.1 (will soon upgrade to 7.0.2).

I've configured 2 distributed caches, both are similar.

Example cache1:
...
<distributed-cache name="infinicache-lbd-imei" mode="SYNC"
async-marshalling="false" owners="2"
     l1-lifespan="600000" l1-cleanup-interval="60000" statistics="false">
   <eviction strategy="LIRS" max-entries="20000000"/>
   <expiration max-idle="86400000" interval="30000"/>
   <persistence passivation="true">
     <file-store
         path="/data1/infini-lbd"
         preload="true" shared="true" purge="false">
       <write-behind/>
     </file-store>
   </persistence>
</distributed-cache>
...

Today, I've seen that one of the two caches suddenly doesn't exist
anymore somehow (JMX access to numberOfEntries returns nothing?!).

After looking into the application logfile, I've found this:

2014-11-19 13:45:47,310 ERROR
[AsyncStoreCoordinator-infinicache-lbd-imei] AsyncCacheWriter.java:267
ISPN000055: Unexpected error in AsyncStoreCoordinator thread.
AsyncCacheWriter is dead!
org.infinispan.util.concurrent.TimeoutException: ISPN000233: Waiting on
work threads latch failed:
java.util.concurrent.CountDownLatch@6e30928d[Count = 1]
         at
org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreCoordinator.workerThreadsAwait(AsyncCacheWriter.java:297)
~[infinispan-embedded.jar:7.0.1.Final]
         at
org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreCoordinator.run(AsyncCacheWriter.java:254)
~[infinispan-embedded.jar:7.0.1.Final]
         at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_72]



No error before nor after this entry. It looks like it's related to
<write-behind/> to me, so I'll remove that setting.

But I'd like to write asynchronously for best performance.

Any help would be appreciated!

-andreas

This email and any attachment may contain confidential information which is intended for use only by the addressee(s) named above. If you received this email by mistake, please notify the sender immediately, and delete the email from your system. You are prohibited from copying, disseminating or otherwise using the email or any attachment.

_______________________________________________
infinispan-dev mailing list
infinispan-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev