[ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.sys... ] Paul Nardone updated ISPN-663: ------------------------------ Attachment: InfinispanPassivationTest.java -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.sys... ] Trustin Lee reassigned ISPN-663: -------------------------------- Assignee: Trustin Lee (was: Manik Surtani) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.sys... ] Trustin Lee commented on ISPN-663: ---------------------------------- On my laptop: 4.0.1-SNAPSHOT: PASSIVATIONLRU10 Time Taken : 90 PASSIVATIONLRU1000 Time Taken : 90 4.2.0-SNAPSHOT PASSIVATIONLRU10 Time Taken : 6380 PASSIVATIONLRU1000 Time Taken : 930 Although incorrect, if I comment out the store() method: PASSIVATIONLRU10 Time Taken : 870 PASSIVATIONLRU1000 Time Taken : 495 Therefore, I need to research: 1) which behavior (calling store() or not) is correct, and 2) why it is still 5~10 times slower without store() Can somebody answer the first one? -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.sys... ] Manik Surtani commented on ISPN-663: ------------------------------------ You are right that eviction from 4.1 onwards doesn't use a background thread since as you said, this is risky. Perhaps there is a way for the JDBM cache store to internally maintain a count of puts, and flush when this count reaches a number or a certain time slice has passed? -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.sys... ] Galder Zamarreño commented on ISPN-663: --------------------------------------- No, I don't think you can assume that. Passivation is about moving data from memory to a persistent disk cos it hasn't been used in a while, but it doesn't mean it doesn't need it any more! So, you have to assume that this is data that might be needed later and hence no loss is allowed. What I mean with this is that with passivation, data is either in memory or cache store. So, even if passivation/eviction occurs in a delayed fashion, it doesn't mean that we evict from memory and we passivate after X time. If eviction/passivation happens in a delayed fashion, it means that after X time, eviction happens and immediately data is passivated, hence no loss occurs. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.sys... ] Trustin Lee commented on ISPN-663: ---------------------------------- 1) The current DefaultDataContainer implementation seems to implement PIGGYBACK thread policy only, and that's probably why the DEFAULT thread policy doesn't trigger the eviction to be executed in a separate thread, hence passivation always piggybacked. Also, Manik said that eviction since 4.1 doesn't use a background thread. Then is DEFAULT thread policy obsolete now? (i.e. always PIGGYBACK) I guess so. The resolution I can think of is to improve DefaultDataContainer and BoundedConcurrentHashMap so that we can resurrect the old behavior (DEFAULT), but in a safe way (i.e. block put() operation if passivation backlog is too large). Maybe we could make PIGGYBACK default and introduce a new thread policy called BACKGROUND. DEFAULT could refer to PIGGYBACK or BACKGROUND instead. 2) On the other side, a unfinished JDBM write can be lost if commit() was not called yet, just like JDBC. Therefore, commit() must be called for any single store() operation. commit() can be delayed only when we are sure the entries associated with the previous uncommitted store() operations were not evicted from the memory yet - the current CacheStore interface is not given with such hint. Adding CacheStore.store(InternalCacheEntry[]) will give the CacheStore enough hint, but it will result in an additional API change - EvictionListener.postEvict() should be changed to accept multiple entries. (e.g. postEvict(Object[] keys, InternalCacheEntry[] values)), which means broken backward compatibility. I think we should go with this option in 5.0 because there is obvious advantage in performance with most CacheStores. WDYT about (1) and (2)? -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.sys... ] Galder Zamarreño commented on ISPN-663: --------------------------------------- Wrt: lost entries if replication/distribution disabled Yes, with passivation, either data is in memory or in cache store, so without clustering data could be lost. Re 1) Only Vladimir/Manik can confirm what the status of DEFAULT is. I'm not sure though how you can introduce a BACKGROUND policy that's intelligent enough to know that there's a passivation backlog. I think what you mean is that the eviction manager or the containers see how many evictions are waiting and send them to a different thread so that it can evict and passivate. However, for this particular case, the issue can be solved by using an asynchronous cache store (http://community.jboss.org/docs/DOC-14629) whereby the actual cache store time is passed to a different thread. This would emulate the previous way eviction worked where passivation happened in a different thread. Re 2) Hmmm, when would commit() not get called? If commit is not called, the operation is not to be written, so the unfinished JDBM write did not complete, so it's fine for it get lost. I'm not sure I understand the API change you suggest. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.sys... ] Trustin Lee commented on ISPN-663: ---------------------------------- I was able to achieve the similar behavior using AsyncStore: AsyncStoreConfig asyncConfig = new AsyncStoreConfig(); asyncConfig.setEnabled(true); ... cacheStoreConfig.setAsyncStoreConfig(asyncConfig); However, because AsyncStore does not delay the store() operations late enough (i.e. performs the enqueued operations ASAP), the test result remains the same. If I put an arbitrary delay in the AsyncStore's processor, I can get the numbers similar to that of 'disabled store()' (#action_12554367). To address this issue, we have to improve AsyncStore to keep the enqueued store() operations and flush them periodically or for every N modifications. (See ISPN-328) Let me continue to investigate on why it is still 5~10 times slower without store(). -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.sys... ] Trustin Lee edited comment on ISPN-663 at 10/6/10 2:27 AM: ----------------------------------------------------------- I was able to achieve the similar behavior using AsyncStore: AsyncStoreConfig asyncConfig = new AsyncStoreConfig(); asyncConfig.setEnabled(true); ... cacheStoreConfig.setAsyncStoreConfig(asyncConfig); However, because AsyncStore does not delay the store() operations late enough (i.e. performs the enqueued operations ASAP), the test result remains the same. If I put an arbitrary delay in the AsyncStore's processor, I can get the numbers similar to that of 'disabled store()'. To address this issue, we have to improve AsyncStore to keep the enqueued store() operations and flush them periodically or for every N modifications. (See ISPN-328) Let me continue to investigate on why it is still 5~10 times slower without store(). was (Author: trustin): I was able to achieve the similar behavior using AsyncStore: AsyncStoreConfig asyncConfig = new AsyncStoreConfig(); asyncConfig.setEnabled(true); ... cacheStoreConfig.setAsyncStoreConfig(asyncConfig); However, because AsyncStore does not delay the store() operations late enough (i.e. performs the enqueued operations ASAP), the test result remains the same. If I put an arbitrary delay in the AsyncStore's processor, I can get the numbers similar to that of 'disabled store()' (<a href="#action_12554367">here</a>). To address this issue, we have to improve AsyncStore to keep the enqueued store() operations and flush them periodically or for every N modifications. (See ISPN-328) Let me continue to investigate on why it is still 5~10 times slower without store(). -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.sys... ] Trustin Lee commented on ISPN-663: ---------------------------------- I was getting suboptimal performance because the new log threshold was set only to appenders. After proper log4j configuration , performance gap is within error range: <category name="org.infinispan"> <priority value="WARN"/> </category> Paul, once ISPN-328 is resolved, the performance of 4.2 will be similar or better than 4.0. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.sys... ] Trustin Lee commented on ISPN-663: ---------------------------------- That's a very good question. I was actually wondering why, too. :) When Cache.put() is called in Paul's test, the stack trace always looks like this: at org.infinispan.loaders.jdbm.JdbmCacheStore.load(JdbmCacheStore.java:135) at org.infinispan.loaders.decorators.AbstractDelegatingStore.load(AbstractDelegatingStore.java:86) at org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:138) at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:78) at org.infinispan.interceptors.ActivationInterceptor.visitPutKeyValueCommand(ActivationInterceptor.java:37) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:172) at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:120) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:87) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:58) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:273) at org.infinispan.CacheDelegate.put(CacheDelegate.java:427) at org.infinispan.CacheSupport.put(CacheSupport.java:28) at test.InfinispanPassivationTest.runTest(InfinispanPassivationTest.java:108) When passivation triggers CacheStore.store(), with AsyncStore enabled, the stack trace looks like the following: at org.infinispan.loaders.jdbm.JdbmCacheStore.store0(JdbmCacheStore.java:300) at org.infinispan.loaders.jdbm.JdbmCacheStore.store(JdbmCacheStore.java:278) at org.infinispan.loaders.decorators.AbstractDelegatingStore.store(AbstractDelegatingStore.java:46) at org.infinispan.loaders.decorators.AsyncStore.applyModificationsSync(AsyncStore.java:205) at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.put(AsyncStore.java:372) at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.innerRun(AsyncStore.java:355) at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.run(AsyncStore.java:273) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) There are two issues: 1) A put() operation results in one load() and one store() operation. Could it be optimized? Instead of loading from CacheStore first, the entry could be stored to the CacheStore first and then loaded back? (e.g. by introducing CacheStore.storeAndLoad()) 2) Since JDBM is not designed to be highly concurrent, HTree.get() and HTree.put() are being guarded by the same lock. That is why using AsyncStore doesn't simply improve the cache operation time. The viable solution would probably be writing a better HTree implementation and ditch JDBM? It would be nice if BDBJE has less restrictive license, but that's not going to happen soon. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.sys... ] Trustin Lee commented on ISPN-663: ---------------------------------- Re 1) Great. Re 2) There's no alternative with such a nice license unfortunately. It might be a good idea to borrow some code from other higher level libraries like Derby and Lucene, because they should be implementing HTree or BTree somehow. JDBM is way too old and we should have a decent disk-based CacheStore implementation optimized for Infinispan to compete with others in terms of performance. We will be able to beat them much easily in a single node scenario with it - others usually have their own optimized storage implementation. Loading happens both in 4.0 and 4.2. The difference is that passivation happens at much later point in 4.0. In 4.0: 1) (Main thread) Measurement begins 2) (Main thread) Lots of load operations as a side effect of put()s. 3) (Main thread) Measurement ends 4) (Passivation thread) Lots of delayed store operations in 4.2 w/o AsyncStore: 1) (Main thread) Measurement begins 2) (Main thread) Lots of load() and store() operations as a side effect of put()s. 3) (Main thread) Measurement ends in 4.2 w/ AsyncStore: 1) (Main thread) Measurement begins 2-a) (Main thread) Lots of load() operations as a side effect of put()s. 2-b) (Async thread) Lots of put() operations (not delayed, runs simultaneously with 2-a) 3) (Main thread) Measurement ends 4) (Async thread) Small number of unfinished put() operations So, the test itself is somewhat flawed, but it shows us a couple points of optimization. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.sys... ] Trustin Lee commented on ISPN-663: ---------------------------------- (For the record for those who read the comments) Yeah, the second optimization is not necessary because we can do batched commit w/ transactional cachestore. JDBM should support multiple simultaneous transaction or we should emulate it. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.s... ] Manik Surtani updated ISPN-663: ------------------------------- Assignee: Manik Surtani (was: Trustin Lee) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.s... ] Galder Zamarreño updated ISPN-663: ---------------------------------- Fix Version/s: 5.0.0.CR1 (was: 5.0.0.BETA2) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.s... ] Manik Surtani updated ISPN-663: ------------------------------- Fix Version/s: 5.0.0.CR3 (was: 5.0.0.CR2) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.s... ] Mircea Markus updated ISPN-663: ------------------------------- Fix Version/s: 5.0.0.FINAL (was: 5.0.0.CR4) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.s... ] Trustin Lee commented on ISPN-663: ---------------------------------- Yes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira