[infinispan-issues] [JBoss JIRA] Commented: (ISPN-663) Eviction with passivation using JdbmCacheStore is 100 times slower in 4.1 vs 4.0

[Trustin Lee (JIRA)]

    [ https://jira.jboss.org/browse/ISPN-663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12554402#action_12554402 ] 

Trustin Lee commented on ISPN-663:
----------------------------------

I observed that store() is also called in 4.0 if I increase the length of the test loop from 1000 to 10000000.  However, 4.0 ends up with OutOfMemoryError with the increased loop count while 4.2 passes the test with minimal heap expansion.

This experiment revealed another difference: 4.0 uses SimpleDataContainer which uses ConcurrentHashMap.  4.2 uses DefaultDataContainer which uses BoundConcurrentHashMap.  BoundConcurrentHashMap calls eviction listener directly, which triggers PassivationManagerImpl.passivate() (this eventually triggers store()), while neither ConcurrentHashMap nor SimpleDataContainer does any eviction / passivation.  Instead, 4.0 uses a ScheduledThreadPoolExecutor to run eviction / passivation in background.

Using ScheduledThreadPoolExecutor for eviction sounds somewhat risky because passivation might not catch up very fast in-memory put()s as demonstrated by the test.  Maybe that's why 4.1 changed how eviction works?  This answers the first question.  To fix this issue:

    1) CacheStore could provide a batch store operation such as CacheStore.store(InternalCacheEntry[] entries) and let user configure batched passivation (e.g. passivate every 128 entries) OR
    2) Run eviction in a background thread as before but make sure the task queue does not grow, using a small bounded queue OR
    3) Make (1) and (2) configurable

Maybe in 5.0?

Let me continue investigating why it is still 5~10 times slower without passivation.

> Eviction with passivation using JdbmCacheStore is 100 times slower in 4.1 vs 4.0
> --------------------------------------------------------------------------------
>
>                 Key: ISPN-663
>                 URL: https://jira.jboss.org/browse/ISPN-663
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Eviction, Loaders and Stores
>    Affects Versions: 4.1.0.Final
>         Environment: Win32 JRE 1.6.0_21
>            Reporter: Paul Nardone
>            Assignee: Trustin Lee
>             Fix For: 4.2.0.BETA1, 4.2.0.Final
>
>         Attachments: InfinispanPassivationTest.java
>
>
> Eviction with passivation enabled using the JdbmCacheStore appears to be significantly slower in 4.1.0.FINAL vs 4.0.0.FINAL.
> The degredation in performance is so signficant to make it impossible  to use
> The  performance issue seems to due as the JdbmCacheStore synching the filesystem via FileDescriptor.sync() or similar which occurs during every object passivation and each passivation occurs as a new object is added beyond the EvictionMaxEntries capacity.
> The attached test inserts 1000 values into two caches
> Both caches use a JdbmCacheStore and  LRU
> PASSIVATIONLRU10 runs with cache with EvictionMaxEntries  10
> PASSIVATIONLRU1000 runs with cache with EvictionMaxEntries  1000
> 4.1.0.FINAL
> PASSIVATIONLRU10 Time Taken : 51704
> PASSIVATIONLRU1000 Time Taken : 4484
> 4.0.0.FINAL
> PASSIVATIONLRU10 Time Taken : 281
> PASSIVATIONLRU1000 Time Taken : 141
> 4.2.0.ALPHA2
> PASSIVATIONLRU10 Time Taken : 51047
> PASSIVATIONLRU1000 Time Taken : 5156

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira