[infinispan-dev] Cachestores performance

[Mircea Markus]

On 2 Jul 2013, at 15:32, Radim Vansa <rvansa at redhat.com> wrote:

> | Given such a scenario, I am not interested at all in synchronous
> | storage. Before we commit into a design which is basically assuming
> | the need for synchronous storage guarantees, I'd like to understand
> | what kind of use case it's aiming to solve.
> 
> You're right that I may miss the wider perspective and that I am too much focused on storing stuff reliably when we have reliability assured by multiple owners.
> 
> | 
> | Only then we would be able to pick a design (or multiple ones); for my
> | use case the proposal from Karsten seems excellent, so I'm wondering
> | why I should be looking for alternatives, and wondering why everyone
> | is still wasting time on different discussions :-D
> 
> Karsten implementation has one major drawback: all the keys are stored in-memory, and there's no search structure for overflow. For use case we've had with bigger keys and very small values this is simply not an option. However, we have the LevelDB JNI implementation for this purpose and the numbers are not bad in non-syncing mode. Do we want to have some efficient no-dependency cache-store? If the question is no, okay, let use recommend LevelDB JNI for this purposes.
I'd say yes: I know users that were put off by using/setting up an external DB to store data on the disk. Some are on this mailing list and may want to comment :-)
> 
> | 
> | I'm pretty sure there is people looking forward for a synch-CacheStore
> | too: if you could nail down such a scenario however I'm pretty sure
> | that some other considerations would not be taken into account (like
> | consistency of data when reactivating a dormant node), so I suspect
> | that just implementing such a component would actually not make any
> | new architecture possible, as you would get blocked by other problems
> | which need to be solved too.. better define all expectations asap!
> | 
> | To me this thread smells of needing the off-heap Direct Memory buffers
> | which I suggested [long time ago] to efficiently offload internal
> | buffers,
> 
> Yes, when it comes to reducing GC pauses, that is the best option. However, you can't have enough RAM to handle everything in-memory (off-heap), persistent storage for overflow is a must-have.
> 
> | but failing to recognise this we're pushing responsibility to
> | an epic level complex CacheStore.. guys let's not forget that a mayor
> | bottleneck of CacheStores today is the SPI it has to implement, we
> | identified several limitations in the contract in the past which
> | prevent a superior efficiency: we're working towards a mayor release
> | now so I'd rather focus on the API changes which will make it possible
> | to get decent performance even without changing any storage engine..
> | I'm pretty sure Cassandra (to pick one) doesn't scale too bad.
> 
> The issues common to all local cache-stores are definitely something to do. I just don't feel enough into the problematics to handle them myself - that's why I jumped onto implementing just another cache store.

Cheers,
-- 
Mircea Markus
Infinispan lead (www.infinispan.org)