[infinispan-dev] CD datastore inserts slow

[Sanne Grinovero]

Hi all,
some context, as this conversation started offlist:
performance problems with CapeDwarf's storage system, as you probably
know based on Infinispan.

So, storing some thousand entities in a single transaction takes a
little more than 1 second, while running each put operation in a
single transaction takes ~50 seconds.

I'm afraid that's nothing new?
When you use indexing, this is expected: it shines much better in
parallel operations than in sequential, individual transactions.

I expect some of the changes we have planned for Infinispan 7 to
improve on this, but it's unlikely to get faster than a single order
of magnitude.

These are some issues which affected my previous performance tests:

https://issues.jboss.org/browse/ISPN-3690
https://issues.jboss.org/browse/ISPN-1764
https://issues.jboss.org/browse/ISPN-3831
https://issues.jboss.org/browse/ISPN-3891
https://issues.jboss.org/browse/ISPN-3905

And of course the Hibernate Search and Lucene upgrades ;-)

(in no particular order, as relevance depends on the test and on what
metric I'm looking to improve)

If you need better figures the best thing we can do is measure your
specific test, and see if we need to add further tasks to my wishlist
or if it's enough to prioritize these.

Currently I think if you expect to use a transaction for each
operation, there is nothing wrong in CD as unfortunately these figures
match my expectations on a non-tuned system.. you could tune things a
bit, like better sizing of packets for JGroups and network stack, but
I wouldn't expect much higher figures without patching things in
Infinispan.

For the record, I'm more concerned that storing "a few thousand"
entities in a single TX takes  a full second: that's not expected, but
my guess is that in this specific case you're not warming up the JVM
nor repeating the test. Having batching enabled, I'd expect this to be
in the order of millions / second.

Sanne

Previous discussion - sorry, the formatting got crazy:

> 17:20:17,640 INFO  [com.google.appengine.tck.benchmark.ObjectifyBenchmarkTest] (default task-1) >>>> Save [2000] time: 1149ms
> 17:21:55,659 INFO  [com.google.appengine.tck.benchmark.ObjectifyBenchmarkTest] (default task-1) >>>> Save [2000] time: 50675ms
> The first time is when whole 2000 is inserted in single Tx,
> the other one is when there is no existing Tx.
> OK, in reality doing non-Tx for whole batch is a no go,
> so on that side we're good.
> But I would still like to see non-Tx do much better then 1sec vs. 50sec (for 2k entities).
> -Ales
> On 28 May 2014, at 17:20, Mircea Markus <mmarkus at redhat.com> wrote:
> Can we isolate the tests somehow?
> On May 28, 2014, at 15:53, Ales Justin <ales.justin at gmail.com> wrote:
> Objectify is a popular GAE ORM, so it does plenty of persistence magic.
> Hence my simple answer is no.
> But it should be easy to take latest CapeDwarf release -- 2.0.0.CR2,
> and run + debug things, and see why it's so slow.
> Either Terry or me is gonna add this test to GAE TCK.
> I'll shoot an email on how to run this against CapeDwarf once it's added.
> Looking at this:
> * https://gist.github.com/alesj/1d24ad24dfbef8b5e12c
> It looks like a problem of Tx usage -- Tx per Cache::put.
> This snapshot only shows the time spent within Infinispan. Out of this time, 9% is spent indeed within cache.put. How much time does the test spend doing non-infinispan related activities? Where I'm trying to get to is: are we sure Infinispan is the culprit?
> It could be that our default config isn't suited to handle such things.
> So we're open for any suggestions.
> ---
> I've added this test to GAE TCK:
> * https://github.com/GoogleCloudPlatform/appengine-tck/blob/master/core/benchmark/src/test/java/com/google/appengine/tck/benchmark/ObjectifyBenchmarkTest.java
> You need CapeDwarf 2.0.0.CR2:
> * http://capedwarf.org/downloads/
> And clone GAE TCK:
> * https://github.com/GoogleCloudPlatform/appengine-tck
> Run CapeDwarf:
> * https://github.com/capedwarf/capedwarf-blue/blob/master/README.md (see (5))
> Then run the ObjectifyBenchmarkTest:
> * cd <GAE_TCK_HOME>
> * mvn clean install
> * cd core/benchmark
> * mvn clean install -Pcapedwarf,benchmark
> Ping me for any issues.
> Thanks!
> -Ales
> Cheers,
> --
> Mircea Markus
> Infinispan lead (www.infinispan.org)