11:42 a.m.
Hi all,
some context, as this conversation started offlist:
performance problems with CapeDwarf's storage system, as you probably
know based on Infinispan.
So, storing some thousand entities in a single transaction takes a
little more than 1 second, while running each put operation in a
single transaction takes ~50 seconds.
I'm afraid that's nothing new?
When you use indexing, this is expected: it shines much better in
parallel operations than in sequential, individual transactions.
I expect some of the changes we have planned for Infinispan 7 to
improve on this, but it's unlikely to get faster than a single order
of magnitude.
These are some issues which affected my previous performance tests:
https://issues.jboss.org/browse/ISPN-3690
https://issues.jboss.org/browse/ISPN-1764
https://issues.jboss.org/browse/ISPN-3831
https://issues.jboss.org/browse/ISPN-3891
https://issues.jboss.org/browse/ISPN-3905
And of course the Hibernate Search and Lucene upgrades ;-)
(in no particular order, as relevance depends on the test and on what
metric I'm looking to improve)
If you need better figures the best thing we can do is measure your
specific test, and see if we need to add further tasks to my wishlist
or if it's enough to prioritize these.
Currently I think if you expect to use a transaction for each
operation, there is nothing wrong in CD as unfortunately these figures
match my expectations on a non-tuned system.. you could tune things a
bit, like better sizing of packets for JGroups and network stack, but
I wouldn't expect much higher figures without patching things in
Infinispan.
For the record, I'm more concerned that storing "a few thousand"
entities in a single TX takes a full second: that's not expected, but
my guess is that in this specific case you're not warming up the JVM
nor repeating the test. Having batching enabled, I'd expect this to be
in the order of millions / second.
Sanne
Previous discussion - sorry, the formatting got crazy:
17:20:17,640 INFO
[com.google.appengine.tck.benchmark.ObjectifyBenchmarkTest] (default task-1)
>>>> Save [2000] time: 1149ms
17:21:55,659 INFO [com.google.appengine.tck.benchmark.ObjectifyBenchmarkTest] (default
task-1) >>>> Save [2000] time: 50675ms
The first time is when whole 2000 is inserted in single Tx,
the other one is when there is no existing Tx.
OK, in reality doing non-Tx for whole batch is a no go,
so on that side we're good.
But I would still like to see non-Tx do much better then 1sec vs. 50sec (for 2k
entities).
-Ales
On 28 May 2014, at 17:20, Mircea Markus <mmarkus(a)redhat.com> wrote:
Can we isolate the tests somehow?
On May 28, 2014, at 15:53, Ales Justin <ales.justin(a)gmail.com> wrote:
Objectify is a popular GAE ORM, so it does plenty of persistence magic.
Hence my simple answer is no.
But it should be easy to take latest CapeDwarf release -- 2.0.0.CR2,
and run + debug things, and see why it's so slow.
Either Terry or me is gonna add this test to GAE TCK.
I'll shoot an email on how to run this against CapeDwarf once it's added.
Looking at this:
* https://gist.github.com/alesj/1d24ad24dfbef8b5e12c
It looks like a problem of Tx usage -- Tx per Cache::put.
This snapshot only shows the time spent within Infinispan. Out of this time, 9% is spent
indeed within cache.put. How much time does the test spend doing non-infinispan related
activities? Where I'm trying to get to is: are we sure Infinispan is the culprit?
It could be that our default config isn't suited to handle such things.
So we're open for any suggestions.
---
I've added this test to GAE TCK:
*
https://github.com/GoogleCloudPlatform/appengine-tck/blob/master/core/ben...
You need CapeDwarf 2.0.0.CR2:
* http://capedwarf.org/downloads/
And clone GAE TCK:
* https://github.com/GoogleCloudPlatform/appengine-tck
Run CapeDwarf:
* https://github.com/capedwarf/capedwarf-blue/blob/master/README.md (see (5))
Then run the ObjectifyBenchmarkTest:
* cd <GAE_TCK_HOME>
* mvn clean install
* cd core/benchmark
* mvn clean install -Pcapedwarf,benchmark
Ping me for any issues.
Thanks!
-Ales
Cheers,
--
Mircea Markus
Infinispan lead (www.infinispan.org)
7:58 a.m.
New subject: CD datastore inserts slow
So, storing some thousand entities in a single transaction takes a
little more than 1 second, while running each put operation in a
single transaction takes ~50 seconds.
For the record, I'm more concerned that storing "a few thousand"
entities in a single TX takes a full second: that's not expected, but
my guess is that in this specific case you're not warming up the JVM
nor repeating the test. Having batching enabled, I'd expect this to be
in the order of millions / second.
Uh, this was actually my bad -- with single Tx it also takes long time ...
(@Sanne -- be concerned, be very concerned ... ;-)
14:46:14,669 INFO [com.google.appengine.tck.benchmark.ObjectifyBenchmarkTest] (default
task-1) >>>> N = 6000
14:46:17,418 INFO [com.google.appengine.tck.benchmark.ObjectifyBenchmarkTest] (default
task-1) >>>> Save [6000] time: 2461ms
14:47:53,450 INFO [com.google.appengine.tck.benchmark.ObjectifyBenchmarkTest] (default
task-1) >>>> Full Tx time: 98576ms
14:50:19,721 INFO [com.google.appengine.tck.benchmark.ObjectifyBenchmarkTest] (default
task-1) >>>> Save [6000] time: 146246ms
The 1st "Save" is fast, as this is just pilling-up the entities within same Tx.
It's the "Full Tx" that is relevant here -- which is ~100sec :-(.
The 2nd "Save" is w/o Tx.
-Ales
10:40 a.m.
New subject: CD datastore inserts slow
Hi Ales,
I don't have CD installed anymore but will try it out. Do you have
some pointers to set up and run this benchmark?
Sanne
On 3 June 2014 13:58, Ales Justin <ales.justin(a)gmail.com> wrote:
So, storing some thousand entities in a single transaction takes a
little more than 1 second, while running each put operation in a
single transaction takes ~50 seconds.
For the record, I'm more concerned that storing "a few thousand"
entities in a single TX takes a full second: that's not expected, but
my guess is that in this specific case you're not warming up the JVM
nor repeating the test. Having batching enabled, I'd expect this to be
in the order of millions / second.
Uh, this was actually my bad -- with single Tx it also takes long time ...
(@Sanne -- be concerned, be very concerned ... ;-)
14:46:14,669 INFO
[com.google.appengine.tck.benchmark.ObjectifyBenchmarkTest] (default task-1)
>>>> N = 6000
14:46:17,418 INFO
[com.google.appengine.tck.benchmark.ObjectifyBenchmarkTest] (default task-1)
>>>> Save [6000] time: 2461ms
14:47:53,450 INFO
[com.google.appengine.tck.benchmark.ObjectifyBenchmarkTest] (default task-1)
>>>> Full Tx time: 98576ms
14:50:19,721 INFO
[com.google.appengine.tck.benchmark.ObjectifyBenchmarkTest] (default task-1)
>>>> Save [6000] time: 146246ms
The 1st "Save" is fast, as this is just pilling-up the entities within same
Tx.
It's the "Full Tx" that is relevant here -- which is ~100sec :-(.
The 2nd "Save" is w/o Tx.
-Ales
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
1:39 p.m.
New subject: CD datastore inserts slow
Here:
---
I've added this test to GAE TCK:
*
https://github.com/GoogleCloudPlatform/appengine-tck/blob/master/core/ben...
You need CapeDwarf 2.0.0.CR2:
* http://capedwarf.org/downloads/
And clone GAE TCK:
* https://github.com/GoogleCloudPlatform/appengine-tck
Run CapeDwarf:
* https://github.com/capedwarf/capedwarf-blue/blob/master/README.md (see (5))
Then run the ObjectifyBenchmarkTest:
* cd <GAE_TCK_HOME>
* mvn clean install
* cd core/benchmark
* mvn clean install -Pcapedwarf,benchmark
Ping me for any issues.
Thanks!
-Ales
On 03 Jun 2014, at 17:40, Sanne Grinovero <sanne(a)infinispan.org> wrote:
Hi Ales,
I don't have CD installed anymore but will try it out. Do you have
some pointers to set up and run this benchmark?
Sanne
11:21 a.m.
New subject: CD datastore inserts slow
Hi Ales,
I've looked at the benchmark.
The main problem is that the high amount of IndexWriter open/close
cycles are quite CPU intensive and also generate a huge amount of
temporary objects; as previously mentioned the easy workaround is to
write objects in batches or transactions, but we can do other
improvements too of course.
My mainline is ~320 seconds when running CapeDwarf as-is; when
reconfiguring the backend to use the more efficient Infinispan
IndexManager (as we had actually previously suggested!) we avoid the
open/close cost but we still have the synchronous flush, so
performance improves but is still not good: 215 seconds.
This last scenario is harder to improve further, as the bottleneck
becomes the chattiness of the Index storage in Infinispan, combined
with the CacheStore usage. We can change the default buffer sizes for
the index to improve it but it would be much more effective if you
could batch multiple changes together.
Also we should look at improving the CacheStore performance, which is
the main bottleneck, although its cost gets amplified by the
inefficient use we do at higher level.. I will try some more
experiments with this.
To compare, I reconfigured it to use local-only NTR backends (not
suited for clustering) and so doing I found another problem: all of
your caches are sharing the same index!
By reconfiguring each cache to use and independent path to isolate the
indexes, the benchmark completes in 20 seconds, and these remaining 20
seconds are all spent in CapeDwarf code, with some more object
allocations from Infinispan core.
So in short, even perfectly tuning indexing away, you can' t get
faster than 20 seconds without making some more changes in CD and
Infinispan Core as well.. would that be an acceptable goal?
Also, question: do you really need both clustering and the persistent
CacheStore for the TCK tests? disabling these aspects would massively
speedup the whole thing.
Sanne
On 3 June 2014 19:39, Ales Justin <ales.justin(a)gmail.com> wrote:
Here:
---
I've added this test to GAE TCK:
*
https://github.com/GoogleCloudPlatform/appengine-tck/blob/master/core/ben...
You need CapeDwarf 2.0.0.CR2:
* http://capedwarf.org/downloads/
And clone GAE TCK:
* https://github.com/GoogleCloudPlatform/appengine-tck
Run CapeDwarf:
* https://github.com/capedwarf/capedwarf-blue/blob/master/README.md (see
(5))
Then run the ObjectifyBenchmarkTest:
* cd <GAE_TCK_HOME>
* mvn clean install
* cd core/benchmark
* mvn clean install -Pcapedwarf,benchmark
Ping me for any issues.
Thanks!
-Ales
On 03 Jun 2014, at 17:40, Sanne Grinovero <sanne(a)infinispan.org> wrote:
Hi Ales,
I don't have CD installed anymore but will try it out. Do you have
some pointers to set up and run this benchmark?
Sanne
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
7:29 a.m.
New subject: CD datastore inserts slow
To compare, I reconfigured it to use local-only NTR backends (not
suited for clustering) and so doing I found another problem: all of
your caches are sharing the same index!
By reconfiguring each cache to use and independent path to isolate the
indexes, the benchmark completes in 20 seconds, and these remaining 20
seconds are all spent in CapeDwarf code, with some more object
allocations from Infinispan core.
How do you configure this index isolation?
(how exactly does this look in standalone.xml?)
So in short, even perfectly tuning indexing away, you can' t get
faster than 20 seconds without making some more changes in CD and
Infinispan Core as well.. would that be an acceptable goal?
20sec vs 320sec definitely sounds better. ;-)
What kind of changes would one have to make in CD?
Also, question: do you really need both clustering and the
persistent
CacheStore for the TCK tests? disabling these aspects would massively
speedup the whole thing.
Wdym?
You mean we could do w/o cache store in tests?
Sure, but we want to be as close as possible to real usage scenario.
-Ales
8:12 a.m.
New subject: CD datastore inserts slow
On 19 June 2014 13:29, Ales Justin <ales.justin(a)gmail.com> wrote:
> To compare, I reconfigured it to use local-only NTR backends
(not
> suited for clustering) and so doing I found another problem: all of
> your caches are sharing the same index!
> By reconfiguring each cache to use and independent path to isolate the
> indexes, the benchmark completes in 20 seconds, and these remaining 20
> seconds are all spent in CapeDwarf code, with some more object
> allocations from Infinispan core.
How do you configure this index isolation?
(how exactly does this look in standalone.xml?)
I did it locally, can send you a pull request.
But to clarify: this would disable clustering! Do you really want
that? the first option is safer, I'd just change the IndexManager.
I can send you a pull request for both variations of the
configuration, just let me know which file it is I have to patch: I
tried to look into the repository for capedwarf-jboss-as, but the
configuration file in there seems quite different than the one in
CapeDwarf blue distribution. So I'm not sure if I'm looking in the
right place, or maybe the wrong branch?
> So in short, even perfectly tuning indexing away, you can' t get
> faster than 20 seconds without making some more changes in CD and
> Infinispan Core as well.. would that be an acceptable goal?
20sec vs 320sec definitely sounds better. ;-)
What kind of changes would one have to make in CD?
General optimisation, the usual stuff: profile it, rinse and repeat. I
didn't look at details but I remember a priority would be to optimise
some HashMap lookups for internal services and components.
> Also, question: do you really need both clustering and the
persistent
> CacheStore for the TCK tests? disabling these aspects would massively
> speedup the whole thing.
Wdym?
You mean we could do w/o cache store in tests?
Sure, but we want to be as close as possible to real usage scenario.
It's probably better to try tuning the CacheStore then, but you'll
always be quite slow as CacheStore(s) significantly slow down
Infinispan.
You should also ask yourself if the CacheStore is really needed.
Sanne
-Ales
8:44 a.m.
New subject: CD datastore inserts slow
>> To compare, I reconfigured it to use local-only NTR backends
(not
>> suited for clustering) and so doing I found another problem: all of
>> your caches are sharing the same index!
>> By reconfiguring each cache to use and independent path to isolate the
>> indexes, the benchmark completes in 20 seconds, and these remaining 20
>> seconds are all spent in CapeDwarf code, with some more object
>> allocations from Infinispan core.
>
> How do you configure this index isolation?
> (how exactly does this look in standalone.xml?)
I did it locally, can send you a pull request.
Please do.
But to clarify: this would disable clustering!
Why or how would this disable clustering?
Do you really want
that?
Probably not. :-)
the first option is safer, I'd just change the IndexManager.
Which IndexManager do we use now?
I can send you a pull request for both variations of the
configuration, just let me know which file it is I have to patch: I
tried to look into the repository for capedwarf-jboss-as, but the
configuration file in there seems quite different than the one in
CapeDwarf blue distribution.
Hmmm, they should be the same, or what's the diff?
*
https://github.com/capedwarf/capedwarf-testsuite/blob/master/integrate-cd...
*
https://github.com/capedwarf/capedwarf-jboss-as/blob/master/build/src/mai...
So I'm not sure if I'm looking in the
right place, or maybe the wrong branch?
The distribution / release is done via Testsuite:
* https://github.com/capedwarf/capedwarf-testsuite
(see -Prelease profile)
General optimisation, the usual stuff: profile it, rinse and repeat.
I
didn't look at details but I remember a priority would be to optimise
some HashMap lookups for internal services and components.
Do you perhaps still have some profile data on which look-ups are this?
It's probably better to try tuning the CacheStore then, but
you'll
always be quite slow as CacheStore(s) significantly slow down
Infinispan.
You should also ask yourself if the CacheStore is really needed.
Well, I'd leave this up to the users.
-Ales
3839
days inactive
3861
days old
infinispan-dev@lists.jboss.org
7 comments
2 participants
participants (2)
-
Ales Justin -
Sanne Grinovero