[infinispan-dev] write-heavy performance degradation between 5.1 and 5.2

[Paul Ferraro]

Good summary Martin.  To this day, I'm not convinced there was ever any
performance regression - but throw some aberrant graphs at management
without sufficient context and things get escalated quickly...

One of the issues with testing across versions is considering the
baseline.  In the case of EAP 6.0 vs 6.1, the non-clustered performance
is actually faster in 6.1.  When this is the case, the cross version
performance comparison should really include a handicap for 6.1, meaning
that we shouldn't be comparing clustering throughput given the same
number of concurrent clients, but rather the clustering throughput given
the same "clustering load", which in 6.1, is actually at a lower number
of clients than 6.0.  When considering the handicap, the perf numbers
are much more in line with the numbers I've seen from Infinispan 5.1 vs
5.2 directly.

On Tue, 2013-05-07 at 14:28 +0200, Martin Gencur wrote:
> Kind of....their configuration is replicated & embedded cache + async 
> replication + batching + file store + passivation off + write-heavy
> 
> We, indeed, do *not* test this configuration for regression (especially 
> the batching + file store + write-heavy). And even if we did, we 
> wouldn't spot the problem easily. The performance regression was 
> revealed in a client stress test where the number of clients is 
> gradually increased. The peak throughput in their tests was higher for 
> 5.2 than for 5.1 - this was OK. They complained about the max. number of 
> clients that can be handled and the throughput after the peak (not 
> before it). And this has to do with JGroups tuning (thread pools etc.) 
> This is a very specific test case and issue and can be revealed only by 
> the client stress tests which are the most time consuming.
> 
> Anyway, we'll try to incorporate this scenario into the regular 
> regression performance testing we're working on so that we catch these 
> problems sooner than the respective version of ISPN gets to AS.
> 
> Martin
> 
> 
> Dne 2.5.2013 19:43, Galder Zamarreño napsal(a):
> > My gut feeling is: AS uses some cache configuration that's affecting performance and it's not currently in the test plan?
> >
> > Cheers,
> >
> > On May 2, 2013, at 12:41 PM, Radim Vansa <rvansa at redhat.com> wrote:
> >
> >> Hi, I've checked it on 4 nodes, stress test 15 minutes with 80% writes. Both reads and writes have improved in 5.2
> >>
> >> READS/sec 247k -> 263k
> >> WRITES/sec 4547 -> 4771 (note: this is average, coordinator has about 2x more writes as it is the lock owner for all locks in replicated mode)
> >> that makes
> >>
> >> Paul, how many operations within transaction do you use? The results above are from single operation (put/get) per transaction (we have improvement in TRANSACTIONS/sec 5655 -> 5936 with this setting), I'll try to rerun with more ops/transaction as well.
> >>
> >> Radim
> >>
> >> ----- Original Message -----
> >> | From: "Paul Ferraro" <paul.ferraro at redhat.com>
> >> | To: "Mircea Markus" <mmarkus at redhat.com>
> >> | Cc: "infinispan" <infinispan-dev at lists.jboss.org>, "Paul Ferraro" <pferraro at redhat.com>
> >> | Sent: Wednesday, May 1, 2013 5:05:39 PM
> >> | Subject: Re: [infinispan-dev] write-heavy performance degradation between	5.1 and 5.2
> >> |
> >> | FYI - here's a summary of EAP 6.0 vs 6.1 performance:
> >> | https://bugzilla.redhat.com/attachment.cgi?id=741896
> >> |
> >> | and here's the relevant BZ:
> >> | https://bugzilla.redhat.com/show_bug.cgi?id=956988
> >> |
> >> | On Wed, 2013-05-01 at 15:55 +0100, Mircea Markus wrote:
> >> | > Hi Martin,
> >> | >
> >> | > Paul mentioned a severe degradation in performance between 5.1 and 5.2 for
> >> | > replicated + embedded + transactional + pessimistic caches, for
> >> | > write-heavy access. Do we have any tests we can run to check this?
> >> | >
> >> | > Cheers,
> >> |
> >> |
> >> | _______________________________________________
> >> | infinispan-dev mailing list
> >> | infinispan-dev at lists.jboss.org
> >> | https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >> |
> >> _______________________________________________
> >> infinispan-dev mailing list
> >> infinispan-dev at lists.jboss.org
> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> > --
> > Galder Zamarreño
> > galder at redhat.com
> > twitter.com/galderz
> >
> > Project Lead, Escalante
> > http://escalante.io
> >
> > Engineer, Infinispan
> > http://infinispan.org
> >
> >
> > _______________________________________________
> > infinispan-dev mailing list
> > infinispan-dev at lists.jboss.org
> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>