[infinispan-dev] New bundler performance

[Dan Berindei]

On Wed, Jun 12, 2013 at 5:39 PM, Radim Vansa <rvansa at redhat.com> wrote:

>
>
> ----- Original Message -----
> | From: "Bela Ban" <bban at redhat.com>
> | To: infinispan-dev at lists.jboss.org
> | Sent: Wednesday, June 12, 2013 3:16:58 PM
> | Subject: Re: [infinispan-dev] New bundler performance
> |
> |
> |
> | On 06/12/2013 04:54 AM, Radim Vansa wrote:
> | > Hi,
> | >
> | > I was going through the commits (running tests on each of them) to
> | > seek the performance regression we've recently discovered and it
> | > seems that our test (replicated udp non-transactional stress test on
> | > 4 nodes) experiences a serious regression on the commit
> | >
> | > ISPN-2848 Use the new bundling mechanism from JGroups 3.3.0
> | > (73da108cdcf9db4f3edbcd6dbda6938d6e45d148)
> | >
> | > The performance drops from about 7800 writes/s to 4800 writes/s, and
> | > from 1.5M reads/s to 1.2M reads/s (having slower reads in replicated
> | > mode is really odd).
> |
> |
> | Is this using sync or async replication ?
>
> Sync
>
> |
> | You could set UDP.bundler_type="old" to see if the old bundler makes a
> | difference.
>
> Yeah, that does not work at all, performance drops very low as everything
> is bundled
>
> |
> | Yes, reads should have *nothing* to do with the bundler as there should
> | not be any communication on reads !
> |
> | Maybe you should instrument the reads with byteman and see if they cause
> | any communication.
>
> Maybe that's really just about overall system behaviour. If we waited for
> some communication, there wouldn't be million reads per second :)
>
>
Higher CPU usage for the write commands will slow down everything else on
the same machine. So as long as the test does reads and writes in parallel,
there will be some interference. I suggest running a read-only test and a
write-only test to get a more reliable picture of the performance for each
of them.



> |
> |
> | > It seems that the bundler is not really as good as we hoped for - it
> | > may be the bottleneck. I have tried to create another bundler which
> | > shares the queue between 4 instances of TransportQueueBundler (so, 4
> | > threads are actually sending the messages which go into one queue)
> | > and the performance mildly improved - to 5200 writes/s, but that's
> | > not enough.
> |
> | I'm currently at the summit, but do you have a test that I could run
> | when I'm back to reproduce the performance issue ?
>
> Would be RadarGun benchmark enough to you? See attachments.
> You have compile Infinispan from snapshot
> (113842c8cf91cbb5a1bbd26e05fab7024fdec081 is the last OK,
> 73da108cdcf9db4f3edbcd6dbda6938d6e45d148 is slower) and compile RadarGun
> from snapshot with
>
> mvn install -Pinfinispan53
>
> (this creates also the plugin using infinispan-core-5.3.0-SNAPSHOT)
>
> Radim
>
> |
> |
> | >
> | > Radim
> | >
> | > Note: you may have seen my conversation with Pedro Ruivo on IRC about
> | > the bundler several days ago, in that time our configuration had old
> | > bundler. This was fixed, but as I have not built Infinispan properly
> | > (something got cached), I have not noticed the regression between
> | > these builds.
> |
> | --
> | Bela Ban
> | Lead JGroups / Clustering Team
> | JBoss
> | _______________________________________________
> | infinispan-dev mailing list
> | infinispan-dev at lists.jboss.org
> | https://lists.jboss.org/mailman/listinfo/infinispan-dev
> |
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20130612/343d3043/attachment.html