On Jun 22, 2016, at 6:56 AM, Filip Brychta <fbrychta(a)redhat.com&gt; wrote: This PR increased throughput 0 -- 200% depending on msg size!!! Performance increase based on msg size: 1 datapoint per request - no change 10 datapoints per request - 100% throughput increase 100 datapoints per request - 200% throughput increase 500 datapoints per request - 180% throughput increase 5000 datapoints per request - 130% throughput increase CPU usage is significantly higher, before it was 70% idle, after it's just 12% idle. Filip ----- Original Message ----- > Hi, > > Today I've been looking at Metrics insertion performance. > > My setup is the following: > - on my laptop I run the Gatling load scenario (which is very similar to > the perf test job scenario) > - I also run a Metrics standalone instance > - on another machine connected to my LAN, I run a single node C* cluster > > In order to avoid problems due to memory constraints, I set min and max > heap size to 2048m. Below this value the server spends significant time > in garbage collection (with high number of concurrent clients). > > I ran Gatling 3 times with different number of clients (think agents): > 100,1000,2000 > ---- > mvn gatling:execute -Dclients=X -Dramp=0 -Dloops=50 > ---- > > From 100 to 1000 virtual clients, the throughput raised accordingly > (x10). From 1000 to 2000, I hit the same kind of plateau Filip was > observing: I got barely 15% throughput increase. None of the machines > had reached cpu/io/memory limits. > > So I ran the 2000 virtual clients test again and jstack (result > attached). As you can see, most of the task handler threads are in state > WAITING inside the > com.datastax.driver.core.HostConnectionPool.awaitAvailableConnection method. > > Then, following the "Monitoring and tuning the pool" section of the > driver doc, I added some code to print the number of open connections, > active requests, and maximum capacity. It confirmed that the maximum > capacity was reached. > > Note that by default with the V3 protocol, the driver creates just one > connection per host and allows a maximum of 1024 concurrent requests. > > After that I added new options in the code to let the user define the > max number of connections per host as well as requests per connection. > > In my tests, I set the maximum connection limit to 10 and maximum > concurrency to 5000. These are arbitrary values, they are simply bigger > than the default. Best values would depend on the environment and should > be determined with testing. > > Then I ran Gatling 3 times again (100, 1000, 2000 virtual clients). This > time throughput increased linearly with the number of clients, from 100 > to 1000 and then 2000. I also tried with 3000, but this time throughput > did not increase as much. This was certainly due to both machines now > using more than 80% cpu. > > I sent a pull request [2] with the changes needed for the new > configuration parameters. > > Hopefully it can help in Filip's tests as well. > > Regards, > > -- > Thomas Segismont > JBoss ON Engineering Team > > [1] > http://datastax.github.io/java-driver/manual/pooling/#monitoring-and-tuni... > [2] https://github.com/hawkular/hawkular-metrics/pull/520 > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev