Re: [Hawkular-dev] Metrics performance testing PR#520 - big perf improvement

Hi, Today I've been looking at Metrics insertion performance. My setup is the following: - on my laptop I run the Gatling load scenario (which is very similar to the perf test job scenario) - I also run a Metrics standalone instance - on another machine connected to my LAN, I run a single node C* cluster In order to avoid problems due to memory constraints, I set min and max heap size to 2048m. Below this value the server spends significant time in garbage collection (with high number of concurrent clients). I ran Gatling 3 times with different number of clients (think agents): 100,1000,2000 ---- mvn gatling:execute -Dclients=X -Dramp=0 -Dloops=50 ---- From 100 to 1000 virtual clients, the throughput raised accordingly (x10). From 1000 to 2000, I hit the same kind of plateau Filip was observing: I got barely 15% throughput increase. None of the machines had reached cpu/io/memory limits. So I ran the 2000 virtual clients test again and jstack (result attached). As you can see, most of the task handler threads are in state WAITING inside the com.datastax.driver.core.HostConnectionPool.awaitAvailableConnection method. Then, following the "Monitoring and tuning the pool" section of the driver doc, I added some code to print the number of open connections, active requests, and maximum capacity. It confirmed that the maximum capacity was reached. Note that by default with the V3 protocol, the driver creates just one connection per host and allows a maximum of 1024 concurrent requests. After that I added new options in the code to let the user define the max number of connections per host as well as requests per connection. In my tests, I set the maximum connection limit to 10 and maximum concurrency to 5000. These are arbitrary values, they are simply bigger than the default. Best values would depend on the environment and should be determined with testing. Then I ran Gatling 3 times again (100, 1000, 2000 virtual clients). This time throughput increased linearly with the number of clients, from 100 to 1000 and then 2000. I also tried with 3000, but this time throughput did not increase as much. This was certainly due to both machines now using more than 80% cpu. I sent a pull request [2] with the changes needed for the new configuration parameters. Hopefully it can help in Filip's tests as well. Regards, -- Thomas Segismont JBoss ON Engineering Team [1] http://datastax.github.io/java-driver/manual/pooling/#monitoring-and-tuni... [2] https://github.com/hawkular/hawkular-metrics/pull/520 _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev