[Hawkular-dev] H-metrics perf test results - Optimal message size

Hello, I tried to find optimal message size for POST requests which will result into the highest total throughput. I tried 1, 100, 500 and 5000 metrics per request. Test: - h-metrics VM: 4 CPU cores, 8GB memory (gradually increasing heap size for WF in each test run - 1GB, 2GB, 4GB, 6GB, 8GB) - cassandra node VM: 2 CPU cores, 4GB memory Summary: - small messages (1 metrics/request) were the worst for all test runs -> total throughput ~ 4500 metrics/sec - medium messages (100 metrics/request) were performing the best almost for all test runs -> total throughput ~ 8600 -- 9000 metrics/sec - bigger messages (500 and 5000 metrics/request) were significantly worst than medium messages when having low heap size (1GB) - bigger messages were worst than medium even for bigger heap sizes but it was getting close - big messages (5000 metrics/request) overloaded h-metrics when having only 1GB heap -> Full GC all the time - when adding additional cassandra node and having sufficient heap size the bigger messages were performing better -> with sufficient heap size (4GB) and cassandra cluster the bigger messages the better performance (measured only up to 5000, I'm not sure if it make sense to try even bigger messages) Details: 4 CPU cores 1GB for WF msg size=1 4400 req/sec -> total throughput = 4400 avg-cpu: %user %nice %system %iowait %steal %idle 68.47 0.00 12.39 0.00 0.67 18.46 msg size=100 87 req/sec -> total throughput = 8700 avg-cpu: %user %nice %system %iowait %steal %idle 24.94 0.00 11.15 0.05 0.28 63.58 msg size=500 13 req/sec -> total throughput = 6500 avg-cpu: %user %nice %system %iowait %steal %idle 41.80 0.00 7.19 0.08 0.15 50.78 msg size=5000 0.879 req/sec -> total throughput = 4395 - GC all the time avg-cpu: %user %nice %system %iowait %steal %idle 66.35 0.00 4.08 0.00 0.08 29.50 =================================== 2GB for WF msg size=1 4574 req/sec -> total throughput = 4574 avg-cpu: %user %nice %system %iowait %steal %idle 67.84 0.00 12.67 0.05 0.87 18.57 msg size=100 87 req/sec -> total throughput = 8700 - GC each 5-6 minutes avg-cpu: %user %nice %system %iowait %steal %idle 26.99 0.00 10.25 0.13 0.21 62.43 msg size=500 16 req/sec -> total throughput = 8000 - GC each 1 min avg-cpu: %user %nice %system %iowait %steal %idle 28.48 0.00 8.22 0.15 0.59 62.55 msg size=5000 1.432 req/sec -> total throughput = 7160 - GC each 30s avg-cpu: %user %nice %system %iowait %steal %idle 36.44 0.00 10.04 0.18 0.08 53.27 =================================== 4GB for WF msg size=1 4498 req/sec -> total throughput = 4498 avg-cpu: %user %nice %system %iowait %steal %idle 63.28 0.00 13.03 0.03 0.58 23.10 msg size=100 90.79 req/sec -> total throughput = 9079 - GC after > 6 minutes avg-cpu: %user %nice %system %iowait %steal %idle 25.86 0.00 10.44 0.08 0.26 63.37 msg size=500 16.07 req/sec -> total throughput = 8035 - GC after > 6 minutes avg-cpu: %user %nice %system %iowait %steal %idle 23.73 0.00 8.18 0.13 0.46 67.50 msg size=5000 1.545 req/sec -> total throughput = 7725 - GC after > 6 minutes avg-cpu: %user %nice %system %iowait %steal %idle 31.35 0.00 10.20 0.05 0.16 58.24 =================================== 6GB for WF msg size=1 4556 req/sec -> total throughput = 4556 avg-cpu: %user %nice %system %iowait %steal %idle 67.59 0.00 12.24 0.15 0.50 19.52 msg size=100 86 req/sec -> total throughput = 8600 avg-cpu: %user %nice %system %iowait %steal %idle 27.40 0.00 10.36 0.13 0.28 61.83 msg size=500 18 req/sec -> total throughput = 9000 avg-cpu: %user %nice %system %iowait %steal %idle 29.30 0.00 8.49 0.18 0.26 61.77 msg size=5000 1.45 req/sec -> total throughput = 7250 avg-cpu: %user %nice %system %iowait %steal %idle 32.26 0.00 11.61 0.34 0.10 55.69 =================================== 8GB for WF msg size=1 4718 req/sec -> total throughput = 4718 avg-cpu: %user %nice %system %iowait %steal %idle 44.19 0.00 17.05 0.33 0.22 38.22 msg size=100 82.27 req/sec -> total throughput = 8227 avg-cpu: %user %nice %system %iowait %steal %idle 30.77 0.00 10.95 0.08 0.13 58.07 msg size=500 17.86 req/sec -> total throughput = 7725 avg-cpu: %user %nice %system %iowait %steal %idle 52.56 20.32 11.16 0.20 0.00 15.76 msg size=5000 failed =================================== 2GB for WF + 2 cassandra nodes msg size=1 total throughput = 6741 msg size=100 total throughput = 12900 msg size=500 total throughput = 11250 msg size=5000 total throughput = 13100 =================================== 4GB for WF + 2 cassandra nodes msg size=1 total throughput = 6533 msg size=100 total throughput = 12280 msg size=500 total throughput = 12650 msg size=5000 total throughput = 15950 Filip