[Hawkular-dev] H-metrics perf test results - Optimal message size

[Filip Brychta]

Hello,
I tried to find optimal message size for POST requests which will result into the highest total throughput.
I tried 1, 100, 500 and 5000 metrics per request.


Test:
- h-metrics VM: 4 CPU cores, 8GB memory (gradually increasing heap size for WF in each test run - 1GB, 2GB, 4GB, 6GB, 8GB)
- cassandra node VM: 2 CPU cores, 4GB memory


Summary:
- small messages (1 metrics/request) were the worst for all test runs -> total throughput ~ 4500 metrics/sec
- medium messages (100 metrics/request) were performing the best almost for all test runs -> total throughput ~ 8600 -- 9000 metrics/sec
- bigger messages (500 and 5000 metrics/request) were significantly worst than medium messages when having low heap size (1GB)
- bigger messages were worst than medium even for bigger heap sizes but it was getting close
- big messages (5000 metrics/request) overloaded h-metrics when having only 1GB heap -> Full GC all the time
- when adding additional cassandra node and having sufficient heap size the bigger messages were performing better -> with sufficient heap size (4GB) and cassandra cluster the bigger messages the better performance (measured only up to 5000, I'm not sure if it make sense to try even bigger messages)


Details:
4 CPU cores

1GB for WF
msg size=1
4400 req/sec -> total throughput = 4400
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          68.47    0.00   12.39    0.00    0.67   18.46

msg size=100
87 req/sec -> total throughput = 8700
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          24.94    0.00   11.15    0.05    0.28   63.58

msg size=500
13 req/sec -> total throughput = 6500
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          41.80    0.00    7.19    0.08    0.15   50.78

msg size=5000
0.879 req/sec -> total throughput = 4395
- GC all the time
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          66.35    0.00    4.08    0.00    0.08   29.50

===================================
2GB for WF
msg size=1
4574 req/sec -> total throughput = 4574
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          67.84    0.00   12.67    0.05    0.87   18.57

msg size=100
87 req/sec -> total throughput = 8700
- GC each 5-6 minutes
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          26.99    0.00   10.25    0.13    0.21   62.43

msg size=500
16 req/sec -> total throughput = 8000
- GC each 1 min
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          28.48    0.00    8.22    0.15    0.59   62.55

msg size=5000
1.432 req/sec -> total throughput = 7160
- GC each 30s
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          36.44    0.00   10.04    0.18    0.08   53.27


===================================
4GB for WF
msg size=1
4498 req/sec -> total throughput = 4498
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          63.28    0.00   13.03    0.03    0.58   23.10

msg size=100
90.79 req/sec -> total throughput = 9079
- GC after > 6 minutes
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          25.86    0.00   10.44    0.08    0.26   63.37

msg size=500
16.07 req/sec -> total throughput = 8035
- GC after > 6 minutes
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          23.73    0.00    8.18    0.13    0.46   67.50

msg size=5000
1.545 req/sec -> total throughput = 7725
- GC after > 6 minutes
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          31.35    0.00   10.20    0.05    0.16   58.24


===================================
6GB for WF
msg size=1
4556 req/sec -> total throughput = 4556
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          67.59    0.00   12.24    0.15    0.50   19.52

msg size=100
86 req/sec -> total throughput = 8600
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          27.40    0.00   10.36    0.13    0.28   61.83
msg size=500
18 req/sec -> total throughput = 9000
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          29.30    0.00    8.49    0.18    0.26   61.77
msg size=5000
1.45 req/sec -> total throughput = 7250
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          32.26    0.00   11.61    0.34    0.10   55.69


===================================
8GB for WF
msg size=1
4718 req/sec -> total throughput = 4718
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          44.19    0.00   17.05    0.33    0.22   38.22

msg size=100
82.27 req/sec -> total throughput = 8227
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          30.77    0.00   10.95    0.08    0.13   58.07
msg size=500
17.86 req/sec -> total throughput = 7725
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          52.56   20.32   11.16    0.20    0.00   15.76
msg size=5000
failed

===================================
2GB for WF + 2 cassandra nodes
msg size=1
total throughput = 6741
msg size=100
total throughput = 12900
msg size=500
total throughput = 11250
msg size=5000
total throughput = 13100

===================================
4GB for WF + 2 cassandra nodes
msg size=1
total throughput = 6533
msg size=100
total throughput = 12280
msg size=500
total throughput = 12650
msg size=5000
total throughput = 15950



Filip