<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Dec 16, 2016, at 9:23 AM, Daniel Miranda <<a href="mailto:danielkza2@gmail.com" class="">danielkza2@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Thank you very much John, that is, indeed very helpful. It seems compression will be exactly what I'm looking for for long-term storage.<br class=""><br class="">I did some tests with KairosDB in the meantime, and it seems it can sustain ~20K data points/s with a 3-node Cassandra cluster of t2.large AWS instances (no provisioned IOPS, just 20GB of standard EBS storage).<br class="">I'll to do some similar tests with Hawkular and report my findings.<br class=""><br class="">Can you share the program that you used for the simulated testing, so that I can try a similar pattern with KairosDB?<br class=""><br class="">Regards,<br class="">Daniel<br class=""></div></div></blockquote></div><br class=""><div class="">The test lives in my repo in branch named generate-data[1]. The test is named GenerateDataITest.java[2].</div><div class=""><br class=""></div><div class="">* Generating raw/uncompressed data</div><div class="">Checkout the generate-data branch from my repo.</div><div class=""><br class=""></div><div class="">`mvn install -DskipTests -Dlicense.skip -Dcheckstyle.skip` (you only need to build a handful of modules, but this is easier since it reduces number of steps)</div><div class=""><br class=""></div><div class="">`cd core/metrics-core-service`</div><div class=""><br class=""></div><div class="">`mvn verify -Dit.test=GenerateDataITest`</div><div class=""><br class=""></div><div class="">This will generate 7 days of raw data for 5,000 metrics with a data point for every 10 seconds. It make take some time to finish. If the test encounters any errors like a write timeout, it will abort.</div><div class=""><br class=""></div><div class="">When the test finishes, run `nodetool drain`.</div><div class=""><br class=""></div><div class="">Measure the size of the <CASSANDRA_DATA_DIR>/hawkulartest/data-* directory.</div><div class=""><br class=""></div><div class="">* Generating compressed data</div><div class="">This will reuse the raw data generated from the previous steps.</div><div class=""><br class=""></div><div class="">Restart Cassandra (has to be restarted since you did a drain)</div><div class=""><br class=""></div><div class="">`mvn verify -Dit.test=GenerateDataITest -Dcompress`</div><div class=""><br class=""></div><div class="">When the test finishes, run `nodetool drain`.</div><div class=""><br class=""></div><div class="">Measure the size of the <CASSANDRA_DATA_DIR>/hawkulartest/data_compressed* directory.</div><div class=""><br class=""></div><div class="">[1] <a href="https://github.com/jsanda/hawkular-metrics/tree/generate-data" class="">https://github.com/jsanda/hawkular-metrics/tree/generate-data</a></div><div class="">[2] <a href="https://github.com/jsanda/hawkular-metrics/blob/77ed5345b145a3c1f1d4c17885d9ebd31a18421b/core/metrics-core-service/src/test/java/org/hawkular/metrics/core/impl/GenerateDataITest.java" class="">https://github.com/jsanda/hawkular-metrics/blob/77ed5345b145a3c1f1d4c17885d9ebd31a18421b/core/metrics-core-service/src/test/java/org/hawkular/metrics/core/impl/GenerateDataITest.java</a></div></body></html>