[Hawkular-dev] Metrics storage usage and compression

[Michael Burman]

Hi,

I managed to delete that data already, so here comes another calculation based on the real values (not theoretical). These come from Cassandra 2.2.8 and the dataset is Gauges of idling MiQ instance Cassandra data..

2168597 values in 1528 keys:

LZ4:     54331865 bytes -> 25.05 bytes per value
Deflate: 28491886 bytes -> 13.14 bytes per value
LZ4 + Gorilla: 1574659 bytes -> 0.726 bytes per value

The last one is too good because the dataset includes a lot of gauges that have "0" as all of their values. In the dataset the Gorilla compression's worst cases are around 5 bytes per value, but as you can see, the 0 valued measurements considerably improve the compression ratio. How realistic measurements these idlings are .. I don't know.

Note: Cassandra 3.8 improves these values, especially with LZ4:

LZ4 -> ~15 bytes per value
Deflate: ~10 bytes per value

In this Openshift example, with just Cassandra upgrade + Deflate we would have improved 25 -> 10.

  - Micke


----- Original Message -----
From: "Heiko W.Rupp" <hrupp at redhat.com>
To: "Discussions around Hawkular development" <hawkular-dev at lists.jboss.org>
Sent: Thursday, July 28, 2016 4:56:04 PM
Subject: Re: [Hawkular-dev] Metrics storage usage and compression

On 27 Jul 2016, at 11:19, Michael Burman wrote:

> When storing to our current data model, the disk space taken by "data" 
> table was 74MB. My prototype uses the method of Facebook's Gorilla 
> paper (same as what for example Prometheus uses), and in this test I 
> used a one day block size (storing one metric's one day data to one 
> row inside Cassandra). The end result was 3,1MB of storage space used. 
> Code can be found from bitbucket.org/burmanm/compress_proto (Golang). 
> I know Prometheus advertises estimated 1.3 bytes per timestamped 
> value, but those

How many bytes to we currently use per timestamped value?
_______________________________________________
hawkular-dev mailing list
hawkular-dev at lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev