9:59 p.m.
Pretty much from the start of the project we have provided configurable data retention.
There is a system-wide default retention that can be set at start up. You can also set the
data retention per tenant as well as per individual metric. Do we need to provide this
fine-grained level of configurability, or is it sufficient to only have a system-wide data
retention which is configurable?
It is worth noting that in OpenShift *only* the system-wide data retention is set.
Recently we have been dealing with a number of production issues including:
* Cassandra crashing with an OutOfMemoryError
* Stats queries failing in Hawkular Metrics due to high read latencies in Cassandra
* Expired data not getting purged in a timely fashion
These issues all involve compaction. In older versions of Hawkular Metrics we were using
the default, size tiered compaction strategy (STCS). Time window compaction strategy
(TWCS) is better suited and for time series data such as our. We are already seeing good
results with some early testing. Using the correct and properly configured compaction
strategy can have a significant impact on several things including:
* I/O usage
* cpu usage
* read performance
* disk usage
TWCS was developed for some very specific use cases which are common use cases with
Cassandra. TWCS is recommended for time series that meet the following criteria:
* append-only writes
* no deletes
* global (i.e., table-wide) TTL
* few out of order writes (at least it is the exception and not the norm)
It is the third bullet which has prompted this email. If we allow/support different TTLs
per tenant and/or per metric we will lose a lot of the benefits of TWCS and likely
continue to struggle with some of the issues we have been facing as of late. If you ask me
exactly how well or poorly will compaction perform using mixed TTLs, I can only speculate.
I simply do not have the bandwidth to test things that C* docs and C* devs say *not* to
do.
I am of the opinion, at least for OpenShift users, that disk usage is much more important
than fine-grained data retentions. A big question I have is, what about outside of
OpenShift? This may be a question for some people not on this list, so I want to make sure
it does reach the right people.
I think we could potentially tie together configurable data retention with rollups. Let’s
say we add support for 15min, 1hr, 6hr, and 24hr rollups where each rollup is stored in
its own table and each larger rollup having a larger retention. Different levels of data
retentions could be used to determine what rollups a tenant has. If a tenant wants a data
retention of a month for example, then that could translate into generating 15min and 1hr
rollups for that tenant.
- John