[Hawkular-dev] [Metrics] How to react on low disk?

Hey, <captain_obvious> so for Hawkular-metrics (and Hawkular) we store the data in a Cassandra database that puts files on a disk, which can get full earlier than expected (and usually on week-ends). And when the disk is full, Metrics does not like it. </captain_obvious> What can we do in this case? I could imagine that on the C* nodes we run a script that uses df to figure out the available space and tries to run some compaction if space gets tight. Of course that does not solve the issue per se, but should give some air to breathe. Right now I fear we are not able to reduce the ttl of a metric/tenant on the fly and have metrics do the right thing - at least if I understood yak correctly. That script should possibly also send an email to an admin. In case that we run Hawkular-full, we can determine the disk space and feed that into Hawkular for Alerts to pick it up and then have the machinery trigger the compaction and send the email.