[Hawkular-dev] [Metrics] How to react on low disk?

[Heiko W.Rupp]

Hey,

<captain_obvious>
so for Hawkular-metrics (and Hawkular) we store the data in a Cassandra 
database that puts files on a disk,
which can get full earlier than expected (and usually on week-ends). And 
when the disk is full, Metrics does not like it.
</captain_obvious>

What can we do in this case?

I could imagine that on the C* nodes we run a script that uses df to 
figure out the available space and tries
to run some compaction if space gets tight.

Of course that does not solve the issue per se, but should give some air 
to breathe.
Right now I fear we are not able to reduce the ttl of a metric/tenant on 
the fly and have metrics do the right thing - at least if I understood 
yak correctly.
That script should possibly also send an email to an admin.

In case that we run Hawkular-full, we can determine the disk space and 
feed that into Hawkular for Alerts to pick it up and then have the 
machinery trigger the compaction and send the email.