[Metrics] How to react on low disk?
by Heiko W.Rupp
Hey,
<captain_obvious>
so for Hawkular-metrics (and Hawkular) we store the data in a Cassandra
database that puts files on a disk,
which can get full earlier than expected (and usually on week-ends). And
when the disk is full, Metrics does not like it.
</captain_obvious>
What can we do in this case?
I could imagine that on the C* nodes we run a script that uses df to
figure out the available space and tries
to run some compaction if space gets tight.
Of course that does not solve the issue per se, but should give some air
to breathe.
Right now I fear we are not able to reduce the ttl of a metric/tenant on
the fly and have metrics do the right thing - at least if I understood
yak correctly.
That script should possibly also send an email to an admin.
In case that we run Hawkular-full, we can determine the disk space and
feed that into Hawkular for Alerts to pick it up and then have the
machinery trigger the compaction and send the email.
8 years, 7 months