[Hawkular-dev] Memory-usage of Hawkular-services

[Heiko W.Rupp]

Update

[ tl;dr: good news ]

Lukas provided me with "connection properties" for hsqldb
that limit the memory usage and this seems stable. I have been
running h-services since Friday afternoon (~40h) with 33 HawkFly
feeds.

Two heap dumps taken yesterday afternoon and this morning
both show a stable heap size of 192M, so I think we are on the
safe side here (with hsqldb, see also below).

On top of just ingesting metrics, I have created poor mans Chaos
Monkey that randomly stops and restarts a container every 5 minutes
(parallel on 2 machines where my feeds live) to also stress the initial
sync when a feeds is (re) connecting.
I also had a (very small) number of alert triggers defined, that
were firing when heap on a hawkfly is > 60k, which happened quite
often.
This was running during the last 36h, so is included in above
heap metrics.

The MBeans for hawkular metrics show an insertion rate of around
33 metrics/s , so a HawkFly corresponds to 1metric/s.

About hsqldb: I am going to shut this down now and test with a
h2-based instance, as h2 is already in WildFly and EAP and
should be used instead of hsqldb.

Some time next week I will also try to instrument HawkFly in a away
that it automatically defines some alert triggers on startup.