[Hawkular-dev] OpenShift: Metric table out of sync issue and heapster

[Matt Wringe]

I have a PR for this https://github.com/mwringe/heapster/commit/fdc28cda148b2b66ac92b659b59e3d4c3223a095

It will always update the metric tags when Heapster is started and doesn't look to see if the metrics have a special tag applied.

I have remove values from metrics_tags_idx and metrics_idx and it looks like now Heapster will repopulate those values when its restarted, which is the behaviour we want.

Even if we properly catch the exception that the write didn't go to both tables, Heapster could be restarted before the update has a chance to complete. This change combined with the fix in https://issues.jboss.org/browse/HWKMETRICS-660 should allow for a much more stable experience here.



----- Original Message -----
> From: "Matt Wringe" <mwringe at redhat.com>
> To: "Discussions around Hawkular development" <hawkular-dev at lists.jboss.org>
> Sent: Wednesday, 24 May, 2017 6:24:13 PM
> Subject: [Hawkular-dev] OpenShift: Metric table out of sync issue and	heapster
> 
> We are still running into issues on machines where the metric tables getting
> out of sync bug has not been fixed for it yet
> (https://issues.jboss.org/browse/HWKMETRICS-660).
> 
> @miburman: would it be possible to update our Heapster sink to always update
> metrics when its started? Eg to remove this check
> https://github.com/kubernetes/heapster/blob/master/metrics/sinks/hawkular/client.go#L45
> 
> I am not sure why we have that check in there in the first place, and I think
> we should always just update the tag definition on each start. That way if
> someone has run into the problem where the tags and metrics definitions get
> out of sync, they can just restart heapster to get things back working
> again.
> 
> Any thoughts?
> 
> - Matt
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>