[Hawkular-dev] OpenShift: Metric table out of sync issue and heapster

[Matt Wringe]

----- Original Message -----
> From: "Michael Burman" <miburman at redhat.com>
> To: "Matt Wringe" <mwringe at redhat.com>
> Cc: "Discussions around Hawkular development" <hawkular-dev at lists.jboss.org>
> Sent: Monday, 29 May, 2017 8:57:48 AM
> Subject: Re: OpenShift: Metric table out of sync issue and heapster
> 
> Hi,
> 
> descriptor_tags are not sent again unless there's a change in the
> Heapster's internal description structure (new metrics have been
> introduced since the last time). The sink assumes that otherwise the
> tags are static for the pod (restart of the pod would result in new
> metrics to be created so there shouldn't be any changes to living pods).

Pods can change even without them being restarted. We store things like labels, which should be updated when the labels on a pod is modified.

> 
>    - Micke
> 
> 
> On 05/29/2017 03:55 PM, Matt Wringe wrote:
> > ----- Original Message -----
> >> From: "Matt Wringe" <mwringe at redhat.com>
> >> To: "Michael Burman" <miburman at redhat.com>
> >> Cc: "Discussions around Hawkular development"
> >> <hawkular-dev at lists.jboss.org>
> >> Sent: Monday, 29 May, 2017 8:52:41 AM
> >> Subject: Re: OpenShift: Metric table out of sync issue and heapster
> >>
> >> ----- Original Message -----
> >>> From: "Michael Burman" <miburman at redhat.com>
> >>> To: "Matt Wringe" <mwringe at redhat.com>, "Discussions around Hawkular
> >>> development" <hawkular-dev at lists.jboss.org>
> >>> Sent: Saturday, 27 May, 2017 2:08:53 AM
> >>> Subject: Re: OpenShift: Metric table out of sync issue and heapster
> >>>
> >>> Hi,
> >>>
> >>> That check is there because it greatly reduces the load on the
> >>> Hawkular-Metrics in an existing environment. Otherwise, after the
> >>> restart, the first time the metrics are sent it is going to make a huge
> >>> amount of writes to HWKMETRICS and this will take time (the operations
> >>> are slower than datapoint writes after all).
> >> The majority (if not all) metrics in Hawkular Metrics right now come from
> >> Heapster, if things are working properly then they should essentially all
> >> have that
> > Ah, I accidentally send this before I was finished. Sorry.
> >
> > What I meant is that in normal situations, don't almost all the metrics
> > have the descriptor tag anyways? And therefore most of the metrics will be
> > resent in a normal operation?
> >
> > All that this changes does it remove the requirement for this tag to be
> > present (since for the most part this indicates that the metrics has an
> > error).
> >
> > Or did I miss something here?
> >
> >>> I'll make the change next week to allows deciding the behavior with a
> >>> command line parameter (in case we fix it in the metrics some other way
> >>> also - or this wasn't the real root cause for some reason).
> >> I am fine with a command line parameter if we think this is the right way
> >> to
> >> go. As long as we provide some sort of mechanism to allow for users to
> >> resolve this issue.
> >>
> >>>
> >>> On 05/25/2017 01:24 AM, Matt Wringe wrote:
> >>>> We are still running into issues on machines where the metric tables
> >>>> getting out of sync bug has not been fixed for it yet
> >>>> (https://issues.jboss.org/browse/HWKMETRICS-660).
> >>>>
> >>>> @miburman: would it be possible to update our Heapster sink to always
> >>>> update metrics when its started? Eg to remove this check
> >>>> https://github.com/kubernetes/heapster/blob/master/metrics/sinks/hawkular/client.go#L45
> >>>>
> >>>> I am not sure why we have that check in there in the first place, and I
> >>>> think we should always just update the tag definition on each start.
> >>>> That
> >>>> way if someone has run into the problem where the tags and metrics
> >>>> definitions get out of sync, they can just restart heapster to get
> >>>> things
> >>>> back working again.
> >>>>
> >>>> Any thoughts?
> >>>>
> >>>> - Matt
> >>>
> 
>