[Hawkular-dev] Inventory and 'transient' servers

[Heiko W.Rupp]

Hey,

when WildFly connects to Inventory for the 1st time, we sync
the EAP information into inventory, which also includes the information
about which metrics are available.

Now when WildFly is deployed into Kubernetes or OpenShift, we
will see that WildFly is started, syncs and then dies at some point
in time, where k8s will not re-use the existing one, but start
a new one, which will have a different FeedId.

This will leave a WildFly in Inventory, that is later detected as
down in Metrics/Alerting, but the one in Inventory will stay
forever. Consequences are
- Inventory will get "full" with no-longer needed information
- Clients will retrieve data about non-"usable" servers

We need to figure out how to deal with this situation like e.g.:
- have inventory listen on k8s events and remove the server
  when k8s removes it (not when it is going down; stopped pods
 can stay around for some time)
- Have a different way of creating the feed id / server id so that
 the state is "re-used". Something like feedId/server name could
 be the name of the deployment config + version + k8s tenant.

Thoughts?