<div dir="ltr"><div><br></div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 1, 2017 at 11:54 PM, John Mazzitelli <span dir="ltr"><<a href="mailto:mazz@redhat.com" target="_blank">mazz@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Spent time trying to figure out how to support WildFly domain mode with the new stuff. It's not going well. Next week I will need to have some discussions on how we want to do this.<br>
<br>
The issues with domain mode (I'll try to be short):<br>
<br>
1) host controllers do not emit JMX metrics for slave servers - just not implemented in WildFly. Need to go to slave servers for their own JMX metrics.<br>
2) slave servers do not have managed interfaces available remotely (can't connect to slaves over remote management interface - so agent can't get any inventory from slaves - have to get all inventory from host controller)<br>
<br>
This means our agent in the host controller needs to collect all inventory for both master host controller and slave servers.<br>
<br>
But we need the slave servers to have a metrics endpoint because its only in the slaves where we can get JMX metrics (remember, host controller can't give us that, we must scrape the slave servers for the JMX metrics).<br>
<br>
If we do not have our agent in slave server (why bother if we can't connect to it over DMR API to get inventory?), we still need to somehow get the P8s JMX Exporter installed in the slaves. But if we just use the raw jmx exporter agent, how do we tell our h-services server/P8s server about the new endpoint that needs to be scraped? And how do we get the jmx exporter yml config to install there?<br>
<br>
So I suppose we should put our agent in a kind of "metrics only" mode in all slave servers so it can expose the JMX exporter and pull down the correct jmx exporter yml from h-services server and have it tell the server to add the scrape endpoint to p8s.<br>
<br>
But because we aren't getting inventory from slave servers, how can our agent tell the server to add the scrape endpoint? Our current implementation only adds scrape endpoints to our P8s server when new agents go into inventory. Workaround: either have agent store a small inventory (just the agent resource itself) which triggers the new scrape endpoint addition, or add a inventory REST endpoint for the agent to call to add the new scrape endpoint manually.<br>
<br>
OK, assume we have all this. How do we link the JMX metrics getting collected from one feed (the agent in the slave server) to the inventory from another feed (the agent in the host controller). Right now, it is assumed the inventory metadata from feed "A" is matched with metrics from the same feed. Now that we have to break that link (feed A's inventory refers to feed B's metrics) we need to figure out how to fix this.<br>
<br>
There are other ancillary issues - like how do I get the correct metadata defined for host controller so it can match resources/metrics from the slaves. I assume that will be "implementation details."<br>
<br>
I'm sure this sounds like gibberish, but that's how convoluted supporting domain mode is going to be.<br></blockquote><div><br></div><div>Mazz,</div><div><br></div><div>What about to promote the host controller agent as the responsible for everything ?</div><div><br></div><div>- Host controller agent will be responsible to collect info of the domain (slave servers) and write into the inventory (Domain, Server, etc).</div><div>- Slave server can have a "metrics-only" endpoint (perhaps same agent, perhaps a *-domain.jar if we want to simplify things).</div><div>- Host controller agent can proxy slave metrics endpoint, so we can control endpoints like /metrics-domain/<slave>/metrics and that is what inventory uses to create the endpoint.</div><div>- Host controller will expose in a proxy way, the metrics endpoint for the slaves.</div><div><br></div><div>This approach focus the complixity in the host controller, but let to not have exceptions and corner use cases in the whole system, the benefit I see is that from MiQ and Hawkular Services we can try to maintain and uniform way.</div><div><br></div><div>So, as you point Host Controllers needs to know the topology of the domain, and each slave should have a local /metrics which can be proxied from the host controller main endpoint.</div><div>On this case, both Host Controllers and Slaves will share same feed, because, they are using the same agent, and that won't break any design use case.</div><div><br></div><div>Also, I think that this proxy has a minimal technical complexity, as the host controller should have all info to expose that.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
(note: none of this attempts to solve how we are going to get MiQ UI to use this new inventory/metrics data - I have no idea if it will be trivial or be convoluted, too)<br>
______________________________<wbr>_________________<br>
hawkular-dev mailing list<br>
<a href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/hawkular-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/hawkular-dev</a><br>
</blockquote></div><br></div></div></div>