Pushing 'hawkular-1275' branches to master
by Matthew Wringe
Unless there is any objection, we would like to merge the hawkular-1275
branches to master and continue to work in the master branch.
Once we are in the master branch we will also be moving towards a one PR
per jira issue. The workflow in jira will require this and will
automatically close the jira when a PR is merged.
Thanks,
Matt Wringe
8 years
OpenShift Deployment
by Matthew Wringe
With the changes that are now going to include Prometheus, how do we want
to deploy this in OpenShift?
We can have a few options:
ALL-IN-ONE CONTAINER
We put both Hawkular Services and Prometheus in the same container.
Pros:
- easy to deploy in plain docker (but this doesn't appear to be a usecase
we are targetting anyways)
- shares the same network connection (even localhost) and ip address (eg
but both services are on the different ports).
- Does't require any special wiring of components.
- Can share the same volume mount
- version of components can't get out of sync.
Cons:
- workflow doesn't work nicely. Docker containers are meant to only run a
single application and running two can cause problems. Eg lifecycle events
would become tricky and require some hacks to get around things.
- can't independently deploy things
- can't reuse or share any existing Prometheus docker containers.
ALL-IN-ONE POD
Hawkular Services and Prometheus are in their own containers, but they are
both deployed within the same pod.
Pros:
- shares the same network connection.
- bound to the same machine (useful if sharing the same hostpath pv) and
don' need to worry about external network configurations (eg firewalls
between OpenShift nodes)
- pvs can be shared or separate.
- lifecycle events will work properly.
Cons:
- lifecycle hooks will mean that both containers will have to pass before
either one will enter the ready state. So if Prometheus is failing for some
reason, Hawkular Services will not be available under the service.
- cannot independently update one container. If we need to deploy a new
container we will need to bring down the whole pod.
- are stuck with a 1:1 ratio between Hawkular Services and Prometheus
SEPARATE PODS
Hawkular Services and Prometheus have their own separate pods.
Pros:
- can independently run components and each component has its own separate
lifecycle
- if in the future we want to cluster Hawkular Services. this will make it
a lot easier and will also allow for running an n:m ratio between Hawkular
Services and Prometheus
- probably the more 'correct' way to deploy things as we don't have a
strong requirement for Hawkular Services and Prometheus to run together.
Cons:
- more complex wiring. We will need to have extra services and routes
created to handle this. This mean more things running and more chances for
things to go wrong. Also more things to configure
- reusing a PV between Hawkular Services and Prometheus could be more
challenging (especially if we are using hostpath pvs). Updating the
Prometheus scrape endpoint may require a new component and container.
8 years
starting wildfly+agent using new script
by John Mazzitelli
Once my PR [1] is merged, we'll have a script to run the wildfly+agent distro easier.
Get the wildfly+agent (build from here [2]) and run the start-with-hawkular.sh script passing in a few arguments.
Usage is:
Usage: ./start-with-hawkular.sh
-d|--domain - to run in domain mode as opposed to standalone mode
-s|--server <server-name> - the host where the Hawkular Server is running
-u|--username <username> - the username to use when logging into the Hawkular Server
-p|--password <password> - the username credentials
[-h|--metrics-host <host/IP>] - the hostname to bind the metrics exporter
* -u and -p override environment variables HAWKULAR_USER and HAWKULAR_PASSWORD.
You can set those environment variables as an alternative to using -u and -p.
* -h will default to '<your machine here>' if not specified.
--John Mazz
[1] https://github.com/hawkular/hawkular-agent/pull/423
[2] https://github.com/hawkular/hawkular-agent/tree/master/hawkular-javaagent...
8 years
need discussion around how we are going to do domain support
by John Mazzitelli
Spent time trying to figure out how to support WildFly domain mode with the new stuff. It's not going well. Next week I will need to have some discussions on how we want to do this.
The issues with domain mode (I'll try to be short):
1) host controllers do not emit JMX metrics for slave servers - just not implemented in WildFly. Need to go to slave servers for their own JMX metrics.
2) slave servers do not have managed interfaces available remotely (can't connect to slaves over remote management interface - so agent can't get any inventory from slaves - have to get all inventory from host controller)
This means our agent in the host controller needs to collect all inventory for both master host controller and slave servers.
But we need the slave servers to have a metrics endpoint because its only in the slaves where we can get JMX metrics (remember, host controller can't give us that, we must scrape the slave servers for the JMX metrics).
If we do not have our agent in slave server (why bother if we can't connect to it over DMR API to get inventory?), we still need to somehow get the P8s JMX Exporter installed in the slaves. But if we just use the raw jmx exporter agent, how do we tell our h-services server/P8s server about the new endpoint that needs to be scraped? And how do we get the jmx exporter yml config to install there?
So I suppose we should put our agent in a kind of "metrics only" mode in all slave servers so it can expose the JMX exporter and pull down the correct jmx exporter yml from h-services server and have it tell the server to add the scrape endpoint to p8s.
But because we aren't getting inventory from slave servers, how can our agent tell the server to add the scrape endpoint? Our current implementation only adds scrape endpoints to our P8s server when new agents go into inventory. Workaround: either have agent store a small inventory (just the agent resource itself) which triggers the new scrape endpoint addition, or add a inventory REST endpoint for the agent to call to add the new scrape endpoint manually.
OK, assume we have all this. How do we link the JMX metrics getting collected from one feed (the agent in the slave server) to the inventory from another feed (the agent in the host controller). Right now, it is assumed the inventory metadata from feed "A" is matched with metrics from the same feed. Now that we have to break that link (feed A's inventory refers to feed B's metrics) we need to figure out how to fix this.
There are other ancillary issues - like how do I get the correct metadata defined for host controller so it can match resources/metrics from the slaves. I assume that will be "implementation details."
I'm sure this sounds like gibberish, but that's how convoluted supporting domain mode is going to be.
(note: none of this attempts to solve how we are going to get MiQ UI to use this new inventory/metrics data - I have no idea if it will be trivial or be convoluted, too)
8 years
