4:44 a.m.
Hey,
[ CC to Federico as he may have some ideas from the Kube/OS side ]
Our QE has opened an interesting case:
https://github.com/ManageIQ/manageiq/issues/9556
where I first thought WTF with that title.
But then when reading further it got more interesting.
Basically what happens is that especially in environments like
Kube/Openshift,
individual containers/appservers are Kettle and not Pets: one goes down,
gets
killed, you start a new one somewhere else.
Now the interesting question for us are (first purely on the Hawkular
side):
- how can we detect that such a container is down and will never come up
with that id again (-> we need to clean it up in inventory)
- can we learn that for a killed container A, a freshly started
container A' is
the replacement to e.g. continue with performance monitoring of the app
or to re-associate relationships with other items in inventory-
(Is that even something we want - again that is Kettle and not Pets
anymore)
- Could eap+embedded agent perhaps store some token in Kube which
is then passed when A' is started so that A' knows it is the new A (e.g.
feed id).
- I guess that would not make much sense anyway, as for an app with
three app servers all would get that same token.
Perhaps we should ignore that use case for now completely and tackle
that differently in the sense that we don't care about 'real' app
servers,
but rather introduce the concept of a 'virtual' server where we only
know
via Kube that it exists and how many of them for a certain application
(which is identified via some tag in Kube). Those virtual servers
deliver
data, but we don't really try to do anything with them 'personally',
but indirectly via Kube interactions (i.e. map the incoming data to the
app and not to an individual server). We would also not store
the individual server in inventory, so there is no need to clean it
up (again, no pet but kettle).
In fact we could just use the feed-id as kube token (or vice versa).
We still need a way to detect that one of those kettle-as is on Kube
and possibly either disable to re-route some of the lifecycle events
onto Kubernetes (start in any case, stop probably does not matter
if he container dies because the appserver inside stops or if kube
just kills it).
--
Reg. Adresse: Red Hat GmbH, Technopark II, Haus C,
Werner-von-Siemens-Ring 14, D-85630 Grasbrunn
Handelsregister: Amtsgericht München HRB 153243
Geschäftsführer: Charles Cachera, Michael Cunningham, Michael O'Neill,
Eric Shander
7:14 a.m.
New subject: Identification of WildFly in container in a Kube/Openshift env
In case you didn't understand the analogy, I believe Heiko meant to use the word
"Cattle" not "Kettle" :-)
I had to look it up - I've not heard the "cattle vs. pets" analogy before -
but I get it now!
----- Original Message -----
Hey,
[ CC to Federico as he may have some ideas from the Kube/OS side ]
Our QE has opened an interesting case:
https://github.com/ManageIQ/manageiq/issues/9556
where I first thought WTF with that title.
But then when reading further it got more interesting.
Basically what happens is that especially in environments like
Kube/Openshift,
individual containers/appservers are Kettle and not Pets: one goes down,
gets
killed, you start a new one somewhere else.
Now the interesting question for us are (first purely on the Hawkular
side):
- how can we detect that such a container is down and will never come up
with that id again (-> we need to clean it up in inventory)
- can we learn that for a killed container A, a freshly started
container A' is
the replacement to e.g. continue with performance monitoring of the app
or to re-associate relationships with other items in inventory-
(Is that even something we want - again that is Kettle and not Pets
anymore)
- Could eap+embedded agent perhaps store some token in Kube which
is then passed when A' is started so that A' knows it is the new A (e.g.
feed id).
- I guess that would not make much sense anyway, as for an app with
three app servers all would get that same token.
Perhaps we should ignore that use case for now completely and tackle
that differently in the sense that we don't care about 'real' app
servers,
but rather introduce the concept of a 'virtual' server where we only
know
via Kube that it exists and how many of them for a certain application
(which is identified via some tag in Kube). Those virtual servers
deliver
data, but we don't really try to do anything with them 'personally',
but indirectly via Kube interactions (i.e. map the incoming data to the
app and not to an individual server). We would also not store
the individual server in inventory, so there is no need to clean it
up (again, no pet but kettle).
In fact we could just use the feed-id as kube token (or vice versa).
We still need a way to detect that one of those kettle-as is on Kube
and possibly either disable to re-route some of the lifecycle events
onto Kubernetes (start in any case, stop probably does not matter
if he container dies because the appserver inside stops or if kube
just kills it).
--
Reg. Adresse: Red Hat GmbH, Technopark II, Haus C,
Werner-von-Siemens-Ring 14, D-85630 Grasbrunn
Handelsregister: Amtsgericht München HRB 153243
Geschäftsführer: Charles Cachera, Michael Cunningham, Michael O'Neill,
Eric Shander
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
10:30 a.m.
No that I know that I should write about Cattle :) I want to pick this
conversation
up again.
In my Docker setup
http://pilhuhn.blogspot.de/2016/06/using-hawkular-services-via-docker.html
I have a WildFly in a container ("hawkfly") which I can start
with `docker run pilhuhn/hawkfly`. When the container stops, I could
restart it (docker start or by
using a --restart=always policy).
Or I just start a new one with `docker run` as above. Actually I can
start dozens that way
to scale up and down.
In the later case I end up with dozens of 'dead' wildfly servers in
Hawkular
inventory (and thus also in MiQ inventory, as the inventory sync can't
remove them if they are still present in Hawkular inventory).
Before I continue I want to list a few use cases that I identified
a) plain WildFly on bare metal/VM. Always same instance, gets started
and stopped many times, keeps state
This is probably what we always did and do (= a pet)
b) WildFly in container
b1) managed by some orchestration system (= cattle )
b2) started is some more ad-hoc way (e.g. docker-compose, manual
docker-run commands) (= in between cattle and pets)
Now we also need to keep in mind that for applications e.g. a bunch of
app servers may
run the same code for load balancing / fault tolerance reasons.
Now about the relationship to our inventory
For a) it is pretty clear that users see the individual App-servers as
long-living installation. When it crashes,
they restart it, but it stays the same install. So we can easily list
and keep it in inventory and also have
some command to be "manually" to clean up once the user decides that
that installation is really no longer needed.
Also as the AS (usually) has full access to the file system, the feed-id
is preserved over restarts.
For b) the situation is different, as images are immutable and
containers have some "local storage", that
is valid only for the same container. So the WF can only remember its
feed id on restarts of the same
container, but not when I docker run a new one as replacement for an old
one.
Now with both Docker and (most probably) also k8s it is possible to get
a stream of events when
containers are stopped and removed. So we could connect to them and make
use of the information.
The other aspect here is what do we do with the collected data. Right
now out approach is very
pet-centric, which is fine for use case a) above.
For containers, we don't want to have dead cattle in inventory for
forever. We may also not want to remove
the collected metrics, as they can be still important.
For these user cases we should probably abstract this to the level of a
flock. We want to monitor
the size of the flocks and also when flock members die and are born, but
no longer try to identify
individual members. We brand them to denote being part of the flock.
With the flock we can still have individual cattle report their metrics
individually, but we
need to have a way to aggregate over the flock. Similar for alerting
purposes, where we
set up the alert definitions on the flock and not individual members.
Alert definitions need
then be applied to all new members.
For inventory, we can when we learn about a member dying just remove if
from the flock
and adjust counters, record an event about it. Similar for new members.
Now the question is how do we learn about cattle in the flock? When
building images
with the agent inside, we can pass an env-variable or agent setting that
a) indicates that this is an agent inside a docker container
b) indicates which flock this belongs to.
Does that make sense?
Heiko
On 3 Jul 2016, at 14:14, John Mazzitelli wrote:
In case you didn't understand the analogy, I believe Heiko meant
to
use the word "Cattle" not "Kettle" :-)
I had to look it up - I've not heard the "cattle vs. pets" analogy
before - but I get it now!
----- Original Message -----
> Hey,
>
> [ CC to Federico as he may have some ideas from the Kube/OS side ]
>
> Our QE has opened an interesting case:
>
> https://github.com/ManageIQ/manageiq/issues/9556
>
> where I first thought WTF with that title.
>
> But then when reading further it got more interesting.
> Basically what happens is that especially in environments like
> Kube/Openshift,
> individual containers/appservers are Kettle and not Pets: one goes
> down,
> gets
> killed, you start a new one somewhere else.
>
> Now the interesting question for us are (first purely on the Hawkular
> side):
> - how can we detect that such a container is down and will never come
> up
> with that id again (-> we need to clean it up in inventory)
> - can we learn that for a killed container A, a freshly started
> container A' is
> the replacement to e.g. continue with performance monitoring of the
> app
> or to re-associate relationships with other items in inventory-
> (Is that even something we want - again that is Kettle and not Pets
> anymore)
> - Could eap+embedded agent perhaps store some token in Kube which
> is then passed when A' is started so that A' knows it is the new A
> (e.g.
> feed id).
> - I guess that would not make much sense anyway, as for an app
> with
> three app servers all would get that same token.
>
> Perhaps we should ignore that use case for now completely and tackle
> that differently in the sense that we don't care about 'real' app
> servers,
> but rather introduce the concept of a 'virtual' server where we only
> know
> via Kube that it exists and how many of them for a certain
> application
> (which is identified via some tag in Kube). Those virtual servers
> deliver
> data, but we don't really try to do anything with them 'personally',
> but indirectly via Kube interactions (i.e. map the incoming data to
> the
> app and not to an individual server). We would also not store
> the individual server in inventory, so there is no need to clean it
> up (again, no pet but kettle).
> In fact we could just use the feed-id as kube token (or vice versa).
> We still need a way to detect that one of those kettle-as is on Kube
> and possibly either disable to re-route some of the lifecycle events
> onto Kubernetes (start in any case, stop probably does not matter
> if he container dies because the appserver inside stops or if kube
> just kills it).
10:58 a.m.
On 22.07.2016 17:30, Heiko W.Rupp wrote:
No that I know that I should write about Cattle :) I want to pick
this
conversation
up again.
Before I share my comments on the individual parts, I have a question.
Do we still see the main monitored subject as the application, or do we
care about the OS/environment? In other words: does it make sense to
have two application instances with the same feed-id? If so, having an
alternate algorithm for containers that comes up with an ID based on
local artifacts would solve most of the issues, I believe.
- Juca.
6:41 a.m.
On 22 Jul 2016, at 17:58, Juraci Paixão Kröhling wrote:
Do we still see the main monitored subject as the application, or do
we
care about the OS/environment? In other words: does it make sense to
have two application instances with the same feed-id? If so, having
I am not sure I understood the question.
Right now the feed-id more or less "identifies the agent". In a case where
one WildFly has one embedded agent, the feed-id more or less also
identifies that server (at least in standalone mode; I don't think Domain+
Docker/K8s makes too much sense, as you now would have 2 competing
orchestration systems, but I may be wrong).
Now what is an "application instance". If it is a WF running a certain .?ar
file, I don't think it does not make sense to have two with the same
feed-id with this current model I described.
an
alternate algorithm for containers that comes up with an ID based on
local artifacts would solve most of the issues, I believe.
Can you describe that please?
Heiko
11:02 a.m.
On 27.07.2016 13:41, Heiko W.Rupp wrote:
On 22 Jul 2016, at 17:58, Juraci Paixão Kröhling wrote:
> Do we still see the main monitored subject as the application, or do we
> care about the OS/environment? In other words: does it make sense to
> have two application instances with the same feed-id? If so, having
I am not sure I understood the question.
IIRC, at the beginning of Hawkular, the main idea was to have metrics on
a per-application basis, instead of a per-deployment/instance. The OS
metrics were an "extra". So, the question is if we still have a focus on
the "per application metrics", no matter how many instances (containers,
VMs, ...) are there for this application. Or if we still need the OS
metrics, even for container environments. If we need the OS metrics,
then we need one feed-id per "application" instance.
Right now the feed-id more or less "identifies the agent".
In a case where
one WildFly has one embedded agent, the feed-id more or less also
identifies that server (at least in standalone mode; I don't think Domain+
Docker/K8s makes too much sense, as you now would have 2 competing
orchestration systems, but I may be wrong).
Now what is an "application instance". If it is a WF running a certain .?ar
file, I don't think it does not make sense to have two with the same
feed-id with this current model I described.
Right, that's my point. For an infra where the application is deployed
on Docker containers, like in OpenShift, it doesn't matter to Hawkular
how many containers are there for a given application: Openshift would
auto scale up and down on demand. In such scenario, the feed-id isn't of
much help, as it would change very often.
> an
> alternate algorithm for containers that comes up with an ID based on
> local artifacts would solve most of the issues, I believe.
Can you describe that please?
Sure: if we assume that the feed-id for an image should remain the same
for all instances of that image (I'm not sure yet that's the case), then
the alternative algorithm could be used when the Agent starts, based on
the checksum for the content of the artifacts found on
standalone/deployments . Example:
- standalone/deployments/foobar.war
sha1sum: 96866e09c9ade125d9f3d3fc9766ccba5961968e
- standalone/deployments/foobar-ds.xml
sha1sum: 644faffab30df784c995fa75e4ff84bcf0857d6e
- feed-id: sha1sum(sha1sum(foobar.war) + sha1sum(foobar-ds.xml))
result: a4f36be49f641967e51e2707c1fe6d0a29c2f682
If there's a new image for the foobar-service, there are two possible
scenarios:
- Only the underlying artifacts were changed (OS updates, EAP updates,
...), but the foobar-service itself is the same. On this scenario, the
resulting sha1sum is the same, so, the same feed-id as before is used.
- foobar-service has changed, no matter if the OS was updated or not:
the sha1sum is new, so, new feed-id.
For non-container deployments, the feed-id could still be generated the
way it is today. A new configuration parameter on standalone.xml would
make sense, on this case, to specify the feed-id algo to use.
- Juca.
11:51 a.m.
- feed-id: sha1sum(sha1sum(foobar.war) + sha1sum(foobar-ds.xml))
result: a4f36be49f641967e51e2707c1fe6d0a29c2f682
Calculating the feed ID based on discovered resources is going to be very difficult if not
impossible with the current implementation - the agent needs the feed ID upfront during
startup before it even runs discovery (i.e. before it even knows about any resources - let
along child resources like deployments) - it requires the feed ID internally to start many
components, it can't talk to hawkular-inventory without it, and it needs the feed ID
to connect to the hawkular cmdgw server.
12:06 p.m.
On 27.07.2016 18:51, John Mazzitelli wrote:
> - feed-id: sha1sum(sha1sum(foobar.war) + sha1sum(foobar-ds.xml))
> result: a4f36be49f641967e51e2707c1fe6d0a29c2f682
Calculating the feed ID based on discovered resources is going to be very difficult if
not impossible with the current implementation - the agent needs the feed ID upfront
during startup before it even runs discovery (i.e. before it even knows about any
resources - let along child resources like deployments) - it requires the feed ID
internally to start many components, it can't talk to hawkular-inventory without it,
and it needs the feed ID to connect to the hawkular cmdgw server.
I considered this, but I thought it wouldn't be a problem to implement a
separate Wildfly extension that would calculate this during the boot and
put the resulting value in a system property for later use by the agent
itself. Would that work?
- Juca.
12:08 p.m.
On 27.07.2016 19:06, Juraci Paixão Kröhling wrote:
On 27.07.2016 18:51, John Mazzitelli wrote:
>> - feed-id: sha1sum(sha1sum(foobar.war) + sha1sum(foobar-ds.xml))
>> result: a4f36be49f641967e51e2707c1fe6d0a29c2f682
>
> Calculating the feed ID based on discovered resources is going to be very difficult
if not impossible with the current implementation - the agent needs the feed ID upfront
during startup before it even runs discovery (i.e. before it even knows about any
resources - let along child resources like deployments) - it requires the feed ID
internally to start many components, it can't talk to hawkular-inventory without it,
and it needs the feed ID to connect to the hawkular cmdgw server.
I considered this, but I thought it wouldn't be a problem to implement a
separate Wildfly extension that would calculate this during the boot and
put the resulting value in a system property for later use by the agent
itself. Would that work?
By the way: I don't think this extension needs to wait for the deployers
either: all it would need to do is to get a VirtualJarInputStream for
each of the deployments, and calculate the sha1sum based on that.
- Juca.
8:31 a.m.
I would say first we should not expect people to be happy to have install a _second_
subsystem extension to just to support Hawkular (having a separate feature pack,
installer, separate <subsystem> in standalone.xml, etc. etc). But we could add more
extension *services* or DUPs (see below) to the Hawkular WildFly Agent subsystem itself
and it just comes with the agent with no additional installation or configs necessary.
Also, this feature of building a feed ID based on content within the app server is going
to have drawbacks when you consider it won't work if the app server's deployments
could ever change (add deployments, remove deployments, or even just patching existing
deployments if the version string is part of the deployment filename [foobar-1.0.war ->
foobar-2.0.war]) So this feature would be very restrictive in who should be using it and
people should be very aware of it in case they turn that feature on when they
shouldn't (because the consequences would be bad - the agent's feed ID would
change underneath of it simply when a deployment changed).
That said, what might be possible (given the restrictions noted above), if we do want to
support such a thing, we could allow a new pre-defined value for
<storage-adapter>'s "feed-id" attribute. We already have one -
"autogenerate". We could have something like
"autogenerate-based-on-deployments". If the agent sees this, it could in theory
add a DUP (DeploymentUnitProcessor) to the series of DUPs in the app server. This DUP of
ours would pass through the deployments, but before it does it looks at the deployment
name and keeps a running sha1 hash, building up the final hash when all deployments are
passed through, put it somewhere the agent can access, and that will be the feed ID. (note
if the "real" deployer that will handle that deployment actually rejects the
deployment, we might have a problem because we'll have hashed on something that
isn't really deployed - I don't know if our DUP can see when a deployment is
rejected down the chain).
We already do this kind of thing in RHQ. We add a DUP [1] and in the DUP (in RHQ anyway)
we look at the name of the deployment and if it isn't rhq.war, the deployment is
rejected [2]. For Hawkular, we just let it pass through and deploy but we'd create the
hash based on the name.
[1]
https://github.com/rhq-project/rhq/blob/master/modules/enterprise/server/...
[2]
https://github.com/rhq-project/rhq/blob/master/modules/enterprise/server/...
----- Original Message -----
On 27.07.2016 19:06, Juraci Paixão Kröhling wrote:
> On 27.07.2016 18:51, John Mazzitelli wrote:
>>> - feed-id: sha1sum(sha1sum(foobar.war) + sha1sum(foobar-ds.xml))
>>> result: a4f36be49f641967e51e2707c1fe6d0a29c2f682
>>
>> Calculating the feed ID based on discovered resources is going to be very
>> difficult if not impossible with the current implementation - the agent
>> needs the feed ID upfront during startup before it even runs discovery
>> (i.e. before it even knows about any resources - let along child
>> resources like deployments) - it requires the feed ID internally to start
>> many components, it can't talk to hawkular-inventory without it, and it
>> needs the feed ID to connect to the hawkular cmdgw server.
>
> I considered this, but I thought it wouldn't be a problem to implement a
> separate Wildfly extension that would calculate this during the boot and
> put the resulting value in a system property for later use by the agent
> itself. Would that work?
By the way: I don't think this extension needs to wait for the deployers
either: all it would need to do is to get a VirtualJarInputStream for
each of the deployments, and calculate the sha1sum based on that.
- Juca.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
9:08 a.m.
On 28.07.2016 15:31, John Mazzitelli wrote:
Also, this feature of building a feed ID based on content within the
app server is going to have drawbacks when you consider it won't work if the app
server's deployments could ever change (add deployments, remove deployments, or even
just patching existing deployments if the version string is part of the deployment
filename [foobar-1.0.war -> foobar-2.0.war]) So this feature would be very restrictive
in who should be using it and people should be very aware of it in case they turn that
feature on when they shouldn't (because the consequences would be bad - the
agent's feed ID would change underneath of it simply when a deployment changed).
On the context of Docker, the deployments will "never" change. The
appropriate way to deploy a new version of the WAR, or to change a DS is
to generate a new image, stop the containers running the old images and
start new containers using the new images.
That said, what might be possible (given the restrictions noted
above), if we do want to support such a thing, we could allow a new pre-defined value for
<storage-adapter>'s "feed-id" attribute. We already have one -
"autogenerate". We could have something like
"autogenerate-based-on-deployments". If the agent sees this, it could in theory
add a DUP (DeploymentUnitProcessor) to the series of DUPs in the app server. This DUP of
ours would pass through the deployments, but before it does it looks at the deployment
name and keeps a running sha1 hash, building up the final hash when all deployments are
passed through, put it somewhere the agent can access, and that will be the feed ID. (note
if the "real" deployer that will handle that deployment actually rejects the
deployment, we might have a problem because we'll have hashed on something that
isn't really deployed - I don't know if our DUP can see when a deployment is
rejected down the chain).
s/deployment name/deployment content/ and we are in agreement :) The
name of the deployment would possibly always be the same
(foobar-service.war), but not its content.
- Juca.
9:25 a.m.
Calculating the feed ID based on discovered resources is going to be very
difficult if not impossible with the current implementation - the agent
needs the feed ID upfront during startup before it even runs discovery
(i.e. before it even knows about any resources - let along child resources
like deployments) - it requires the feed ID internally to start many
components, it can't talk to hawkular-inventory without it, and it needs
the feed ID to connect to the hawkular cmdgw server.
Updating the feed name ex post:
https://git.io/vKAl2
Calculating the feed ID based on discovered resources... - identity hashes
in inventory, Lukas knows
On Wed, Jul 27, 2016 at 6:51 PM, John Mazzitelli <mazz(a)redhat.com> wrote:
> - feed-id: sha1sum(sha1sum(foobar.war) +
sha1sum(foobar-ds.xml))
> result: a4f36be49f641967e51e2707c1fe6d0a29c2f682
Calculating the feed ID based on discovered resources is going to be very
difficult if not impossible with the current implementation - the agent
needs the feed ID upfront during startup before it even runs discovery
(i.e. before it even knows about any resources - let along child resources
like deployments) - it requires the feed ID internally to start many
components, it can't talk to hawkular-inventory without it, and it needs
the feed ID to connect to the hawkular cmdgw server.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
10:58 a.m.
Calculating the feed ID based on discovered resources... - identity
hashes
in inventory, Lukas knows
chicken and the egg - agent needs to know its feed ID before it can even put anything into
inventory.
But the fact an agent can change its feed ID via inventory REST call is interesting.
However, that might not fly because its more than Inventory we are worried about. The
agent stores things in Hawkular *Metrics* that refer to the (now old) feed ID (like metric
IDs and metric tag names and values).
I don't like this talk of changing feed IDs at runtime. This goes against everything
we talked about and designed when we started. Feeds identify an agent - and a feed ID
should never change (at least, that's what we said up until now, and everything is
implemented in the agent with that assumption in mind). Allowing an agent to change its
feed ID during its lifetime is going to cause problems - probably even problems we
haven't thought of yet.
IMO, feed IDs need to be locked. An agent, once it determines its feed ID and registers
it, should never change that feed ID.
11:34 a.m.
Hi,
Is there any reason the agent can't use container as identification?
HOSTNAME="hawkular-cassandra-1-6o1y8"
I guess that's exported to all Openshift containers at least (I don't have
Kubernetes instance to check). Unfortunately the image id was not there, but that hostname
includes the container-name + generated id. If you deploy additional instances, you get a
new id, redeploy - new id.
- Micke
----- Original Message -----
From: "John Mazzitelli" <mazz(a)redhat.com>
To: "Discussions around Hawkular development"
<hawkular-dev(a)lists.jboss.org>
Sent: Thursday, July 28, 2016 6:58:30 PM
Subject: Re: [Hawkular-dev] Identification of WildFly in container in a Kube/Openshift
env
Calculating the feed ID based on discovered resources... - identity
hashes
in inventory, Lukas knows
chicken and the egg - agent needs to know its feed ID before it can even put anything into
inventory.
But the fact an agent can change its feed ID via inventory REST call is interesting.
However, that might not fly because its more than Inventory we are worried about. The
agent stores things in Hawkular *Metrics* that refer to the (now old) feed ID (like metric
IDs and metric tag names and values).
I don't like this talk of changing feed IDs at runtime. This goes against everything
we talked about and designed when we started. Feeds identify an agent - and a feed ID
should never change (at least, that's what we said up until now, and everything is
implemented in the agent with that assumption in mind). Allowing an agent to change its
feed ID during its lifetime is going to cause problems - probably even problems we
haven't thought of yet.
IMO, feed IDs need to be locked. An agent, once it determines its feed ID and registers
it, should never change that feed ID.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
12:08 p.m.
Is there any reason the agent can't use container as
identification?
HOSTNAME="hawkular-cassandra-1-6o1y8"
Things like that would work well.
We could even allow the agent to support notation like this:
<storage-adapter feed-id="%envvar(HOSTNAME)" ... />
or whatever syntax makes sense
1:09 p.m.
Are multiple servers/agents per container a possibility? If so we'd
need to append a differentiator to the container identifier.
Also, with respect to a feedid based on deployments, isn't it possible
that multiple containers run the same deployments for load-balancing
purposes? Wouldn't that result in multiple agents running with the same
feedid? Anyway, It seems if we could derive a feedid from the
container, with perhaps a new token like autogenerate-from-container,
that could be a nice way to easily identify a feed for a container.
This doesn't solve the issue of replacing a stopped container with a new
container, thus generating new inventory and orphaning the existing
inventory. But in that situation I think we may just want to look for a
decent way to kill that inventory, like TTL or something.
On 7/28/2016 1:08 PM, John Mazzitelli wrote:
> Is there any reason the agent can't use container as
identification?
>
> HOSTNAME="hawkular-cassandra-1-6o1y8"
Things like that would work well.
We could even allow the agent to support notation like this:
<storage-adapter feed-id="%envvar(HOSTNAME)" ... />
or whatever syntax makes sense
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
1:22 p.m.
Hi,
Running multiple servers per pod sounds like a defeat of the whole idea of docker ;). If
you run multiple instances of this same pod, each one will have a different hostname (the
hawkular-cassandra would stay the same, but the part after it would change).
- Micke
----- Original Message -----
From: "Jay Shaughnessy" <jshaughn(a)redhat.com>
To: hawkular-dev(a)lists.jboss.org
Sent: Friday, July 29, 2016 9:09:02 PM
Subject: Re: [Hawkular-dev] Identification of WildFly in container in a Kube/Openshift
env
Are multiple servers/agents per container a possibility? If so we'd need to append a
differentiator to the container identifier.
Also, with respect to a feedid based on deployments, isn't it possible that multiple
containers run the same deployments for load-balancing purposes? Wouldn't that result
in multiple agents running with the same feedid? Anyway, It seems if we could derive a
feedid from the container, with perhaps a new token like autogenerate-from-container, that
could be a nice way to easily identify a feed for a container. This doesn't solve the
issue of replacing a stopped container with a new container, thus generating new inventory
and orphaning the existing inventory. But in that situation I think we may just want to
look for a decent way to kill that inventory, like TTL or something.
On 7/28/2016 1:08 PM, John Mazzitelli wrote:
Is there any reason the agent can't use container as identification?
HOSTNAME="hawkular-cassandra-1-6o1y8"
Things like that would work well.
We could even allow the agent to support notation like this:
<storage-adapter feed-id="%envvar(HOSTNAME)" ... />
or whatever syntax makes sense
_______________________________________________
hawkular-dev mailing list hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
9:20 a.m.
On 27 Jul 2016, at 17:02, Juraci Paixão Kröhling wrote:
Sure: if we assume that the feed-id for an image should remain the
same for all instances of that image (I'm not sure yet that's the
case), then the alternative algorithm could be used when the Agent
starts, based on the checksum for the content of the artifacts found
on standalone/deployments . Example:
- standalone/deployments/foobar.war
sha1sum: 96866e09c9ade125d9f3d3fc9766ccba5961968e
There is one case we need to consider here:
we have an image with foobar.war;v1 with a certain sha.
Now the user creates a new image with foobar.war;v2.
For the user this is logically the same thing but with a
different version.
We should not treat that as different applications.
In this case we would rather need to record the fact
that v1 was replaced by v2 (by putting a tag) and
continue the already recorded metrics for the same
(logical) application.
9:31 a.m.
On 28.07.2016 16:20, Heiko W.Rupp wrote:
There is one case we need to consider here:
we have an image with foobar.war;v1 with a certain sha.
Now the user creates a new image with foobar.war;v2.
For the user this is logically the same thing but with a
different version.
We should not treat that as different applications.
In this case we would rather need to record the fact
that v1 was replaced by v2 (by putting a tag) and
continue the already recorded metrics for the same
(logical) application.
Do you mean that the contents of v1 are different of that for v2? If so,
only the feed-id will be different. Everything else is as it is today:
it's the same case as if the company had deployed a new VM with the v2,
router some traffic to it for A/B testing, and shut down the v1 once
they are happy with v2.
If it is important to have consistent feed-ids across the whole life of
the application, then I see no other option than to "force" the consumer
to specify a feed-id on the image building process.
- Juca.
10:14 a.m.
On 28 Jul 2016, at 15:31, Juraci Paixão Kröhling wrote:
Do you mean that the contents of v1 are different of that for v2?
Yes.
If it is important to have consistent feed-ids across the whole life
of the application, then I see no other option than to "force" the
consumer to specify a feed-id on the image building process.
I did not say that :)
What I said is that for the 'foobar' application we want to have a
continuity of the recorded data even if the version changed.
In my post some days ago I wrote:
For these user cases we should probably abstract this to the level
of a flock. We want to monitor the size of the flocks and also when
flock members die and are born, but no longer try to identify
individual members. We brand them to denote being part of the flock.
We can put labels on the deployments and those labels are reported
back into the Hawkular. So even if the feed id changes we can still
identify that the server/node belongs to 'foobar'.
11:29 a.m.
Heiko,
OpenShift source-to-image (S2I) does what you just describe. The DeploymentConfig
('app' config or cattle breeder if you will) remains static.
Very likely that in a production OpenShift setup there will be a persistent storage for
the WF instance. So regardless of where the WF pod has lived we can still identify its
former self. No?
Viet
----- Original Message -----
From: "Heiko W.Rupp" <hrupp(a)redhat.com>
To: "Juraci Paixão Kröhling" <jpkroehling(a)redhat.com>
Cc: "Discussions around Hawkular development"
<hawkular-dev(a)lists.jboss.org>
Sent: Thursday, July 28, 2016 7:20:59 AM
Subject: Re: [Hawkular-dev] Identification of WildFly in container in a Kube/Openshift
env
On 27 Jul 2016, at 17:02, Juraci Paixão Kröhling wrote:
Sure: if we assume that the feed-id for an image should remain the
same for all instances of that image (I'm not sure yet that's the
case), then the alternative algorithm could be used when the Agent
starts, based on the checksum for the content of the artifacts found
on standalone/deployments . Example:
- standalone/deployments/foobar.war
sha1sum: 96866e09c9ade125d9f3d3fc9766ccba5961968e
There is one case we need to consider here:
we have an image with foobar.war;v1 with a certain sha.
Now the user creates a new image with foobar.war;v2.
For the user this is logically the same thing but with a
different version.
We should not treat that as different applications.
In this case we would rather need to record the fact
that v1 was replaced by v2 (by putting a tag) and
continue the already recorded metrics for the same
(logical) application.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
7:19 a.m.
New subject: Identification of WildFly in container in a Kube/Openshift env
Is there some mechanism by which the agent can know if it (or the EAP server it is
managing) is running inside a container? I'm thinking of something analogous to
/etc/machine-id - perhaps when running in a container, Kube sets some environment
variable, file system token, something?
If there is a way to know, then the agent can just write some resource config property
somewhere to say "this 'thing' is running in a container." So when that
"thing" goes down, the server-side can be notified and do special things (clean
up inventr? send an alert?
----- Original Message -----
Hey,
[ CC to Federico as he may have some ideas from the Kube/OS side ]
Our QE has opened an interesting case:
https://github.com/ManageIQ/manageiq/issues/9556
where I first thought WTF with that title.
But then when reading further it got more interesting.
Basically what happens is that especially in environments like
Kube/Openshift,
individual containers/appservers are Kettle and not Pets: one goes down,
gets
killed, you start a new one somewhere else.
Now the interesting question for us are (first purely on the Hawkular
side):
- how can we detect that such a container is down and will never come up
with that id again (-> we need to clean it up in inventory)
- can we learn that for a killed container A, a freshly started
container A' is
the replacement to e.g. continue with performance monitoring of the app
or to re-associate relationships with other items in inventory-
(Is that even something we want - again that is Kettle and not Pets
anymore)
- Could eap+embedded agent perhaps store some token in Kube which
is then passed when A' is started so that A' knows it is the new A (e.g.
feed id).
- I guess that would not make much sense anyway, as for an app with
three app servers all would get that same token.
Perhaps we should ignore that use case for now completely and tackle
that differently in the sense that we don't care about 'real' app
servers,
but rather introduce the concept of a 'virtual' server where we only
know
via Kube that it exists and how many of them for a certain application
(which is identified via some tag in Kube). Those virtual servers
deliver
data, but we don't really try to do anything with them 'personally',
but indirectly via Kube interactions (i.e. map the incoming data to the
app and not to an individual server). We would also not store
the individual server in inventory, so there is no need to clean it
up (again, no pet but kettle).
In fact we could just use the feed-id as kube token (or vice versa).
We still need a way to detect that one of those kettle-as is on Kube
and possibly either disable to re-route some of the lifecycle events
onto Kubernetes (start in any case, stop probably does not matter
if he container dies because the appserver inside stops or if kube
just kills it).
--
Reg. Adresse: Red Hat GmbH, Technopark II, Haus C,
Werner-von-Siemens-Ring 14, D-85630 Grasbrunn
Handelsregister: Amtsgericht München HRB 153243
Geschäftsführer: Charles Cachera, Michael Cunningham, Michael O'Neill,
Eric Shander
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
1:17 a.m.
cattle vs pet monitoring is something I struggle with TBH...
It doesn't make much sense to keep all data about all elements of the
cattle as you are less interested by the performance of 1 member but more
about the overall performance.
With auto scaling, new containers are created/removed. You add one, you
remove it, you read one, there is no continuation unlike when you restart a
server... A configuration change is also not a continuation anymore, it's a
whole new image, whole new container (in good practice at least)
IMO we should keep thinking about those, and think more in terms of
collection for the cases when Middleware is running in (immutable)
containers...
Thomas
On Sun, Jul 3, 2016 at 11:44 AM, Heiko W.Rupp <hrupp(a)redhat.com> wrote:
Hey,
[ CC to Federico as he may have some ideas from the Kube/OS side ]
Our QE has opened an interesting case:
https://github.com/ManageIQ/manageiq/issues/9556
where I first thought WTF with that title.
But then when reading further it got more interesting.
Basically what happens is that especially in environments like
Kube/Openshift,
individual containers/appservers are Kettle and not Pets: one goes down,
gets
killed, you start a new one somewhere else.
Now the interesting question for us are (first purely on the Hawkular
side):
- how can we detect that such a container is down and will never come up
with that id again (-> we need to clean it up in inventory)
- can we learn that for a killed container A, a freshly started
container A' is
the replacement to e.g. continue with performance monitoring of the app
or to re-associate relationships with other items in inventory-
(Is that even something we want - again that is Kettle and not Pets
anymore)
- Could eap+embedded agent perhaps store some token in Kube which
is then passed when A' is started so that A' knows it is the new A (e.g.
feed id).
- I guess that would not make much sense anyway, as for an app with
three app servers all would get that same token.
Perhaps we should ignore that use case for now completely and tackle
that differently in the sense that we don't care about 'real' app
servers,
but rather introduce the concept of a 'virtual' server where we only
know
via Kube that it exists and how many of them for a certain application
(which is identified via some tag in Kube). Those virtual servers
deliver
data, but we don't really try to do anything with them 'personally',
but indirectly via Kube interactions (i.e. map the incoming data to the
app and not to an individual server). We would also not store
the individual server in inventory, so there is no need to clean it
up (again, no pet but kettle).
In fact we could just use the feed-id as kube token (or vice versa).
We still need a way to detect that one of those kettle-as is on Kube
and possibly either disable to re-route some of the lifecycle events
onto Kubernetes (start in any case, stop probably does not matter
if he container dies because the appserver inside stops or if kube
just kills it).
--
Reg. Adresse: Red Hat GmbH, Technopark II, Haus C,
Werner-von-Siemens-Ring 14, D-85630 Grasbrunn
Handelsregister: Amtsgericht München HRB 153243
Geschäftsführer: Charles Cachera, Michael Cunningham, Michael O'Neill,
Eric Shander
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
6:59 a.m.
On 5 Jul 2016, at 8:17, Thomas Heute wrote:
cattle vs pet monitoring is something I struggle with TBH...
It doesn't make much sense to keep all data about all elements of the
cattle as you are less interested by the performance of 1 member but more
about the overall performance.
There is a grey area in my opinion: suppose you have 2 containers
running. One dies and another one is spun up.
In this case I would still want to see the collected data of the dead
container with some sort of marker that it died and got replaced.
3047
days inactive
3094
days old
24 comments
8 participants
participants (8)
-
Heiko W.Rupp -
Jay Shaughnessy -
Jiri Kremser -
John Mazzitelli -
Juraci Paixão Kröhling -
Michael Burman -
Thomas Heute -
Viet Nguyen