September 2015 - hawkular-dev - Jboss List Archives

Coming to inventory 0.4.0 - bulk create of entities

by Lukas Krejci

Hi all, pending a peer-review is a new feature in inventory to support bulk creation of entities (in a single transaction). So far, our REST API was rather granular and concentrated on working with single entities or even their individual relationships. This was quite costly especially during create, update or delete operations, where the clients would need to do stuff one-by-one. This is being addressed in Inventory 0.4.0 (hopefully coming to Hawkular Alpha5) with a new REST endpoint called "/bulk". For 0.4.0 we will only support bulk creation with update and delete coming in the next versions. The bulk endpoint accepts a rather complex javascript object that can be used to specify what is being created and where. The best way to explain it is by an example: { "/t;tenant": { "environment": [ { "id": "env1", "properties": {"key": "value"} }, { "id": "env2", "properties": {"key": "value2"} } ], "resourceType": [ { "id": "URL" } ], "metricType": [ { "id": "ResponseTime" } ] }, "/t;tenant/rt;URL": { "data": [ { "role": "configurationSchema", "value": { "title": "URL config schema", "description": "A json schema describing configuration of an URL", "type": "string" } } ], "operationType": [ { "id": "ping" } ] }, "/t;tenant/e;env1": { "resource": [ { "id": "url1", "resourceTypePath": "/t;tenant/rt;URL" } ], "metric": [ { "id": "url1_responseTime", "metricTypePath": "/t;tenant/mt;ResponseTime" } ] }, "/t;tenant/e;env1/r;url1": { "data": [ { "role": "configuration", "value": "http://redhat.com" } ], "relationship": [ { "name": "incorporates", "otherEnd": "/t;tenant/e;env1/m;url1_responseTime", "direction": "outgoing" } ] } } The javascript object above would cause creation of 2 new environments, "env1" and "env2", under the tenant "tenant", a new resource type called "URL" and a new metric type called "ResponseTime". When the URL resource type is created, a configuration schema is created for it (and the configuration schema prescribes that the configuration of the URL is merely a string). Additionally, an operation called "ping" is defined for the resource type. After that a new resource, "url1" is added to environment "env1" with the "URL" resource type and a metric called "url1_responseTime" is also added to the environment. Finally a configuration is added to the resource and the resource is configured to incorporate the previously created metric. The structure of the javascript object is following: 1) The top-level keys are paths to the inventory entities *under* which the new entities should be added, e.g. the first key "/t;tenant" means that the value of that key describes what should be created directly under a tenant called "tenant". The second key, "/t;tenant/rt;URL", tells the REST API that the value of that key describes what should be added "under" that resource type. The structure of what kinds of entities can contain what kinds of entities can be found in the documentation [1]. Notice that the order of the keys in the object is significant. 2) The second-level keys describe what type of data to create under a given parent (notice that you can create more than 1 type of data under some entity types). The values of those keys are arrays of actual blueprint objects describing the new entities to be created (each entity type can have different format). The blueprint objects are the same as the ones passed in to the granular REST endpoints that already exist. The response always has the 201 - Created status code and is a JSON object with the types of entities created as keys and values are again objects with keys being the canonical paths to the entities created and values are HTTP status codes describing the result of the creation (201, 409, etc.). E.g. for the above example the following would be a response: { "environment": { "/t;tenant/e;env1": 201, "/t;tenant/e;env2": 201 }, "resourceType": { "/t;tenant/rt;URL": 201 } "metricType": { "/t;tenant/mt;ResponseTime": 201 }, "data": { "/t;tenant/rt;URL/d;configurationSchema": 201, "/t;tenant/e;env1/r;url1/d;configuration": 201 }, "operationType": { "/t;tenant/rt;URL/ot;ping": 201 }, "resource": { "/t;tenant/e;env1/r;url1": 201 }, "metric": { "/t;tenant/e;env1/m;url1_responseTime": 201 }, "relationship": { "<some-random-id>": 201 } } Hope you will find this useful, Lukas [1] http://www.hawkular.org/docs/components/inventory/index.html#inventory-or...

9 years, 3 months

3
6
0 / 0

how to identify UI clients and which should get responses

by John Mazzitelli

I'm sending this here because I need the input from several people (specifically Heiko, Juca, and Peter, but others are free to chime in, too). Right now, the UI sends messages over the websocket to the hawkular server to make request for things. That UI request is immediately authenticated via hawkular-accounts (typically the UI sends the keycloak token that we authentiate). At this point, we put the message on the bus so the message can be routed to the proper kettle instance so the message can be forwarded to the feed over the feed websocket connection. Once the feed processes the request, a response may be sent back (asynchronously from the request) - if a response is sent back from the feed, it goes to the kettle instance via websockets, the kettle puts it on the bus, where the listener processing the UI client responses will take the bus message and forward it to the UI over the websocket. Phew. OK, my question is - how does the UI identify itself? We need some string identifier so the response message can be targeted to the proper UI client (today, we don't have UI identifiers, so we just send responses to EVERY UI client but only those UI clients connected to the kettle that the feed is connected to - not good in a clustered environment and not good for security purposes - some UIs will be different tenants and shouldn't see these messages). Here's some use cases to illustrate the problem: 1) I'm logged into the UI using two different browser tabs. In one, I send a "deploy an application" request. Once the response comes back - do I show the response in both tabs? What if I closed the tab that performed the deploy? What should happen? 2) I am logged in kettle #1 as "jdoe" and someone else is logged in kettle #2 as the same "jdoe". I deploy an application. Should that other person get the response that a new app was deployed, too? 3) I am logged into the UI and I deploy an application. I log out (perhaps close the browser) and log back in. Then the deployment finishes and the response comes in - should I get that response (even though its a new browser instance and a new websocket connection?) The way I am thinking about this is Google Drive. I could be logged into my Google Drive account on any number of browsers or tabs - when I make a change to a document in Drive, that change is immediately reflected in every browser/tab I have open to Google Drive, not just in the browser that made the change. Is this the kind of thing we want? Or do we only want the browser that is making the requests get the responses? One thing I am thinking is hawkular-accounts gives us some common ID (not the token) that we assign a UI client and that ID floats in all requests and responses. The bad thing about this is we'll never know the UI client ID until they send their first request. Another thing I am thinking is have it work just like feeds - a feed connects via the URL ws://host/hawkular-command-gateway/feed/<FEED ID>. We could let each UI generate their own id and connect via ws://host/hawkular-command-gateway/ui/<UI ID>. The good thing here is the UI knows its ID immediately (since it generated it) and we know it from the start without even getting an initial message (and we can link it with that websocket session easily). Bad thing is only that UI will be able to get responses - it won't work the "Google Drive way" where a user logged into N browsers will get responses in all N browsers. Thoughts?

9 years, 3 months

3
3
0 / 0

RfC: Remove / rename servers from / in inventory

by Heiko W.Rupp

Hey, currently we can add (WF) servers to inventory easily, but we can't get rid of them anymore. Similarly it is not really clear what should happen if a serve changes its name. In WildFly 10 supposedly a new uuid kind of thing should appear, that would identify each WF uniquely and which could serve as the "primary key", so that the name is just a display attribute that can be changed at will, so that the renaming piece is sort of solved. Now to the semantics of "uninventory" (not deletion of physical servers). In RHQ we had uninventory that removed the server from inventory. After a discovery scan happened, it showed up again and could be re-inventoried. During uninventory, all collected metrics etc. and alert definitions for the resource were wiped (or prepared for wiping). This (wiping of recorded metrics) has more than one time been mentioned as undesired e.g. in case that the uninventory happened due to bad luck. Also in RHQ there were issues when e.g. the machine where a WF-server was running on got renamed. Some questions that come to mind: - Should we wipe data when a server gets uninventoried? - How can we prevent it from entering inventory directly after an uninventory again? - what states do we need (data wiping may take too long to be done synchronously)? - would users perhaps want a data dump to external storage before wiping? - How can we make sure that servers without that uuid can still be identified even after rename (of the machine)? - How can we make sure that servers without that uuid can still be identified even when they are moved to a different pod Heiko

9 years, 3 months

4
5
0 / 0

[Metrics] HWKMETRICS-125 Use JBoss Logging as in all other Hawkular components

by Thomas Segismont

Hi, I created a pull request for $subject. https://github.com/hawkular/hawkular-metrics/pull/352 I'm writing to share some implementation details. I followed some Hawkular wide conventions (which I gathered from mailing list threads, so tell me if it's wrong): - put logging stuff into a 'log' package - in the message logger interface, method names should start with the log level [1] - logger instances should be named 'log' [2] - only put messages which need an id (level INFO and above) in logger interfaces [3] I tried to draw up some Metrics specific conventions (some may not be specific, I just found no mention of them in the threads): - group all messages in one class for maintainability [4] - instead of creating a logger instance in the message logger interface, use XXXLogging helper class to get a logger instance with proper category (inspired by what Hibernate does) [5] - use existing message logger instance to log trace/debug messages instead of creating a separate logger instance [6] The logging 'projectCode' is 'HAWKMETRICS'. Does that fit everyone? Note that until we no longer need to support EAP 6.4, we must not use primitive arguments in logging interfaces. [7] Also, logback is now the logging backend for tests. It allows to set log level with a system property while still having a default value. [8][9] No Maven filtering/replace dance involved. If there's no issue with all of the above, then maybe we should not wait too long before merging, because a bunch of files are impacted. Thanks, Thomas [1] https://github.com/hawkular/hawkular-metrics/pull/352/files#diff-75a85495... [2] https://github.com/hawkular/hawkular-metrics/pull/352/files#diff-26d97c76... [3] https://github.com/hawkular/hawkular-metrics/pull/352/files#diff-1354be09... [4] https://github.com/hawkular/hawkular-metrics/pull/352/files#diff-75a85495... [5] https://github.com/hawkular/hawkular-metrics/pull/352/files#diff-fb0691f5... [6] https://github.com/hawkular/hawkular-metrics/pull/352/files#diff-04e1ad2a... [7] http://lists.jboss.org/pipermail/hawkular-dev/2015-March/000378.html [8] https://github.com/hawkular/hawkular-metrics/pull/352/files#diff-e0d878a7... [9] https://github.com/hawkular/hawkular-metrics/pull/352/files#diff-e0d878a7...

9 years, 3 months

2
1
0 / 0

Resource Path view in UI

by mike thompson

Now that we have switched to Resource Paths (canonical paths from Inventory) it has come up that displaying the resource path is not very readable and requires parsing to make sense of it. Please see the current use case as example: https://issues.jboss.org/secure/attachment/12392747/AlertCenter2.jpg <https://issues.jboss.org/secure/attachment/12392747/AlertCenter2.jpg> https://issues.jboss.org/secure/attachment/12392582/Alert-detailsnu.jpg <https://issues.jboss.org/secure/attachment/12392582/Alert-detailsnu.jpg> An example resource path: /t;28026b36-8fe4-4332-84c8-524e173a68bf/e;test/f;localhost/r;localhost~Local~~ To further break the above example into its pieces: tenantId: /t;28026b36-8fe4-4332-84c8-524e173a68bf environment: test feed: localhost resourceId: localhost The following snippet explaining the resource path was extracted from: http://www.hawkular.org/docs/components/inventory/index.html <http://www.hawkular.org/docs/components/inventory/index.html> Canonical Paths A canonical path follows the contains relationships from a tenant down to the entity in question. The canonical path has a form illustrated by the following example: /t;tenant-id/e;env-id/r;resource-id The above example is a canonical path to a resource with ID resource-id which is located in environment env-id which is inside a tenant tenant-id. The type specifiers in the individual path segments can be these: t - tenant e - environment rt - resource type mt - metric type f - feed r - resource m - metric And as I understand it, the resource path also contains the parent resource prepended to this. So one can always figure out where the resource came from and can uniquely identify a resource. A resource path is just one kind of canonical path (the others are given in the above except). What fields should we parse out and display in the UI as individual fields? I’ll start this conversation by assuming that the environment and tenant are not very useful to display. Then what about feed? And since resource is what we are after that one is obvious as needed. WDYT? And then there is the question of how Ancestry path should be viewed — but for now lets just stick to the above question of how should resources be viewed in the UI? Ancestry Path is another topic for later. Please refer to Jira: https://issues.jboss.org/browse/HAWKULAR-605 <https://issues.jboss.org/browse/HAWKULAR-605>

9 years, 3 months

5
4
0 / 0

Keycloak Authentication token

by mike thompson

Juca, Is there an api to get the bearer token set by KeyCloak? This seems like something we should have easy access to now that we are using Web sockets that have their own authentication. The current persona contains: createdAt: "2015-09-10T14:51:06.173Z" id: "28026b36-8fe4-4332-84c8-524e173a68bf” name: "John Doe" updatedAt: "2015-09-10T14:51:06.173Z” Can we get authenticationToken added to this? Or is there another better way?

9 years, 3 months

2
1
0 / 0

Hawkular Alerts 0.2.0.Final released

by Jay Shaughnessy

Hawkular 0.2.0.Final has been Released. It includes the following work in Jira: https://issues.jboss.org/issues/?jql=project%20%3D%20HWKALERTS%20AND%20fi... Additionally, it contains changes for Hawkular WF9 support and a fix for Alert pagination. The next scheduled version is 0.3.0.Final sometime in the next few weeks. -Jay

9 years, 3 months

1
1
0 / 0

article: "My Philosophy on Alerting"

by John Mazzitelli

For the alerting folks: https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0Yf... By Rob Ewaschuk, an ex-Site Reliability Engineer at Google.

9 years, 3 months

1
0
0 / 0

Bucketed data issues when no datapoints fall in the bucket span.

by Matt Wringe

I am running into a few issues when dealing with bucketed data when there is no datapoints which happen to fall within a particular bucket size. For instance, lets say I am getting data every 30 seconds. If my bucket spans at least 30 seconds than everything works fine and all my buckets contain non-empty data. But, if I were to 'zoom in' a bit and ask for a bucket span less than 30 seconds, then some of the buckets would return with empty data. Or, if my data does not come in perfectly for each of those 30 second fragments (say the timestamps are plus or minus a couple of seconds) then I could also end up with some buckets containing empty data as well. This causes some strange charting behaviour. If a bucket is empty, then its just not displayed in the chart. This means we can end up with non-continuous lines in the charts, which is not desired. Anyone know what should be done in this situation? Its not really that there is no data there and nothing should be graphed, its just that the bucket span is too small to show anything in there. Should charting just ignore empty buckets if they are not at the start or the end? Should hawkular metrics perform calculations to figure out what the missing bucket values should be? Any ideas here?

9 years, 3 months

3
3
0 / 0

Heads up: Accounts URL is changing!

by Juraci Paixão Kröhling

Folks, As reported via HAWKULAR-454 , it seems that the context I use on Accounts backend is not in line with the context used by other modules. Hence, Accounts' backend URL will be changing! - If you only depend on the accounts-api module, then you might just need to update to the latest Accounts version with this change (1.0.12.Final, to be released). - If you do REST calls to /hawkular-accounts, you'll need to change it to /hawkular/accounts . - This change will happen for Accounts 1.0.12.Final . Changes in Hawkular proper (aka kettle) will be performed by me . The plan is to release 1.0.12.Final at most by the end of the next week, but hopefully on Tuesday. The only difference between 1.0.11.Final and .12.Final will be this context path change. It would be a good idea to have your component updated to .11.Final by then, to make sure any possible new issues are solely related to the context path change. - Juca.

9 years, 3 months

2
2
0 / 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

hawkular-dev September 2015