Maven, Continuous and Reproducible
by Peter Palaga
Hi *,
I think I found a way how to make our process more continuous, while
staying with stock Maven and Components in separate git repositories.
The core of the idea is that we actually do not need to release the
components because they are not our deliverables (yes, except for
Metrics). If we were able to declare the dependencies using git
revisions and build them on the fly, we could get rid of both SNAPSHOTS
and releases.
I have written srcdeps-maven-plugin [1] today that does exactly that:
* It collects dependencies with versions matching
{whatever}-SRC-{git-commit-hash}
* It checks out {git-commit-hash} of their sources to
~/.m2/dependency-sources
* It changes the version in the sources to
{whatever}-SRC-{git-commit-hash}
* Builds the artifacts and installs them to the local repository
* All the above happens before the dependency resolution starts
so it is fully transparent for the rest of Maven.
* The -SRC- dependencies are build only if they are not found in local
repository - so they prolong the build only when upgrading.
How to try it out:
#Make sure that you have maven 3.2.5 or newer:
mvn -version
# build the plugin
cd ~/git
git checkout https://github.com/ppalaga/srcdeps-maven-plugin.git
cd srcdeps-maven-plugin
mvn clean install
# build hawkular with the plugin
cd ../hawkular
git remote add ppalaga https://github.com/ppalaga/hawkular.git
git fetch ppalaga
git checkout -b 150819-srcdeps
mvn clean install ...
[1] https://github.com/ppalaga/srcdeps-maven-plugin
9 years, 3 months
Extra back-end for our front-end
by Viliam Rockai
Hey all,
As we put more and more functionality into Hawkular, the more and more
logic is put into the front-end itself, too. From my perspective, we're
dealing with too much stuff that doesn't even belong to the front-end
anyway. I believe we should create a new back-end module, which would
connect all the existing modules with the front-end. This module may
even be maintained by the front-end team, since it'll evolve and mutate
very fast - to suit the UI needs. There are several use-cases which call
for such module. For example to create all the alert entities needed for
alerting you need to create a:
* Trigger (per alerting "entity")
* Dampening (one or more)
* Condition (one or more)
Each of those has its separate endpoint and creation of each may fail.
So basically we should (we don't do that yet, we just assume everything
went all right) even deal with transactions in the front-end right now.
>From the UI perspective, this is just a single entity "Alert
definition". Another question is - when should those objects be created?
Once you add an URL - that's pretty easy (for URLS). But, in my opinion,
for JVMs they should be created once the resource is discovered, not the
1st time the resource is accessed by UI. And the situation will be more
complicated with new resources. Being able to get some predefined sets
of metrics would be another nice feature. I.e., right now, we do 3
requests (User, maximum, committed) to get the data for the "JVM Heap
Usage" graph (and other graphs, too). Being able to process those data
on a server and send it the the front-end in one bulk would be pretty
neat.
I believe adding a new module to the back-end, which role would be to
connect the UI to the rest of Hawkular services would be beneficial for
performance, testability and robustness of the whole app.
WDYT?
Viliam
9 years, 3 months
Add JDBC Driver support in Agent
by Peter Palaga
Hi *,
there is a requirement for Agent to support adding JDBC drivers
https://issues.jboss.org/browse/HAWKULAR-517
The named Jira does not name any standalone/domain/local/remote related
limitations for the functionality, hence the functionality should be
there for all modes.
As already noticed on IRC, WildFly 9's jboss-cli.sh offers the
possibility to add a module, but it works only for the local machine in
standalone mode. Remote servers or domain mode are not supported.
So clearly, for now, we can support the standalone local mode using the
same means as jboss-cli, but how should we proceed further?
Should we
(a) wait for WildFly to implement the missing functionality, or
(b) contribute to Wildfly or
(c) create a minimal Agent to install on remote servers to provide the
functionality missing in WildFly itself (as proposed by Heiko)?
Thanks,
Peter
9 years, 3 months
hawkular-accounts integration with websocket stuff in kettle
by John Mazzitelli
Juca,
I think the time is coming close when I need to wrap authentication around the websockets stuff we have in the server. Right now, any client can connect to our websocket server running in kettle. I know you mentioned that KeyCloak doesn't have an official integration with websockets today, but can we make calls to your accounts API directly to do things like logging in?
In our websockets code, I believe we can get access to the headers that were passed in by the UI client when connecting to websockets (javax.websocket.Session.getRequestParameterMap() is what I assume we can use). So a UI or feed could pass in username/password (just like a normal HTTP client would to connect to our metrics or inventory REST API). We would just need to manually take those username/password credentials and ask accounts to verify the credentials and perhaps give us back a UserPrincipal or whatever it is you can give us.
We will eventually need to use this user principal or whatever to logically link that UI client (and the requests it makes to the server) to the responses of those requests (so those responses can be sent back to the proper client)
Ideas?
--John Mazz
9 years, 4 months
[Metrics] Evaluating performance of #groupBy vs #window for stats computation
by Thomas Segismont
Hi,
Recently, we've ported to RxJava the stats computation code, a part not
fully "reactive" yet. The intention was to be able to compute statistics
on raw data for large period of times, with low requirements on memory
(the initial implementation needed all data loaded at once in a list).
If you're interested in the $subject and don't known what stats
computation is, it's an API metrics provides to give the user stats
(min/max/avg/median/95th percentile) on raw data. The stats are given in
buckets (portions of time). As an example, it's the API you use if you
want monthly stats on a year of data, or hourly stats for a day, ... etc.
The implementation makes use of the Observable#groupBy operator. This
allows to determine the bucket each data point belongs to, and then we
#collect the points in each bucket to compute the stats.
https://github.com/hawkular/hawkular-metrics/blob/master/core/metrics-cor...
As we were talking on a related topic (stats for sliding windows), John
wondered if a solution based on Observable#window would perform equally.
So I've implemented this solution and instrumented the code to determine
where the system was spending the execution time. Then I tried both
solutions with different bucket sizes, metric resolutions, numbers of
buckets.
Here are my observations.
#groupBy and #window behave about the same in our case (I've tried with
a few buckets up to a thousand buckets)
Response time increases linearly with the number of data points loaded
from the C*. Response time here means time elapsed between the moment
Wildfly invokes the JAX-RS handler method and we resume the AsyncResponse.
The bottleneck is the data loading and mapping: 95% of execution time.
C* Row to object mapping is nearly 50% of it.
And my conclusions.
When working with non overlapping buckets (not sliding windows),
readability should determine the use of #groupBy or #window, not
performance.
It seems possible to get a better response time. Currently we transform
a ResultSet into an Observable<Row> with a simple call to
Observable#from (ResultSet is an Iterable<Row>). By default, the C*
driver fetches a page of data, and only when it's entirely consumed,
fetches the following one. But ResultSet has a #fetchMoreResults method
which we could use to fetch pages ahead of time (see
http://docs.datastax.com/en/drivers/java/2.0/com/datastax/driver/core/Res...).
Then, while Rx computation threads would spend time on mapping a Row to
a DataPoint<T>, the C* driver could load more Rows.
Attached are:
- a patch of the changes implemented for testing
- some of the results
Regards,
Thomas
9 years, 4 months
MS4 demo / release
by Thomas Heute
So in the end, with the absence of application deployment, what is left
to demo/announce ?
- Access portal features
- Alert settings
Anything else ?
Thomas
9 years, 4 months
bus/agent master will break things temporarily
by John Mazzitelli
As per a conversation betweek heiko, mike, and myself, I'm merging stuff into master branches of bus and agent that will break things temporarily. This is to start getting HAWKULAR-530 in place. Once completely, hopefully we can work on HWKAGENT-5/HAWKULAR-546 working, but that might not be possible for the wed. release.
9 years, 4 months
inventory and agent status
by Jiri Kremser
Hi,
I'll be on PTO whole next week, so I am sending the status what is done and what still needs to be done here:
Inventory 0.3.0 changed the way it deals with escaped characters, we use the @Encoded annotation in resteasy for path params. This is because of the fact one can pass a path to the resources that contains slashes as a resource separators, but the resourceIds can have also slashes in their ids.
This works quite ok, but if the resourceId contains a slash at the end, it doesn't work.
example:
/inventory/test/resources/parent%2res/child%2fres (ok)
/inventory/test/resources/parent%2f (not - ok) the last slash is lost, even though it's part of the id
To make it short, agent can't work with the latest inventory and I prepared 2 scenarios:
1) just fix the escaping and forget about all the new features introduced in inventory/agent:
to make this work just merge https://github.com/hawkular/hawkular-agent/pull/38/ into agent, release a new version of agent
+ merge the https://github.com/hawkular/hawkular/tree/dev/inventory-0.3.0.Final into hawkular master with the new version of agent (currently there is agent@snapshot)
2) on the agent side repo merge the https://github.com/hawkular/hawkular-agent/tree/mazz/hwkagent-5-resourcec... which contains the logic for inserting the resource configs grabbed from DMR into inventory, has support for the resource hierarchy, fixes the thing from 1) and moves the resource types and metric types in the inventory topology under the feed (this is questionable, because UI has certain issues with it like "give me all the Wildflies" must be translated into "give me all the feeds and for each feed ask for all the WF resources" - it's done here http://git.io/vsRXr, but still).
next step for this scenario to work is to merge the PR on the ui-services http://git.io/vs8Jj and do the appropriate changes in the hawkular/hawkuar ui, I started this here http://git.io/vsRXr all in all UI works, I saw just some errors when going to the details page for the urls, but it's definitely fixable within next week, oh and it depends on my own ui-services version, so once the my PR is merged into ui-services, this (http://git.io/vs04A) can go away
I vote for 2) of course :]
Just a comment to the resource hierarchy:
currently, the agent considers all the resources equal (flat structure) and the hierarchy is captured by the prefixes, like "~/", "~/foo", "~/foo~bar", etc.
The change in the https://github.com/hawkular/hawkular-agent/tree/mazz/hwkagent-5-resourcec... branch preserves the ids, but adds the real structure among them on the inventory level.
2lkrejci: this is bad wrt resource hierarchy rest api https://github.com/angular/angular.js/issues/1388 (but they say it should be possible somehow using http://git.io/vs02W)
sorry for lack of the structure, it was more a brain dump.
jk
9 years, 4 months
