Thanks for the response, Thomas. I have more questions inline. Thanks for humoring me. What do the collectors submit? It seems like there are two options: a) periodically capture metrics; or b) capture every “event of interest” whenever it occurs IIUC, monitoring something like JMX-enabled system would likely be (a), but (b) is really where the value is. Yes, (b) is more invasive but it let’s you capture every possible activity and derive metrics accurately without losing transient spikes. The problem with (a) is that you might miss important cases, since each captured metric represents a measurement at a single instant in time. Consider monitoring a system periodically (e.g., every 15 seconds) to obtain some metric (e.g., the size of a db connection pool, etc). If a spike occurs and is resolved *within a single interval*, then the captured metric will never reflect this, and any derived aggregate (max, min, aggregate) will also not reflect the spike. Examples of (b) include low-level system information like "create database connection”, “closed database connection”, “start new thread”, “stop thread”, etc., but it also includes recording business-related events, such as “received request from user X from IP address A to read resource R”. The latter is insanely valuable, because customers can build downstream services that consume Hawkular output and get valuable business insight and perform business analytics. Those custom services can do all kinds of interesting things, such as automatically rate-limit requests from a specific user, bill based upon usage, audit access for compliance purposes, prevent DoS by rejecting requests from suspect IP addresses, etc. This is what is called “event sourcing”, and it records all interesting events and forms the basis for a lot of near-real-time analytics and monitoring solutions. And, an architecture designed for this purpose can do things will persist every event in a persistent transaction log, and services will consume this stream/log, do work, and output to other persistent logs/streams. These streams and services form a workflow, but it’s not quite the same as a traditional pub/sub set of services because each service consume the events at their own pace. The net effect is that you can take a service down for maintenance (or it can crash), and when it comes back up it will simply continue where it left off. Upstream services are unaffected, and downstream services just see a lull in events but otherwise are unaffected. Plus, you never lose any events or any data, and all aggregates and windowing (done in downstream services) are entirely accurate. You can even set up separate streams for windowed metrics (e.g., minute, hour, day, week, etc.) Here’s an article that describes how LinkedIn is using stream-oriented processing to capture metrics and events for their systems, and how they derive benefit for the business: http://engineering.linkedin.com/samza/real-time-insights-linkedins-perfor... <http://engineering.linkedin.com/samza/real-time-insights-linkedins-perfor...;. They process 500 billion events per day (yes, billion). Here are a couple of other links: http://blog.confluent.io/2015/01/29/making-sense-of-stream-processing/ <http://blog.confluent.io/2015/01/29/making-sense-of-stream-processing/>;, http://www.slideshare.net/InfoQ/samza-in-linkedin-how-linkedin-processes-... <http://www.slideshare.net/InfoQ/samza-in-linkedin-how-linkedin-processes-...;, http://scale-out-blog.blogspot.co.uk/2014/02/why-arent-all-data-immutable... <http://scale-out-blog.blogspot.co.uk/2014/02/why-arent-all-data-immutable...;, and the ever-essential http://engineering.linkedin.com/distributed-systems/log-what-every-softwa... <http://engineering.linkedin.com/distributed-systems/log-what-every-softwa... Sounds good. How will the work be split up across the cluster? Are the incoming messages stored before they’re processed? Sounds good. So I understand that the collectors will drop any buffered data, since that will be unsent if the monitored system (or external collectors) crash. But what happens if Hawkular suffers a partial or total crash? What data is lost? What happens to data that was in-process when the crash occurred? Stream-based architectures are interesting because they often can handle more load, partition it more effectively, and are more durable.

Just posted today: Parse.ly’s time series backend is log-oriented and uses the same technology I discussed earlier. But they have details about data volumes and how they aggregate/window data. Might be useful. http://blog.parsely.com/post/1633/mage/ <http://blog.parsely.com/post/1633/mage/>

Le 13/02/2015 18:43, Randall Hauch a écrit : The collectors I've talked about earlier fall in category (a). You need (a) *and* (b) to build a comprehensive view of what is/was going on your system. RHQ has always been (a) only. Hawkular will focus on (a) initially. We have plans for "events" storage in Hawkular metrics. You can already add tags to your numeric data, but without "events" you can't build something like you describe above. An Hawkular metrics server is web server (Wildfly) in front of a Cassandra cluster. So you'd distribute the work with a load balancer. I'm not sure I understand your second question. If you lose Wildfly servers or Cassandra nodes then the rest of cluster should continue to work. As a Hawkular metrics client, you can't say what happened to your data if you lose the connection before you get a response with a status code. When a Wildfly server crashes, your data may be still being unmarshaled, or might have reached the Cassandra driver, or have been written to the Cassandra logs. We have no plan ATM to implement log-based event processors ourselves. In the beginning, we'll focus on existing and wide-spread collectors (generally buffer-based) and in-house collectors like the wildfly-monitor.

My question is simply this: is the raw incoming data persisted before any other computations or analyses are done? The answer dictates whether Hawkular has the potential to lose data in a crash. The only reason I bring this up is that monitoring (like data storage) is one of those use cases where you can’t really afford to drop or lose any data, since for monitoring that lost data might be the critical evidence you need to understand why something bad happened to your monitored system. Let’s say that a collector sends some data that will ultimately be processed by Hawkular, including updating various windows (e.g., minutely, hourly, daily, etc.), before persisting. If Hawkular crashes before that data is completely processed, then that data will be lost forever. If Hawkular performs some of the work (e.g., just updating the minutely window) before crashing, then the data appears in only some of the generated/derived information. BTW, in this traditional approach, using a transaction will simply ensure that all of your derived/generated data is consistent; it doesn’t ensure that your data is processed. If, however, Hawkular were to persist the collected data *before* processing anything, then as long as the incoming data were persisted then if Hawkular crashed and was restarted it might be able to just continue where it left off, without losing any data or computations. There are lots of ways of ensuring this, but it’s not likely they will all perform well. The stream-oriented architectures make it possible to not lose any data, because every computation or bit of processing is done in services/jobs that read from and write to (usually persistent) streams. The architecture can ensure at-least-once (and sometimes an exactly-once) semantics for every bit of data passing thru each stream. Of course, this is done without transactions and often with partitioned streams to allow things to scale quite well and handle very large volumes of data.

I think that Spark's streaming API, particularly the window operations, could be an effective way to do computations in real time as data as ingested. The problem though with introducing something like Spark or Storm is that it increases the operational complexity of hawkular itself which I think we want to avoid if possible.

Exactly. Use Spark Streaming or even Storm would increase the installation and operational complexity but it would give you a lot of really easy management of your workflow. Each of the stream processors consume a stream and do one thing (e.g., aggregate the last 60 seconds of raw data, or aggregate the last 60 minutes of either the raw data or the 60-second windows, etc.). The different outputs can still get written to whatever storage you want; stream-oriented processing just changes *how* you process the incoming data. An alternative approach is to use Apache Kafka directly. There are pros and cons, but the benefit is that the services that do the computations would be microservices (no, really - just really small, single-threaded processes that do a single thing) that could be easily deployed across the cluster. If anyone is interested this approach, ping me and I’ll point you to a prototype that does this (not for analytics). BTW, a stream-processing approach does not limit you to live data. In fact, quite the opposite. Many people use stream processing for ingesting large volumes of live data, but lots of other people use it in “big data” as an alternative to batch processing (often map-reduce).