<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Feb 18, 2015, at 6:50 AM, Heiko Braun <<a href="mailto:ike.braun@googlemail.com" class="">ike.braun@googlemail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On 18 Feb 2015, at 13:43, John Sanda <<a href="mailto:jsanda@redhat.com" class="">jsanda@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 16px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I think that Spark's streaming API, particularly the window operations, could be an effective way to do computations in real time as data as ingested</span></div></blockquote></div><br class=""><div class="">+1</div><div class=""><br class=""></div><div class="">not only for processing the streams, but also for any kind of post processing needed. plus it would supply the abstractions to run computations across large number of nodes.</div><div class=""><br class=""></div></div></div></blockquote><br class=""></div><div>Exactly. Use Spark Streaming or even Storm would increase the installation and operational complexity but it would give you a lot of really easy management of your workflow. Each of the stream processors consume a stream and do one thing (e.g., aggregate the last 60 seconds of raw data, or aggregate the last 60 minutes of either the raw data or the 60-second windows, etc.). The different outputs can still get written to whatever storage you want; stream-oriented processing just changes *how* you process the incoming data.</div><div><br class=""></div><div>An alternative approach is to use Apache Kafka directly. There are pros and cons, but the benefit is that the services that do the computations would be microservices (no, really - just really small, single-threaded processes that do a single thing) that could be easily deployed across the cluster. If anyone is interested this approach, ping me and I’ll point you to a prototype that does this (not for analytics).</div><div><br class=""></div><div>BTW, a stream-processing approach does not limit you to live data. In fact, quite the opposite. Many people use stream processing for ingesting large volumes of live data, but lots of other people use it in “big data” as an alternative to batch processing (often map-reduce).</div><br class=""></body></html>