<div dir="ltr"><div>Hi All, </div><div>it will interested to give cep capabilities to infinispan caches,</div><div>I have some comments, </div><div class="gmail_extra"><br><div class="gmail_quote">On 17 March 2014 13:00, Jonathan Halliday <span dir="ltr"><<a href="mailto:jonathan.halliday@redhat.com" target="_blank">jonathan.halliday@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
Alongside recent talk of integrating infinispan with hadoop batch<br>
processing, there has been some discussion of using the data grid<br>
alongside an event stream processing system.<br>
<br>
There are several directions we could consider here. In approximate<br>
order of increasing complexity these are:<br>
<br>
- Allow bi-directional flow of events, such that listeners on the cache<br>
can be used to cause events in the processing engine, or events in the<br>
processing engine can update the cache.<br></blockquote><div><br></div><div>To catch events from cache, I propose to develop a simple infinispanSource for flume ( <a href="http://flume.apache.org" target="_blank">http://flume.apache.org</a> ),</div>
<div>using this infinispanSource, one can listen any cache for updates or inserts and redirect this events to either a cep engine or other destination. </div><div><br></div><div>Updating cache will be similar, we may have a infinispanSink for flume and if any application that needs to update any cache via sending events, he can use infinispanSink in its application.</div>
<div><br></div><div>Actually, developing such flume components we will have a change data capture tool ( <a href="http://en.wikipedia.org/wiki/Change_data_capture">http://en.wikipedia.org/wiki/Change_data_capture</a> ) for infinispan. CDC tools are vital for complex event processing integrations and I think this will be a good starting point.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
- Allow the cache to be used to hold lookup data for reference from user<br>
code running the processing engine, to speed up joining streamed events<br>
to what would otherwise be data tables on disk.<br>
<br></blockquote><div>Actually it is important to cache some rdms table into memory in such systems and sync this cache periodically from rdms table to be up-to-date. I think this requirement can be achived via infinispan's "cache loader"s .</div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
- Integrate with the processing engine itself, such that infinispan can<br>
be used to store items that would otherwise occupy precious RAM. This<br>
one is probably only viable with the cooperation of the stream<br>
processing system, so I'll base further discussion on Drools Fusion.<br>
<br>
The engine uses memory for a) rules, i.e. processing logic. Some of this<br>
is infrequently accessed. Think of a decision tree in which some<br>
branches are traversed more than others. So, opportunities to swap bits<br>
out to cache perhaps. b) state, particularly sliding windows. Again<br>
some data is infrequently accessed. For many sliding window calculations<br>
in particular (e.g. running average), only the head and tail of the<br>
window are actually used. The events in-between can be swapped out.<br></blockquote><div><br></div><div>Holding states are the most important case. In this requirement off-heap cache will be a must. ( Ben Coton is implementing Peter Lawrey's hugemaps into infinispan for off-heap cache you may know )</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Of course these integrations require the stream processing engine to be<br>
written to support such operations - careful handling of object<br>
references is needed. Currently the engine doesn't work that way -<br>
everything is focussed on speed at the expense of memory.</blockquote><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
- Borrow some ideas from the event processing DSLs, such that the data<br>
grid query engine can independently support continuous (standing)<br>
queries rather than just one-off queries. Arguably this is reinventing<br>
the wheel, but for simple use cases it may be preferable to run the<br>
stream processing logic directly in the grid rather than deploying a<br>
dedicated event stream processing system.</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I think it's probably going to<br>
require supporting lists as a first class construct alongside maps<br>
though. There are various cludges possible here, including the brute<br>
force approach of faking continuous query by re-executing a one-off<br>
query on each mutation, but they tend to be inefficient. There is also<br>
the thorny problem of supporting a (potentially distributed) clock,<br>
since a lot of use cases need to reference the passage of time in the<br>
query e.g. 'send event to listener if avg in last N minutes > x'.<br>
<span><font color="#888888"><br>
<br>
<br></font></span></blockquote><div>regards</div><div> </div><div>Yavuz Gökırmak</div><div><ul style="margin:0px;padding:0px 0px 8px;border:0px;outline:0px;font-size:12px;font-family:Arial,sans-serif;vertical-align:baseline;list-style:none;line-height:17px;display:table-cell;width:505px;color:rgb(51,51,51)">
<li style="margin:0px;padding:8px 12px 2px 0px;border:0px;outline:0px;font-weight:inherit;font-style:inherit;font-size:11px;font-family:inherit;vertical-align:baseline;font-variant:inherit;line-height:1.2em"><dl class="" style="margin:0px;padding:0px;border:0px;outline:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:baseline;font-variant:inherit;line-height:inherit;word-wrap:break-word">
<dd style="margin:0px;padding:2px 9px 1px 0px;border:0px;outline:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:top;font-variant:inherit;line-height:inherit;display:inline-block;zoom:1"><a href="http://tr.linkedin.com/pub/yavuz-gokirmak/20/a11/23b/" title="View public profile" name="webProfileURL" style="margin:0px 10px 0px 0px;padding:0px 0px 0px 19px;border:0px;outline:0px;font-weight:inherit;font-style:inherit;font-family:inherit;vertical-align:middle;text-decoration:none;color:rgb(102,102,102);font-variant:inherit;line-height:inherit;display:inline-block;zoom:1;background-image:url(http://s.c.lnkd.licdn.com/scds/common/u/images/apps/profile/sprite/sprite_profile_top_card_v8.png);background-repeat:no-repeat no-repeat">tr.linkedin.com/pub/yavuz-gokirmak/20/a11/23b/</a></dd>
</dl></li></ul></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span><font color="#888888">
Jonathan Halliday<br>
Core developer, JBoss.<br>
<br>
--<br>
Registered in England and Wales under Company Registration No. 03798903<br>
Directors: Michael Cunningham (USA), Paul Hickey (Ireland), Matt Parson<br>
(USA), Charlie Peters (USA)<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br>
</font></span></blockquote></div><br></div></div>