<div dir="ltr">On Fri, Dec 9, 2016 at 9:13 AM, Radim Vansa <span dir="ltr"><<a href="mailto:rvansa@redhat.com" target="_blank">rvansa@redhat.com</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
1) 'cache that would persist the events with a monotonically increasing id'<br>
<br>
I assume that you mean globally (for all entries) monotonous. How will<br>
you obtain such ID? Currently, commands have unique IDs that are<br>
<Address, Long> where the number part is monotonous per node. That's<br>
easy to achieve. But introducing globally monotonous counter means that<br>
there will be a single contention point. (you can introduce another<br>
contention points by adding backups, but this is probably unnecessary as<br>
you can find out the last id from the indexed cache data). Per-segment<br>
monotonous would be probably more scalabe, though that increases complexity.<br></blockquote><div><br></div><div>Having it per segment would imply only operations involving the same key would be ordered, <br></div><div>probably it's fine for most cases. <br><br>Could this order be affected during topology changes though? As I could observe, there is a small <br>window where there is more than 1 primary owner for a given key due to the fact that the CH propagation<br></div><div>is not complete.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
2) 'The write to the event log would be async in order to not affect<br>
normal data writes'<br>
<br>
Who should write to the cache?<br>
a) originator - what if originator crashes (despite the change has been<br>
added)? Besides, originator would have to do (async) RPC to primary<br>
owner (which will be the primary owner of the event, too).<br>
b) primary owner - with triangle, primary does not really know if the<br>
change has been written on backup. Piggybacking that info won't be<br>
trivial - we don't want to send another message explicitly. But even if<br>
we get the confirmation, since the write to event cache is async, if the<br>
primary owner crashes before replicating the event to backup, we lost<br>
the event<br>
c) all owners, but locally - that will require more complex<br>
reconciliation if the event did really happen on all surviving nodes or<br>
not. And backups could have some trouble to resolve order, too.<br>
<br>
IIUC clustered listeners are called from primary owner before the change<br>
is really confirmed on backups (@Pedro correct me if I am wrong,<br>
please), but for this reliable event cache you need higher level of<br>
consistency.<br></blockquote><div><br>Async writes to a cache event log would not provide the best of guarantees, agreed.<br><br></div><div>OTOH, to have the writes done synchronously, it'd be hard to avoid extra RPCs.<br>Some can be prevented by using a KeyPartitioner similar to the one used on the AffinityIndexManager [1],<br>so that Segment(K) = Segment(KE), being K the key and KE the related event log key.<br><br>Still RPCs would happen to replicate events, and as you pointed out, it is not trivial to piggyback this on the triangle'd <br>data RPCs.<br><br></div><div>I'm starting to think that an extra cache to store events is overkill.<br><br></div><div>An alternative could be to bypass the event log cache altogether and store the events on the Lucene index directly.<br>For this a custom interceptor would write them to a local index when it's "safe" to do so, similar to what the QueryInterceptor <br>does with the Index.ALL flag, but only writing on primary + backup, more like to a hypothetical "Index.OWNER" setup.<br></div><div><br>This index does not necessarily need to be stored in extra caches (like the Infinispan directory does) but can use a local MMap <br>based directory, making it OS cache friendly. At event consumption time, though, broadcast queries to the primary owners would be <br>needed to collect the events on each of the nodes and merge them before serving to the clients. <br></div><div><br></div><div><br></div><div>[1] <a href="https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/distribution/ch/impl/AffinityPartitioner.java" target="_blank">https://github.com/infinispan/<wbr>infinispan/blob/master/core/sr<wbr>c/main/java/org/infinispan/dis<wbr>tribution/ch/impl/AffinityPart<wbr>itioner.java</a><br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
3) The log will also have to filter out retried operations (based on<br>
command ID - though this can be indexed, too). Though, I would prefer to<br>
see per-event command-id log to deal with retries properly.<br>
<br>
4) Client should pull data, but I would keep push notifications that<br>
'something happened' (throttled on server). There could be use case for<br>
rarely updated caches, and polling the servers would be excessive there.<br>
<br>
Radim<br></blockquote><div><br><br></div><div>Makes sense, the push could be a notification that the event log changed and the <br>client would them proceed with its normal pull.<br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="m_1805473862695814679m_211782213323611962m_-4280587775476328259m_-9116492687533281564m_-3009646764263810743m_6496743631215548118m_5648078105759408914m_7424953911122507266gmail-m_4215348147783816758gmail-im m_1805473862695814679m_211782213323611962m_-4280587775476328259m_-9116492687533281564m_-3009646764263810743m_6496743631215548118m_5648078105759408914m_7424953911122507266gmail-m_4215348147783816758gmail-HOEnZb"><br>
><br>
><br>
> [1]<br>
> <a href="https://github.com/infinispan/infinispan/wiki/Remote-Listeners-improvement-proposal" rel="noreferrer" target="_blank">https://github.com/infinispan/<wbr>infinispan/wiki/Remote-Listene<wbr>rs-improvement-proposal</a><br>
><br>
> Thanks,<br>
> Gustavo<br>
><br>
><br>
><br>
</span><span class="m_1805473862695814679m_211782213323611962m_-4280587775476328259m_-9116492687533281564m_-3009646764263810743m_6496743631215548118m_5648078105759408914m_7424953911122507266gmail-m_4215348147783816758gmail-im m_1805473862695814679m_211782213323611962m_-4280587775476328259m_-9116492687533281564m_-3009646764263810743m_6496743631215548118m_5648078105759408914m_7424953911122507266gmail-m_4215348147783816758gmail-HOEnZb">> ______________________________<wbr>_________________<br>
> infinispan-dev mailing list<br>
> <a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailma<wbr>n/listinfo/infinispan-dev</a><br>
<br>
<br>
</span><span class="m_1805473862695814679m_211782213323611962m_-4280587775476328259m_-9116492687533281564m_-3009646764263810743m_6496743631215548118m_5648078105759408914m_7424953911122507266gmail-m_4215348147783816758gmail-HOEnZb"><font color="#888888">--<br>
Radim Vansa <<a href="mailto:rvansa@redhat.com" target="_blank">rvansa@redhat.com</a>><br>
JBoss Performance Team<br>
</font></span><div class="m_1805473862695814679m_211782213323611962m_-4280587775476328259m_-9116492687533281564m_-3009646764263810743m_6496743631215548118m_5648078105759408914m_7424953911122507266gmail-m_4215348147783816758gmail-HOEnZb"><div class="m_1805473862695814679m_211782213323611962m_-4280587775476328259m_-9116492687533281564m_-3009646764263810743m_6496743631215548118m_5648078105759408914m_7424953911122507266gmail-m_4215348147783816758gmail-h5"><br>
______________________________<wbr>_________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org" target="_blank">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/mailma<wbr>n/listinfo/infinispan-dev</a><br>
</div></div></blockquote></div><br></div></div>