Thanks Manik.<div><br></div><div>We want to avoid transactions because of the additional perceptible overhead. Our opinion is that the use case does not literally involve multiple datastores/ caches and hence transactions should be avoided as far as possible. Do you have different thoughts/ inputs?</div>
<div><br></div><div>If we go with Lucene option - is there a way to calculate the memory footprint required for the indices based upon the field length etc.</div><div> </div><div>Kapil<br><br><div class="gmail_quote">On Wed, Sep 14, 2011 at 8:22 AM, Manik Surtani <span dir="ltr"><<a href="mailto:manik@jboss.org">manik@jboss.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div style="word-wrap:break-word">Hi Kapil<div><br></div><div>After reading through this again, it is indeed an interesting use case. My comments inline:</div>
<div><br><div><div class="im"><div>On 9 Sep 2011, at 05:23, kapil nayar wrote:</div><br></div><div class="im"><blockquote type="cite">We have two data sets {A1, A2, A3...} and {B1, B2, B3...}<br>Each B has some associated data {C1, C2, C3....} which has 1:1 mapping. <br>
<br>The mappings would be something like (assume that C would be stored along side B):<br>
A1-> B1, B2<br>A2-> B3, B5<br>A3-> B4, B6, B7<br><br>Now, we would need the following indexes:<br>A->B and B->A<br><br>Notice, that both are unique mappings. However, as shown A has multiple mappings to B.<br>
The big-table type of data structure allow this and make it pretty easy off the shelf.<br>
<br>Now, I am trying to explore if we can implement these mappings with Infinispan.<br>We may need a basic multi-map - to store multiple values for the same key in the cache.<br>
<br>1. The "get" would return the complete list of the values.<br>2. The "put" would add the new value without replacing the existing value.<br>3. The "remove" would remove a specific value or optionally all values associated with the key.<br>
4. These operations (especially "put") on the same key can occur simultaneously from multiple nodes.<br><br>I know there is an atomic map option in Infinispan which may be applicable, but AFAIK it requires transactions (which we want to avoid..).<br>
</blockquote><div><br></div></div><div>The AtomicMap does do this, but will lock the entire map for any operation. We're working on a FineGrainedMap as well, which will allow concurrent updates to contents within the map. See <a href="https://issues.jboss.org/browse/ISPN-1115" target="_blank">https://issues.jboss.org/browse/ISPN-1115</a></div>
<div><br></div><div>However this too is likely to require JTA transactions for consistency. Could you explain why you wish to avoid transactions?</div><div class="im"><br><blockquote type="cite">
<br>Alternatively, perhaps Infinispan (in combination with lucene) can be used.<br>1. We should be able to create data structure {B, C} and store A-> {B,C} with indexes defined for B.<br>2. Also, the key A could be structured as a combination of A+B to store multiple entries like A1B1->{B1,C1} and A1B2->{B2,C2}. Lucene would allow wild carded searches. e.g. To look for all A1 values we could do something like A1* which should return both A1B1 and A2B2....I may be making some assumptions here (feel free to correct!)<br>
</blockquote><div><br></div></div><div>Yes, this should be possible.</div><div class="im"><br><blockquote type="cite">
3. There seems to be one bottleneck though - since the cache mode is "distribution", it seems it is mandatory to use a backend DB to store these indexes and moreover the DB needs to be shared. This requirement actually seems to defeat the purpose of using Infinispan.<br>
</blockquote><div><br></div></div><div>Not necessarily. You can configure Lucene to store indexes in a replicated Infinispan cache as well. This means the indexes are globally available, and in-memory. You would need a lot of memory though! :)</div>
<div><br></div><div>Cheers</div><div>Manik</div><div>--</div></div><div class="im"><div><div><div>Manik Surtani</div><div><a href="mailto:manik@jboss.org" target="_blank">manik@jboss.org</a></div><div><a href="http://twitter.com/maniksurtani" target="_blank">twitter.com/maniksurtani</a></div>
<div><br></div><div>Lead, Infinispan</div><div><a href="http://www.infinispan.org" target="_blank">http://www.infinispan.org</a></div><div><br></div></div><br>
</div>
<br></div></div></div><br>_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a><br></blockquote></div><br></div>