Re: [infinispan-dev] Feedback on Infinispan patch

I'm Sorry, I'm no legal: this is what I understood in a chat but I could be wrong; I'll ask again for clarifications. Whatever the real reason to deny it as-is, it appears to be possible to include it in Lucene if this API/Impl split could happen, I don't mean to push anyone for this, just make it clear that the option was offered; as a personal opinion I would favour it. Sanne 2009/9/24 Jeff Ramsdale <jeff.ramsdale(a)gmail.com&gt;: > Untrue. Apache-licensed projects are certainly able to include LGPL > components. However, the Apache FOUNDATION does not allow LPGL > components in its own projects. This is not a limitation of the Apache > license, this is a matter of Foundation policy. As Compass is not a > product of the Apache Foundation it shouldn't be an issue (unless, of > course, they have their own policy against including LGPL components). > > -jeff > > On Thu, Sep 24, 2009 at 1:24 PM, Sanne Grinovero > <sanne.grinovero(a)gmail.com&gt; wrote: >> Compass is Apache License, as Lucene is. >> Some Apache members would love to have this inside Lucene, but it >> appears to be a no-go because of the dependency on Infinispan >> (incompatible license); so either Compass can't have it either or it >> could be included into Lucene. >> >> There wouldn't be any problems if the code could depend on an Apache >> licensed API... if it's not out of question here to split into >> API/Impl. >> >> Sanne >> >> 2009/9/24 Manik Surtani <manik(a)jboss.org&gt;: >>> >>> On 24 Sep 2009, at 17:36, Jeff Ramsdale wrote: >>> >>>> Another alternative would be to see if the Compass Project would be >>>> interested in hosting it: http://www.compass-project.org/ >>> >>> Good idea.  Anybody have traction with the compass folk to propose this? >>> >>>> Even if Infinispan ends up hosting it there might be value in doing >>>> some cross-pollination with the Compass folks since this aligns >>>> directly with what they are working on. >>>> >>>> -jeff >>>> >>>> 2009/9/24 Manik Surtani <manik(a)jboss.org&gt;: >>>>> Minorly off topic, but rather than working with patches, do we want >>>>> this >>>>> Directory impl in source control somewhere? >>>>> Being dependent on LGPL, it won't be accepted into Lucene's >>>>> contribs.  If it >>>>> doesn't depend on any Hibernate Search code, I could host it in >>>>> Infinispan's >>>>> SVN repo... >>>>> >>>>> On 23 Sep 2009, at 13:58, Łukasz Moreń wrote: >>>>> >>>>> I agree that Infinispan case is not much different from >>>>> RamDirectory. The >>>>> major difference is that in RD (also FileDirectory) changes are not >>>>> batched >>>>> like in ID. If I do not wrap changes in InfinispanDirectory(simple >>>>> remove >>>>> tx.begin() from obtain() method and tx.commit() from release() in >>>>> InfinispanLock), and immediately commit every change made by IW it >>>>> works >>>>> well. Hovewer it makes indexing really slower, because of frequent >>>>> replication to other nodes. >>>>> Sanne it's good remark that IW commit is kind of flush. >>>>> >>>>> I've attached patch with InfinispanDirectory, failing test is >>>>> testDirectoryWithMultipleThreads in InfinispanDirectoryTest class. >>>>> It fails >>>>> randomly. I think problem is Infinispan commit on lockRelease() in >>>>> org.apache.lucene.index.IndexWriter (line 1658) is after IW commit >>>>> () (line >>>>> 1654). >>>>> >>>>>> Is it because, the IndexWriter only clean files if no indexReaders >>>>>> are >>>>>> reading them (how would that be detected)? >>>>> >>>>> It can happen if IndexWriter clean file, and IndexReader try to >>>>> access that >>>>> cleaned file. >>>>> >>>>> 2009/9/23 Sanne Grinovero <sanne.grinovero(a)gmail.com&gt; >>>>>> >>>>>> I agree It should work the same way; The IndexWriter cleans files >>>>>> whenever it likes to, it doesn't try to detect readers, and this >>>>>> shouldn't have any effect on the working of readers. >>>>>> The IndexReader opens the "SegmentsInfo" first, and immediately >>>>>> after** gets a reference to the segments listed in this >>>>>> SegmentsInfo. >>>>>> No IndexWriter will ever change an existing segment, only add new >>>>>> files or eventually delete old ones (segments merge,optimize). >>>>>> The deletion of segments is the interesting subject: when using >>>>>> Files >>>>>> it uses "delete at last close", which works because the IR needing >>>>>> it >>>>>> have it opened already**; when using the RAMDirectory they have a >>>>>> reference preventing garbage collection. >>>>>> >>>>>> ( the two "**" are assuming the same event occurred correctly, >>>>>> otherwise an exception is thrown at opening) >>>>>> >>>>>> When using Infinispan it shouldn't be much different than the >>>>>> RAMDirectory? so even if the needed segment is deleted, the IR >>>>>> holds a >>>>>> reference to the Java object locally since it was opened. >>>>>> >>>>>>  Łukcasz, do you have some failing test? >>>>>> >>>>>> Sanne >>>>>> >>>>>> 2009/9/23 Emmanuel Bernard <emmanuel(a)hibernate.org&gt;: >>>>>>> Conceptually I don't understand why it does work in a pure file >>>>>>> system >>>>>>> directory (ie IndexReader can go and process queries with the >>>>>>> IndexWriter >>>>>>> goes about its business) and not when using Infinispan. >>>>>>> Is it because, the IndexWriter only clean files if no >>>>>>> indexReaders are >>>>>>> reading them (how would that be detected)? >>>>>>> On 22 sept. 09, at 20:46, Łukasz Moreń wrote: >>>>>>> >>>>>>> I need to provide this same lifecycle for IndexWriter as for >>>>>>> Infinispan >>>>>>> tx - >>>>>>> IW is created: tx is started, IW is commited: tx is commited. It >>>>>>> assures >>>>>>> that IndexReader doesn't read old data from directory. >>>>>>> Infinispan transaction can be started when IW acquires the lock, >>>>>>> but its >>>>>>> commit on IW lock release, as it is done so far, causes a problem: >>>>>>> >>>>>>> index writer close { >>>>>>>   index writer commit(); //changes are visible for IndexReaders >>>>>>> >>>>>>>        //Index reader starts reading here, i.e. tries to access >>>>>>> file "A" >>>>>>> >>>>>>>   index writer lockRelease(); //changes in Infinispan directory are >>>>>>> commited, file "A" was removed, IndexReader cannot find it and >>>>>>> crashes >>>>>>> } >>>>>>> >>>>>>> I think Infinispan tx have to be commited just before IW commit, >>>>>>> and the >>>>>>> problem is where to put in code. >>>>>>> >>>>>>> W dniu 22 września 2009 18:24 użytkownik Emmanuel Bernard >>>>>>> <emmanuel(a)hibernate.org&gt; napisał: >>>>>>>> >>>>>>>> Can you explain in more details what is going on. >>>>>>>> Aside from that Workspace has been Sanne's baby lately so he >>>>>>>> will be >>>>>>>> the >>>>>>>> best to see what design will work in HSearch. That being said, I >>>>>>>> don't >>>>>>>> like >>>>>>>> the idea of subclassing / overriding very much. In my >>>>>>>> experience, it >>>>>>>> has >>>>>>>> lead to more bad and unmaintainable code than anything else. >>>>>>>> On 22 sept. 09, at 02:16, Łukasz Moreń wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Thanks for explanation. >>>>>>>> Maybe better I will concentrate on the first release and postpone >>>>>>>> distributed writing. >>>>>>>> >>>>>>>> There is already LockStrategy that uses Infinispan. With using >>>>>>>> it I was >>>>>>>> wrapping changes made by IndexWriter in Infinispan transaction, >>>>>>>> because >>>>>>>> of >>>>>>>> performance reasons - >>>>>>>> on lock obtaining transaction was started, on lock release >>>>>>>> transaction >>>>>>>> was >>>>>>>> commited. Hovewer Ispn transaction commit on lock release is not >>>>>>>> good >>>>>>>> idea >>>>>>>> since IndexWriter calls index commit before lock is released(and >>>>>>>> ispn >>>>>>>> transaction is committed). >>>>>>>> I was thinking to override Workspace class and getIndexWriter >>>>>>>> (start >>>>>>>> infinispan tx), commitIndexWriter (commit tx) methods to wrap >>>>>>>> IndexWrite >>>>>>>> lifecycle, but this needs few other changes. Some other ideas? >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Lukasz >>>>>>>> >>>>>>>> 2009/9/21 Sanne Grinovero <sanne.grinovero(a)gmail.com&gt; >>>>>>>>> >>>>>>>>> Hi Łukasz, >>>>>>>>> you've rightful concerns, because the way the IndexWriter tries >>>>>>>>> to >>>>>>>>> achieve the lock >>>>>>>>> that will bring some trouble; As far as I remember we decided >>>>>>>>> in this >>>>>>>>> first release >>>>>>>>> to avoid multiple writer nodes because of this reasons >>>>>>>>> (that's written in your docs?) >>>>>>>>> >>>>>>>>> Actually it shouldn't be very hard to do, as the LockStrategy is >>>>>>>>> pluggable (see changes from HSEARCH-345) >>>>>>>>> and you could implement one delegating to an Infinispan eager >>>>>>>>> lock on >>>>>>>>> some key, >>>>>>>>> like the default LockStrategy takes a file lock in the index >>>>>>>>> directory. >>>>>>>>> >>>>>>>>> Maybe it's simpler to support this distributed writing instead of >>>>>>>>> sending the queue to some single >>>>>>>>> (elected) node? Would be cool, as the Document Analysis effort >>>>>>>>> would >>>>>>>>> be distributed, >>>>>>>>> but I have no idea if this would be more or less efficient than a >>>>>>>>> single node writing; it could >>>>>>>>> bring some huge data transfers along the wire during segments >>>>>>>>> merging >>>>>>>>> (basically fetching >>>>>>>>> the whole index data at each node performing a segment merge); >>>>>>>>> maybe >>>>>>>>> you'll need to >>>>>>>>> play with IndexWriter settings ( >>>>>>>>> >>>>>>>>> >>>>>>>>> http://docs.jboss.org/hibernate/stable/search/reference/en/html_single/#l... >>>>>>>>> ) >>>>>>>>> probably need to find the sweet spot for "merge_factor". >>>>>>>>> I just saw now that MergePolicy is now re-implementable, but I >>>>>>>>> hope >>>>>>>>> that won't be needed. >>>>>>>>> >>>>>>>>> Sanne >>>>>>>>> >>>>>>>>> 2009/9/21 Łukasz Moreń <lukasz.moren(a)gmail.com&gt;: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I'm wondering if it is reasonable to have multiple threads/nodes >>>>>>>>>> that >>>>>>>>>> modifies indexes in Lucene Directory based on Infinispan? Let's >>>>>>>>>> assume >>>>>>>>>> that >>>>>>>>>> two nodes try to update index in this same time. First one >>>>>>>>>> creates >>>>>>>>>> IndexWriter and obtains >>>>>>>>>> write lock. There is high propability that second node throws >>>>>>>>>> LockObtainFailedException (as one IndexWriter is allowed on >>>>>>>>>> single >>>>>>>>>> index) >>>>>>>>>> and index is not modified. How is that? Should be always only >>>>>>>>>> one >>>>>>>>>> node >>>>>>>>>> that >>>>>>>>>> makes changes in >>>>>>>>>> the index? >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Lukasz >>>>>>>>>> >>>>>>>>>> W dniu 15 września 2009 01:39 użytkownik Łukasz Moreń >>>>>>>>>> <lukasz.moren(a)gmail.com&gt; napisał: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> With using JMeter I wanted to check if Infinispan dir does not >>>>>>>>>>> crash >>>>>>>>>>> under >>>>>>>>>>> heavy load in "real" use and check performance in comparison >>>>>>>>>>> with >>>>>>>>>>> none/other >>>>>>>>>>> directories. >>>>>>>>>>> However appeared problem when multiple IndexWriters tries to >>>>>>>>>>> modify >>>>>>>>>>> index >>>>>>>>>>> (test InfinispanDirectoryTest) - random deadlocks, and Lucene >>>>>>>>>>> exceptions. >>>>>>>>>>> IndexWriter tries to access files in index that were removed >>>>>>>>>>> before. >>>>>>>>>>> I'm >>>>>>>>>>> looking into it, but not having good idea. >>>>>>>>>>> >>>>>>>>>>> Concerning the last part, I think similar thing is done in >>>>>>>>>>> InfinispanDirectoryProviderTest. Many threads are making >>>>>>>>>>> changes >>>>>>>>>>> and >>>>>>>>>>> searching (not checking if db is in sync with index). >>>>>>>>>>> If threads finish their work, with Lucene query I'm checking if >>>>>>>>>>> index >>>>>>>>>>> contains as many results as expected. Maybe you meant something >>>>>>>>>>> else? >>>>>>>>>>> Would be good to run each node in different VM. >>>>>>>>>>> >>>>>>>>>>>> Great ! Looking forward to it. What state are things in at the >>>>>>>>>>>> moment >>>>>>>>>>>> if I want to play around with it ? >>>>>>>>>>> >>>>>>>>>>> Should work with with one master(updates index) and one many >>>>>>>>>>> slave >>>>>>>>>>> nodes >>>>>>>>>>> (sends changes to master). I tried with one master and one >>>>>>>>>>> slave >>>>>>>>>>> (both >>>>>>>>>>> with >>>>>>>>>>> jms and jgroups backend) and worked ok. Still fails if multiple >>>>>>>>>>> nodes >>>>>>>>>>> want >>>>>>>>>>> to modify index. >>>>>>>>>>> >>>>>>>>>>> I've attached patch with current version. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Łukasz >>>>>>>>>>> >>>>>>>>>>> 2009/9/13 Michael Neale <michael.neale(a)gmail.com&gt; >>>>>>>>>>>> >>>>>>>>>>>> Great ! Looking forward to it. What state are things in at the >>>>>>>>>>>> moment >>>>>>>>>>>> if I want to play around with it ? >>>>>>>>>>>> >>>>>>>>>>>> Sent from my phone. >>>>>>>>>>>> >>>>>>>>>>>> On 13/09/2009, at 7:26 PM, Sanne Grinovero >>>>>>>>>>>> <sanne.grinovero(a)gmail.com&gt; >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> 2009/9/12 Michael Neale <michael.neale(a)gmail.com&gt;: >>>>>>>>>>>>>> That does sounds pretty cool. Would be nice if the lucene >>>>>>>>>>>>>> indexes >>>>>>>>>>>>>> could scale along with how people will want to use >>>>>>>>>>>>>> infinispan. >>>>>>>>>>>>>> Probably worth playing with. >>>>>>>>>>>>> >>>>>>>>>>>>> Sure, this is the goal of Łukasz's work; We know compass has >>>>>>>>>>>>> some good Directories, but we're building our own as one >>>>>>>>>>>>> based >>>>>>>>>>>>> on Infinispan is not yet available. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sent from my phone. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 13/09/2009, at 8:37 AM, Jeff Ramsdale >>>>>>>>>>>>>> <jeff.ramsdale(a)gmail.com&gt; >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm afraid I haven't followed the Infinispan-Lucene >>>>>>>>>>>>>>> implementation >>>>>>>>>>>>>>> closely, but have you looked at the Compass Project? >>>>>>>>>>>>>>> (http://www.compass-project.org/overview.html) It >>>>>>>>>>>>>>> provides a >>>>>>>>>>>>>>> simplified interface to Lucene (optional) as well as >>>>>>>>>>>>>>> Directory >>>>>>>>>>>>>>> implementations built on Terracotta, Gigaspaces and >>>>>>>>>>>>>>> Coherence. >>>>>>>>>>>>>>> The >>>>>>>>>>>>>>> latter, in particular, might be a useful guide for the >>>>>>>>>>>>>>> Infinispan >>>>>>>>>>>>>>> implementation. I believe it's mature enough to have solved >>>>>>>>>>>>>>> many >>>>>>>>>>>>>>> of >>>>>>>>>>>>>>> the most difficult problems of implementing Directory on a >>>>>>>>>>>>>>> distributed >>>>>>>>>>>>>>> Map. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If someone has any experience with Compass (particularly >>>>>>>>>>>>>>> it's >>>>>>>>>>>>>>> Directory implementations) I'd be interested in hearing >>>>>>>>>>>>>>> about >>>>>>>>>>>>>>> it... >>>>>>>>>>>>>>> It's Apache 2.0 licensed, btw. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -jeff >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> infinispan-dev mailing list >>>>>>>>>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> infinispan-dev mailing list >>>>>>>>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> infinispan-dev mailing list >>>>>>>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> infinispan-dev mailing list >>>>>>>>>>>> infinispan-dev(a)lists.jboss.org >>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> < >>>>> InfinispanDirectoryProvider_22_09_2009 >>>>> .patch>_______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>>> -- >>>>> Manik Surtani >>>>> manik(a)jboss.org >>>>> Lead, Infinispan >>>>> Lead, JBoss Cache >>>>> http://www.infinispan.org >>>>> http://www.jbosscache.org >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> -- >>> Manik Surtani >>> manik(a)jboss.org >>> Lead, Infinispan >>> Lead, JBoss Cache >>> http://www.infinispan.org >>> http://www.jbosscache.org >>> >>> >>> >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev