Re: [hibernate-dev] [infinispan-dev] Feedback on Infinispan patch
by Łukasz Moreń
I agree that Infinispan case is not much different from RamDirectory. The
major difference is that in RD (also FileDirectory) changes are not batched
like in ID. If I do not wrap changes in InfinispanDirectory(simple remove
tx.begin() from obtain() method and tx.commit() from release() in
InfinispanLock), and immediately commit every change made by IW it works
well. Hovewer it makes indexing really slower, because of frequent
replication to other nodes.
Sanne it's good remark that IW commit is kind of flush.
I've attached patch with InfinispanDirectory, failing test is
testDirectoryWithMultipleThreads in InfinispanDirectoryTest class. It fails
randomly. I think problem is Infinispan commit on lockRelease() in
org.apache.lucene.index.IndexWriter (line 1658) is after IW commit() (line
1654).
Is it because, the IndexWriter only clean files if no indexReaders are
> reading them (how would that be detected)?
>
It can happen if IndexWriter clean file, and IndexReader try to access that
cleaned file.
2009/9/23 Sanne Grinovero <sanne.grinovero(a)gmail.com>
> I agree It should work the same way; The IndexWriter cleans files
> whenever it likes to, it doesn't try to detect readers, and this
> shouldn't have any effect on the working of readers.
> The IndexReader opens the "SegmentsInfo" first, and immediately
> after** gets a reference to the segments listed in this SegmentsInfo.
> No IndexWriter will ever change an existing segment, only add new
> files or eventually delete old ones (segments merge,optimize).
> The deletion of segments is the interesting subject: when using Files
> it uses "delete at last close", which works because the IR needing it
> have it opened already**; when using the RAMDirectory they have a
> reference preventing garbage collection.
>
> ( the two "**" are assuming the same event occurred correctly,
> otherwise an exception is thrown at opening)
>
> When using Infinispan it shouldn't be much different than the
> RAMDirectory? so even if the needed segment is deleted, the IR holds a
> reference to the Java object locally since it was opened.
>
> Łukcasz, do you have some failing test?
>
> Sanne
>
> 2009/9/23 Emmanuel Bernard <emmanuel(a)hibernate.org>:
> > Conceptually I don't understand why it does work in a pure file system
> > directory (ie IndexReader can go and process queries with the IndexWriter
> > goes about its business) and not when using Infinispan.
> > Is it because, the IndexWriter only clean files if no indexReaders are
> > reading them (how would that be detected)?
> > On 22 sept. 09, at 20:46, Łukasz Moreń wrote:
> >
> > I need to provide this same lifecycle for IndexWriter as for Infinispan
> tx -
> > IW is created: tx is started, IW is commited: tx is commited. It assures
> > that IndexReader doesn't read old data from directory.
> > Infinispan transaction can be started when IW acquires the lock, but its
> > commit on IW lock release, as it is done so far, causes a problem:
> >
> > index writer close {
> > index writer commit(); //changes are visible for IndexReaders
> >
> > //Index reader starts reading here, i.e. tries to access file "A"
> >
> > index writer lockRelease(); //changes in Infinispan directory are
> > commited, file "A" was removed, IndexReader cannot find it and crashes
> > }
> >
> > I think Infinispan tx have to be commited just before IW commit, and the
> > problem is where to put in code.
> >
> > W dniu 22 września 2009 18:24 użytkownik Emmanuel Bernard
> > <emmanuel(a)hibernate.org> napisał:
> >>
> >> Can you explain in more details what is going on.
> >> Aside from that Workspace has been Sanne's baby lately so he will be the
> >> best to see what design will work in HSearch. That being said, I don't
> like
> >> the idea of subclassing / overriding very much. In my experience, it has
> >> lead to more bad and unmaintainable code than anything else.
> >> On 22 sept. 09, at 02:16, Łukasz Moreń wrote:
> >>
> >> Hi,
> >>
> >> Thanks for explanation.
> >> Maybe better I will concentrate on the first release and postpone
> >> distributed writing.
> >>
> >> There is already LockStrategy that uses Infinispan. With using it I was
> >> wrapping changes made by IndexWriter in Infinispan transaction, because
> of
> >> performance reasons -
> >> on lock obtaining transaction was started, on lock release transaction
> was
> >> commited. Hovewer Ispn transaction commit on lock release is not good
> idea
> >> since IndexWriter calls index commit before lock is released(and ispn
> >> transaction is committed).
> >> I was thinking to override Workspace class and getIndexWriter(start
> >> infinispan tx), commitIndexWriter (commit tx) methods to wrap IndexWrite
> >> lifecycle, but this needs few other changes. Some other ideas?
> >>
> >> Cheers,
> >> Lukasz
> >>
> >> 2009/9/21 Sanne Grinovero <sanne.grinovero(a)gmail.com>
> >>>
> >>> Hi Łukasz,
> >>> you've rightful concerns, because the way the IndexWriter tries to
> >>> achieve the lock
> >>> that will bring some trouble; As far as I remember we decided in this
> >>> first release
> >>> to avoid multiple writer nodes because of this reasons
> >>> (that's written in your docs?)
> >>>
> >>> Actually it shouldn't be very hard to do, as the LockStrategy is
> >>> pluggable (see changes from HSEARCH-345)
> >>> and you could implement one delegating to an Infinispan eager lock on
> >>> some key,
> >>> like the default LockStrategy takes a file lock in the index directory.
> >>>
> >>> Maybe it's simpler to support this distributed writing instead of
> >>> sending the queue to some single
> >>> (elected) node? Would be cool, as the Document Analysis effort would
> >>> be distributed,
> >>> but I have no idea if this would be more or less efficient than a
> >>> single node writing; it could
> >>> bring some huge data transfers along the wire during segments merging
> >>> (basically fetching
> >>> the whole index data at each node performing a segment merge); maybe
> >>> you'll need to
> >>> play with IndexWriter settings (
> >>>
> >>>
> http://docs.jboss.org/hibernate/stable/search/reference/en/html_single/#l...
> >>> )
> >>> probably need to find the sweet spot for "merge_factor".
> >>> I just saw now that MergePolicy is now re-implementable, but I hope
> >>> that won't be needed.
> >>>
> >>> Sanne
> >>>
> >>> 2009/9/21 Łukasz Moreń <lukasz.moren(a)gmail.com>:
> >>> > Hi,
> >>> >
> >>> > I'm wondering if it is reasonable to have multiple threads/nodes that
> >>> > modifies indexes in Lucene Directory based on Infinispan? Let's
> assume
> >>> > that
> >>> > two nodes try to update index in this same time. First one creates
> >>> > IndexWriter and obtains
> >>> > write lock. There is high propability that second node throws
> >>> > LockObtainFailedException (as one IndexWriter is allowed on single
> >>> > index)
> >>> > and index is not modified. How is that? Should be always only one
> node
> >>> > that
> >>> > makes changes in
> >>> > the index?
> >>> >
> >>> > Cheers,
> >>> > Lukasz
> >>> >
> >>> > W dniu 15 września 2009 01:39 użytkownik Łukasz Moreń
> >>> > <lukasz.moren(a)gmail.com> napisał:
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> With using JMeter I wanted to check if Infinispan dir does not crash
> >>> >> under
> >>> >> heavy load in "real" use and check performance in comparison with
> >>> >> none/other
> >>> >> directories.
> >>> >> However appeared problem when multiple IndexWriters tries to modify
> >>> >> index
> >>> >> (test InfinispanDirectoryTest) - random deadlocks, and Lucene
> >>> >> exceptions.
> >>> >> IndexWriter tries to access files in index that were removed before.
> >>> >> I'm
> >>> >> looking into it, but not having good idea.
> >>> >>
> >>> >> Concerning the last part, I think similar thing is done in
> >>> >> InfinispanDirectoryProviderTest. Many threads are making changes and
> >>> >> searching (not checking if db is in sync with index).
> >>> >> If threads finish their work, with Lucene query I'm checking if
> index
> >>> >> contains as many results as expected. Maybe you meant something
> else?
> >>> >> Would be good to run each node in different VM.
> >>> >>
> >>> >>> Great ! Looking forward to it. What state are things in at the
> moment
> >>> >>> if I want to play around with it ?
> >>> >>
> >>> >> Should work with with one master(updates index) and one many slave
> >>> >> nodes
> >>> >> (sends changes to master). I tried with one master and one slave
> (both
> >>> >> with
> >>> >> jms and jgroups backend) and worked ok. Still fails if multiple
> nodes
> >>> >> want
> >>> >> to modify index.
> >>> >>
> >>> >> I've attached patch with current version.
> >>> >>
> >>> >> Cheers,
> >>> >> Łukasz
> >>> >>
> >>> >> 2009/9/13 Michael Neale <michael.neale(a)gmail.com>
> >>> >>>
> >>> >>> Great ! Looking forward to it. What state are things in at the
> moment
> >>> >>> if I want to play around with it ?
> >>> >>>
> >>> >>> Sent from my phone.
> >>> >>>
> >>> >>> On 13/09/2009, at 7:26 PM, Sanne Grinovero
> >>> >>> <sanne.grinovero(a)gmail.com>
> >>> >>> wrote:
> >>> >>>
> >>> >>> > 2009/9/12 Michael Neale <michael.neale(a)gmail.com>:
> >>> >>> >> That does sounds pretty cool. Would be nice if the lucene
> indexes
> >>> >>> >> could scale along with how people will want to use infinispan.
> >>> >>> >> Probably worth playing with.
> >>> >>> >
> >>> >>> > Sure, this is the goal of Łukasz's work; We know compass has
> >>> >>> > some good Directories, but we're building our own as one based
> >>> >>> > on Infinispan is not yet available.
> >>> >>> >
> >>> >>> >>
> >>> >>> >> Sent from my phone.
> >>> >>> >>
> >>> >>> >> On 13/09/2009, at 8:37 AM, Jeff Ramsdale <
> jeff.ramsdale(a)gmail.com>
> >>> >>> >> wrote:
> >>> >>> >>
> >>> >>> >>> I'm afraid I haven't followed the Infinispan-Lucene
> >>> >>> >>> implementation
> >>> >>> >>> closely, but have you looked at the Compass Project?
> >>> >>> >>> (http://www.compass-project.org/overview.html) It provides a
> >>> >>> >>> simplified interface to Lucene (optional) as well as Directory
> >>> >>> >>> implementations built on Terracotta, Gigaspaces and Coherence.
> >>> >>> >>> The
> >>> >>> >>> latter, in particular, might be a useful guide for the
> Infinispan
> >>> >>> >>> implementation. I believe it's mature enough to have solved
> many
> >>> >>> >>> of
> >>> >>> >>> the most difficult problems of implementing Directory on a
> >>> >>> >>> distributed
> >>> >>> >>> Map.
> >>> >>> >>>
> >>> >>> >>> If someone has any experience with Compass (particularly it's
> >>> >>> >>> Directory implementations) I'd be interested in hearing about
> >>> >>> >>> it...
> >>> >>> >>> It's Apache 2.0 licensed, btw.
> >>> >>> >>>
> >>> >>> >>> -jeff
> >>> >>> >>> _______________________________________________
> >>> >>> >>> infinispan-dev mailing list
> >>> >>> >>> infinispan-dev(a)lists.jboss.org
> >>> >>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>> >>> >> _______________________________________________
> >>> >>> >> infinispan-dev mailing list
> >>> >>> >> infinispan-dev(a)lists.jboss.org
> >>> >>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>> >>> >>
> >>> >>> >
> >>> >>> > _______________________________________________
> >>> >>> > infinispan-dev mailing list
> >>> >>> > infinispan-dev(a)lists.jboss.org
> >>> >>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>> >>>
> >>> >>> _______________________________________________
> >>> >>> infinispan-dev mailing list
> >>> >>> infinispan-dev(a)lists.jboss.org
> >>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>> >
> >>> >
> >>
> >>
> >
> >
> >
>
14 years, 7 months
Patch for HHH-2308: Adjusting the Outer Join Predicate using Criteria Query
by Craig Christophel
Hello,
I've submitted a patch which updates Scott Van Wart's effort with unit tests and documentation changes (en-US) for HHH-2308. This patch allows users to add Criterion to Criteria aliasing similar to the WITH clause in hql.
List result = session.createCriteria( Student.class )
.createAlias( "preferredCourse", "pc", Criteria.LEFT_JOIN, Restrictions.eq("pc.courseCode", "HIB-A") )
.setProjection( Property.forName("pc.courseCode") )
.addOrder(Order.asc("pc.courseCode"))
.list();
This is useful when sorting large and or complex data inside the database when you intend to only retrieve X number of records from the result-set.
Without this patch we are forced to perform the same query as follows:
1. Query without join constraint and store results.
2. Query with an inner join and store results.
3. Manually sort in memory with java.
This effectively doubles the workload on the database as the queries are complex even without this join. It also causes some concern as to memory constraints because the full number of allowable data elements must be loaded twice. I'd love to be able to use HQL to solve this problem, however the search is very dynamic and the Criteria interface has helped to solve the problem much more elegantly.
The file hibernate-outer-join-criteria-trunk.diff attached to this defect is the latest example on trunk (3.5)
Thanks,
Craig Christophel
14 years, 7 months
Re: [hibernate-dev] [infinispan-dev] Benchmarking 2nd level cache providers?
by Galder Zamarreno
On 08/26/2009 05:48 PM, Mircea Markus wrote:
> Can't we write a plugin for the CacheBenchmark fwk?
We could potentially do so but we'd be missing loads of things and we'd
have to spend a fair bit of time replicating the way Hibernate uses the
cache which is not straightforward by any means. Take in account as well
that there're several caches and cache usages involved here: entity,
collection, query and timestamps and each has a different usage pattern.
Bottom line: I don't think trying to emulate this is worth the effort.
What is needed here is a Hibernate/JPA load/perf test environment where
the most common Hibernate usage patterns are exercised and then you swap
2nd level cache providers. This is really what counts to the users: how
quickly I can persist N entities, how quick I can read N times an entity
that I've just persisted, how fast I can execute same query N
times...etc, these are the use cases where the cache provider must show off.
I haven't heard anything from the HB team wrt to this, so I assume they
don't have anything like this.
>
> On Aug 19, 2009, at 1:14 PM, Galder Zamarreno wrote:
>
>> While talking to Manik online, the topic of 2nd level cache benchmarking
>> came up and was wondering if there's a way to benchmark different 2nd
>> level cache providers in Hibernate?
>>
>> Cheers,
>> --
>> Galder Zamarreño
>> Sr. Software Engineer
>> Infinispan, JBoss Cache
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
14 years, 7 months
Returned mail: Data format error
by Automatic Email Delivery Software
The original message was received at Thu, 24 Sep 2009 07:44:12 -0400
from lists.jboss.org [218.184.51.19]
----- The following addresses had permanent fatal errors -----
<hibernate-dev(a)lists.jboss.org>
----- Transcript of session follows -----
... while talking to host 123.212.204.180:
>>> DATA
<<< 400-aturner; %MAIL-E-OPENOUT, error opening !AS as output
<<< 400
14 years, 7 months
Annotations patch for column read/write expressions (HHH-272)
by Rob Hasselbaum
I'm working on an patch to expose HHH-272 functionality (custom column read
and write expressions).in Annotations. At Steve and Emmanuel's request, I'm
doing this by creating a new @o.h.a.Column annotation to supplement the
@javax.persistence.Column annotation, similar to what's in place today for
@Table. This will allow for future extension beyond this one use case,
eventually.
It's getting a bit dicey, though, because of the variety of places
@j.p.Column can show up--for example, inside @AttributeOverride and
@Columns. The question is how best to embed the companion @o.h.a.Column
instances in these contexts. Create a @o.h.a.AttributeOverride and
@o.h.a.AttributeOverrides? A totally separate grouping mechanism? It's a
little easier with @Columns because that's already in the org.hibernate
namespace hierarchy, so it can be modified. Thoughts?
-Rob
14 years, 7 months
Re: [hibernate-dev] [infinispan-dev] Feedback on Infinispan patch
by Łukasz Moreń
Hi,
Thanks for explanation.
Maybe better I will concentrate on the first release and postpone
distributed writing.
There is already LockStrategy that uses Infinispan. With using it I was
wrapping changes made by IndexWriter in Infinispan transaction, because of
performance reasons -
on lock obtaining transaction was started, on lock release transaction was
commited. Hovewer Ispn transaction commit on lock release is not good idea
since IndexWriter calls index commit before lock is released(and ispn
transaction is committed).
I was thinking to override Workspace class and getIndexWriter(start
infinispan tx), commitIndexWriter (commit tx) methods to wrap IndexWrite
lifecycle, but this needs few other changes. Some other ideas?
Cheers,
Lukasz
2009/9/21 Sanne Grinovero <sanne.grinovero(a)gmail.com>
> Hi Łukasz,
> you've rightful concerns, because the way the IndexWriter tries to
> achieve the lock
> that will bring some trouble; As far as I remember we decided in this
> first release
> to avoid multiple writer nodes because of this reasons
> (that's written in your docs?)
>
> Actually it shouldn't be very hard to do, as the LockStrategy is
> pluggable (see changes from HSEARCH-345)
> and you could implement one delegating to an Infinispan eager lock on some
> key,
> like the default LockStrategy takes a file lock in the index directory.
>
> Maybe it's simpler to support this distributed writing instead of
> sending the queue to some single
> (elected) node? Would be cool, as the Document Analysis effort would
> be distributed,
> but I have no idea if this would be more or less efficient than a
> single node writing; it could
> bring some huge data transfers along the wire during segments merging
> (basically fetching
> the whole index data at each node performing a segment merge); maybe
> you'll need to
> play with IndexWriter settings (
>
> http://docs.jboss.org/hibernate/stable/search/reference/en/html_single/#l...
> )
> probably need to find the sweet spot for "merge_factor".
> I just saw now that MergePolicy is now re-implementable, but I hope
> that won't be needed.
>
> Sanne
>
> 2009/9/21 Łukasz Moreń <lukasz.moren(a)gmail.com>:
> > Hi,
> >
> > I'm wondering if it is reasonable to have multiple threads/nodes that
> > modifies indexes in Lucene Directory based on Infinispan? Let's assume
> that
> > two nodes try to update index in this same time. First one creates
> > IndexWriter and obtains
> > write lock. There is high propability that second node throws
> > LockObtainFailedException (as one IndexWriter is allowed on single index)
> > and index is not modified. How is that? Should be always only one node
> that
> > makes changes in
> > the index?
> >
> > Cheers,
> > Lukasz
> >
> > W dniu 15 września 2009 01:39 użytkownik Łukasz Moreń
> > <lukasz.moren(a)gmail.com> napisał:
> >>
> >> Hi,
> >>
> >> With using JMeter I wanted to check if Infinispan dir does not crash
> under
> >> heavy load in "real" use and check performance in comparison with
> none/other
> >> directories.
> >> However appeared problem when multiple IndexWriters tries to modify
> index
> >> (test InfinispanDirectoryTest) - random deadlocks, and Lucene
> exceptions.
> >> IndexWriter tries to access files in index that were removed before. I'm
> >> looking into it, but not having good idea.
> >>
> >> Concerning the last part, I think similar thing is done in
> >> InfinispanDirectoryProviderTest. Many threads are making changes and
> >> searching (not checking if db is in sync with index).
> >> If threads finish their work, with Lucene query I'm checking if index
> >> contains as many results as expected. Maybe you meant something else?
> >> Would be good to run each node in different VM.
> >>
> >>> Great ! Looking forward to it. What state are things in at the moment
> >>> if I want to play around with it ?
> >>
> >> Should work with with one master(updates index) and one many slave nodes
> >> (sends changes to master). I tried with one master and one slave (both
> with
> >> jms and jgroups backend) and worked ok. Still fails if multiple nodes
> want
> >> to modify index.
> >>
> >> I've attached patch with current version.
> >>
> >> Cheers,
> >> Łukasz
> >>
> >> 2009/9/13 Michael Neale <michael.neale(a)gmail.com>
> >>>
> >>> Great ! Looking forward to it. What state are things in at the moment
> >>> if I want to play around with it ?
> >>>
> >>> Sent from my phone.
> >>>
> >>> On 13/09/2009, at 7:26 PM, Sanne Grinovero <sanne.grinovero(a)gmail.com>
> >>> wrote:
> >>>
> >>> > 2009/9/12 Michael Neale <michael.neale(a)gmail.com>:
> >>> >> That does sounds pretty cool. Would be nice if the lucene indexes
> >>> >> could scale along with how people will want to use infinispan.
> >>> >> Probably worth playing with.
> >>> >
> >>> > Sure, this is the goal of Łukasz's work; We know compass has
> >>> > some good Directories, but we're building our own as one based
> >>> > on Infinispan is not yet available.
> >>> >
> >>> >>
> >>> >> Sent from my phone.
> >>> >>
> >>> >> On 13/09/2009, at 8:37 AM, Jeff Ramsdale <jeff.ramsdale(a)gmail.com>
> >>> >> wrote:
> >>> >>
> >>> >>> I'm afraid I haven't followed the Infinispan-Lucene implementation
> >>> >>> closely, but have you looked at the Compass Project?
> >>> >>> (http://www.compass-project.org/overview.html) It provides a
> >>> >>> simplified interface to Lucene (optional) as well as Directory
> >>> >>> implementations built on Terracotta, Gigaspaces and Coherence. The
> >>> >>> latter, in particular, might be a useful guide for the Infinispan
> >>> >>> implementation. I believe it's mature enough to have solved many of
> >>> >>> the most difficult problems of implementing Directory on a
> >>> >>> distributed
> >>> >>> Map.
> >>> >>>
> >>> >>> If someone has any experience with Compass (particularly it's
> >>> >>> Directory implementations) I'd be interested in hearing about it...
> >>> >>> It's Apache 2.0 licensed, btw.
> >>> >>>
> >>> >>> -jeff
> >>> >>> _______________________________________________
> >>> >>> infinispan-dev mailing list
> >>> >>> infinispan-dev(a)lists.jboss.org
> >>> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>> >> _______________________________________________
> >>> >> infinispan-dev mailing list
> >>> >> infinispan-dev(a)lists.jboss.org
> >>> >> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>> >>
> >>> >
> >>> > _______________________________________________
> >>> > infinispan-dev mailing list
> >>> > infinispan-dev(a)lists.jboss.org
> >>> > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>>
> >>> _______________________________________________
> >>> infinispan-dev mailing list
> >>> infinispan-dev(a)lists.jboss.org
> >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >
> >
>
14 years, 7 months
Compilation failure on entitymanager
by Emmanuel Bernard
[INFO]
------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Compilation failure
/Users/manu/projects/trunk/core/entitymanager/src/main/java/org/
hibernate/ejb/QueryImpl.java:[89,45] cannot find symbol
symbol : method getParameterMetadata()
location: class org.hibernate.impl.AbstractQueryImpl
/Users/manu/projects/trunk/core/entitymanager/src/main/java/org/
hibernate/ejb/QueryImpl.java:[91,14] cannot find symbol
symbol : method getParameterMetadata()
location: class org.hibernate.impl.AbstractQueryImpl
/Users/manu/projects/trunk/core/entitymanager/src/main/java/org/
hibernate/ejb/QueryImpl.java:[108,34] cannot find symbol
symbol : method getParameterMetadata()
location: class org.hibernate.impl.AbstractQueryImpl
/Users/manu/projects/trunk/core/entitymanager/src/main/java/org/
hibernate/ejb/QueryImpl.java:[110,14] cannot find symbol
symbol : method getParameterMetadata()
location: class org.hibernate.impl.AbstractQueryImpl
14 years, 7 months
