Hi, Well, this time, I'm not sure we will be able to win the round by ourselves so I'm asking for advice and guidance... Following our recent patch submissions on the subject, we have one problem left: when we're doing a mass import (note: this mass import is really part of the application so we can't just disable Hibernate Search during it), we have the following pattern: Transaction { for (items partitioned by 100) { session.flush(); session.clear(); do work for these 100 items(); } } -> indexing is done at the end of the transaction by Hibernate Search automatically. Note that we really need to flush the session otherwise it really becomes too slow. >From what we understand, when we clear the session, the entities are detached from the session so the LuceneWorks executed when the transaction is commited are working on detached entities. Are we correct on this? The fact is that following the addition of fetch = FetchType.LAZY in certain relations, we start to have the following problems. Both are related though. I'm pretty sure we missed something as the problem seems obvious but isn't reproducible without all the steps we have in our import. We thought it might also be related to the fact that some entities might come from the cache. * Problem 1: it doesn't throw an exception but it doesn't sound right * This code in AbstractDocumentBuilder.appendContainedInWorkForInstance() is triggered as the session of the PersistentSet it's working on is null: try { collection = getActualCollection( member, value ); collection.size(); //load it } catch ( Exception e ) { if ( e.getClass().getName().contains( "org.hibernate.LazyInitializationException" ) ) { /* A deleted entity not having its collection initialized * leads to a LIE because the collection is no longer attached to the session * * But that's ok as the collection update event has been processed before * or the fk would have been cleared and thus triggering the cleaning */ collection = null; } } -> the collection is then completely ignored but shouldn't be as we are not working on a deleted entity. * Problem 2: this time, the exception is not caught and it breaks our mass import * When Hibernate Search tries to index an entity, it throws a LazyInitException in HibernateStatelessInitializer.unproxy() as, one more time, the session of the lazy proxy is null: it triggers the throw new LazyInitializationException( "could not initialize proxy - no Session" ); in AbstractLazyInitializer.initialize(). The fact is that the trick we added in the unproxy() method of HibernateLoadingSessionInitializer recently (see https://github.com/openwide-java/hibernate-search/commit/2f7844857bfafb95... and next commit on this file) cannot work as we don't have a session in Hibernate*Stateless*Initializer. We're kinda stuck here and I'm pretty sure we're missing something. Any idea? Thanks in advance. -- Guillaume _______________________________________________ hibernate-dev mailing list hibernate-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev

Is it transactionally safe?

From what I read, it's not.

On Mon, Dec 10, 2012 at 9:07 PM, Guillaume Smet <guillaume.smet(a)gmail.com&gt; wrote: > On Mon, Dec 10, 2012 at 8:27 PM, Guillaume Smet > <guillaume.smet(a)gmail.com&gt; wrote: >> Is it transactionally safe? > > From what I read, it's not. > > Do you see any way to get this type of pattern working in a > transactionally safe way? Hmmm, just handwaving at the moment but couldn't we imagine that, instead of flushing to the indexes, we could flush to a "buffer" of Lucene documents. I think it could reduce the memory footprint compared to keeping all the entities in memory while allowing us to keep the operation transactional? And it might allow us to get the above case working. Perhaps, flushing to a queue of Lucene documents isn't the right idea but we could prepare/"compile" the LuceneWork to whatever is convenient for this particular operation (Lucene document, query for deletion...), get rid of the entity and all the entity related stuff and apply the works at the commit of the transaction as it's already done. I'm pretty sure it could be an interesting compromise - if doable. Any thoughts? -- Guillaume _______________________________________________ hibernate-dev mailing list hibernate-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev

2# I didn't see Guillaume asking about queries, so assuming he would just need a large write buffer where to keep the current Documents; this is a simpler patch as we already do build such a plan during any index flush / commit. Have a look into org.hibernate.search.backend.impl.BatchedQueueingProcessor a simple way would be to have the method _performWorks_ to "shelf" the WorkQueue in a list rather than committing it, a more sophisticated patch (not sure if it's worth it) might want to reuse the same workQueue.