[hibernate-issues] [Hibernate-JIRA] Commented: (HSEARCH-515) Lucene AlreadyClosedExceptions During Batch Processing With ASYNC Workers

[Sanne Grinovero (JIRA)]

    [ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=36805#action_36805 ] 

Sanne Grinovero commented on HSEARCH-515:
-----------------------------------------

a summary of the problem: when using async the production of Work might be faster than the consumer (IndexWriting); if this high load endures for a time long enough to get the asynchronous executor "pile up" more work than what the queue can store, it's Executor strategy is going to delegate the Runnable to be performed by the caller: this way under high load the async backend might see more than a single thread using the Workspace.
The Workspace is synchronized, but it's still possible that one thread closes the IndexWriter while the other thread just got it and was going to use it.

options:
* Define a very long (Long.maxvalue instead of 1000) for the queue size. Easy but we could end up using lots of memory, not a resources friendly approach.
* change the strategy from "caller runs" to "block and wait for space", but this is option is not available in Java5 Executor, we could code a para-executor using the blocking queues as I did for the MassIndexer - not loving this solution as it's a bit intrusive on the current design.
* even when "async=true" have the async work internally wait for the indexwriter steps. Easy to implement, does hurt a bit on performance as the threads are going to handle more waits on locks. Should consider that when the backend threads are all busy because of this lock, they might fill up their own limited queue and show up the same effect up to the user thread, way more unlikely but still not impossible to have concurrency on the Workspace.
* The Workspace is threadsafe already, we could use a reference counter on IndexWriter to make sure we don't close a IW being used by another thread, but as the work is all a queue it should be performed serially, not in parallel anyway, so imho we should prevent the Workspace from being used concurrently.

I'm thinking that option 2 is the only one providing a totally safe solution, but I need opinions and some time to experiment with it.
We could have a temporary workaround as option 3, imho it's very unlikely to happen in these cases, and increasing the queue size too (opt 1) as it's size would be less stressed.


> Lucene AlreadyClosedExceptions During Batch Processing With ASYNC Workers
> -------------------------------------------------------------------------
>
>                 Key: HSEARCH-515
>                 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-515
>             Project: Hibernate Search
>          Issue Type: Bug
>          Components: engine
>    Affects Versions: 3.1.0.GA, 3.1.1.GA, 3.2.0.Beta1, 3.2.0.CR1
>            Reporter: j nadler
>            Priority: Critical
>             Fix For: 3.2.0
>
>         Attachments: HSEARCH-515-reformatted-failing.log
>
>
> We have a batch job that updates 10,000 preexisting User entities in a single transaction; each User has an @IndexedEmbedded UserDetails entity which is also updated during the job run.  When run with hibernate.search.worker.execution=sync, life is good.
> When we run with async, indexing intermittently fails with a Lucene AlreadyClosedException.  There is no other exception logged even with TRACE logging enabled.
> We did not see this issue with 3.1.1GA but Sanne has noted that it is possible the exceptions were occurring and were not being logged in that version; I cannot exclude this possibility.
> Posted a log file to http://www.attensa.com/download/files/attensa.log  the first exception is at timestamp [2010-04-21 11:23:45.992,236829]
> Associated forum discussion here: https://forum.hibernate.org/viewtopic.php?f=9&t=1004022&p=2429214#p2429214

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira