[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-515?pag... ] Sanne Grinovero commented on HSEARCH-515: ----------------------------------------- is just one entity indexed, or is UserDetails indexed too? I need to understand if these logs are relevant to a single index or about more than one index -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-515?pag... ] Sanne Grinovero updated HSEARCH-515: ------------------------------------ Attachment: HSEARCH-515-reformatted-failing.log attaching the log here; removed some unneeded details for readability (full date, log level, blank lines) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-515?pag... ] Sanne Grinovero commented on HSEARCH-515: ----------------------------------------- a summary of the problem: when using async the production of Work might be faster than the consumer (IndexWriting); if this high load endures for a time long enough to get the asynchronous executor "pile up" more work than what the queue can store, it's Executor strategy is going to delegate the Runnable to be performed by the caller: this way under high load the async backend might see more than a single thread using the Workspace. The Workspace is synchronized, but it's still possible that one thread closes the IndexWriter while the other thread just got it and was going to use it. options: * Define a very long (Long.maxvalue instead of 1000) for the queue size. Easy but we could end up using lots of memory, not a resources friendly approach. * change the strategy from "caller runs" to "block and wait for space", but this is option is not available in Java5 Executor, we could code a para-executor using the blocking queues as I did for the MassIndexer - not loving this solution as it's a bit intrusive on the current design. * even when "async=true" have the async work internally wait for the indexwriter steps. Easy to implement, does hurt a bit on performance as the threads are going to handle more waits on locks. Should consider that when the backend threads are all busy because of this lock, they might fill up their own limited queue and show up the same effect up to the user thread, way more unlikely but still not impossible to have concurrency on the Workspace. * The Workspace is threadsafe already, we could use a reference counter on IndexWriter to make sure we don't close a IW being used by another thread, but as the work is all a queue it should be performed serially, not in parallel anyway, so imho we should prevent the Workspace from being used concurrently. I'm thinking that option 2 is the only one providing a totally safe solution, but I need opinions and some time to experiment with it. We could have a temporary workaround as option 3, imho it's very unlikely to happen in these cases, and increasing the queue size too (opt 1) as it's size would be less stressed. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-515?pag... ] Sanne Grinovero reassigned HSEARCH-515: --------------------------------------- Assignee: Sanne Grinovero -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-515?pag... ] j nadler commented on HSEARCH-515: ---------------------------------- Sanne thanks so much. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira