[Hibernate-JIRA] Created: (HSEARCH-598) MassIndexer freezes when pool size is too low

[Hibernate-JIRA] (HSEARCH-1257)...

[Hibernate-JIRA] Created:...

I D (JIRA)

Saturday, 25 September 2010 Sat, 25 Sep '10

2:52 p.m.

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

Show replies by thread

Sanne Grinovero (JIRA)

Saturday, 25 September Sat, 25 Sep

5:08 p.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Sanne Grinovero updated HSEARCH-598: ------------------------------------ Fix Version/s: 3.3.0.Beta2 3.2.2 Component/s: documentation right, I'm not sure if we can "detect" the problem before freezing, but at least mentioning it in the documentation should be done.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Fix For: 3.2.2, 3.3.0.Beta2 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

-- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://opensource.atlassian.com/projects/hibernate/secure/Administrators.... - For more information on JIRA, see: http://www.atlassian.com/software/jira

Emmanuel Bernard (JIRA)

Friday, 8 October Fri, 8 Oct

7:19 p.m.

New subject: [Hibernate-JIRA] Assigned: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Emmanuel Bernard reassigned HSEARCH-598: ---------------------------------------- Assignee: Sanne Grinovero Yes the documentation + the FAQ seem like a good place. The annoying thing is that you cannot necessarily predict how many concurrent DB connection you will need. Do you have an idea of why this is happening? Meaning if the system can recover from this resource starving? AFAIR we have 2 or 3 queues that in practice use the database.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.Beta2 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

I D (JIRA)

Saturday, 9 October Sat, 9 Oct

11:35 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] I D commented on HSEARCH-598: ----------------------------- The inconsistency of the freeze (e.g. when maxPoolSize=15) indicates a race condition. It is possible that some of the queues / threads can recover from unavailable connections, whereas the others cannot. Like a game of musical chairs: when only robust threads are left standing with no available connections - there is no freeze; when a dumb thread is left standing - we get a freeze. This is just a theory.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.Beta2 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

I D (JIRA)

12:46 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] I D commented on HSEARCH-598: ----------------------------- Just got my first freeze with maxPoolSize=20. Moved to 30. This is not good. Is there any value that's guaranteed to be safe? Please provide a constant value, or if this value is dependent on the size of the data-set - I'd love to know how it can be computed.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.Beta2 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Sanne Grinovero (JIRA)

Friday, 15 October Fri, 15 Oct

8:29 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Sanne Grinovero commented on HSEARCH-598: ----------------------------------------- hi, the dataset size is not relevant, the number of threads enabled on the indexer are. each thread might potentially need an hibernate session, and this number varies with the actual data but of course will never be above the number of threads. Until we find a better way, to be safe always have a higher number of connections than the total of each set of threads of the massindexer, and it won't hurt to have a couple of spare ones. I'll figure out what the exact numbers are and document it, but I don't think we can actually prevent users from setting this wrong - still looking for ideas and suggestions are welcome.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.Beta2 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

I D (JIRA)

11:56 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] I D commented on HSEARCH-598: ----------------------------- It appears that the total number of threads is 7, unless otherwise configured: - MassIndexerImpl.objectLoadingThreads defaults to 2 - MassIndexerImpl.collectionLoadingThreads defaults to 4 - The main thread However, I've seen MassIndexer use over 20 connections, so either it's creating more threads than it's configured to create, or it's leaking threads, or some of the threads are using more than one session / connection each.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.Beta2 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Emmanuel Bernard (JIRA)

4:06 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Emmanuel Bernard commented on HSEARCH-598: ------------------------------------------ I wonder if we could detect the starvation process and log warnings explaining the probable cause. I could imagine two cases, the first being the connection provider raising an exception, in his case all is good as the cause is well understood though we need doc), the second would be to have a timeout on the operation surrounding the connection gathering. It's just raw ideas at this point

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.Beta2 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Emmanuel Bernard (JIRA)

9:08 p.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Emmanuel Bernard updated HSEARCH-598: ------------------------------------- Fix Version/s: (was: 3.3.0.Beta2) 3.3.0.Beta3

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.Beta3 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Emmanuel Bernard (JIRA)

Saturday, 30 October Sat, 30 Oct

1:12 a.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-598) MassIndexer freezes when pool size is too low

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.CR1 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Michael Mogley (JIRA)

2:52 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Michael Mogley commented on HSEARCH-598: ---------------------------------------- I have hit this issue myself, and it has prevented me from using the mass-indexer. From my own tests, it seems like the indexer was leaking connections, because no matter how large a connection pool I started with, eventually it would hit the limit. It seems like a good solution might be to add another configuration parameter to set the maximum number of connection the indexer attempts to invoke. This setting would then serve as a ceiling for other related settings. For example, if I set maxConnection = 10, loadingThreads = 20, fetchingThreads = 100, the maxConnections setting would effectively floor the other two values in a way that ensures only 10 connections are ever opened at any one time.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.CR1 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Sanne Grinovero (JIRA)

3:12 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Sanne Grinovero commented on HSEARCH-598: ----------------------------------------- the problem is that it's exactly what happens: an 11th connection is needed and the pool refuses to hand it over. So you get a deadlock. You're saying that having size(maxConnetion) > loadingThread+fetchinThreads it still locks? Could it be another issue, or my diagnose wrong?

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.CR1 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Michael Mogley (JIRA)

3:18 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Michael Mogley commented on HSEARCH-598: ---------------------------------------- Yes, that's what I'm saying. The code seems to be leaking connections. But also, from an api standpoint, it's not clear to me as a user that loading + fetching threads translates into the max number of connections. That requires me to know more about the internals of the mass indexer than I care to. Why not have a third parameter that's clearly named, such as I suggested, that keeps the other two parameters at bay?

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.CR1 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Sanne Grinovero (JIRA)

6:54 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Sanne Grinovero commented on HSEARCH-598: ----------------------------------------- ah sorry, now I understand better; still not all your proposal. An explicit parameter might work, but the code would just to the math and throw an exception? (I totally agree that the API user shouldn't need to be aware of this)

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.CR1 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Michael Mogley (JIRA)

7:37 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Michael Mogley commented on HSEARCH-598: ---------------------------------------- Or the code could do the math and see if it can limit the loading/fetching threads based on some built-in heuristic. In this case, it would also issue a warning, explaining the situation and the steps it took. In case the maxConnections setting makes it impossible to set the load/fetch threads to any reasonable number (e.g., maxConnections = 0), it would then throw an appropriate error on startup.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.CR1 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Sanne Grinovero (JIRA)

7:59 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Sanne Grinovero commented on HSEARCH-598: ----------------------------------------- how's this built-in heuristics looking like? you have a proposal? I'm afraid the the resulting API would look similar to: indexer.fetchingThreads(10).loadingThreads(5).verifyTotalThreasIsLessThan(12) --> exception "10 + 5 is higher than 12! we're going to deadlock!" this doesn't seem very useful to me - so my guess is that I didn't understand what you mean with doing the math? I think I'm going to see if the internals can be refactored to avoid the deadlock - seems easier than this heuristics, but I'd be very pleased if you came up with a good idea.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.CR1 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Michael Mogley (JIRA)

8:06 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Michael Mogley commented on HSEARCH-598: ---------------------------------------- I'm saying this: // The maximum number connections this indexer can have open at once. indexer.setMaxConnections(10); // Internally checks maxConnections. 7 < 10? Ok to proceed. indexer.fetchingThreads(7); // Internally checks maxConnections. 7 already used. Limit 5 to 3. indexer.loadingThreads(5); The heuristic would attempt to still create the indexer, but limit the number of threads to the max number of connections specified at the outset. Alternatively, you could always just throw an exception if the math doesn't work out. But I think having it try to configure itself and issue a warning would be more friendly.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0.CR1 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Emmanuel Bernard (JIRA)

Monday, 8 November Mon, 8 Nov

7:46 p.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Emmanuel Bernard updated HSEARCH-598: ------------------------------------- Fix Version/s: (was: 3.3.0.CR1) 3.3.0

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.3.0 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Emmanuel Bernard (JIRA)

Wednesday, 8 December Wed, 8 Dec

7:21 p.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Emmanuel Bernard updated HSEARCH-598: ------------------------------------- Fix Version/s: (was: 3.3.0.CR2) 3.4.0

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.4.0 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Sanne Grinovero (JIRA)

Monday, 14 February Mon, 14 Feb

8:27 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Sanne Grinovero commented on HSEARCH-598: ----------------------------------------- Another case: https://forum.hibernate.org/viewtopic.php?f=9&t=1009524 I don't think we can detect this, but I could re-arrange the order of thread start to start first the minimum number of threads needed to keep it alive, and then gradually attempt to increase them - optimistically. If after that some threads fail to make progress, we won't know that nor be able to warn. It would work just fine, just suboptimal for performance; depending on point of view, it might be considered optimal: if we can't have more connections, that's the best it can do.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.4.0 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Adrien (JIRA)

Thursday, 31 March Thu, 31 Mar

1:31 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Adrien commented on HSEARCH-598: -------------------------------- I have the same problem and here what I found : Using the mass indexer with 34 entities, it launch 34 BatchIndexingWorkspace. Even with threadsToLoadObjects(1) and threadsForSubsequentFetching(1), each BatchIndexingWorkspace take 3 connections (one for entity consumer, one for identy consumer and one for identity producer). So a total of 102 connections. The problem comes from the fact that if the identity consumer take the last connection first and then wait for its identity producer, the identity producer waits for a connection that will never be given back. Double lock. One way to go around this problem would be not to launch the indexer for all the entity at the same time. To control the executor in BatchCoordinator.doBatchWork(). Any other idea ?

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.4.0 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Adrien (JIRA)

Friday, 1 April Fri, 1 Apr

5:20 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Adrien commented on HSEARCH-598: -------------------------------- I fixed the problem for me by limiting the number of threads launched in executor in BatchCoordinator.doBatchWork(). I could give you a patch to allow to configure this number of threads (default to number of enity) is you agree to the idea. But it wont fixed the true dead lock. One solution to really avoid the dead lock even with lots of threads it to obtain the connection not in the consumer/producer threads, but in the workspace threads. We may do that by changing OptionallyWrapInJTATransaction : If we obtain the session in the constructor of instead of in the run(), it will be executed in the workspace thread. We also need to start the transaction in the constructor because the database connection is only open there. I'm just wondering if opening and closing the transaction in different threads will be ok with JTA transaction.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.4.0 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Sanne Grinovero (JIRA)

6:49 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Sanne Grinovero commented on HSEARCH-598: ----------------------------------------- Hi Adrien, thanks for all the insight. I don't think it's safe to open the connection in a different thread, as you say some containers might have issues with that; I'm not sure I could ask for opinions of colleagues but it doesn't feel right, let's explore alternatives first. Also remember that opening a Session doesn't open a connection, and that a Session isn't designed to be passed to a different thread - so even hacking in an eager connection acquire there might be other issues. About your case, you have 34 different indexed entities? I think that reducing each worker to a single thread is likely going to hurt overall performance, did you consider launching the MassIndexer on each type sequentially? You can have it reindex a single type only - wondering if we should rather add an option to have it perform the different indexations in sequence, or in a maximum configurable parallelism. A patch would be awesome. Other possible approaches: * Have a background thread monitor advancement, possible killing off threads when a lock is spotted, so dynamically reducing parallelism to find a sweet spot. * Have the jobs use a comment Executor, to control global thread amounts (requires some care to have the same pipeline fully active) * Revert the design in such a way that each stage is able to handle all different entities - shouldn't be hard, and my original design was like that but it had a small performance overhead compared to the current pipelines. Could be fun to play with some work-stealing pattern, in which each thread starts working on the first part of the pipeline to switch stealing work from the subsequent phases when the output queue of the first stage is full. This way we could have a single pool of threads working on the first part only, and doing some of the subsequent work when he can't make progress in the first stage -> no deadlock possible, likely even better performance assuming the context switch decision can be written with efficient code.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.4.0 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Sanne Grinovero (JIRA)

6:49 p.m.

New subject: [Hibernate-JIRA] Issue Comment Edited: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Sanne Grinovero edited comment on HSEARCH-598 at 4/1/11 4:48 AM: ----------------------------------------------------------------- Hi Adrien, thanks for all the insight. I don't think it's safe to open the connection in a different thread, as you say some containers might have issues with that; I'm not sure I could ask for opinions on this but it doesn't feel right, let's explore alternatives first. Also remember that opening a Session doesn't open a connection, and that a Session isn't designed to be passed to a different thread - so even hacking in an eager connection acquire there might be other issues. About your case, you have 34 different indexed entities? I think that reducing each worker to a single thread is likely going to hurt overall performance, did you consider launching the MassIndexer on each type sequentially? You can have it reindex a single type only - wondering if we should rather add an option to have it perform the different indexations in sequence, or in a maximum configurable parallelism. A patch would be awesome. Other possible approaches: * Have a background thread monitor advancement, possible killing off threads when a lock is spotted, so dynamically reducing parallelism to find a sweet spot. * Have the jobs use a comment Executor, to control global thread amounts (requires some care to have the same pipeline fully active) * Revert the design in such a way that each stage is able to handle all different entities - shouldn't be hard, and my original design was like that but it had a small performance overhead compared to the current pipelines. Could be fun to play with some work-stealing pattern, in which each thread starts working on the first part of the pipeline to switch stealing work from the subsequent phases when the output queue of the first stage is full. This way we could have a single pool of threads working on the first part only, and doing some of the subsequent work when he can't make progress in the first stage -> no deadlock possible, likely even better performance assuming the context switch decision can be written with efficient code. was (Author: sanne): Hi Adrien, thanks for all the insight. I don't think it's safe to open the connection in a different thread, as you say some containers might have issues with that; I'm not sure I could ask for opinions of colleagues but it doesn't feel right, let's explore alternatives first. Also remember that opening a Session doesn't open a connection, and that a Session isn't designed to be passed to a different thread - so even hacking in an eager connection acquire there might be other issues. About your case, you have 34 different indexed entities? I think that reducing each worker to a single thread is likely going to hurt overall performance, did you consider launching the MassIndexer on each type sequentially? You can have it reindex a single type only - wondering if we should rather add an option to have it perform the different indexations in sequence, or in a maximum configurable parallelism. A patch would be awesome. Other possible approaches: * Have a background thread monitor advancement, possible killing off threads when a lock is spotted, so dynamically reducing parallelism to find a sweet spot. * Have the jobs use a comment Executor, to control global thread amounts (requires some care to have the same pipeline fully active) * Revert the design in such a way that each stage is able to handle all different entities - shouldn't be hard, and my original design was like that but it had a small performance overhead compared to the current pipelines. Could be fun to play with some work-stealing pattern, in which each thread starts working on the first part of the pipeline to switch stealing work from the subsequent phases when the output queue of the first stage is full. This way we could have a single pool of threads working on the first part only, and doing some of the subsequent work when he can't make progress in the first stage -> no deadlock possible, likely even better performance assuming the context switch decision can be written with efficient code.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.4.0 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Adrien (JIRA)

Monday, 4 April Mon, 4 Apr

11:29 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Adrien commented on HSEARCH-598: -------------------------------- How about doing all the work in one connection, since we only do read only stuff ? We obtain a connection with session.doWork() in MassIndexerImpl, pass the connection to all the threads and create session in them with sessionFactory.openSession(Connection). This way we may have all the threads we want without risking dead lock or even delay waiting for a connection.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.4.0 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Sanne Grinovero (JIRA)

Tuesday, 5 April Tue, 5 Apr

12:06 a.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Sanne Grinovero commented on HSEARCH-598: ----------------------------------------- Interesting thought, but I'm afraid that many java.sql.Connection implementations are not threadsafe in practice, even if they should be by spec; seems to look for trouble. Performance-wise, I really wanted to use more connections, as the bottleneck in indexing is usually not the CPU but the delay coming from lazy loading and the many network roundtrips to the DB.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.2.2, 3.4.0 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Emmanuel Bernard (JIRA)

Monday, 18 April Mon, 18 Apr

9:03 p.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-598) MassIndexer freezes when pool size is too low

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 3.5 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

Sanne Grinovero (JIRA)

Tuesday, 6 September Tue, 6 Sep

11:07 p.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Sanne Grinovero updated HSEARCH-598: ------------------------------------ Fix Version/s: (was: 4.0) 4.0.0.Beta2

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 4.0.0.Beta2 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

-- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

Sanne Grinovero (JIRA)

Thursday, 29 September Thu, 29 Sep

2:17 a.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Sanne Grinovero updated HSEARCH-598: ------------------------------------ Fix Version/s: (was: 4.0.0.Beta2) 4.0.0.CR1

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 4.0.0.CR1 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

-- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

Emmanuel Bernard (JIRA)

Thursday, 13 October Thu, 13 Oct

9:20 p.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Emmanuel Bernard updated HSEARCH-598: ------------------------------------- Fix Version/s: (was: 4.0.0.CR1) 4.0

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 4.0 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

-- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

Sanne Grinovero (JIRA)

Wednesday, 19 October Wed, 19 Oct

6:51 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Sanne Grinovero commented on HSEARCH-598: ----------------------------------------- More input about this: https://forum.hibernate.org/viewtopic.php?f=9&t=1013012 I've described some possible work-arounds there.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 4.0 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

-- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

Laubrino (JIRA)

Wednesday, 14 December Wed, 14 Dec

5:05 p.m.

New subject: [Hibernate-JIRA] Commented: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Laubrino commented on HSEARCH-598: ---------------------------------- I experienced same problem here. MassIndexer always hangs at about 95% of reindexing.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 4.0.0.Final In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

-- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

Laubrino (JIRA)

Tuesday, 20 December Tue, 20 Dec

6:29 p.m.

New subject: [Hibernate-JIRA] Issue Comment Edited: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Laubrino edited comment on HSEARCH-598 at 12/20/11 3:27 AM: ------------------------------------------------------------ I experienced same problem here. MassIndexer always hangs at about 95% of reindexing. EDITED: IT WAS MY FAULT. Sory, the problem was somewhere else. The indexing process reached outofmemory error in my lazy field. But the error was nowhere reported. It just hanged. Then I set all thread settings to 1 (threadsForIndexWriter, threadsForSubsequentFetching, threadsToLoadObjects) and the error appeared in on of my logs. So I fixed it and the MassIndexer works fine was (Author: laubrino): I experienced same problem here. MassIndexer always hangs at about 95% of reindexing.

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero Fix For: 4.1 In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

-- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

Emmanuel Bernard (JIRA)

Friday, 23 December Fri, 23 Dec

11:40 p.m.

New subject: [Hibernate-JIRA] Updated: (HSEARCH-598) MassIndexer freezes when pool size is too low

[ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598?pag... ] Emmanuel Bernard updated HSEARCH-598: ------------------------------------- Fix Version/s: (was: 4.1)

...

MassIndexer freezes when pool size is too low --------------------------------------------- Key: HSEARCH-598 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-598 Project: Hibernate Search Issue Type: Bug Components: documentation, massindexer Affects Versions: 3.2.1 Environment: Hibernate Core 3.5.4, PostgreSQL 8.4.4 Reporter: I D Assignee: Sanne Grinovero In our application we use db connection pooling at the servlet container level - Jetty instantiates a com.mchange.v2.c3p0.ComboPooledDataSource. We've disabled Hibernate's connection pooling to avoid multiple connection pools. Soon after starting to use MassIndexer we noticed that it SOMETIMES freezes during operation - startAndWait() just hangs indefinitely. After some experimentation, we realized that during this freeze the connection pool is maxed out and all the 15 connections (c3p0's default value for maxPoolSize is 15) are active. We therefore experimented with various values for maxPoolSize and found that 10 or less always seems to cause freezes, whereas 20 or more seems to work fine consistently. In between is a grey area, where the freeze occurs inconsistently (this grey area may of course extend to maxPoolSize<=10 and/or maxPoolSize>=20, since our tests only provide a partial statistical sample). If this is expected behavior, the minimal pool size / number of required connections should be well documented.

-- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

Kai Moritz (JIRA)

Tuesday, 8 January Tue, 8 Jan

8:40 p.m.

New subject: [Hibernate-JIRA] (HSEARCH-598) MassIndexer freezes when pool size is too low

Sanne Grinovero (JIRA)

Wednesday, 9 January Wed, 9 Jan

3:15 a.m.

New subject: [Hibernate-JIRA] (HSEARCH-598) MassIndexer freezes when pool size is too low

Kai Moritz (JIRA)

8:21 a.m.

New subject: [Hibernate-JIRA] (HSEARCH-598) MassIndexer freezes when pool size is too low

Sanne Grinovero (JIRA)

8:45 p.m.

New subject: [Hibernate-JIRA] (HSEARCH-598) MassIndexer freezes when pool size is too low

4145

days inactive

4982

days old

hibernate-issues@lists.jboss.org

Manage subscription

37 comments

7 participants

tags (0)

participants (7)

Adrien (JIRA)
Emmanuel Bernard (JIRA)
I D (JIRA)
Kai Moritz (JIRA)
Laubrino (JIRA)
Michael Mogley (JIRA)
Sanne Grinovero (JIRA)

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

[Hibernate-JIRA] Created: (HSEARCH-598) MassIndexer freezes when pool size is too low