Lock in eviction.Region:putNodeEvent() -------------------------------------- Key: JBCACHE-1332 URL: http://jira.jboss.com/jira/browse/JBCACHE-1332 Project: JBoss Cache Issue Type: Bug Security Level: Public(Everyone can see) Components: Eviction Affects Versions: 1.4.1.SP1 Environment: JBoss 4.05 Hibernate 3.2.6.ga Reporter: Nicolas Lebreton Assigned To: Manik Surtani Priority: Critical Attachments: jboss-service.xml, ThreadDumps.zip JBoss TreeCache is used as cache provider. Please find attached jboss-service.xml configuration file. The second level cache is actitated, and several read-only classes are cacheable: <cache usage="read-only"/> The query cache is activated, and few queries are cacheable: <query name="ActorType.listAll.named" cacheable="true"> We encounter an issue. It can be reproduced on an environment with only few users connected (less than 10). For those users, screen is refreshed every 30 seconds through Ajax requests. Application is running in a cluster. On the other hand, this issue occurs with only one node started. A JBoss is started. Less than 24 hours later, application is becoming unresponsive. JBoss then needs to be recycled. And it works. Before stoping the node, I have taken several thread dumps separated by at least 30 seconds. Please find attached 10th, 11th, 25th's output files. We noticed thread locks in these 3 snapshots. The stack trace of the culprit was always the same (please see attached outputs for more details): # java.lang.Object:wait # waiting on-->EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue(0x9331fd60) # java.lang.Object:wait(Object.java:474) # EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue:put # locked-->EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue(0x9331fd60) # locked-->java.lang.Object(0x9331fe78) # org.jboss.cache.eviction.Region:putNodeEvent(Region.java:141) # org.jboss.cache.interceptors.EvictionInterceptor:doEventUpdatesOnRegionManager(EvictionInterceptor.java:145) # org.jboss.cache.interceptors.EvictionInterceptor:updateNode(EvictionInterceptor.java:118) # org.jboss.cache.interceptors.EvictionInterceptor:invoke(EvictionInterceptor.java:93) Nicolas

Lock in eviction.Region:putNodeEvent() -------------------------------------- Key: JBCACHE-1332 URL: http://jira.jboss.com/jira/browse/JBCACHE-1332 Project: JBoss Cache Issue Type: Bug Security Level: Public(Everyone can see) Components: Eviction Affects Versions: 1.4.1.SP1 Environment: JBoss 4.05 Hibernate 3.2.6.ga Reporter: Nicolas Lebreton Assigned To: Manik Surtani Priority: Critical Fix For: 1.4.X Attachments: 2008-04-10-molene.nexotelematics.com_susan_node2_Percentage db connections in use.jpg, jboss-service.xml, ThreadDumps.zip JBoss TreeCache is used as cache provider. Please find attached jboss-service.xml configuration file. The second level cache is actitated, and several read-only classes are cacheable: <cache usage="read-only"/> The query cache is activated, and few queries are cacheable: <query name="ActorType.listAll.named" cacheable="true"> We encounter an issue. It can be reproduced on an environment with only few users connected (less than 10). For those users, screen is refreshed every 30 seconds through Ajax requests. Application is running in a cluster. On the other hand, this issue occurs with only one node started. A JBoss is started. Less than 24 hours later, application is becoming unresponsive. JBoss then needs to be recycled. And it works. Before stoping the node, I have taken several thread dumps separated by at least 30 seconds. Please find attached 10th, 11th, 25th's output files. We noticed thread locks in these 3 snapshots. The stack trace of the culprit was always the same (please see attached outputs for more details): # java.lang.Object:wait # waiting on-->EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue(0x9331fd60) # java.lang.Object:wait(Object.java:474) # EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue:put # locked-->EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue(0x9331fd60) # locked-->java.lang.Object(0x9331fe78) # org.jboss.cache.eviction.Region:putNodeEvent(Region.java:141) # org.jboss.cache.interceptors.EvictionInterceptor:doEventUpdatesOnRegionManager(EvictionInterceptor.java:145) # org.jboss.cache.interceptors.EvictionInterceptor:updateNode(EvictionInterceptor.java:118) # org.jboss.cache.interceptors.EvictionInterceptor:invoke(EvictionInterceptor.java:93) Nicolas

Lock in eviction.Region:putNodeEvent() -------------------------------------- Key: JBCACHE-1332 URL: http://jira.jboss.com/jira/browse/JBCACHE-1332 Project: JBoss Cache Issue Type: Bug Security Level: Public(Everyone can see) Components: Eviction Affects Versions: 1.4.1.SP1 Environment: JBoss 4.05 Hibernate 3.2.6.ga Reporter: Nicolas Lebreton Assigned To: Manik Surtani Priority: Critical Fix For: 1.4.X Attachments: 2008-04-10-molene.nexotelematics.com_susan_node2_Percentage db connections in use.jpg, 2008-04-25 locktree.svg, jboss-service.xml, ThreadDumps.zip JBoss TreeCache is used as cache provider. Please find attached jboss-service.xml configuration file. The second level cache is actitated, and several read-only classes are cacheable: <cache usage="read-only"/> The query cache is activated, and few queries are cacheable: <query name="ActorType.listAll.named" cacheable="true"> We encounter an issue. It can be reproduced on an environment with only few users connected (less than 10). For those users, screen is refreshed every 30 seconds through Ajax requests. Application is running in a cluster. On the other hand, this issue occurs with only one node started. A JBoss is started. Less than 24 hours later, application is becoming unresponsive. JBoss then needs to be recycled. And it works. Before stoping the node, I have taken several thread dumps separated by at least 30 seconds. Please find attached 10th, 11th, 25th's output files. We noticed thread locks in these 3 snapshots. The stack trace of the culprit was always the same (please see attached outputs for more details): # java.lang.Object:wait # waiting on-->EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue(0x9331fd60) # java.lang.Object:wait(Object.java:474) # EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue:put # locked-->EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue(0x9331fd60) # locked-->java.lang.Object(0x9331fe78) # org.jboss.cache.eviction.Region:putNodeEvent(Region.java:141) # org.jboss.cache.interceptors.EvictionInterceptor:doEventUpdatesOnRegionManager(EvictionInterceptor.java:145) # org.jboss.cache.interceptors.EvictionInterceptor:updateNode(EvictionInterceptor.java:118) # org.jboss.cache.interceptors.EvictionInterceptor:invoke(EvictionInterceptor.java:93) Nicolas

Lock in eviction.Region:putNodeEvent() -------------------------------------- Key: JBCACHE-1332 URL: http://jira.jboss.com/jira/browse/JBCACHE-1332 Project: JBoss Cache Issue Type: Bug Security Level: Public(Everyone can see) Components: Eviction Affects Versions: 1.4.1.SP1 Environment: JBoss 4.05 Hibernate 3.2.6.ga Reporter: Nicolas Lebreton Assigned To: Manik Surtani Priority: Critical Fix For: 1.4.X Attachments: 2008-04-10-molene.nexotelematics.com_susan_node2_Percentage db connections in use.jpg, 2008-04-25 locktree.svg, jboss-service.xml, ThreadDumps.zip JBoss TreeCache is used as cache provider. Please find attached jboss-service.xml configuration file. The second level cache is actitated, and several read-only classes are cacheable: <cache usage="read-only"/> The query cache is activated, and few queries are cacheable: <query name="ActorType.listAll.named" cacheable="true"> We encounter an issue. It can be reproduced on an environment with only few users connected (less than 10). For those users, screen is refreshed every 30 seconds through Ajax requests. Application is running in a cluster. On the other hand, this issue occurs with only one node started. A JBoss is started. Less than 24 hours later, application is becoming unresponsive. JBoss then needs to be recycled. And it works. Before stoping the node, I have taken several thread dumps separated by at least 30 seconds. Please find attached 10th, 11th, 25th's output files. We noticed thread locks in these 3 snapshots. The stack trace of the culprit was always the same (please see attached outputs for more details): # java.lang.Object:wait # waiting on-->EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue(0x9331fd60) # java.lang.Object:wait(Object.java:474) # EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue:put # locked-->EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue(0x9331fd60) # locked-->java.lang.Object(0x9331fe78) # org.jboss.cache.eviction.Region:putNodeEvent(Region.java:141) # org.jboss.cache.interceptors.EvictionInterceptor:doEventUpdatesOnRegionManager(EvictionInterceptor.java:145) # org.jboss.cache.interceptors.EvictionInterceptor:updateNode(EvictionInterceptor.java:118) # org.jboss.cache.interceptors.EvictionInterceptor:invoke(EvictionInterceptor.java:93) Nicolas

Lock in eviction.Region:putNodeEvent() -------------------------------------- Key: JBCACHE-1332 URL: http://jira.jboss.com/jira/browse/JBCACHE-1332 Project: JBoss Cache Issue Type: Bug Security Level: Public(Everyone can see) Components: Eviction Affects Versions: 1.4.1.SP1 Environment: JBoss 4.05 Hibernate 3.2.6.ga Reporter: Nicolas Lebreton Assigned To: Manik Surtani Priority: Critical Fix For: 1.4.X Attachments: 2008-04-10-molene.nexotelematics.com_susan_node2_Percentage db connections in use.jpg, 2008-04-25 locktree.svg, jboss-service.xml, ThreadDumps.zip JBoss TreeCache is used as cache provider. Please find attached jboss-service.xml configuration file. The second level cache is actitated, and several read-only classes are cacheable: <cache usage="read-only"/> The query cache is activated, and few queries are cacheable: <query name="ActorType.listAll.named" cacheable="true"> We encounter an issue. It can be reproduced on an environment with only few users connected (less than 10). For those users, screen is refreshed every 30 seconds through Ajax requests. Application is running in a cluster. On the other hand, this issue occurs with only one node started. A JBoss is started. Less than 24 hours later, application is becoming unresponsive. JBoss then needs to be recycled. And it works. Before stoping the node, I have taken several thread dumps separated by at least 30 seconds. Please find attached 10th, 11th, 25th's output files. We noticed thread locks in these 3 snapshots. The stack trace of the culprit was always the same (please see attached outputs for more details): # java.lang.Object:wait # waiting on-->EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue(0x9331fd60) # java.lang.Object:wait(Object.java:474) # EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue:put # locked-->EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue(0x9331fd60) # locked-->java.lang.Object(0x9331fe78) # org.jboss.cache.eviction.Region:putNodeEvent(Region.java:141) # org.jboss.cache.interceptors.EvictionInterceptor:doEventUpdatesOnRegionManager(EvictionInterceptor.java:145) # org.jboss.cache.interceptors.EvictionInterceptor:updateNode(EvictionInterceptor.java:118) # org.jboss.cache.interceptors.EvictionInterceptor:invoke(EvictionInterceptor.java:93) Nicolas