9:02 p.m.
I am experiencing what appears to be a deadlock situation in JBoss Cache 1.4.1.SP3. After
some period of time with the app up and running in JBoss AS 4.0.5 I am seeing one thread
holding a couple of monitor locks in BoundedLinkedQueue and a multitude of threads waiting
on the lock. The application grinds to a halt with users experiencing hanging HTTP
responses.
In the latest scenario I have 28 HTTP threads,
26 are blocked (see Stack #1).
1 thread holds 2 locks (see Stack #2).
1 thread is still free (see Stack #3).
A Google search indicates that BoundedLinkedQueue has had issues with race conditions in
the past,
http://altair.cs.oswego.edu/pipermail/concurrency-interest/2004-September...,
although the scenario seems slightly different.
Stack #1
"http-0.0.0.0-8080-12" daemon prio=1 tid=0x08c5bd58 nid=0x26e0 waiting for
monitor entry [0x97cfc000..0x97cff030]
| at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.put(Unknown Source)
| - waiting to lock <0xb4291690> (a java.lang.Object)
| at org.jboss.cache.eviction.Region.putNodeEvent(Region.java:141)
| at
org.jboss.cache.interceptors.EvictionInterceptor.doEventUpdatesOnRegionManager(EvictionInterceptor.java:149)
| at
org.jboss.cache.interceptors.EvictionInterceptor.updateNode(EvictionInterceptor.java:122)
| at
org.jboss.cache.interceptors.EvictionInterceptor.invoke(EvictionInterceptor.java:97)
| at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
| at
org.jboss.cache.interceptors.OptimisticCreateIfNotExistsInterceptor.invoke(OptimisticCreateIfNotExistsInterceptor.java:69)
| at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
| at
org.jboss.cache.interceptors.OptimisticValidatorInterceptor.invoke(OptimisticValidatorInterceptor.java:87)
| at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
| at
org.jboss.cache.interceptors.OptimisticLockingInterceptor.invoke(OptimisticLockingInterceptor.java:126)
| at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
| at
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
| at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
| at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
| at
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
| at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
| at org.jboss.cache.TreeCache.get(TreeCache.java:3627)
| at org.jboss.cache.TreeCache.get(TreeCache.java:3608)
|
Stack #2
"http-0.0.0.0-8080-11" daemon prio=1 tid=0x087dab18 nid=0x2691 in Object.wait()
[0x976f0000..0x976f3030]
| at java.lang.Object.wait(Native Method)
| at java.lang.Object.wait(Object.java:474)
| at EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue.put(Unknown Source)
| - locked <0xb42914e8> (a EDU.oswego.cs.dl.util.concurrent.BoundedLinkedQueue)
| - locked <0xb4291690> (a java.lang.Object)
| at org.jboss.cache.eviction.Region.putNodeEvent(Region.java:141)
| at
org.jboss.cache.interceptors.EvictionInterceptor.doEventUpdatesOnRegionManager(EvictionInterceptor.java:149)
| at
org.jboss.cache.interceptors.EvictionInterceptor.updateNode(EvictionInterceptor.java:122)
| at
org.jboss.cache.interceptors.EvictionInterceptor.invoke(EvictionInterceptor.java:97)
| at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
| at
org.jboss.cache.interceptors.OptimisticCreateIfNotExistsInterceptor.invoke(OptimisticCreateIfNotExistsInterceptor.java:69)
| at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
| at
org.jboss.cache.interceptors.OptimisticValidatorInterceptor.invoke(OptimisticValidatorInterceptor.java:87)
| at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
| at
org.jboss.cache.interceptors.OptimisticLockingInterceptor.invoke(OptimisticLockingInterceptor.java:126)
| at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
| at
org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:365)
| at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:160)
| at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
| at
org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:138)
| at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5863)
| at org.jboss.cache.TreeCache.get(TreeCache.java:3627)
| at org.jboss.cache.TreeCache.get(TreeCache.java:3608)
|
Stack #3
"http-0.0.0.0-8080-27" daemon prio=1 tid=0x09643720 nid=0x2fc0 in Object.wait()
[0x978f7000..0x978f71b0]
| at java.lang.Object.wait(Native Method)
| - waiting on <0xc3c65dc0> (a
org.apache.tomcat.util.net.MasterSlaveWorkerThread)
| at java.lang.Object.wait(Object.java:474)
| at
org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
| - locked <0xc3c65dc0> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
| at
org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
| at java.lang.Thread.run(Thread.java:595)
|
TreeCache Configuration
<server>
| <classpath codebase="./lib" archives="jboss-cache.jar,
jgroups.jar"/>
| <mbean code="org.jboss.cache.TreeCache"
name="jboss.cache:service=TreeCache">
| <depends>jboss:service=Naming</depends>
| <depends>jboss:service=TransactionManager</depends>
| <attribute
name="ClusterName">FCXp_hibernate</attribute>
| <attribute name="CacheMode">LOCAL</attribute>
| <attribute name="SyncReplTimeout">10000</attribute>
| <attribute
name="LockAcquisitionTimeout">15000</attribute>
| <attribute name="FetchInMemoryState">false</attribute>
| <attribute
name="NodeLockingScheme">OPTIMISTIC</attribute>
|
| <attribute name="EvictionPolicyClass">
org.jboss.cache.eviction.LRUPolicy </attribute>
| <attribute name="EvictionPolicyConfig">
| <config>
| <attribute
name="wakeUpIntervalSeconds">10000</attribute>
| <!-- Cache wide default -->
| <region name="/_default_">
| <attribute
name="maxNodes">200000</attribute>
| <attribute
name="timeToIdleSeconds">10000</attribute>
| </region>
| </config>
| </attribute>
| </mbean>
| </server>
View the original post :
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4065227#...
Reply to the post :
http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&a...