[jboss-user] [JBoss Cache: Core Edition] - lots of TimeoutException

[g9yuayon]

Hi, 

I've been getting a lot of TimeoutExceptions when the method Cache#get() is called. My cluster is set up as below:
0. The cluster runs on JBoss Cache 3.1.0 GA. The locking scheme is MVCC. The isolation level is READ_COMMITTED. 
1. Two nodes, N1, and N2. Each node is a different machine. Each node has a cache instance.  
2. The node N1 only reads from cache. If it doesn't find an entry from its local cache, it will read from N2 via a ClusteredCacheLoader with synchronous replication.  
3. The node N2 both reads from and writes to its cache.

If I understand it correctly, a read operation doesn't not aquire a lock, let alone a write-lock. So, why would I get many TimeoutExceptions when Cache#get() was called, as the stack trace shows? Could anyone please let me what possibly could cause this problem, and how I could solve it? 

org.jboss.cache.lock.TimeoutException: Unable to acquire lock on Fqn [/parentNode/key] after [20000] milliseconds for requestor [Thread[pool-1-thread-37,5,main]]! Lock held by [null]
    at org.jboss.cache.mvcc.MVCCNodeHelper.acquireLock(MVCCNodeHelper.java:157)
    at org.jboss.cache.mvcc.MVCCNodeHelper.wrapNodeForWriting(MVCCNodeHelper.java:252)
    at org.jboss.cache.mvcc.MVCCNodeHelper.wrapNodeForWriting(MVCCNodeHelper.java:184)
    at org.jboss.cache.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:300)
    at org.jboss.cache.interceptors.CacheLoaderInterceptor.visitGetKeyValueCommand(CacheLoaderInterceptor.java:165)
    at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
    at org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:110)
    at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
    at org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:110)
    at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:301)
    at org.jboss.cache.interceptors.TxInterceptor.handleDefault(TxInterceptor.java:283)
    at org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:110)
    at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:77)
    at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178)
    at org.jboss.cache.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:130)
    at org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:110)
    at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97)
    at org.jboss.cache.interceptors.InterceptorChain.invoke(InterceptorChain.java:287)
    at org.jboss.cache.invocation.CacheInvocationDelegate.get(CacheInvocationDelegate.java:444)


I also noticed quite many RejectedExceptions when there were many TimeoutExceptions. Are they somehow correlated? 


java.util.concurrent.RejectedExecutionException
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1759)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92)
    at org.jboss.cache.marshall.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:206)
    at org.jboss.cache.RPCManagerImpl.callRemoteMethods(RPCManagerImpl.java:748)
    at org.jboss.cache.loader.ClusteredCacheLoader.callRemote(ClusteredCacheLoader.java:135)
    at org.jboss.cache.loader.ClusteredCacheLoader.get0(ClusteredCacheLoader.java:195)
    at org.jboss.cache.loader.ClusteredCacheLoader.get(ClusteredCacheLoader.java:182)
    at org.jboss.cache.loader.ChainingCacheLoader.get(ChainingCacheLoader.java:111)
    at org.jboss.cache.interceptors.CacheLoaderInterceptor.loadData(CacheLoaderInterceptor.java:496)
    at org.jboss.cache.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:315)
    at org.jboss.cache.interceptors.CacheLoaderInterceptor.visitGetKeyValueCommand(CacheLoaderInterceptor.java:165)
    at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
    at org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:110)
    at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
    at org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:110)
    at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:301)
    at org.jboss.cache.interceptors.TxInterceptor.handleDefault(TxInterceptor.java:283)
    at org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:110)
    at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:77)
    at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178)
    at org.jboss.cache.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:130)
    at org.jboss.cache.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:110)
    at org.jboss.cache.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:97)
    at org.jboss.cache.interceptors.InterceptorChain.invoke(InterceptorChain.java:287)
    at org.jboss.cache.invocation.CacheInvocationDelegate.get(CacheInvocationDelegate.java:444)
 



View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4236820#4236820

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4236820