[jboss-user] [JBoss Cache] - RejectedExecutionException on distributed cache environment

[Girish Adat]

Girish Adat [http://community.jboss.org/people/girishadat] replied to the discussion

"RejectedExecutionException on distributed cache environment"

To view the discussion, visit: http://community.jboss.org/message/544018#544018

--------------------------------------------------------------
I am also getting similar exceptions as follows.

1) Following exception occured during the run without calling any API of cache in current node. This happened 21 times.

ERROR [org.jboss.cache.cluster.ReplicationQueue] failed replicating 3 elements in replication queue
java.util.concurrent.RejectedExecutionException
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1477)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:384)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:856)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:45)
    at org.jboss.cache.marshall.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:210)
    at org.jboss.cache.RPCManagerImpl.callRemoteMethods(RPCManagerImpl.java:744)
    at org.jboss.cache.RPCManagerImpl.callRemoteMethods(RPCManagerImpl.java:712)
    at org.jboss.cache.RPCManagerImpl.callRemoteMethods(RPCManagerImpl.java:717)
    at org.jboss.cache.cluster.ReplicationQueue.flush(ReplicationQueue.java:170)
    at org.jboss.cache.cluster.ReplicationQueue$2.run(ReplicationQueue.java:114)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)

2) Following happened when I tried to call the removeNode operation of core cache. This happened for ~10K times on a run with approximately 6 million removeNode() operations. But I think each time the executor skipped the entire replication queue containing large number of elements. We are using removeNode() core cache API to prevent the internalFqns to stay in cache, after the pojocache.detach() operation.

ERROR [org.jboss.cache.cluster.ReplicationQueue] failed replicating 500 elements in replication queue

java.util.concurrent.RejectedExecutionException
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1477)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:384)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:856)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:45)
    at org.jboss.cache.marshall.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:210)
    at org.jboss.cache.RPCManagerImpl.callRemoteMethods(RPCManagerImpl.java:744)
    at org.jboss.cache.RPCManagerImpl.callRemoteMethods(RPCManagerImpl.java:712)
    at org.jboss.cache.RPCManagerImpl.callRemoteMethods(RPCManagerImpl.java:717)
    at org.jboss.cache.cluster.ReplicationQueue.flush(ReplicationQueue.java:170)
    at org.jboss.cache.cluster.ReplicationQueue.add(ReplicationQueue.java:145)
    at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:145)
    at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:135)
    at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:107)
    at org.jboss.cache.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:160)
    at org.jboss.cache.interceptors.ReplicationInterceptor.visitRemoveNodeCommand(ReplicationInterceptor.java:125)
    at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
    at org.jboss.cache.commands.AbstractVisitor.visitRemoveNodeCommand(AbstractVisitor.java:75)
    at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:301)
    at org.jboss.cache.interceptors.TxInterceptor.handleDefault(TxInterceptor.java:283)
    at org.jboss.cache.commands.AbstractVisitor.visitRemoveNodeCommand(AbstractVisitor.java:75)
    at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
    at org.jboss.cache.commands.AbstractVisitor.visitRemoveNodeCommand(AbstractVisitor.java:75)
    at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
    at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
    at org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178)
    at org.jboss.cache.interceptors.InvocationContextInterceptor.visitRemoveNodeCommand(InvocationContextInterceptor.java:88)
    at org.jboss.cache.commands.write.RemoveNodeCommand.acceptVisitor(RemoveNodeCommand.java:125)
    at org.jboss.cache.interceptors.InterceptorChain.invoke(InterceptorChain.java:287)
    at org.jboss.cache.invocation.CacheInvocationDelegate.removeNode(CacheInvocationDelegate.java:478)
    at org.jboss.cache.invocation.CacheInvocationDelegate.removeNode(CacheInvocationDelegate.java:485)
    at [removed custom code from stack trace]

I am using CoreCache 3.2.4 GA, with JGroups 2.7.0 GA. POJO Cache version is 3.0.3 GA. I am also using 4.6.0 GA of JBossTS as transaction manager for distributed transactions.

--------------------------------------------------------------

Reply to this message by going to Community
[http://community.jboss.org/message/544018#544018]

Start a new discussion in JBoss Cache at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2052]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/jboss-user/attachments/20100520/525cb89f/attachment-0001.html