[jboss-user] [JBoss Cache: Core Edition] - Eviction queue fills up causing threads to block

[Jeremy Stone]

We have a problem whereby we have seen the following message logged continuously on a customer's installation of our software that uses JBoss Cache with Hibernate:

Warning putNodeEvent(): eviction node event queue size is at 98% threshold value of capacity: 200000 Region: /query/entity/by_name You will need to reduce the wakeUpIntervalSeconds parameter.

We have seen this only once so far. It does not appear to recur after a server restart.

A stack dump shows many threads waiting to put nodes on to the eviction queue. The following is representative...
            
Thread: pool-21-thread-4 : priority:5, demon:false, threadId:144, threadState:WAITING, lockName:java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject at 584b2d
  | 
  | sun.misc.Unsafe.park(Native Method)
  | java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  | java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
  | java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
  | org.jboss.cache.RegionImpl.registerEvictionEvent(RegionImpl.java:249)
  | org.jboss.cache.RegionImpl.registerEvictionEvent(RegionImpl.java:234)
  | org.jboss.cache.interceptors.EvictionInterceptor.registerEvictionEventToRegionManager(EvictionInterceptor.java:252)
  | ...etc
  | org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:118)
  | org.hibernate.impl.SessionImpl.list(SessionImpl.java:1596)
  | org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:306)
  | org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:328)
  | ...etc
  |                 
The EvictionTimer thread appears just to waiting for something to do...

Thread: EvictionTimer-0 : priority:5, demon:true, threadId:88, threadState:WAITING, lockName:java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject at 353fbf
  | 
  | sun.misc.Unsafe.park(Native Method)
  | java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  | java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
  | java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
  | java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
  | java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
  | java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  | java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  | java.lang.Thread.run(Thread.java:619)

The only inkling as to what is occurring is that perhaps an eviction algorithm (silently) threw an unchecked exception causing the EvictionTimerTask Task to no longer be scheduled.
(see http://java.sun.com/javase/6/docs/api/java/util/concurrent/ScheduledExecutorService.html#scheduleWithFixedDelay(java.lang.Runnable,%20long,%20long,%20java.util.concurrent.TimeUnit) 
which states '...If any execution of the task encounters an exception, subsequent executions are suppressed....').

Even if this is not the cause of our problem then EvictionTimerTask seems susceptible to this behaviour. Maybe you should consider using something like the Spring Framework DelegatingExceptionProofRunnable
(http://static.springsource.org/spring/docs/2.5.6/api/org/springframework/scheduling/support/DelegatingExceptionProofRunnable.html).

Any thoughts on this?

View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4242437#4242437

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4242437