[jboss-jira] [JBoss JIRA] Reopened: (JGRP-654) Scheduler thread goes in infinite loop after getting interrupted by the priority task

Scheduler thread goes in infinite loop after getting interrupted by the priority task ------------------------------------------------------------------------------------- Key: JGRP-654 URL: http://jira.jboss.com/jira/browse/JGRP-654 Project: JGroups Issue Type: Bug Affects Versions: 2.6.1 Environment: Linux, JGroups v. 2.6.1 Reporter: vivek v Assigned To: Bela Ban Fix For: 2.7, 2.6.2 This seems like a bug in Scheduler class. The scheduler thread goes into infinite loop and never recovers after being interrupted by the priority task (upon deadlock detection). Here is what I think happens (with both deadlock_detection and concurrent_processing enabled for RpcDispatcher), 1) The tasks are added to the Scheduler Queue 2) If the deadlock_detection is turned on and there is a kind of deadlock (2-way method calls) the RequestCorrelator.receiveMessage() calls scheduler.addPrio(req) method 3) The addPrio(Runnable task) method adds the priority task to the head of the queue and calls the interrupt method on the Scheduler thread. 4) In the run() method of Scheduler we peek the queue, take the task from head and run it 5) "sched_thread.isInterrupted()" in the run method returns true and throws InterruptedException 6) Since, interrupted exception is thrown we bypass the "queue.removeElement(current_task)" - and the task is not removed from the queue. 7) The catch block of InterruptedException sets the current_task.suspended=true, but never resets the interrupt flag 8) Now we keep going through the same loop for the same task in the queue in the run() method as we never resets the interrupt flag. I hope this is a correct diagnosis of this "TimeoutException" problem we have been seeing for last 2-3 months. I'm not sure if there are some nested Rpc calls in our system, but with deadlock-detection turned on they should be resolved. As tasks are never moved out of the queue, all the tasks just keep adding up (I see my queue over 2300 elements) and thus, we get TimeoutException for all the incoming calls. I think we need to call "sched_thread.interrupted()" in the InterruptedException catch block so we can remove items from the queue as we process them. I've tried with this change (added Thread.interrupted at the end of InterruptedException catch block) in my local jgroups-all.jar and it seems to work fine for a long time now (3-4 hours, earlier it used to fail within 2 hours). I haven't found any workaround for this problem if the deadlock detection logic is exercised.