[jboss-user] [Remoting] - Re: EJB3/Socket invoker - connection timeouts

[gurgstar]

Not sure how we got into this state, but its clearly a deadlock...

(1) SocketServerInvoker is stuck on clientpool.wait()
(2) maxPoolSize ServerThreads are all stuck on this.wait()
(3) At least one ServerThread must call clientpool.notify() to wake up the SocketServerInvoker
(4) SocketServerInvoker must be woken up in order to call notify on at least one ServerThread.

The deadlock is clear enough, but not sure how we got there other than what appears to be a very small chance of a lost wakeup...

ServerSocketInvoker:

  |         while (thread == null) {
  |             synchronized (threadpool) {
  |                 if (threadpool.size() > 0) {
  |                     thread = (ServerThread) threadpool.removeFirst();
  |                 }
  |             }
  |             if (thread == null) {
  |                 synchronized (clientpool) {
  |                     if (clientpool.size() < maxPoolSize) {
  |                         thread = new ServerThread(socket, this, clientpool, threadpool, getTimeout(), serverSocketClass);
  |                         newThread = true;
  |                     }
  |                     if (thread == null) {
  |                         clientpool.evict();
  |                         if (trace) {
  |                             log.trace("Waiting for a thread...");
  |                         }
  |                         clientpool.wait();
  |                         if (trace) {
  |                             log.trace("Notified of available thread");
  |                         }
  |                     }
  |                 }
  |             }
  |         }
  | 
ServerThread:

  |                synchronized (this) {
  |                    synchronized (clientpool) {
  |                        synchronized (threadpool) {
  |                            if (shutdown) {
  |                                invoker = null;
  |                                return; // exit thread
  |                            } else {
  |                                clientpool.remove(this);
  |                                threadpool.add(this);
  |                                Thread.interrupted(); // clear any interruption so that we can be pooled.
  |                                clientpool.notify();  //273
  |                            }
  |                        }
  |                    }
  |                    try {
  |                        log.debug("begin thread wait");
  |                        this.wait();                      // 280
  |                        log.debug("WAKEUP in SERVER THREAD");
  |                    } catch (InterruptedException e) {
  |                        if (shutdown) {
  |                            invoker = null;
  |                            return; // exit thread
  |                        }
  | 
  |                        throw e;
  |                    }
  |                }
  | 

There is the potential for lost wakeups in both directions.  The ServerSocketInvoker can miss the ServerThread notifies, and the ServerThreads can miss the ServerSocketInvoker's notifies.

This is how the ServerSocketInvoker could miss the notifies form the ServerThreads (seems very unlikely):

(1) ServerSocketInvoker sees that the threadpool is empty
(2) All the ServerThreads then finish processing and call clientpool.notify()
(3) ServerSocketInvoker calls clientpool.wait() and will wait forever cause it missed the notifies

This is how the ServerThreads could miss the notifies from the ServerSocketInvoker (seems much more likely):

(1) A ServerThread finishes processing, calls clientpool.notify() and is then suspended by the scheduler
(2) ServerSocketInvoker wakes up and calls ServerThread.notify
(3) The ServerThread then calls this.wait() and waits forever because it missed the ServerSocketInvoker's notify

Each time this happens, it would take a ServerThread out of comission.  Eventually you'd run out of ServerThreads.






View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4057930#4057930

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4057930