[jboss-dev-forums] [Design of Messaging on JBoss (Messaging/JBoss)] - Re: PingStressTests hanging

[clebert.suconic@jboss.com]

Trustin, If I could have some help with this:

I've been running org.jboss.messaging.tests.stress.remote.PingStressTest (change getNumberOfIterations to 100).

One thread will fail with Ping, while other threads are also trying to close the connections.

The Pinger thread will be waiting the Executors to shutdown, but a thread will stay alive forever. (Look at my previous post).

Looking at NioWorker, I can see that selector.keys() is not empty hence the shutdown check is never made.

I'm not sure if there is something that should be closed after the ping failure.

If you want to verify what thread is holding the executor, this is the patch for the hack I've done here locally. (which I don't intend to commit it.. it's just for debug purposes):



  | Index: src/main/org/jboss/messaging/utils/JBMThreadFactory.java
  | ===================================================================
  | --- src/main/org/jboss/messaging/utils/JBMThreadFactory.java	(revision 6014)
  | +++ src/main/org/jboss/messaging/utils/JBMThreadFactory.java	(working copy)
  | @@ -37,7 +37,31 @@
  |  
  |     private final AtomicInteger threadCount = new AtomicInteger(0);
  |  
  | +   private final ConcurrentSet<Thread> threads = new ConcurrentHashSet<Thread>();
  | +   
  | +   private final String groupName;
  | +   
  | +   
  | +   public void debug()
  | +   {
  | +      if (threads.isEmpty())
  | +      {
  | +         System.out.println("Thread factory is empty");
  | +      }
  | +      for (Thread t : threads)
  | +      {
  | +         System.out.println("This thread is still created by the factory : " + t + "  name = " + t.getName());
  | +         StackTraceElement elements[] =  t.getStackTrace();
  | +         for (StackTraceElement trace : elements)
  | +         {
  | +            System.out.println("Trace : " + trace);
  | +         }
  | +      }
  | +   }
  | +   
  | +
  |     private final int threadPriority;
  | +
  |     public JBMThreadFactory(final String groupName)
  |     {
  |        this(groupName, Thread.NORM_PRIORITY);
  | @@ -47,11 +71,27 @@
  |     {
  |        group = new ThreadGroup(groupName + "-" + System.identityHashCode(this));
  |        threadPriority = priority;
  | +      this.groupName = groupName;
  |     }
  |  
  |     public Thread newThread(final Runnable command)
  |     {
  | -      Thread t = new Thread(group, command, "Thread-" + threadCount.getAndIncrement() +
  | +      Runnable myRunnable = new      Runnable()
  | +      {
  | +         public void run()
  | +         {
  | +            try
  | +            {
  | +               command.run();
  | +            }
  | +            finally
  | +            {
  | +               threads.remove(Thread.currentThread());
  | +            }
  | +         }
  | +      };
  | +      
  | +      Thread t = new Thread(group, myRunnable, "Thread-" + threadCount.getAndIncrement() +
  |                                              " (group:" +
  |                                              group.getName() +
  |                                              ")");
  | @@ -59,6 +99,7 @@
  |        // Don't want to prevent VM from exiting
  |        t.setDaemon(true);
  |        t.setPriority(threadPriority);
  | +      threads.add(t);
  |        return t;
  |     }
  |  }
  | Index: src/main/org/jboss/messaging/integration/transports/netty/NettyConnector.java
  | ===================================================================
  | --- src/main/org/jboss/messaging/integration/transports/netty/NettyConnector.java	(revision 6014)
  | +++ src/main/org/jboss/messaging/integration/transports/netty/NettyConnector.java	(working copy)
  | @@ -90,7 +90,10 @@
  |  public class NettyConnector implements Connector
  |  {
  |     // Constants -----------------------------------------------------
  | +   
  | +   static JBMThreadFactory workerFactory = new JBMThreadFactory("jbm-netty-connector-worker-threads");
  |  
  | +
  |     private static final Logger log = Logger.getLogger(NettyConnector.class);
  |  
  |     // Attributes ----------------------------------------------------
  | @@ -236,7 +239,7 @@
  |           return;
  |        }
  |  
  | -      workerExecutor = Executors.newCachedThreadPool(new JBMThreadFactory("jbm-netty-connector-worker-threads"));
  | +      workerExecutor = Executors.newCachedThreadPool(workerFactory);
  |        if (useServlet)
  |        {
  |           bossExecutor = Executors.newCachedThreadPool(new JBMThreadFactory("jbm-netty-connector-boss-threads"));
  | @@ -360,10 +363,14 @@
  |           {
  |              try
  |              {
  | -               if (workerExecutor.awaitTermination(1, TimeUnit.SECONDS))
  | +               if (workerExecutor.awaitTermination(5, TimeUnit.SECONDS))
  |                 {
  |                    break;
  |                 }
  | +               else
  | +               {
  | +                  workerFactory.debug();
  | +               }
  |              }
  |              catch (InterruptedException e)
  |              {
  | 



View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4215504#4215504

Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4215504