[Design of Messaging on JBoss (Messaging/JBoss)] - Re: Another DeadLock, something around InVM
by clebert.suconic@jboss.com
I was trying to replicate this on a test.
I came up with a test, but it hangs with a different thread dump.
it would probably be a good test.
It starts a bunch of threads, all of them missing pings, and all of them closing the session.
I'm playing with a random wait between the createSession and close:
Thread.sleep(PING_INTERVAL * (RandomUtil.randomPositiveInt() % 3));
So, I could eventually have session.close and failure detection being called at the same time.
WDYT ? :
| /*
| * Test the client triggering failure due to no pong received in time
| */
| public void testMultiThreadOpenAndCloses() throws Exception
| {
| final TransportConfiguration transportConfig = new TransportConfiguration("org.jboss.messaging.integration.transports.netty.NettyConnectorFactory");
|
| Interceptor noPongInterceptor = new Interceptor()
| {
| public boolean intercept(final Packet packet, final RemotingConnection conn) throws MessagingException
| {
| log.info("In interceptor, packet is " + packet.getType());
| if (packet.getType() == PacketImpl.PING)
| {
| log.info("Ignoring Ping packet.. it will be dropped");
| return false;
| }
| else
| {
| return true;
| }
| }
| };
|
| messagingService.getServer().getRemotingService().addInterceptor(noPongInterceptor);
| final int numberOfSessions = 30;
| final int numberOfThreads = 8;
| final CountDownLatch flagStart = new CountDownLatch(1);
| final CountDownLatch flagAligned = new CountDownLatch(numberOfThreads);
|
| final ClientSessionFactory csf = new ClientSessionFactoryImpl(transportConfig,
| null,
| DEFAULT_CONNECTION_LOAD_BALANCING_POLICY_CLASS_NAME,
| PING_INTERVAL,
| (long)(PING_INTERVAL * 1.5),
| DEFAULT_CALL_TIMEOUT,
| DEFAULT_CONSUMER_WINDOW_SIZE,
| DEFAULT_CONSUMER_MAX_RATE,
| DEFAULT_SEND_WINDOW_SIZE,
| DEFAULT_PRODUCER_MAX_RATE,
| DEFAULT_MIN_LARGE_MESSAGE_SIZE,
| DEFAULT_BLOCK_ON_ACKNOWLEDGE,
| DEFAULT_BLOCK_ON_NON_PERSISTENT_SEND,
| DEFAULT_BLOCK_ON_PERSISTENT_SEND,
| DEFAULT_AUTO_GROUP,
| DEFAULT_MAX_CONNECTIONS,
| DEFAULT_PRE_ACKNOWLEDGE,
| DEFAULT_ACK_BATCH_SIZE,
| DEFAULT_RETRY_INTERVAL,
| DEFAULT_RETRY_INTERVAL_MULTIPLIER,
| DEFAULT_MAX_RETRIES_BEFORE_FAILOVER,
| DEFAULT_MAX_RETRIES_AFTER_FAILOVER);
|
| class LocalThread extends Thread
| {
| Throwable failure;
|
| @Override
| public void run()
| {
| try
| {
| // Start all at once to make concurrency worst
| flagAligned.countDown();
| flagStart.await();
| for (int i = 0; i < numberOfSessions; i++)
| {
|
| ClientSession session = csf.createSession(false, false, false);
|
| Thread.sleep(PING_INTERVAL * (RandomUtil.randomPositiveInt() % 3));
|
| session.close();
| }
| }
| catch (Throwable e)
| {
| failure = e;
| }
| }
| };
|
| LocalThread threads[] = new LocalThread[numberOfThreads];
|
| for (int i = 0; i < numberOfThreads; i++)
| {
| threads = new LocalThread();
| threads.start();
| }
|
| flagAligned.await();
| flagStart.countDown();
|
| Throwable e = null;
| for (LocalThread t : threads)
| {
| t.join();
| if (t.failure != null)
| {
| e = t.failure;
| }
| }
|
| if (e != null)
| {
| throw new Exception("Test Failed", e);
| }
|
| }
|
|
View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4215008#4215008
Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4215008
17 years, 1 month
[Design of Messaging on JBoss (Messaging/JBoss)] - Another DeadLock, something around InVM
by clebert.suconic@jboss.com
*maybe* related to JORAM hangs:
| [junit] Found one Java-level deadlock:
| [junit] =============================
| [junit] "Thread-11705":
| [junit] waiting to lock monitor 0xa8838124 (object 0xb276fcd8, a java.lang.Object),
| [junit] which is held by "Thread-11703"
| [junit] "Thread-11703":
| [junit] waiting to lock monitor 0xa88381e4 (object 0xb279d280, a org.jboss.messaging.core.remoting.impl.invm.InVMConnection),
| [junit] which is held by "Thread-11705"
| [junit]
| [junit] Java stack information for the threads listed above:
| [junit] ===================================================
| [junit] "Thread-11705":
| [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:867)
| [junit] - waiting to lock <0xb276fcd8> (a java.lang.Object)
| [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
| [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnector$Listener.connectionDestroyed(InVMConnector.java:187)
| [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:96)
| [junit] - locked <0xb279d280> (a org.jboss.messaging.core.remoting.impl.invm.InVMConnection)
| [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnector.disconnect(InVMConnector.java:161)
| [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMAcceptor$Listener$1.run(InVMAcceptor.java:167)
| [junit] "Thread-11703":
| [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:86)
| [junit] - waiting to lock <0xb279d280> (a org.jboss.messaging.core.remoting.impl.invm.InVMConnection)
| [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.internalClose(RemotingConnectionImpl.java:557)
| [junit] at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:423)
| [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.failConnection(ConnectionManagerImpl.java:871)
| [junit] - locked <0xb276fcd8> (a java.lang.Object)
| [junit] at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.connectionDestroyed(ConnectionManagerImpl.java:197)
| [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnector$Listener.connectionDestroyed(InVMConnector.java:187)
| [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:96)
| [junit] - locked <0xb2775b98> (a org.jboss.messaging.core.remoting.impl.invm.InVMConnection)
| [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMConnector.disconnect(InVMConnector.java:161)
| [junit] at org.jboss.messaging.core.remoting.impl.invm.InVMAcceptor$Listener$1.run(InVMAcceptor.java:167)
|
View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4214990#4214990
Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4214990
17 years, 1 month
[Design the new POJO MicroContainer] - Re: InterruptedException not being cleared?
by david.lloyd@jboss.com
This trivial patch gets clears the interrupt status, tolerates interrupts, and then restores the interrupt status in a finally block. Is there a reason to not do it this way? Otherwise I'll commit it, if you give the OK. Of course the forums will mangle this, but...
| Index: classloader/src/main/java/org/jboss/classloader/spi/base/ClassLoaderManager.java
| ===================================================================
| --- classloader/src/main/java/org/jboss/classloader/spi/base/ClassLoaderManager.java (revision 85240)
| +++ classloader/src/main/java/org/jboss/classloader/spi/base/ClassLoaderManager.java (working copy)
| @@ -185,133 +185,141 @@
| * Process the next task
| *
| * @param thread the thread
| * @param task the task
| * @throws InterruptedException if it is interrupted
| */
| private static void nextTask(Thread thread, ClassLoadingTask task) throws InterruptedException
| {
| - boolean trace = log.isTraceEnabled();
| - if (trace)
| - log.trace("Next task thread=" + thread + " task=" + task);
| -
| - List<ThreadTask> taskList = loadTasksByThread.get(thread);
| - synchronized (taskList)
| + boolean intr = Thread.interrupted();
| + try
| {
| - // There may not be any ThreadTasks
| - while (taskList.isEmpty() && task.getThreadTaskCount() != 0 )
| + boolean trace = log.isTraceEnabled();
| + if (trace)
| + log.trace("Next task thread=" + thread + " task=" + task);
| +
| + List<ThreadTask> taskList = loadTasksByThread.get(thread);
| + synchronized (taskList)
| {
| - /* There are no more tasks for the calling thread to execute, so the
| - calling thread must wait until the task.threadTaskCount reaches 0
| - */
| - if (trace)
| - log.trace("Begin nextTask(WAIT_ON_EVENT), task="+task);
| - try
| + // There may not be any ThreadTasks
| + while (taskList.isEmpty() && task.getThreadTaskCount() != 0 )
| {
| - task.waitOnEvent();
| - taskList.wait();
| + /* There are no more tasks for the calling thread to execute, so the
| + calling thread must wait until the task.threadTaskCount reaches 0
| + */
| + if (trace)
| + log.trace("Begin nextTask(WAIT_ON_EVENT), task="+task);
| + try
| + {
| + task.waitOnEvent();
| + taskList.wait();
| + if (trace)
| + log.trace("nextTask(WAIT_ON_EVENT), notified, task="+task);
| + }
| + catch(InterruptedException e)
| + {
| + if( trace )
| + log.trace("nextTask(WAIT_ON_EVENT), interrupted, task="+task, e);
| + intr = true;
| + }
| }
| - catch(InterruptedException e)
| +
| + if (trace)
| + log.trace("Continue nextTask(" + taskList.size()+"), task="+task);
| +
| + // See if the task is complete
| + if (task.getThreadTaskCount() == 0)
| {
| - if( trace )
| - log.trace("nextTask(WAIT_ON_EVENT), interrupted, task="+task, e);
| - // Abort this task t
| - throw e;
| + task.finish();
| + log.trace("End nextTask(FINISHED), task="+task);
| + return;
| }
| - if (trace)
| - log.trace("nextTask(WAIT_ON_EVENT), notified, task="+task);
| }
|
| + ThreadTask threadTask = taskList.remove(0);
| + ClassLoadingTask loadTask = threadTask.getLoadTask();
| if (trace)
| - log.trace("Continue nextTask(" + taskList.size()+"), task="+task);
| + log.trace("Begin nextTask(" + taskList.size() + "), loadTask=" + loadTask);
|
| - // See if the task is complete
| - if (task.getThreadTaskCount() == 0)
| + try
| {
| - task.finish();
| - log.trace("End nextTask(FINISHED), task="+task);
| - return;
| - }
| - }
| -
| - ThreadTask threadTask = taskList.remove(0);
| - ClassLoadingTask loadTask = threadTask.getLoadTask();
| - if (trace)
| - log.trace("Begin nextTask(" + taskList.size() + "), loadTask=" + loadTask);
| -
| - try
| - {
| - Thread taskThread = threadTask.getThread();
| - if (taskThread == null)
| - {
| - /* This is a task that has been reassigned back to the original
| - requesting thread ClassLoadingTask, so a new ThreadTask must
| - be scheduled.
| - */
| - if (trace)
| - log.trace("Rescheduling threadTask=" + threadTask);
| - scheduleTask(loadTask, threadTask.getLoader(), true);
| + Thread taskThread = threadTask.getThread();
| + if (taskThread == null)
| + {
| + /* This is a task that has been reassigned back to the original
| + requesting thread ClassLoadingTask, so a new ThreadTask must
| + be scheduled.
| + */
| + if (trace)
| + log.trace("Rescheduling threadTask=" + threadTask);
| + scheduleTask(loadTask, threadTask.getLoader(), true);
| + }
| + else
| + {
| + if (trace)
| + log.trace("Running threadTask=" + threadTask);
| + threadTask.run();
| + }
| }
| - else
| + catch (Throwable e)
| {
| if (trace)
| - log.trace("Running threadTask=" + threadTask);
| - threadTask.run();
| - }
| - }
| - catch (Throwable e)
| - {
| - if (trace)
| - log.trace("Run failed with exception", e);
| - boolean retry = e instanceof ClassCircularityError || e.getClass().equals(LinkageError.class);
| - if (retry && loadTask.incrementNumCCE() < MAX_CCE)
| - {
| - /* Reschedule this task after all existing tasks to allow the
| - current load tasks which are conflicting to complete.
| - */
| - try
| + log.trace("Run failed with exception", e);
| + boolean retry = e instanceof ClassCircularityError || e.getClass().equals(LinkageError.class);
| + if (retry && loadTask.incrementNumCCE() < MAX_CCE)
| {
| - // Reschedule and update the loadTask.threadTaskCount
| - scheduleTask(loadTask, threadTask.getLoader(), true);
| + /* Reschedule this task after all existing tasks to allow the
| + current load tasks which are conflicting to complete.
| + */
| + try
| + {
| + // Reschedule and update the loadTask.threadTaskCount
| + scheduleTask(loadTask, threadTask.getLoader(), true);
| + }
| + catch (Throwable ex)
| + {
| + loadTask.setLoadError(ex);
| + log.warn("Failed to reschedule task after CCE", ex);
| + }
| + if (trace)
| + log.trace("Post CCE state, loadTask=" + loadTask);
| }
| - catch (Throwable ex)
| + else
| {
| - loadTask.setLoadError(ex);
| - log.warn("Failed to reschedule task after CCE", ex);
| + loadTask.setLoadError(e);
| }
| - if (trace)
| - log.trace("Post CCE state, loadTask=" + loadTask);
| }
| - else
| + finally
| {
| - loadTask.setLoadError(e);
| + // Release any lock on the classloader
| + if (threadTask.isReleaseInNextTask())
| + threadTask.getClassLoader().unlock(false);
| }
| - }
| - finally
| - {
| - // Release any lock on the classloader
| - if (threadTask.isReleaseInNextTask())
| - threadTask.getClassLoader().unlock(false);
| - }
|
| - // If the ThreadTasks are complete mark the ClassLoadingTask finished
| - if (loadTask.getThreadTaskCount() == 0)
| - {
| - List<ThreadTask> loadTaskThreadTasks = loadTasksByThread.get(loadTask.getRequestingThread());
| - synchronized (loadTaskThreadTasks)
| + // If the ThreadTasks are complete mark the ClassLoadingTask finished
| + if (loadTask.getThreadTaskCount() == 0)
| {
| - if( trace )
| - log.trace("Notifying task of thread completion, loadTask:"+loadTask);
| - task.finish();
| - loadTaskThreadTasks.notify();
| + List<ThreadTask> loadTaskThreadTasks = loadTasksByThread.get(loadTask.getRequestingThread());
| + synchronized (loadTaskThreadTasks)
| + {
| + if( trace )
| + log.trace("Notifying task of thread completion, loadTask:"+loadTask);
| + task.finish();
| + loadTaskThreadTasks.notify();
| + }
| }
| + if (trace)
| + log.trace("End nextTask(" + taskList.size()+ "), loadTask=" + loadTask);
| + }
| + finally
| + {
| + if (intr)
| + Thread.currentThread().interrupt();
| }
| - if (trace)
| - log.trace("End nextTask(" + taskList.size()+ "), loadTask=" + loadTask);
| }
|
| /**
| * Invoked to create a ThreadTask to assign a thread to the task of
| * loading the class of ClassLoadingTask.
| *
| * @param task the classloading task
| * @param loader the loader
|
View the original post : http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4214969#4214969
Reply to the post : http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4214969
17 years, 1 month