[JBoss JIRA] Created: (JBMESSAGING-893) Failover valve logic flawed, causing MultiThreadFailoverTest.testMultiThreadFailoverSeveralThreadsTransacted() to loop forever
by Ovidiu Feodorov (JIRA)
Failover valve logic flawed, causing MultiThreadFailoverTest.testMultiThreadFailoverSeveralThreadsTransacted() to loop forever
------------------------------------------------------------------------------------------------------------------------------
Key: JBMESSAGING-893
URL: http://jira.jboss.com/jira/browse/JBMESSAGING-893
Project: JBoss Messaging
Issue Type: Sub-task
Reporter: Ovidiu Feodorov
Assigned To: Clebert Suconic
Priority: Blocker
Fix For: 1.2.0.GA
This is an analysis of the MultiThreadFailoverTest.testMultiThreadFailoverSeveralThreadsTransacted() failure
5 producers and 10 consumers concurrently send/receive messages transactionally, using a connection to server 1.
At 20:18:23,484, the test kills server 1:
20:18:23,484 INFO @main [MultiThreadFailoverTest] Killing server 1
At 20:18:24,156, the first failure is detected, LocalThreadConsumer-6 that was sending a transaction to the server detects remoting failure:
20:18:24,156 DEBUG @LocalThreadConsumer-6 [MicroSocketClientInvoker] SocketClientInvoker[faa03e, bisocket://localhost:2808] got SocketException java.net.SocketException: Connection reset
LocalThreadConsumer-6 triggers FailoverValveInterceptor and notifies FailoverCommandCenter of failure, that uses it to close the valve:
20:18:24,234 DEBUG @LocalThreadConsumer-6 [FailoverValveInterceptor] FailoverValve.ConnectionDelegate[5, SID=1] detected network failure, putting sendTransaction() on hold until failover completes
20:18:24,234 DEBUG @LocalThreadConsumer-6 [FailoverCommandCenter] failure detected by FailoverValve.ConnectionDelegate[5, SID=1]
20:18:24,234 TRACE @LocalThreadConsumer-6 [FailoverCommandCenter] FailoverCommandCenter[ConnectionState[5]] broadcasting FAILURE_DETECTED
20:18:24,234 DEBUG @LocalThreadConsumer-6 [FailoverCommandCenter] FailoverCommandCenter[ConnectionState[5]] initiating client-side failover
20:18:24,234 DEBUG @LocalThreadConsumer-6 [FailoverValve] FailoverValve[connectionID=5] closing ...
However, the valve cannot be closed at this moment, since it's "busy", the write lock is taken by other threads, so LocalThreadConsumer-6 starts looping trying to close it.
Concurrently, various other threads detect failure, in this order: LocalThreadConsumer-2 (20:18:25,234), @Thread-117 (connection validator thread, 20:18:25,328), LocalThreadConsumer-0 (20:18:26,234), LocalThreadProducer-1 (20:18:27,250), LocalThreadProducer-3 (20:18:28,250), LocalThreadProducer-2 (20:18:29,250), LocalThreadConsumer-9 (20:18:30,359), LocalThreadProducer-4 (20:18:31,359 ), LocalThreadConsumer-1 (20:18:32,375).
They cannot close the valve, so they start looping.
At 20:18:33,281, LocalThreadProducer-0 detects failure, and successfully closes the valve, and the client side fail-over begins:
20:18:33,281 DEBUG @LocalThreadProducer-0 [MicroSocketClientInvoker] SocketClientInvoker[faa03e, bisocket://localhost:2808] got Exception java.net.ConnectException: Connection refused: connect, creation attempt took 906 ms
20:18:33,281 DEBUG @LocalThreadProducer-0 [FailoverValveInterceptor] FailoverValve.ConnectionDelegate[5, SID=1] detected network failure, putting sendTransaction() on hold until failover completes
20:18:33,281 DEBUG @LocalThreadProducer-0 [FailoverCommandCenter] failure detected by FailoverValve.ConnectionDelegate[5, SID=1]
20:18:33,281 TRACE @LocalThreadProducer-0 [FailoverCommandCenter] FailoverCommandCenter[ConnectionState[5]] broadcasting FAILURE_DETECTED
20:18:33,281 DEBUG @LocalThreadProducer-0 [FailoverCommandCenter] FailoverCommandCenter[ConnectionState[5]] initiating client-side failover
20:18:33,281 DEBUG @LocalThreadProducer-0 [FailoverValve] FailoverValve[connectionID=5] closing ...
20:18:33,281 DEBUG @LocalThreadProducer-0 [FailoverValve] FailoverValve[connectionID=5] closed
...
20:18:34,281 TRACE @LocalThreadProducer-0 [FailoverCommandCenter] FailoverCommandCenter[ConnectionState[5]] broadcasting FAILOVER_STARTED
20:18:34,281 TRACE @LocalThreadProducer-0 [ClientLogInterceptor] invoking ClusteredConnectionFactoryDelegate[SIDs={0,2}].createConnectionDelegate(null, *****, 1)
...
However, the threads that were trying to close the valve are still spinning, and they will, forever.
For example:
20:18:40,250 WARN @LocalThreadConsumer-2 [FailoverValve] FailoverValve[connectionID=6] could not close, trying again ...
java.lang.Exception
at org.jboss.jms.client.FailoverValve.close(FailoverValve.java:160)
at org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:80)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 3 months
[JBoss JIRA] Created: (JBREM-707) Fix handling of OPTIONS invocations in CoyoteInvoker
by Ron Sigal (JIRA)
Fix handling of OPTIONS invocations in CoyoteInvoker
----------------------------------------------------
Key: JBREM-707
URL: http://jira.jboss.com/jira/browse/JBREM-707
Project: JBoss Remoting
Issue Type: Task
Security Level: Public (Everyone can see)
Affects Versions: 2.2.0.Beta1 (Bluto)
Reporter: Ron Sigal
Assigned To: Ron Sigal
Fix For: 2.2.0.Beta1 (Bluto)
Currently, OPTIONS invocations have an empty body, although the www.w3.org HTTP/1.1 protocol document says that in the future OPTIONS messages might have a body. CoyoteInvoker.versionedRead() should handle an OPTIONS invocation with empty body the same way it handles a GET or HEAD invocation, that is, create a new InvocationRequest.
Failure to handle OPTIONS invocations properly cause a failure in org.jboss.test.remoting.transport.http.method.HTTPInvokerTestClient.testOptionsInvocation() when it is run with JBoss serialization.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 3 months
[JBoss JIRA] Resolved: (JBAS-2617) ej3.deployer contains too many jars
by Carlo de Wolf (JIRA)
[ http://jira.jboss.com/jira/browse/JBAS-2617?page=all ]
Carlo de Wolf resolved JBAS-2617.
---------------------------------
Resolution: Done
> ej3.deployer contains too many jars
> -----------------------------------
>
> Key: JBAS-2617
> URL: http://jira.jboss.com/jira/browse/JBAS-2617
> Project: JBoss Application Server
> Issue Type: Task
> Security Level: Public(Everyone can see)
> Components: EJB3
> Reporter: Scott M Stark
> Assigned To: Carlo de Wolf
> Fix For: JBossAS-5.0.0.Beta2, JBossAS-4.2.0.CR1
>
>
> There are numerous duplicate and seemingly unused jars in the current ejb3.deployer:
> [starksm@banshee9100 jboss-4.0]$ ls build/output/jboss-4.0.4beta/server/default
> /deploy/ejb3.deployer/
> META-INF/ commons-lang-1.0.jar*
> antlr-2.7.5H3.jar* commons-logging.jar*
> asm-attrs.jar* commons-pool.jar*
> asm.jar* dom4j.jar*
> cglib-2.1.1.jar* ejb3-persistence.jar*
> commons-beanutils.jar* hibernate-annotations.jar*
> commons-codec-1.2.jar* hibernate-entitymanager.jar*
> commons-collections.jar* hibernate3.jar*
> commons-digester-1.6.jar* jaxen-1.1-beta-4.jar*
> commons-discovery.jar* jboss-annotations-ejb3.jar*
> commons-fileupload.jar* jboss-ejb3.jar*
> commons-httpclient.jar* jboss-ejb3x.jar*
> Why all these are needed and which ones should be factored out into the server lib directory need to be defined. There needs to be a distinction between the ejb3.deployer bundled in the app server vs a standalone bundle for installing into earlier releases.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 3 months
[JBoss JIRA] Updated: (JBAS-2617) ej3.deployer contains too many jars
by Carlo de Wolf (JIRA)
[ http://jira.jboss.com/jira/browse/JBAS-2617?page=all ]
Carlo de Wolf updated JBAS-2617:
--------------------------------
Fix Version/s: JBossAS-5.0.0.Beta2
> ej3.deployer contains too many jars
> -----------------------------------
>
> Key: JBAS-2617
> URL: http://jira.jboss.com/jira/browse/JBAS-2617
> Project: JBoss Application Server
> Issue Type: Task
> Security Level: Public(Everyone can see)
> Components: EJB3
> Reporter: Scott M Stark
> Assigned To: Carlo de Wolf
> Fix For: JBossAS-4.2.0.CR1, JBossAS-5.0.0.Beta2
>
>
> There are numerous duplicate and seemingly unused jars in the current ejb3.deployer:
> [starksm@banshee9100 jboss-4.0]$ ls build/output/jboss-4.0.4beta/server/default
> /deploy/ejb3.deployer/
> META-INF/ commons-lang-1.0.jar*
> antlr-2.7.5H3.jar* commons-logging.jar*
> asm-attrs.jar* commons-pool.jar*
> asm.jar* dom4j.jar*
> cglib-2.1.1.jar* ejb3-persistence.jar*
> commons-beanutils.jar* hibernate-annotations.jar*
> commons-codec-1.2.jar* hibernate-entitymanager.jar*
> commons-collections.jar* hibernate3.jar*
> commons-digester-1.6.jar* jaxen-1.1-beta-4.jar*
> commons-discovery.jar* jboss-annotations-ejb3.jar*
> commons-fileupload.jar* jboss-ejb3.jar*
> commons-httpclient.jar* jboss-ejb3x.jar*
> Why all these are needed and which ones should be factored out into the server lib directory need to be defined. There needs to be a distinction between the ejb3.deployer bundled in the app server vs a standalone bundle for installing into earlier releases.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
19 years, 3 months