[JBoss JIRA] (JBWEB-276) Using non-blocking thread pool causes excessive RejectedExecutionExceptions to be logged
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/JBWEB-276?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on JBWEB-276:
-----------------------------------------------
James Livingston <jlivings(a)redhat.com> changed the Status of [bug 985191|https://bugzilla.redhat.com/show_bug.cgi?id=985191] from NEW to MODIFIED
> Using non-blocking thread pool causes excessive RejectedExecutionExceptions to be logged
> ----------------------------------------------------------------------------------------
>
> Key: JBWEB-276
> URL: https://issues.jboss.org/browse/JBWEB-276
> Project: JBoss Web
> Issue Type: Enhancement
> Security Level: Public(Everyone can see)
> Affects Versions: JBossWeb-7.2.0.Beta1
> Reporter: James Livingston
> Assignee: Remy Maucherat
> Attachments: JBWEB-276-1.patch, JBWEB-276-2.patch
>
>
> When a finite-queue thread pool is used to process web requests, the queue being full can either block or drop the request. If you use a pool which drops requests, a RejectedExecutionException is thrown.
> Currently JIoEndpoint catches that exception the same as other ones such as being unable to create threads. When using a non-blocking pool, the task being rejected is not a critical error like other exceptions so should not be logged at ERROR level.
> Under excessive load, these exception will be continuously generated as requests are dropped, which cases a large amount of logging. Setting it to DEBUG level would stop that, but it would hide the potentially useful messages by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (WFLY-789) HornetQ native library(AIO) are loaded by default - for transaction manager using hornetq object store
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/WFLY-789?page=com.atlassian.jira.plugin.s... ]
RH Bugzilla Integration commented on WFLY-789:
----------------------------------------------
Ivo Studensky <istudens(a)redhat.com> changed the Status of [bug 913117|https://bugzilla.redhat.com/show_bug.cgi?id=913117] from ASSIGNED to POST
> HornetQ native library(AIO) are loaded by default - for transaction manager using hornetq object store
> ------------------------------------------------------------------------------------------------------
>
> Key: WFLY-789
> URL: https://issues.jboss.org/browse/WFLY-789
> Project: WildFly
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 8.0.0.Alpha1
> Reporter: Ivo Studensky
> Assignee: Ivo Studensky
> Fix For: 8.0.0.Beta1
>
>
> The issue cloned for upstream from an EAP6 bugzilla.
> original description:
> {quote}
> It seems that in case that of using hornetq for implementation of object store (configuration of transaction subsystem via <use-hornetq-store/> tag) the native library (AIO) is used automatically without necessity of user confirmation.
> For EAP 6.0.1 was decided that native library can't be used without explicit customer configuration request. Please check discussion on bug #900591.
> The NIO should be used until customer defined otherwise.
> The configuration option is missing in TM and after the adding it the EAP configuration should reflect fact that the default behaviour should be NIO.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months
[JBoss JIRA] (WFLY-1966) Hangs in DomainControllerMigrationTestCase
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFLY-1966?page=com.atlassian.jira.plugin.... ]
Brian Stansberry commented on WFLY-1966:
----------------------------------------
Problem is due to an issue unmarshalling the Subject on the slave HC. See
http://brontes.lab.eng.brq.redhat.com/repository/download/WF_DomainOnlyLi...
04:20:47,552 INFO [org.jboss.as.protocol] (Remoting "failover-h3:MANAGEMENT" task-16) handling org.jboss.as.protocol.mgmt.ManagementRequestHeader@f9ed49 using org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler@498870
04:20:47,557 INFO [org.jboss.as.controller.management-operation] (Remoting "failover-h3:MANAGEMENT" task-16) Requested subject for 1166537054
04:20:47,564 INFO [org.jboss.as.protocol.connection] (Remoting "failover-h3:MANAGEMENT" task-1) received response for 4
04:20:47,565 INFO [org.jboss.as.protocol] (Remoting "failover-h3:MANAGEMENT" task-1) handling org.jboss.as.protocol.mgmt.ManagementResponseHeader@736a7e using org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$GetSubjectResponseHandler@154abc4
04:20:47,654 ERROR [org.jboss.as.protocol] (Remoting "failover-h3:MANAGEMENT" task-1) failed handling org.jboss.as.protocol.mgmt.ManagementResponseHeader@736a7e using org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$GetSubjectResponseHandler@154abc4: java.io.IOException: JBAS013469: Unable to unmarshall Subject received for request.
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$GetSubjectResponseHandler.handleRequest(TransactionalProtocolOperationHandler.java:184) [wildfly-controller-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:258) [wildfly-protocol-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleRequest(AbstractMessageHandler.java:222) [wildfly-protocol-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:116) [wildfly-protocol-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
at org.jboss.as.protocol.mgmt.ManagementChannelReceiver$1.handleMessage(ManagementChannelReceiver.java:56) [wildfly-protocol-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
at org.jboss.as.protocol.mgmt.ManagementChannelReceiver.handleMessage(ManagementChannelReceiver.java:84) [wildfly-protocol-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:451) [jboss-remoting-4.0.0.Beta1.jar:4.0.0.Beta1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
Caused by: java.lang.ClassNotFoundException: org.jboss.as.core.security.RealmUser from [Module "org.jboss.as.host-controller:main" from local module loader @1160a60 (finder: local module finder @3514a (roots: /opt/buildAgent/work/e3fca795beba699e/testsuite/domain/../../build/target/wildfly-8.0.0.Beta1-SNAPSHOT/modules,/opt/buildAgent/work/e3fca795beba699e/testsuite/domain/../../build/target/wildfly-8.0.0.Beta1-SNAPSHOT/modules/system/layers/base))]
at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:197) [jboss-modules.jar:1.3.0.Beta3]
at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:443) [jboss-modules.jar:1.3.0.Beta3]
at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:431) [jboss-modules.jar:1.3.0.Beta3]
at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:373) [jboss-modules.jar:1.3.0.Beta3]
at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:118) [jboss-modules.jar:1.3.0.Beta3]
at java.lang.Class.forName0(Native Method) [rt.jar:1.7.0_25]
at java.lang.Class.forName(Class.java:270) [rt.jar:1.7.0_25]
at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:135) [jboss-marshalling-1.3.18.GA.jar:1.3.18.GA]
at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:116) [jboss-marshalling-1.3.18.GA.jar:1.3.18.GA]
at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:947)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1259)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadCollectionObject(RiverUnmarshaller.java:184)
at org.jboss.marshalling.river.RiverUnmarshaller.readCollectionData(RiverUnmarshaller.java:777)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:662)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.18.GA.jar:1.3.18.GA]
at org.jboss.marshalling.river.RiverObjectInputStream.readFields(RiverObjectInputStream.java:129)
at javax.security.auth.Subject$SecureSet.readObject(Subject.java:1297) [rt.jar:1.7.0_25]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
at org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:218) [jboss-marshalling-1.3.18.GA.jar:1.3.18.GA]
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1629)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1290)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1732)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1648)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1612)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1290)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1732)
at org.jboss.marshalling.river.RiverObjectInputStream.defaultReadObject(RiverObjectInputStream.java:73)
at javax.security.auth.Subject.readObject(Subject.java:947) [rt.jar:1.7.0_25]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
at org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:218) [jboss-marshalling-1.3.18.GA.jar:1.3.18.GA]
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1629)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1290)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:72) [jboss-marshalling-1.3.18.GA.jar:1.3.18.GA]
at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$GetSubjectResponseHandler.handleRequest(TransactionalProtocolOperationHandler.java:182) [wildfly-controller-8.0.0.Beta1-SNAPSHOT.jar:8.0.0.Beta1-SNAPSHOT]
... 9 more
I have no idea why that is intermittent.
Another issue is why the catch of that exception didn't result in proper cleanup.
> Hangs in DomainControllerMigrationTestCase
> ------------------------------------------
>
> Key: WFLY-1966
> URL: https://issues.jboss.org/browse/WFLY-1966
> Project: WildFly
> Issue Type: Bug
> Components: Domain Management
> Reporter: Brian Stansberry
> Fix For: 8.0.0.Beta1
>
> Attachments: thread-dump.txt
>
>
> DomainControllerMigrationTestCase is intermittently hanging recently.
> I will attach thread dumps from a recent hang.
> The test driver is hanging here:
> "main" prio=10 tid=0xb7605c00 nid=0x7656 in Object.wait() [0xb7786000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xa99833f8> (a org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl)
> at java.lang.Object.wait(Object.java:503)
> at org.jboss.threads.AsyncFutureTask.await(AsyncFutureTask.java:192)
> - eliminated <0xa99833f8> (a org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl)
> at org.jboss.threads.AsyncFutureTask.get(AsyncFutureTask.java:266)
> - locked <0xa99833f8> (a org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl)
> at org.jboss.as.controller.client.impl.AbstractDelegatingAsyncFuture.get(AbstractDelegatingAsyncFuture.java:100)
> at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127)
> at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:76)
> at org.jboss.as.controller.client.helpers.domain.impl.DomainClientImpl.execute(DomainClientImpl.java:86)
> at org.jboss.as.test.integration.domain.management.util.DomainLifecycleUtil.executeForResult(DomainLifecycleUtil.java:580)
> at org.jboss.as.test.integration.domain.DomainControllerMigrationTestCase.testDCFailover(DomainControllerMigrationTestCase.java:206)
> The master HC process is stuck here:
> "management-handler-thread - 2" prio=10 tid=0x8be50400 nid=0x1a92 waiting on condition [0x87604000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xaa032fd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.jboss.as.controller.remote.BlockingQueueOperationListener.retrievePreparedOperation(BlockingQueueOperationListener.java:84)
> at org.jboss.as.domain.controller.operations.coordination.DomainSlaveHandler.execute(DomainSlaveHandler.java:117)
> at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:608)
> at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:486)
> at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:275)
> at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:270)
> at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:257)
> at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:142)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:205)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:110)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$2.run(ModelControllerClientOperationHandler.java:157)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$2.run(ModelControllerClientOperationHandler.java:153)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:153)
> at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:296)
> at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:518)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> at org.jboss.threads.JBossThread.run(JBossThread.java:122)
> I see nothing relevant in the other thread dumps.
> The master HC is blocking waiting for a response from the slave HC.
> Just before the client makes the request that triggers this, it has reloaded the slave HC. The test is not robust in terms of how it checks that the slave is ready before continuing on with the test; it merely connects directly to the slave checks that the slave's root resource can be read. That can succeed relatively early in boot even before the slave has registered with the master.
> Fixing that may prevent the hangs, but it would likely paper over whatever problem is causing the hangs. A client invoking multi-host operations while a slave is booting and registering is a use case that can't hang.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 8 months