[JBoss JIRA] (WFLY-12815) Wildfly 13 - Thread local state corrupted by deployed application explosion during session timeout leading to WELD-001304 - More than one context active for scope type javax.enterprise.context.SessionScoped
by NUNO GODINHO DE MATOS (Jira)
[ https://issues.redhat.com/browse/WFLY-12815?page=com.atlassian.jira.plugi... ]
NUNO GODINHO DE MATOS commented on WFLY-12815:
----------------------------------------------
Hi,
Just to let you know we have gotten more reports on this issue.
So we are providing to projects experiencing this issue our "hack" patch to fix the thread local corruption of the thread:
[^sourceCodeToSendToWildfly.7z]
I am adding a sketch of the listener class that we use to work-around this problem, underlying the underlying "reasons" of our blow ups during session desctroyed are discovered.
IT would be good to not need to implement such work around with these hacky listeners which can stop working from version to verison of wildfly.
> Wildfly 13 - Thread local state corrupted by deployed application explosion during session timeout leading to WELD-001304 - More than one context active for scope type javax.enterprise.context.SessionScoped
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-12815
> URL: https://issues.redhat.com/browse/WFLY-12815
> Project: WildFly
> Issue Type: Bug
> Components: CDI / Weld
> Affects Versions: 13.0.0.Final
> Environment: Environment independent the issue, it is purely a logical problem
> Reporter: NUNO GODINHO DE MATOS
> Assignee: Matěj Novotný
> Priority: Major
> Attachments: sourceCodeToSendToWildfly.7z
>
>
> The full description of the problem can be seen in stack overflow.
> Please consulder the issue:
> https://stackoverflow.com/questions/58930939/wildflt-13-weld-001304-more-...
> SUMMARY:
> (1) Setup you wildfly to have a session timeout of 1 minute - so that you can esaily make your http sessions timeout
> (2) Program in your WAR application a sessionDestroyed listener that will be broken.
> In our case whenever the session is timing out, we have some code that explodes in wildfly and not in weblogic because it expected for the RequestScope context to be active, but apparently in wildfly when Undertow to start killing of a session the request scope context is not made active so that caused our session destroyed handling to break
> (3) Do this sufficient amount of times to corrupted as may threads in the thread pool as possible
> (4) Now try to interact with your application making use of some session scoped beans .
> If you travel to ay sort of view that makes use of a session scoped bean that thread will be broken with the exception that multiple session scope context implementation are active.
> But this exception will only come out and aply if the thread handling the HTTP request is one of the threads that in the past were used by undertow to handle the session timeout.
> The only threads that have been corrupted forever are those that had a broken sessin timeout
> Explanation for the issue:
> - When the session timeout is being orchestrated by underdow, wildfly is activating a special HttpSessionDescrutionContext and making it active.
> This ACTIVE TRUE/FALSE flag is a ThreadLocal variable.
> So the activation of the scope context is marked on the thread itself.
> - When the thread blows up the thread context will remain for as long at the thread lives
> - in a future request the flag had that thread local variable active already.
> So when the BeanManagaerImpl is hunting to the one and only active http session context it finds the traditional happy path http session context active plust the DestructionSession context that was activated in a previous call.
> All of the illustrative stack traces that facilitate the comprehention of the issue are shown in the stack overflow thread.
> I am of the oppinion that errors like this can happen in the deployed applications.
> It would not hurt if wildfly would somehow be able to ensure that the thread that hand an explosion in a previous request is not corrupted when it is used to handle new requests.
> Many thanks for having a look.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (WFLY-12815) Wildfly 13 - Thread local state corrupted by deployed application explosion during session timeout leading to WELD-001304 - More than one context active for scope type javax.enterprise.context.SessionScoped
by NUNO GODINHO DE MATOS (Jira)
[ https://issues.redhat.com/browse/WFLY-12815?page=com.atlassian.jira.plugi... ]
NUNO GODINHO DE MATOS updated WFLY-12815:
-----------------------------------------
Attachment: sourceCodeToSendToWildfly.7z
> Wildfly 13 - Thread local state corrupted by deployed application explosion during session timeout leading to WELD-001304 - More than one context active for scope type javax.enterprise.context.SessionScoped
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-12815
> URL: https://issues.redhat.com/browse/WFLY-12815
> Project: WildFly
> Issue Type: Bug
> Components: CDI / Weld
> Affects Versions: 13.0.0.Final
> Environment: Environment independent the issue, it is purely a logical problem
> Reporter: NUNO GODINHO DE MATOS
> Assignee: Matěj Novotný
> Priority: Major
> Attachments: sourceCodeToSendToWildfly.7z
>
>
> The full description of the problem can be seen in stack overflow.
> Please consulder the issue:
> https://stackoverflow.com/questions/58930939/wildflt-13-weld-001304-more-...
> SUMMARY:
> (1) Setup you wildfly to have a session timeout of 1 minute - so that you can esaily make your http sessions timeout
> (2) Program in your WAR application a sessionDestroyed listener that will be broken.
> In our case whenever the session is timing out, we have some code that explodes in wildfly and not in weblogic because it expected for the RequestScope context to be active, but apparently in wildfly when Undertow to start killing of a session the request scope context is not made active so that caused our session destroyed handling to break
> (3) Do this sufficient amount of times to corrupted as may threads in the thread pool as possible
> (4) Now try to interact with your application making use of some session scoped beans .
> If you travel to ay sort of view that makes use of a session scoped bean that thread will be broken with the exception that multiple session scope context implementation are active.
> But this exception will only come out and aply if the thread handling the HTTP request is one of the threads that in the past were used by undertow to handle the session timeout.
> The only threads that have been corrupted forever are those that had a broken sessin timeout
> Explanation for the issue:
> - When the session timeout is being orchestrated by underdow, wildfly is activating a special HttpSessionDescrutionContext and making it active.
> This ACTIVE TRUE/FALSE flag is a ThreadLocal variable.
> So the activation of the scope context is marked on the thread itself.
> - When the thread blows up the thread context will remain for as long at the thread lives
> - in a future request the flag had that thread local variable active already.
> So when the BeanManagaerImpl is hunting to the one and only active http session context it finds the traditional happy path http session context active plust the DestructionSession context that was activated in a previous call.
> All of the illustrative stack traces that facilitate the comprehention of the issue are shown in the stack overflow thread.
> I am of the oppinion that errors like this can happen in the deployed applications.
> It would not hurt if wildfly would somehow be able to ensure that the thread that hand an explosion in a previous request is not corrupted when it is used to handle new requests.
> Many thanks for having a look.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (WFLY-13686) Deadlock on Wildfly startup
by K G (Jira)
K G created WFLY-13686:
--------------------------
Summary: Deadlock on Wildfly startup
Key: WFLY-13686
URL: https://issues.redhat.com/browse/WFLY-13686
Project: WildFly
Issue Type: Bug
Components: EJB
Affects Versions: 20.0.1.Final
Reporter: K G
Assignee: Cheng Fang
On Wildfly startup there can be a deadlock related to ejb/singleton access and more specifically: StartupAwaitInterceptor and ContainerManagedConcurrencyInterceptor. This can happen when there is a too early client request (occurring during app startup) or a request caused by thread running in managed executor (that's what happened to me). A thread that is blocked by StartupAwaitInterceptor also holds a lock from ContainerManagedConcurrencyInterceptor and blocks other threads. This is related to the following pull request, link to the comment: [https://github.com/wildfly/wildfly/pull/9009#issuecomment-656147415] .
I guess possible solution is to change interceptors ordering. Other possibility is to add "privileged" flag (see pull request for explanation) to threads from managed thread factory but in this case a too early client request could also cause a dealock.
Scenario of deadlock (description copied from pull request's comment):
* startup singleton A's initialization starts and completes successfully
* startup singleton B is initializing and during that it starts a task X via managedThreadExecutor
* X wants to access A and is blocked by StartupCountdown.await
* meanwhile B continues initializing and wants to access A but X already holds a lock on A (I can see ContainerManagedConcurrencyInterceptor.processInvocation in the tread dump) hence after 5000ms B's initialization fails as well as whole deployment
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (WFCORE-5053) Prometheus JMX exporter jar, stops starting the wildfly server.
by Rakesh Kumar Shah (Jira)
[ https://issues.redhat.com/browse/WFCORE-5053?page=com.atlassian.jira.plug... ]
Rakesh Kumar Shah commented on WFCORE-5053:
-------------------------------------------
Hi James,
Can I get nightly jars to verify the fix at our ends? So when it is available, we can consume it right away.
What is the best way we can consume only this bug related changes?
Thank you
-Rakesh
> Prometheus JMX exporter jar, stops starting the wildfly server.
> ----------------------------------------------------------------
>
> Key: WFCORE-5053
> URL: https://issues.redhat.com/browse/WFCORE-5053
> Project: WildFly Core
> Issue Type: Bug
> Components: Logging
> Reporter: Rakesh Kumar Shah
> Assignee: James Perkins
> Priority: Major
> Fix For: 13.0.0.Beta1
>
>
> Trying to integrate the Prometheus JMX exporter with wildfly. After spending lot of time and work around it, wildfly is not starting. It throws following exception.
> {code}
> java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager
> at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at java.logging/java.util.logging.LogManager$1.run(LogManager.java:239)
> at java.logging/java.util.logging.LogManager$1.run(LogManager.java:223)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.logging/java.util.logging.LogManager.<clinit>(LogManager.java:223)
> at java.logging/java.util.logging.Logger.demandLogger(Logger.java:648)
> at java.logging/java.util.logging.Logger.getLogger(Logger.java:717)
> at java.logging/java.util.logging.Logger.getLogger(Logger.java:701)
> at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.<clinit>(JmxCollector.java:39)
> at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:29)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
> at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
> WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager
> Exception in thread "main" java.lang.NoClassDefFoundError: org/jboss/logmanager/Level
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:398)
> at org.jboss.modules.Module.run(Module.java:340)
> at org.jboss.modules.Module.run(Module.java:320)
> at org.jboss.modules.Main.main(Main.java:617)
> Caused by: java.lang.ClassNotFoundException: org.jboss.logmanager.Level
> at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at org.jboss.modules.JDKSpecific.getSystemClass(JDKSpecific.java:183)
> at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:395)
> at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (WFCORE-4845) Allow Remote Debug Connections on Java 9+ With StandaloneCommandBuilder
by Katarina Hermanova (Jira)
[ https://issues.redhat.com/browse/WFCORE-4845?page=com.atlassian.jira.plug... ]
Katarina Hermanova reassigned WFCORE-4845:
------------------------------------------
Assignee: Katarina Hermanova
> Allow Remote Debug Connections on Java 9+ With StandaloneCommandBuilder
> -----------------------------------------------------------------------
>
> Key: WFCORE-4845
> URL: https://issues.redhat.com/browse/WFCORE-4845
> Project: WildFly Core
> Issue Type: Enhancement
> Components: Launcher
> Reporter: Josh Fisher
> Assignee: Katarina Hermanova
> Priority: Minor
>
> When using StandaloneCommandBuilder#setDebug method to set the debug VM argument it will use this format:
> -agentlib:jdwp=transport=dt_socket,server=y,suspend=$suspend,address=$debugPort
> On Java 9+, this will only accept connections from the localhost, while in Java 8 this will accept connections from any host. It is possible to support remote connections on Java 9+ by configuring the argument in the Java options list instead of using setDebug as a workaround.
> While debug connections from remote hosts are not especially common, there are some use cases where it may be desired. For consistency between Java versions, we should consider using the wildcard "*" host so the debug argument format is:
> -agentlib:jdwp=transport=dt_socket,server=y,suspend=$suspend,address=*:$debugPort
>
> I believe one way this could be achieved is by modifying the DEBUG_FORMAT to:
> DEBUG_FORMAT = "-agentlib:jdwp=transport=dt_socket,server=y,suspend=%s,address=%s"
> then in setDebug check the environment VM and format the address accordingly.
> {code:java}
> public StandaloneCommandBuilder setDebug(boolean suspend, int port) {
> final String address;
> if (environment.getJvm().isModular()){
> address = "*:" + port;
> } else {
> address = String.valueOf(port);
> }
> debugArg = String.format(DEBUG_FORMAT, (suspend ? "y" : "n"), address);
> return this;
> }
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (WFLY-13684) Intermittent failures in MixedDomainDeployment700TestCase.testJsfWorks
by Brian Stansberry (Jira)
[ https://issues.redhat.com/browse/WFLY-13684?page=com.atlassian.jira.plugi... ]
Brian Stansberry commented on WFLY-13684:
-----------------------------------------
[~ehugonnet] But, the failure is on the legacy HC, not the current DC, so changes in the current DC should not cause runtime issues on the legacy HC.
I regarded https://issues.apache.org/jira/browse/ARTEMIS-444 as the cause of WFLY-12649.
> Intermittent failures in MixedDomainDeployment700TestCase.testJsfWorks
> ----------------------------------------------------------------------
>
> Key: WFLY-13684
> URL: https://issues.redhat.com/browse/WFLY-13684
> Project: WildFly
> Issue Type: Bug
> Components: JMS, Management
> Reporter: Brian Stansberry
> Assignee: Emmanuel Hugonnet
> Priority: Major
>
> Lots of failures starting July 16, 2020: https://ci.wildfly.org/project.html?projectId=WF&buildTypeId=&tab=testDet...
> This is the first failure in a set of 12-13 failures.
> Odd thing is nothing was merged into master after July 12, so why these started failing is unclear.
> WFLY-12649 was a similar issue-, but there it was known that the version of Artemis of EAP 7.0.0 did not work properly on OpenJ9-. This current issue is affecting various jobs running different OpenJDK versions.
> I'll send up a PR ignoring this test.
> The problem details:
> {code}
> java.lang.AssertionError: {"WFLYDC0074: Operation failed or was rolled back on all servers. Server failures:" => {"server-group" => {"other-server-group" => {"host" => {"slave" => {"server-one" => {"WFLYCTL0062: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-1" => {
> "WFLYCTL0180: Services with missing/unavailable dependencies" => ["jboss.naming.context.java.module.test.test.DefaultJMSConnectionFactory is missing [jboss.naming.context.java.jboss.DefaultJMSConnectionFactory]"],
> "WFLYCTL0288: One or more services were unable to start due to one or more indirect dependencies not being available." => {
> "Services that were unable to start:" => [
> "jboss.deployment.unit.\"test.war\".CdiValidatorFactoryService",
> "jboss.deployment.unit.\"test.war\".WeldStartService",
> "jboss.deployment.unit.\"test.war\".component.\"com.sun.faces.config.ConfigureListener\".START",
> "jboss.deployment.unit.\"test.war\".component.\"com.sun.faces.config.ConfigureListener\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"javax.faces.webapp.FacetTag\".START",
> "jboss.deployment.unit.\"test.war\".component.\"javax.faces.webapp.FacetTag\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV\".START",
> "jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.ScriptFreeTLV\".START",
> "jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.ScriptFreeTLV\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldInitialListener\".START",
> "jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldInitialListener\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldTerminalListener\".START",
> "jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldTerminalListener\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".deploymentCompleteService",
> "jboss.deployment.unit.\"test.war\".jndiDependencyService",
> "jboss.undertow.deployment.default-server.default-host./test",
> "jboss.undertow.deployment.default-server.default-host./test.UndertowDeploymentInfoService"
> ],
> "Services that may be the cause:" => [
> "jboss.clustering.singleton.server.default",
> "jboss.naming.context.java.jboss.DefaultJMSConnectionFactory"
> ]
> }
> }}}}}}}}}
> {code}
> In the server.log of the /host=slave/server=server-one server:
> {code}
> 2020-07-17 08:48:06,358 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/journal,bindingsDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/bindings,largeMessagesDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/largemessages,pagingDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/paging)
> 2020-07-17 08:48:06,378 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221012: Using AIO Journal
> 2020-07-17 08:48:06,478 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
> 2020-07-17 08:48:06,481 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
> 2020-07-17 08:48:06,509 WARN [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ222010: Critical IO Error, shutting down the server. file=AIOSequentialFile:/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/journal/activemq-data-1.amq.tmp, message=Cannot open file:Invalid argument: java.io.IOException: Cannot open file:Invalid argument
> at org.apache.activemq.artemis.jlibaio.LibaioContext.open(Native Method)
> at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:291)
> at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:274)
> at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.open(AIOSequentialFile.java:133)
> at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile0(JournalFilesRepository.java:590)
> at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile(JournalFilesRepository.java:543)
> at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.ensureMinFiles(JournalFilesRepository.java:206)
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.setUpCurrentFile(JournalImpl.java:2652)
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1728)
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1126)
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1110)
> at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:1260)
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:1701)
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1595)
> at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:60)
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:393)
> at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:381)
> at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:199)
> at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:63)
> at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:97)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (WFLY-13684) Intermittent failures in MixedDomainDeployment700TestCase.testJsfWorks
by Brian Stansberry (Jira)
[ https://issues.redhat.com/browse/WFLY-13684?page=com.atlassian.jira.plugi... ]
Brian Stansberry updated WFLY-13684:
------------------------------------
Description:
Lots of failures starting July 16, 2020: https://ci.wildfly.org/project.html?projectId=WF&buildTypeId=&tab=testDet...
This is the first failure in a set of 12-13 failures.
Odd thing is nothing was merged into master after July 12, so why these started failing is unclear.
WFLY-12649 was a similar issue-, but there it was known that the version of Artemis of EAP 7.0.0 did not work properly on OpenJ9-. This current issue is affecting various jobs running different OpenJDK versions.
I'll send up a PR ignoring this test.
The problem details:
{code}
java.lang.AssertionError: {"WFLYDC0074: Operation failed or was rolled back on all servers. Server failures:" => {"server-group" => {"other-server-group" => {"host" => {"slave" => {"server-one" => {"WFLYCTL0062: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-1" => {
"WFLYCTL0180: Services with missing/unavailable dependencies" => ["jboss.naming.context.java.module.test.test.DefaultJMSConnectionFactory is missing [jboss.naming.context.java.jboss.DefaultJMSConnectionFactory]"],
"WFLYCTL0288: One or more services were unable to start due to one or more indirect dependencies not being available." => {
"Services that were unable to start:" => [
"jboss.deployment.unit.\"test.war\".CdiValidatorFactoryService",
"jboss.deployment.unit.\"test.war\".WeldStartService",
"jboss.deployment.unit.\"test.war\".component.\"com.sun.faces.config.ConfigureListener\".START",
"jboss.deployment.unit.\"test.war\".component.\"com.sun.faces.config.ConfigureListener\".WeldInstantiator",
"jboss.deployment.unit.\"test.war\".component.\"javax.faces.webapp.FacetTag\".START",
"jboss.deployment.unit.\"test.war\".component.\"javax.faces.webapp.FacetTag\".WeldInstantiator",
"jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV\".START",
"jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV\".WeldInstantiator",
"jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.ScriptFreeTLV\".START",
"jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.ScriptFreeTLV\".WeldInstantiator",
"jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldInitialListener\".START",
"jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldInitialListener\".WeldInstantiator",
"jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldTerminalListener\".START",
"jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldTerminalListener\".WeldInstantiator",
"jboss.deployment.unit.\"test.war\".deploymentCompleteService",
"jboss.deployment.unit.\"test.war\".jndiDependencyService",
"jboss.undertow.deployment.default-server.default-host./test",
"jboss.undertow.deployment.default-server.default-host./test.UndertowDeploymentInfoService"
],
"Services that may be the cause:" => [
"jboss.clustering.singleton.server.default",
"jboss.naming.context.java.jboss.DefaultJMSConnectionFactory"
]
}
}}}}}}}}}
{code}
In the server.log of the /host=slave/server=server-one server:
{code}
2020-07-17 08:48:06,358 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/journal,bindingsDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/bindings,largeMessagesDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/largemessages,pagingDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/paging)
2020-07-17 08:48:06,378 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221012: Using AIO Journal
2020-07-17 08:48:06,478 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-07-17 08:48:06,481 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2020-07-17 08:48:06,509 WARN [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ222010: Critical IO Error, shutting down the server. file=AIOSequentialFile:/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/journal/activemq-data-1.amq.tmp, message=Cannot open file:Invalid argument: java.io.IOException: Cannot open file:Invalid argument
at org.apache.activemq.artemis.jlibaio.LibaioContext.open(Native Method)
at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:291)
at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:274)
at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.open(AIOSequentialFile.java:133)
at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile0(JournalFilesRepository.java:590)
at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile(JournalFilesRepository.java:543)
at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.ensureMinFiles(JournalFilesRepository.java:206)
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.setUpCurrentFile(JournalImpl.java:2652)
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1728)
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1126)
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1110)
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:1260)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:1701)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1595)
at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:60)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:393)
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:381)
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:199)
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:63)
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
{code}
was:
Lots of failures starting July 16, 2020: https://ci.wildfly.org/project.html?projectId=WF&buildTypeId=&tab=testDet...
This is the first failure in a set of 12-13 failures.
Odd thing is nothing was merged into master after July 12, so why these started failing is unclear.
WFLY-12649 was a similar issue, but there it was known that the version of Artemis of EAP 7.0.0 did not work properly on OpenJ9. This current issue is affecting various jobs running different OpenJDK versions.
I'll send up a PR ignoring this test.
The problem details:
{code}
java.lang.AssertionError: {"WFLYDC0074: Operation failed or was rolled back on all servers. Server failures:" => {"server-group" => {"other-server-group" => {"host" => {"slave" => {"server-one" => {"WFLYCTL0062: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-1" => {
"WFLYCTL0180: Services with missing/unavailable dependencies" => ["jboss.naming.context.java.module.test.test.DefaultJMSConnectionFactory is missing [jboss.naming.context.java.jboss.DefaultJMSConnectionFactory]"],
"WFLYCTL0288: One or more services were unable to start due to one or more indirect dependencies not being available." => {
"Services that were unable to start:" => [
"jboss.deployment.unit.\"test.war\".CdiValidatorFactoryService",
"jboss.deployment.unit.\"test.war\".WeldStartService",
"jboss.deployment.unit.\"test.war\".component.\"com.sun.faces.config.ConfigureListener\".START",
"jboss.deployment.unit.\"test.war\".component.\"com.sun.faces.config.ConfigureListener\".WeldInstantiator",
"jboss.deployment.unit.\"test.war\".component.\"javax.faces.webapp.FacetTag\".START",
"jboss.deployment.unit.\"test.war\".component.\"javax.faces.webapp.FacetTag\".WeldInstantiator",
"jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV\".START",
"jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV\".WeldInstantiator",
"jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.ScriptFreeTLV\".START",
"jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.ScriptFreeTLV\".WeldInstantiator",
"jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldInitialListener\".START",
"jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldInitialListener\".WeldInstantiator",
"jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldTerminalListener\".START",
"jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldTerminalListener\".WeldInstantiator",
"jboss.deployment.unit.\"test.war\".deploymentCompleteService",
"jboss.deployment.unit.\"test.war\".jndiDependencyService",
"jboss.undertow.deployment.default-server.default-host./test",
"jboss.undertow.deployment.default-server.default-host./test.UndertowDeploymentInfoService"
],
"Services that may be the cause:" => [
"jboss.clustering.singleton.server.default",
"jboss.naming.context.java.jboss.DefaultJMSConnectionFactory"
]
}
}}}}}}}}}
{code}
In the server.log of the /host=slave/server=server-one server:
{code}
2020-07-17 08:48:06,358 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/journal,bindingsDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/bindings,largeMessagesDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/largemessages,pagingDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/paging)
2020-07-17 08:48:06,378 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221012: Using AIO Journal
2020-07-17 08:48:06,478 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2020-07-17 08:48:06,481 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2020-07-17 08:48:06,509 WARN [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ222010: Critical IO Error, shutting down the server. file=AIOSequentialFile:/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/journal/activemq-data-1.amq.tmp, message=Cannot open file:Invalid argument: java.io.IOException: Cannot open file:Invalid argument
at org.apache.activemq.artemis.jlibaio.LibaioContext.open(Native Method)
at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:291)
at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:274)
at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.open(AIOSequentialFile.java:133)
at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile0(JournalFilesRepository.java:590)
at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile(JournalFilesRepository.java:543)
at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.ensureMinFiles(JournalFilesRepository.java:206)
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.setUpCurrentFile(JournalImpl.java:2652)
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1728)
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1126)
at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1110)
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:1260)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:1701)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1595)
at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:60)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:393)
at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:381)
at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:199)
at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:63)
at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
{code}
> Intermittent failures in MixedDomainDeployment700TestCase.testJsfWorks
> ----------------------------------------------------------------------
>
> Key: WFLY-13684
> URL: https://issues.redhat.com/browse/WFLY-13684
> Project: WildFly
> Issue Type: Bug
> Components: JMS, Management
> Reporter: Brian Stansberry
> Assignee: Emmanuel Hugonnet
> Priority: Major
>
> Lots of failures starting July 16, 2020: https://ci.wildfly.org/project.html?projectId=WF&buildTypeId=&tab=testDet...
> This is the first failure in a set of 12-13 failures.
> Odd thing is nothing was merged into master after July 12, so why these started failing is unclear.
> WFLY-12649 was a similar issue-, but there it was known that the version of Artemis of EAP 7.0.0 did not work properly on OpenJ9-. This current issue is affecting various jobs running different OpenJDK versions.
> I'll send up a PR ignoring this test.
> The problem details:
> {code}
> java.lang.AssertionError: {"WFLYDC0074: Operation failed or was rolled back on all servers. Server failures:" => {"server-group" => {"other-server-group" => {"host" => {"slave" => {"server-one" => {"WFLYCTL0062: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-1" => {
> "WFLYCTL0180: Services with missing/unavailable dependencies" => ["jboss.naming.context.java.module.test.test.DefaultJMSConnectionFactory is missing [jboss.naming.context.java.jboss.DefaultJMSConnectionFactory]"],
> "WFLYCTL0288: One or more services were unable to start due to one or more indirect dependencies not being available." => {
> "Services that were unable to start:" => [
> "jboss.deployment.unit.\"test.war\".CdiValidatorFactoryService",
> "jboss.deployment.unit.\"test.war\".WeldStartService",
> "jboss.deployment.unit.\"test.war\".component.\"com.sun.faces.config.ConfigureListener\".START",
> "jboss.deployment.unit.\"test.war\".component.\"com.sun.faces.config.ConfigureListener\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"javax.faces.webapp.FacetTag\".START",
> "jboss.deployment.unit.\"test.war\".component.\"javax.faces.webapp.FacetTag\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV\".START",
> "jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.ScriptFreeTLV\".START",
> "jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.ScriptFreeTLV\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldInitialListener\".START",
> "jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldInitialListener\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldTerminalListener\".START",
> "jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldTerminalListener\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".deploymentCompleteService",
> "jboss.deployment.unit.\"test.war\".jndiDependencyService",
> "jboss.undertow.deployment.default-server.default-host./test",
> "jboss.undertow.deployment.default-server.default-host./test.UndertowDeploymentInfoService"
> ],
> "Services that may be the cause:" => [
> "jboss.clustering.singleton.server.default",
> "jboss.naming.context.java.jboss.DefaultJMSConnectionFactory"
> ]
> }
> }}}}}}}}}
> {code}
> In the server.log of the /host=slave/server=server-one server:
> {code}
> 2020-07-17 08:48:06,358 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/journal,bindingsDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/bindings,largeMessagesDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/largemessages,pagingDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/paging)
> 2020-07-17 08:48:06,378 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221012: Using AIO Journal
> 2020-07-17 08:48:06,478 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
> 2020-07-17 08:48:06,481 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
> 2020-07-17 08:48:06,509 WARN [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ222010: Critical IO Error, shutting down the server. file=AIOSequentialFile:/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/journal/activemq-data-1.amq.tmp, message=Cannot open file:Invalid argument: java.io.IOException: Cannot open file:Invalid argument
> at org.apache.activemq.artemis.jlibaio.LibaioContext.open(Native Method)
> at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:291)
> at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:274)
> at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.open(AIOSequentialFile.java:133)
> at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile0(JournalFilesRepository.java:590)
> at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile(JournalFilesRepository.java:543)
> at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.ensureMinFiles(JournalFilesRepository.java:206)
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.setUpCurrentFile(JournalImpl.java:2652)
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1728)
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1126)
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1110)
> at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:1260)
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:1701)
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1595)
> at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:60)
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:393)
> at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:381)
> at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:199)
> at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:63)
> at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:97)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (WFLY-13684) Intermittent failures in MixedDomainDeployment700TestCase.testJsfWorks
by Brian Stansberry (Jira)
[ https://issues.redhat.com/browse/WFLY-13684?page=com.atlassian.jira.plugi... ]
Brian Stansberry commented on WFLY-13684:
-----------------------------------------
[~ehugonnet] It started happening on July 16. https://github.com/wildfly/wildfly/pull/13383 was merged in June 26, so failures didn't show up for a while. But perhaps it is relevant and some OS update happened later that surfaced the problem.
> Intermittent failures in MixedDomainDeployment700TestCase.testJsfWorks
> ----------------------------------------------------------------------
>
> Key: WFLY-13684
> URL: https://issues.redhat.com/browse/WFLY-13684
> Project: WildFly
> Issue Type: Bug
> Components: JMS, Management
> Reporter: Brian Stansberry
> Assignee: Emmanuel Hugonnet
> Priority: Major
>
> Lots of failures starting July 16, 2020: https://ci.wildfly.org/project.html?projectId=WF&buildTypeId=&tab=testDet...
> This is the first failure in a set of 12-13 failures.
> Odd thing is nothing was merged into master after July 12, so why these started failing is unclear.
> WFLY-12649 was a similar issue, but there it was known that the version of Artemis of EAP 7.0.0 did not work properly on OpenJ9. This current issue is affecting various jobs running different OpenJDK versions.
> I'll send up a PR ignoring this test.
> The problem details:
> {code}
> java.lang.AssertionError: {"WFLYDC0074: Operation failed or was rolled back on all servers. Server failures:" => {"server-group" => {"other-server-group" => {"host" => {"slave" => {"server-one" => {"WFLYCTL0062: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-1" => {
> "WFLYCTL0180: Services with missing/unavailable dependencies" => ["jboss.naming.context.java.module.test.test.DefaultJMSConnectionFactory is missing [jboss.naming.context.java.jboss.DefaultJMSConnectionFactory]"],
> "WFLYCTL0288: One or more services were unable to start due to one or more indirect dependencies not being available." => {
> "Services that were unable to start:" => [
> "jboss.deployment.unit.\"test.war\".CdiValidatorFactoryService",
> "jboss.deployment.unit.\"test.war\".WeldStartService",
> "jboss.deployment.unit.\"test.war\".component.\"com.sun.faces.config.ConfigureListener\".START",
> "jboss.deployment.unit.\"test.war\".component.\"com.sun.faces.config.ConfigureListener\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"javax.faces.webapp.FacetTag\".START",
> "jboss.deployment.unit.\"test.war\".component.\"javax.faces.webapp.FacetTag\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV\".START",
> "jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.ScriptFreeTLV\".START",
> "jboss.deployment.unit.\"test.war\".component.\"javax.servlet.jsp.jstl.tlv.ScriptFreeTLV\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldInitialListener\".START",
> "jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldInitialListener\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldTerminalListener\".START",
> "jboss.deployment.unit.\"test.war\".component.\"org.jboss.weld.servlet.WeldTerminalListener\".WeldInstantiator",
> "jboss.deployment.unit.\"test.war\".deploymentCompleteService",
> "jboss.deployment.unit.\"test.war\".jndiDependencyService",
> "jboss.undertow.deployment.default-server.default-host./test",
> "jboss.undertow.deployment.default-server.default-host./test.UndertowDeploymentInfoService"
> ],
> "Services that may be the cause:" => [
> "jboss.clustering.singleton.server.default",
> "jboss.naming.context.java.jboss.DefaultJMSConnectionFactory"
> ]
> }
> }}}}}}}}}
> {code}
> In the server.log of the /host=slave/server=server-one server:
> {code}
> 2020-07-17 08:48:06,358 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/journal,bindingsDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/bindings,largeMessagesDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/largemessages,pagingDirectory=/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/paging)
> 2020-07-17 08:48:06,378 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221012: Using AIO Journal
> 2020-07-17 08:48:06,478 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
> 2020-07-17 08:48:06,481 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
> 2020-07-17 08:48:06,509 WARN [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 42) AMQ222010: Critical IO Error, shutting down the server. file=AIOSequentialFile:/store/work/tc-work/f5da3564a57e9d74/testsuite/mixed-domain/target/domains/MixedDomain700TestSuite/slave/servers/server-one/data/activemq/journal/activemq-data-1.amq.tmp, message=Cannot open file:Invalid argument: java.io.IOException: Cannot open file:Invalid argument
> at org.apache.activemq.artemis.jlibaio.LibaioContext.open(Native Method)
> at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:291)
> at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:274)
> at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.open(AIOSequentialFile.java:133)
> at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile0(JournalFilesRepository.java:590)
> at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile(JournalFilesRepository.java:543)
> at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.ensureMinFiles(JournalFilesRepository.java:206)
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.setUpCurrentFile(JournalImpl.java:2652)
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1728)
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1126)
> at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1110)
> at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:1260)
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:1701)
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1595)
> at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:60)
> at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:393)
> at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:381)
> at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:199)
> at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:63)
> at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:97)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months