[JBoss JIRA] (ISPN-10138) JMS ClassCastException
by worldline_ge_dev worldline_ge_dev (Jira)
[ https://issues.jboss.org/browse/ISPN-10138?page=com.atlassian.jira.plugin... ]
worldline_ge_dev worldline_ge_dev commented on ISPN-10138:
----------------------------------------------------------
thanks for the explanation, could be the reason. Unfortunately, it is too complex to drill this down to a simple example application and we don't use CompletableFuture anymore, so currently the problem seems solved for us. It is then more a JBoss classloading problem? It seems to me very strange that classloading behaviour depends on whether I start two applications simultaneously or one after the other. Even when starting simultaneously the error could maybe occur at random
> JMS ClassCastException
> ----------------------
>
> Key: ISPN-10138
> URL: https://issues.jboss.org/browse/ISPN-10138
> Project: Infinispan
> Issue Type: Bug
> Components: Listeners
> Affects Versions: 9.4.4.Final
> Environment: JBoss 6.4.10.GA
> Infinispan 9.4.4.FINAL
> hornetq 2.3.25.FINAL
> jdk 1.8.0_92
> Reporter: worldline_ge_dev worldline_ge_dev
> Priority: Minor
>
> We have a malfunction that we think is a bug but are not sure whether it is a JBoss, HornetQ or Infinspan issue. Here is the situation:
> Two war applications on one JBoss, one sends data into a JMS queue, the other consumes them. On startup, each application registers in a replicated Infinispan cache.
> Infinispan subsystem is excluded in jboss-deployment-structure.xml and libs are in the lib folder of the applications. When we start JBoss and both applications are deployed simultaneously,
> all is okay. When first the consuming app is deployed and then the sending app we observe following error in last line of the following code in MessageListener:
> {code:java}
> public void process(Message inMessage) {
> DistributionMetadata metadata = null;
> log.info("DistributionMetadata.class.getClassLoader().toString(): " + DistributionMetadata.class.getClassLoader().toString());
> try {
> Object obj = ((ObjectMessage) inMessage).getObject();
> log.info("obj.getClass().getClassLoader().toString(): " + obj.getClass().getClassLoader().toString());
> metadata = (DistributionMetadata) obj;
> {code}
> INFO 12:01:41,805 (Thread-9 (HornetQ-client-global-threads-1054694434)) (JmsConsumer.java:process:214) -
> DistributionMetadata.class.getClassLoader().toString(): ModuleClassLoader for Module "deployment.msp.war:main" from Service Module Loader
> obj.getClass().getClassLoader().toString(): ModuleClassLoader for Module "deployment.bvn-idx-routing.war:main" from Service Module Loader
> ERROR 12:01:41,805 (Thread-9 (HornetQ-client-global-threads-1054694434)) (JmsConsumer.java:process:244) -FAILED: com.equensworldline.jms.entities.DistributionMetadata cannot be cast to com.equensworldline.jms.entities.DistributionMetadata: java.lang.ClassCastException: com.equensworldline.jms.entities.DistributionMetadata cannot be cast to com.equensworldline.jms.entities.DistributionMetadata
> at com.equensworldline.jms.api.external.JmsConsumer.process(JmsConsumer.java:217)
> at com.equensworldline.correlationidmgmt.jee.CorrelationIDMessageListener.onMessage(CorrelationIDMessageListener.java:32)
> at org.hornetq.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:100)
> at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1123)
> at org.hornetq.core.client.impl.ClientConsumerImpl.access$500(ClientConsumerImpl.java:57)
> at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1258)
> at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> You see the DistributionMetadata class is loaded by two different classloaders which causes this error.
> The remedy we found is very strange and has on first sight nothing to do with the failure:
> We have registered an Infinispan cachelistener and within the @CacheEntryCreated event we start a JMS listener and access the cache. Because we observed org.infinispan.util.concurrent.TimeoutException
> we make this asynchronously following the advice in [https://developer.jboss.org/thread/268919|https://developer.jboss.org/thr...].
> {code:java}
> CompletableFuture.runAsync(() -> regService.startListener(event.getValue()));
> {code}
> When we change back to a synchronous call, everything works and the ClassCastException does not occur. It seems, either Infinispan, HornetQ or JBoss does something queer with the classloaders in the described situation
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years
[JBoss JIRA] (ISPN-10133) ModuleLoadError: Alias module org.jboss.marshalling is referencing not existing module
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10133?page=com.atlassian.jira.plugin... ]
Dan Berindei commented on ISPN-10133:
-------------------------------------
[~NadirX] looking at the error message again, it's not from a proper server starting up, but from a {{vault}} macro defined in {{build-testsuite.xml}}.
AFAICT the failure should not happen in the upstream build, only in the product build, because the Brew Maven repository's SSL certificate isn't valid without the Red Hat root CA (see JDG-2799).
> ModuleLoadError: Alias module org.jboss.marshalling is referencing not existing module
> --------------------------------------------------------------------------------------
>
> Key: ISPN-10133
> URL: https://issues.jboss.org/browse/ISPN-10133
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.12.Final, 10.0.0.Beta3
> Environment: openjdk11-windows2012
> Reporter: Tristan Tarrant
> Assignee: Tristan Tarrant
> Priority: Critical
> Fix For: 10.0.0.Beta4, 9.4.13.Final
>
>
> We can reproduce the issue running the job https://jdg-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/JDG-2755-jdg-func...
> I attached the full log because I am not sure that this is the root cause. Feel free to add a comment.
> {noformat}
> The ' characters around the executable and arguments are
> not part of the command.
> [java] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 -Djboss.modules.system.pkgs=com.sun.crypto.provider
> [java] Exception in thread "main" org.jboss.modules.ModuleLoadError: Alias module org.jboss.marshalling is referencing not existing module
> [java] at org.jboss.modules.ModuleLoadException.toError(ModuleLoadException.java:74)
> [java] at org.jboss.modules.Module.getPathsUnchecked(Module.java:1608)
> [java] at org.jboss.modules.Module.loadModuleClass(Module.java:726)
> [java] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:247)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
> [java] at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
> [java] at java.base/java.lang.ClassLoader.defineClass1(Native Method)
> [java] at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
> [java] at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:423)
> [java] at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:519)
> [java] at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:339)
> [java] at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:126)
> [java] at org.jboss.modules.Module.loadModuleClass(Module.java:731)
> [java] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:247)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
> [java] at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
> [java] at java.base/java.lang.Class.forName0(Native Method)
> [java] at java.base/java.lang.Class.forName(Class.java:398)
> [java] at org.jboss.modules.Module.run(Module.java:338)
> [java] at org.jboss.modules.Module.run(Module.java:320)
> [java] at org.jboss.modules.Main.main(Main.java:593)
> [ant] Exiting C:\home\jenkins\workspace\JDG-AAAA-jdg-func-ispn-testsuite-reproducer\0ab58ada\infinispan\server\integration\testsuite\build-testsuite.xml.
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years
[JBoss JIRA] (ISPN-10133) ModuleLoadError: Alias module org.jboss.marshalling is referencing not existing module
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10133?page=com.atlassian.jira.plugin... ]
Dan Berindei edited comment on ISPN-10133 at 4/22/19 12:09 PM:
---------------------------------------------------------------
[~NadirX] looking at the error message again, it's not from a proper server starting up, but from a {{vault}} macro defined in {{build-testsuite.xml}}.
AFAICT the failure should not happen in the upstream build, only in the product build, because the Brew Maven repository's SSL certificate isn't valid without the Red Hat root CA (see JDG-2797).
was (Author: dan.berindei):
[~NadirX] looking at the error message again, it's not from a proper server starting up, but from a {{vault}} macro defined in {{build-testsuite.xml}}.
AFAICT the failure should not happen in the upstream build, only in the product build, because the Brew Maven repository's SSL certificate isn't valid without the Red Hat root CA (see JDG-2799).
> ModuleLoadError: Alias module org.jboss.marshalling is referencing not existing module
> --------------------------------------------------------------------------------------
>
> Key: ISPN-10133
> URL: https://issues.jboss.org/browse/ISPN-10133
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 9.4.12.Final, 10.0.0.Beta3
> Environment: openjdk11-windows2012
> Reporter: Tristan Tarrant
> Assignee: Tristan Tarrant
> Priority: Critical
> Fix For: 10.0.0.Beta4, 9.4.13.Final
>
>
> We can reproduce the issue running the job https://jdg-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/JDG-2755-jdg-func...
> I attached the full log because I am not sure that this is the root cause. Feel free to add a comment.
> {noformat}
> The ' characters around the executable and arguments are
> not part of the command.
> [java] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 -Djboss.modules.system.pkgs=com.sun.crypto.provider
> [java] Exception in thread "main" org.jboss.modules.ModuleLoadError: Alias module org.jboss.marshalling is referencing not existing module
> [java] at org.jboss.modules.ModuleLoadException.toError(ModuleLoadException.java:74)
> [java] at org.jboss.modules.Module.getPathsUnchecked(Module.java:1608)
> [java] at org.jboss.modules.Module.loadModuleClass(Module.java:726)
> [java] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:247)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
> [java] at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
> [java] at java.base/java.lang.ClassLoader.defineClass1(Native Method)
> [java] at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
> [java] at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:423)
> [java] at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:519)
> [java] at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:339)
> [java] at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:126)
> [java] at org.jboss.modules.Module.loadModuleClass(Module.java:731)
> [java] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:247)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
> [java] at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
> [java] at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
> [java] at java.base/java.lang.Class.forName0(Native Method)
> [java] at java.base/java.lang.Class.forName(Class.java:398)
> [java] at org.jboss.modules.Module.run(Module.java:338)
> [java] at org.jboss.modules.Module.run(Module.java:320)
> [java] at org.jboss.modules.Main.main(Main.java:593)
> [ant] Exiting C:\home\jenkins\workspace\JDG-AAAA-jdg-func-ispn-testsuite-reproducer\0ab58ada\infinispan\server\integration\testsuite\build-testsuite.xml.
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years
[JBoss JIRA] (ISPN-10138) JMS ClassCastException
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10138?page=com.atlassian.jira.plugin... ]
Dan Berindei commented on ISPN-10138:
-------------------------------------
Can you reproduce the problem with a small application and attach it here? The code in the description doesn't seem Infinispan-related at all.
My guess would be that when you register a listener in JMS, JMS uses the thread context classloader to deserialize messages. {{CompletableFuture.runAsync()}} runs your code on a {{ForkJoinPool.commonPool()}} thread, which is not managed by the application service and doesn't have a proper thread context classloader. The first application to call {{CompletableFuture.runAsync()}} creates a common-pool thread with its classloader set as the thread-context classloader, and the 2nd application to call {{CompletableFuture.runAsync()}} reuses the thread with the wrong thread-context classloader. When the applications start in parallel, each of them gets its own common-pool thread, so they have the proper thread-context classloader.
> JMS ClassCastException
> ----------------------
>
> Key: ISPN-10138
> URL: https://issues.jboss.org/browse/ISPN-10138
> Project: Infinispan
> Issue Type: Bug
> Components: Listeners
> Affects Versions: 9.4.4.Final
> Environment: JBoss 6.4.10.GA
> Infinispan 9.4.4.FINAL
> hornetq 2.3.25.FINAL
> jdk 1.8.0_92
> Reporter: worldline_ge_dev worldline_ge_dev
> Priority: Minor
>
> We have a malfunction that we think is a bug but are not sure whether it is a JBoss, HornetQ or Infinspan issue. Here is the situation:
> Two war applications on one JBoss, one sends data into a JMS queue, the other consumes them. On startup, each application registers in a replicated Infinispan cache.
> Infinispan subsystem is excluded in jboss-deployment-structure.xml and libs are in the lib folder of the applications. When we start JBoss and both applications are deployed simultaneously,
> all is okay. When first the consuming app is deployed and then the sending app we observe following error in last line of the following code in MessageListener:
> {code:java}
> public void process(Message inMessage) {
> DistributionMetadata metadata = null;
> log.info("DistributionMetadata.class.getClassLoader().toString(): " + DistributionMetadata.class.getClassLoader().toString());
> try {
> Object obj = ((ObjectMessage) inMessage).getObject();
> log.info("obj.getClass().getClassLoader().toString(): " + obj.getClass().getClassLoader().toString());
> metadata = (DistributionMetadata) obj;
> {code}
> INFO 12:01:41,805 (Thread-9 (HornetQ-client-global-threads-1054694434)) (JmsConsumer.java:process:214) -
> DistributionMetadata.class.getClassLoader().toString(): ModuleClassLoader for Module "deployment.msp.war:main" from Service Module Loader
> obj.getClass().getClassLoader().toString(): ModuleClassLoader for Module "deployment.bvn-idx-routing.war:main" from Service Module Loader
> ERROR 12:01:41,805 (Thread-9 (HornetQ-client-global-threads-1054694434)) (JmsConsumer.java:process:244) -FAILED: com.equensworldline.jms.entities.DistributionMetadata cannot be cast to com.equensworldline.jms.entities.DistributionMetadata: java.lang.ClassCastException: com.equensworldline.jms.entities.DistributionMetadata cannot be cast to com.equensworldline.jms.entities.DistributionMetadata
> at com.equensworldline.jms.api.external.JmsConsumer.process(JmsConsumer.java:217)
> at com.equensworldline.correlationidmgmt.jee.CorrelationIDMessageListener.onMessage(CorrelationIDMessageListener.java:32)
> at org.hornetq.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:100)
> at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1123)
> at org.hornetq.core.client.impl.ClientConsumerImpl.access$500(ClientConsumerImpl.java:57)
> at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1258)
> at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> You see the DistributionMetadata class is loaded by two different classloaders which causes this error.
> The remedy we found is very strange and has on first sight nothing to do with the failure:
> We have registered an Infinispan cachelistener and within the @CacheEntryCreated event we start a JMS listener and access the cache. Because we observed org.infinispan.util.concurrent.TimeoutException
> we make this asynchronously following the advice in [https://developer.jboss.org/thread/268919|https://developer.jboss.org/thr...].
> {code:java}
> CompletableFuture.runAsync(() -> regService.startListener(event.getValue()));
> {code}
> When we change back to a synchronous call, everything works and the ClassCastException does not occur. It seems, either Infinispan, HornetQ or JBoss does something queer with the classloaders in the described situation
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years