[JBoss JIRA] (WFLY-10274) [pretesting] NPE during server shutdown when using scattered cache
by Michal Vinkler (JIRA)
[ https://issues.jboss.org/browse/WFLY-10274?page=com.atlassian.jira.plugin... ]
Michal Vinkler updated WFLY-10274:
----------------------------------
Summary: [pretesting] NPE during server shutdown when using scattered cache (was: NPE during server shutdown when using scattered cache)
> [pretesting] NPE during server shutdown when using scattered cache
> ------------------------------------------------------------------
>
> Key: WFLY-10274
> URL: https://issues.jboss.org/browse/WFLY-10274
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 13.0.0.Beta1
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
>
> We hit NPE when running tests for RFE EAP7-867.
> EAP distribution was built from https://github.com/pferraro/wildfly/tree/scattered .
> Test description: Positive stress test (no failover), 4-node EAP cluster, clients: starting with 400 clients in the beginning, raising the number of clients to 6000 in the end of the test.
> During clean server shutdown in the end of the test, server logged NPE and got stuck:
> {code}
> [JBossINF] [0m[31m07:55:57,643 ERROR [org.infinispan.scattered.impl.ScatteredStateConsumerImpl] (thread-200,ejb,dev214) ISPN000471: Failed processing values received from remote node during rebalance.: java.lang.NullPointerException
> [JBossINF] at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.applyValues(ScatteredStateConsumerImpl.java:505)
> [JBossINF] at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.lambda$getValuesAndApply$8(ScatteredStateConsumerImpl.java:475)
> [JBossINF] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> [JBossINF] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> [JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> [JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:66)
> [JBossINF] at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:56)
> [JBossINF] at org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:35)
> [JBossINF] at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:53)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1304)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1207)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1342)
> [JBossINF] at org.jgroups.JChannel.up(JChannel.java:819)
> [JBossINF] at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:134)
> [JBossINF] at org.jgroups.stack.Protocol.up(Protocol.java:340)
> [JBossINF] at org.jgroups.protocols.FORK.up(FORK.java:134)
> [JBossINF] at org.jgroups.protocols.FRAG3.up(FRAG3.java:166)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
> [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
> [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
> [JBossINF] at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
> [JBossINF] at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
> [JBossINF] at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
> [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
> [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
> [JBossINF] at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
> [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
> [JBossINF] at org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
> [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:267)
> [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
> [JBossINF] at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> {code}
> Scattered cache was configured with bias-lifespan="0".
> Server configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-ses...
> Server link:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-ses...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (WFLY-10109) [Artemis 2.x upgrade] MDB connected to remote server does not get activated when using discovery group with JGroups
by Jeff Mesnil (JIRA)
[ https://issues.jboss.org/browse/WFLY-10109?page=com.atlassian.jira.plugin... ]
Jeff Mesnil commented on WFLY-10109:
------------------------------------
[~eduda] I fail to reproduce the issue locally and there is not a lot to glance from the failure at https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eduda-verifier.... Could the issue be environmental?
However, there might be missing information from Artemis because the configuration you use should have reported some errors.
The configuration for node2 (which hosts the MDB) is:
{code}
<subsystem xmlns="urn:jboss:domain:messaging-activemq:3.1">
<server name="default" persistence-enabled="true" id-cache-size="20000">
...
<discovery-group name="dg-group1" jgroups-cluster="activemq-cluster" refresh-timeout="10000"/>
<pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" discovery-group="dg-group1" ha="false" min-large-message-size="102400" block-on-acknowledge="false" retry-interval="2000" retry-interval-multiplier="1.0" reconnect-attempts="-1" transaction="xa" min-pool-size="-1" max-pool-size="-1">
<inbound-config rebalance-connections="true"/>
</pooled-connection-factory>
</server>
</subsystem>
{code}
The MDB is activated at 14:03:45
{code}
14:03:45,583 INFO [org.jboss.as.ejb3] (MSC service thread 1-7) WFLYEJB0042: Started message driven bean 'mdb1' with 'activemq-ra.rar' resource adapter
{code}
The RA configuration uses the default initial-connect-attempts of 1 and the discovery initial-wait-timeout is the default 10 seconds.
However, there is no info or warning before the server is shutdown at 14:04:36 (51 seconds later):
{code}
14:04:36,336 WARN [org.apache.activemq.artemis.ra] (default-threads - 1) AMQ152005: Failure in broker activation org.apache.activemq.artemis.ra.inflow.ActiveMQActivationSpec(ra=org.wildfly.extension.messaging.activemq.ActiveMQResourceAdapter@e77d2b4c destination=jms/queue/InQueue destinationType=javax.jms.Queue ack=Auto-acknowledge durable=false clientID=null user=null maxSession=15): ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ119012: Timed out waiting to receive initial broadcast from cluster]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:761) [artemis-core-client-2.5.0.jar:2.6.0-SNAPSHOT]
at org.apache.activemq.artemis.ra.inflow.ActiveMQActivation.setup(ActiveMQActivation.java:307) [artemis-ra-2.5.0.jar:2.6.0-SNAPSHOT]
at org.apache.activemq.artemis.ra.inflow.ActiveMQActivation$SetupActivation.run(ActiveMQActivation.java:696) [artemis-ra-2.5.0.jar:2.6.0-SNAPSHOT]
at org.jboss.jca.core.workmanager.WorkWrapper.runWork(WorkWrapper.java:445)
at org.jboss.as.connector.services.workmanager.WildflyWorkWrapper.runWork(WildflyWorkWrapper.java:69)
at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:223)
at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:29)
at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:789)
at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:44)
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:809)
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_162]
at org.jboss.threads.JBossThread.run(JBossThread.java:485)
{code}
I don't understand why Artemis did not throw this error 10 seconds after the MDB activation.
It seems that the call to org.apache.activemq.artemis.core.cluster.DiscoveryGroup#waitForBroadcast was unblocked when the node2 server was shutdown and not after the 10 seconds timeout
> [Artemis 2.x upgrade] MDB connected to remote server does not get activated when using discovery group with JGroups
> --------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10109
> URL: https://issues.jboss.org/browse/WFLY-10109
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: feature-branch-blocker
> Attachments: log
>
>
> Sometimes happens that MDB does not start to consume messages from remote server. There are not warning or error messages in log.
> Test Scenario:
> * Start server with deployed InQueue and OutQueue
> * Start 2nd server with configured pooled-connection-factory to connect to 1st server
> * Deploy MDB to 2nd server - mdb is consuming messages from InQueue and sending to OutQueue
> * Start sending messages to InQueue to 1st server
> * Let MDB processing messages from InQueue
> * Start consuming messages from OutQueue from 1st server
> Pass Criteria: There is the same number of sent and received messages
> Actual Result: Sometimes happens that Mdb does not start to consume messages.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (WFLY-10269) RemoteStore should use unresolved hostnames to accommodate dynamic environments
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-10269?page=com.atlassian.jira.plugin... ]
Radoslav Husar commented on WFLY-10269:
---------------------------------------
[~kabirkhan] I noticed the title was misleading, implying like this would be a new option. Fixed the title now.
> RemoteStore should use unresolved hostnames to accommodate dynamic environments
> -------------------------------------------------------------------------------
>
> Key: WFLY-10269
> URL: https://issues.jboss.org/browse/WFLY-10269
> Project: WildFly
> Issue Type: Enhancement
> Components: Clustering
> Affects Versions: 11.0.0.Final
> Reporter: Marek Posolda
> Assignee: Paul Ferraro
>
> Use-case: We have openshift environment with RHSSO server connected to JDG server through HotRod protocol. RHSSO is connected to JDG through the RemoteStore . Configuration on RHSSO side is like this:
> Socket binding in standalone-ha.xml
> {code}
> <outbound-socket-binding name="remote-cache">
> <remote-destination host="jdg-app-hotrod.infinispan.svc" port="11222"/>
> </outbound-socket-binding>
> {code}
> And remote-store something like this:
> {code}
> <replicated-cache name="work" mode="SYNC">
> <remote-store cache="work" remote-servers="remote-cache" passivation="false" fetch-state="false" purge="false" preload="false" shared="true">
> </remote-store>
> </replicated-cache>
> {code}
> Let's assume that JDG needs to be restarted. This usually causes that service "jdg-app-hotrod.infinispan.svc" will be available under different IP address. For example previous IP of "jdg-app-hotrod.infinispan.svc" is "172.30.247.78" . After restart, it is changed to "172.30.28.27" .
> The issue is, that HotRod client won't be able to see this change and will still try to connect to old address.
> Why?: The org.jboss.as.clustering.infinispan.subsystem.RemoteStoreBuilder method "accept" always pass resolved address to infinispan: https://github.com/wildfly/wildfly/blob/master/clustering/infinispan/exte... . So infinispan will receive just static IP address.
> If there is an option for RemoteStoreBuilder to pass the "unresolved" hostname, it will help. Infinispan itself has support for "unresolved" dynamic hostnames thanks to the JIRA: ISPN-7955 , so it is already able to dynamically adapt to changed IP.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years