[JBoss JIRA] (ISPN-7092) Failed to broadcast asynchronous command
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-7092?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-7092:
----------------------------------
Fix Version/s: 10.0.1.Final
(was: 10.0.0.Final)
> Failed to broadcast asynchronous command
> ----------------------------------------
>
> Key: ISPN-7092
> URL: https://issues.jboss.org/browse/ISPN-7092
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.4.Final
> Reporter: Radoslav Husar
> Assignee: Dan Berindei
> Priority: Minor
> Labels: infinispan_clean_shutdown
> Fix For: 10.0.1.Final
>
>
> Occured on server at the end of server stopping.
> Scenarios affected:
> eap-7x-failover-ejb-ejbremote-jvmkill-dist-sync
> eap-7x-failover-ejb-ejbremote-jvmkill-repl-async
> eap-7x-failover-ejb-ejbremote-jvmkill-repl-sync
> eap-7x-failover-ejb-ejbremote-shutdown-dist-async
> eap-7x-failover-ejb-ejbremote-shutdown-dist-sync-tcpStack
> eap-7x-failover-ejb-ejbremote-shutdown-repl-sync
> eap-7x-failover-ejb-ejbremote-undeploy-repl-sync
> eap-7x-failover-ejb-ejbservlet-shutdown-repl-async
> eap-7x-failover-ejb-ejbstateless-jvmkill-repl-async
> eap-7x-failover-http-granular-shutdown-repl-sync
> eap-7x-failover-http-session-shutdown-repl-async
> eap-7x-failover-http-session-shutdown-repl-async-tcpStack
> eap-7x-failover-http-session-shutdown-repl-sync
> eap-7x-failover-jvmkill-ha-ss
> WARN message has two variants.
> *Variant 1*:
> {code}
> 15:01:15,157 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel web
> [JBossINF] [0m[0m15:01:15,157 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel web
> [JBossINF] [0m[0m15:01:15,158 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [JBossINF] [0m[33m15:01:15,158 WARN [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread--p7-t13) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=REBALANCE_CONFIRM, sender=dev214, joinInfo=null, topologyId=35, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=12}: org.infinispan.commons.CacheException: Failed to broadcast asynchronous command: CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=STABLE_TOPOLOGY_UPDATE, sender=dev213, joinInfo=null, topologyId=36, rebalanceId=17, currentCH=DefaultConsistentHash{ns=256, owners = (2)[dev214: 124+132, dev215: 132+124]}, pendingCH=null, availabilityMode=null, actualMembers=[dev214, dev215], throwable=null, viewId=12}
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:595)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastStableTopologyUpdate(ClusterTopologyManagerImpl.java:615)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.startQueuedRebalance(ClusterCacheStatus.java:633)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.endRebalance(ClusterCacheStatus.java:373)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.doConfirmRebalance(ClusterCacheStatus.java:310)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:263)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:176)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:153)
> [JBossINF] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:159)
> [JBossINF] at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:199)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: java.lang.IllegalStateException: channel is not connected
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:158)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:138)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:578)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:532)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:592)
> [JBossINF] ... 13 more
> [JBossINF] Caused by: java.lang.IllegalStateException: channel is not connected
> [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:682)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:172)
> [JBossINF] at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:325)
> [JBossINF] at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:76)
> [JBossINF] at org.jgroups.blocks.Request.execute(Request.java:67)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:373)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:386)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:407)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:133)
> [JBossINF] ... 16 more
> {code}
> *Variant 2*:
> {code}
> 15:53:27,821 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t19) ISPN000336: Finished cluster-wide rebalance for cache routing, topology id = 33
> [JBossINF] [0m[0m15:53:27,829 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> [JBossINF] [0m[0m15:53:27,830 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel web
> [JBossINF] [0m[0m15:53:27,830 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel web
> [JBossINF] [0m[0m15:53:27,821 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t14) ISPN000336: Finished cluster-wide rebalance for cache clusterbench-ee7.ear.clusterbench-ee7-web-default.war, topology id = 39
> [JBossINF] [0m[0m15:53:27,834 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [JBossINF] [0m[33m15:53:27,832 WARN [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread--p6-t14) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=REBALANCE_CONFIRM, sender=dev215, joinInfo=null, topologyId=39, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=12}: org.infinispan.commons.CacheException: Failed to broadcast asynchronous command: CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=CH_UPDATE, sender=dev213, joinInfo=null, topologyId=40, rebalanceId=21, currentCH=DefaultConsistentHash{ns=256, owners = (2)[dev214: 129+127, dev215: 127+129]}, pendingCH=null, availabilityMode=null, actualMembers=[dev214, dev215], throwable=null, viewId=12}
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:595)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastTopologyUpdate(ClusterTopologyManagerImpl.java:606)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.endRebalance(ClusterCacheStatus.java:369)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.doConfirmRebalance(ClusterCacheStatus.java:310)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:263)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:176)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:153)
> [JBossINF] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:159)
> [JBossINF] at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:199)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: java.lang.NullPointerException
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:611)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:532)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:592)
> [JBossINF] ... 12 more
> {code}
> Link to server log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months
[JBoss JIRA] (ISPN-7139) Consistent prefix in property names of Hot Rod client configurations
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-7139?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-7139:
----------------------------------
Fix Version/s: 10.0.1.Final
(was: 10.0.0.Final)
> Consistent prefix in property names of Hot Rod client configurations
> --------------------------------------------------------------------
>
> Key: ISPN-7139
> URL: https://issues.jboss.org/browse/ISPN-7139
> Project: Infinispan
> Issue Type: Enhancement
> Components: Configuration, Remote Protocols
> Reporter: Sanne GRINOVERO
> Priority: Major
> Fix For: 10.0.1.Final
>
>
> Most Hot Rod configuration properties use the prefix {{infinispan.client.hotrod.}} which makes it easy to recognize them.
> Some properties don't use the prefix though, such as all those related to connection pooling, e.g. {{maxActive}}, {{maxTotal}}, ..
> This makes it hard to read Hot Rod specific configuration properties from alternative sources. In particular we'd like to embed such properties in an Hibernate configuration file (for Hibernate OGM) but the fact that some properties need special handling forces us to keep a list of these to have special care, rather than being able to use a prefix like it is done in most such cases.
> It should be easy to prefix all the properties, and deprecate the names.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months
[JBoss JIRA] (ISPN-7159) CompleteShutdownDistRetryTest.testRetryAfterCompleteShutdown random failures
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-7159?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-7159:
----------------------------------
Fix Version/s: 10.0.1.Final
(was: 10.0.0.Final)
> CompleteShutdownDistRetryTest.testRetryAfterCompleteShutdown random failures
> ----------------------------------------------------------------------------
>
> Key: ISPN-7159
> URL: https://issues.jboss.org/browse/ISPN-7159
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Server
> Affects Versions: 9.0.0.Alpha4
> Reporter: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 10.0.1.Final
>
>
> {noformat}
> java.lang.AssertionError: expected:<two> but was:<null>
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:101)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:108)
> at org.infinispan.client.hotrod.retry.CompleteShutdownDistRetryTest.testRetryAfterCompleteShutdown(CompleteShutdownDistRetryTest.java:58)
> {noformat}
> I think the problem is that killing node 0 can make node 2's key move from node 2 to node 1, and then it's lost after node 1 is also killed.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months
[JBoss JIRA] (ISPN-7167) ExampleConfigsIT.testTopologyConfigRackAttribute random failures
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-7167?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-7167:
----------------------------------
Fix Version/s: 10.0.1.Final
(was: 10.0.0.Final)
> ExampleConfigsIT.testTopologyConfigRackAttribute random failures
> -----------------------------------------------------------------
>
> Key: ISPN-7167
> URL: https://issues.jboss.org/browse/ISPN-7167
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Server
> Affects Versions: 9.0.0.Alpha4
> Reporter: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 10.0.1.Final
>
>
> {noformat}
> java.lang.AssertionError: Unexpected number of entries in server1: 0
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.infinispan.server.test.configs.ExampleConfigsIT.verifyTopologyHinting(ExampleConfigsIT.java:482)
> at org.infinispan.server.test.configs.ExampleConfigsIT.testTopologyConfigRackAttribute(ExampleConfigsIT.java:419)
> {noformat}
> The test is also inconsistent about the names of the servers, sometimes it starts counting from 0 and sometimes it starts counting from 1.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months
[JBoss JIRA] (ISPN-7302) ClusteredConditionalCommandTest.testPutIfAbsentOnNonOwnerWithIgnoreReturnValues random failures
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-7302?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-7302:
----------------------------------
Fix Version/s: 10.0.1.Final
(was: 10.0.0.Final)
> ClusteredConditionalCommandTest.testPutIfAbsentOnNonOwnerWithIgnoreReturnValues random failures
> -----------------------------------------------------------------------------------------------
>
> Key: ISPN-7302
> URL: https://issues.jboss.org/browse/ISPN-7302
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 9.0.0.Beta1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 10.0.1.Final
>
>
> When the backup owner replies before the primary had a chance to load the value from persistence, the number of loads from the primary can be {{0}} at the time the test checks it.
> {noformat}
> 00:21:48,941 TRACE (testng-Test:[]) [JGroupsTransport] dests=[Test-NodeB-56150, Test-NodeA-13512], command=ClusteredGetCommand{key=Test-key, flags=[IGNORE_RETURN_VALUES]}, mode=WAIT_FOR_VALID_RESPONSE, timeout=15000
> 00:21:48,942 TRACE (jgroups-7,Test-NodeB-56150:[]) [InvocationContextInterceptor] Invoked with command GetCacheEntryCommand {key=Test-key, flags=[SKIP_REMOTE_LOOKUP, IGNORE_RETURN_VALUES]} and InvocationContext [org.infinispan.context.impl.NonTxInvocationContext@5f931934]
> 00:21:48,943 TRACE (jgroups-7,Test-NodeB-56150:[]) [EntryFactoryImpl] Wrap Test-key for read. Entry=NullCacheEntry{}
> 00:21:48,946 TRACE (jgroups-8,Test-NodeA-13512:[]) [InvocationContextInterceptor] Invoked with command GetCacheEntryCommand {key=Test-key, flags=[SKIP_REMOTE_LOOKUP, IGNORE_RETURN_VALUES]} and InvocationContext [org.infinispan.context.impl.NonTxInvocationContext@265c6c97]
> 00:21:48,946 TRACE (jgroups-8,Test-NodeA-13512:[]) [PersistenceUtil] Loaded MarshalledEntryImpl{keyBytes=null, valueBytes=null, metadataBytes=null, key=Test-key, value=Test-value1, metadata=null, marshaller=org.infinispan.marshall.core.GlobalMarshaller@1aee2408} for key Test-key from persistence.
> 00:21:48,946 TRACE (jgroups-8,Test-NodeA-13512:[]) [CommandAwareRpcDispatcher] About to send back response SuccessfulResponse{responseValue=ImmortalCacheValue {value=Test-value1}} for command ClusteredGetCommand{key=Test-key, flags=[IGNORE_RETURN_VALUES]}
> 00:21:48,948 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.persistence.ClusteredTxConditionalCommandTest.testPutIfAbsentOnNonOwnerWithIgnoreReturnValues[tx=true]
> java.lang.AssertionError: primary owner load expected:<1> but was:<0>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59) ~[testng-6.8.8.jar:?]
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364) ~[testng-6.8.8.jar:?]
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80) ~[testng-6.8.8.jar:?]
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:170) ~[testng-6.8.8.jar:?]
> at org.infinispan.persistence.ClusteredTxConditionalCommandTest.assertLoadAfterOperation(ClusteredTxConditionalCommandTest.java:46) ~[test-classes/:?]
> at org.infinispan.persistence.ClusteredConditionalCommandTest.doTest(ClusteredConditionalCommandTest.java:121) ~[test-classes/:?]
> at org.infinispan.persistence.ClusteredConditionalCommandTest.testPutIfAbsentOnNonOwnerWithIgnoreReturnValues(ClusteredConditionalCommandTest.java:242) ~[test-classes/:?]
> 00:21:48,953 TRACE (jgroups-7,Test-NodeB-56150:[]) [PersistenceUtil] Loaded null for key Test-key from persistence.
> 00:21:48,953 TRACE (jgroups-7,Test-NodeB-56150:[]) [CommandAwareRpcDispatcher] About to send back response SuccessfulResponse{responseValue=null} for command ClusteredGetCommand{key=Test-key, flags=[IGNORE_RETURN_VALUES]}
> {noformat}
> Possible fix: replace {{assertEquals}} with {{eventuallyEquals}}.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months