[JBoss JIRA] (ISPN-7092) Failed to broadcast asynchronous command
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-7092?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-7092:
----------------------------------
Fix Version/s: 10.0.0.Beta5
(was: 10.0.0.Beta4)
> Failed to broadcast asynchronous command
> ----------------------------------------
>
> Key: ISPN-7092
> URL: https://issues.jboss.org/browse/ISPN-7092
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.4.Final
> Reporter: Radoslav Husar
> Assignee: Dan Berindei
> Priority: Minor
> Labels: infinispan_clean_shutdown
> Fix For: 10.0.0.Beta5
>
>
> Occured on server at the end of server stopping.
> Scenarios affected:
> eap-7x-failover-ejb-ejbremote-jvmkill-dist-sync
> eap-7x-failover-ejb-ejbremote-jvmkill-repl-async
> eap-7x-failover-ejb-ejbremote-jvmkill-repl-sync
> eap-7x-failover-ejb-ejbremote-shutdown-dist-async
> eap-7x-failover-ejb-ejbremote-shutdown-dist-sync-tcpStack
> eap-7x-failover-ejb-ejbremote-shutdown-repl-sync
> eap-7x-failover-ejb-ejbremote-undeploy-repl-sync
> eap-7x-failover-ejb-ejbservlet-shutdown-repl-async
> eap-7x-failover-ejb-ejbstateless-jvmkill-repl-async
> eap-7x-failover-http-granular-shutdown-repl-sync
> eap-7x-failover-http-session-shutdown-repl-async
> eap-7x-failover-http-session-shutdown-repl-async-tcpStack
> eap-7x-failover-http-session-shutdown-repl-sync
> eap-7x-failover-jvmkill-ha-ss
> WARN message has two variants.
> *Variant 1*:
> {code}
> 15:01:15,157 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel web
> [JBossINF] [0m[0m15:01:15,157 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel web
> [JBossINF] [0m[0m15:01:15,158 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [JBossINF] [0m[33m15:01:15,158 WARN [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread--p7-t13) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=REBALANCE_CONFIRM, sender=dev214, joinInfo=null, topologyId=35, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=12}: org.infinispan.commons.CacheException: Failed to broadcast asynchronous command: CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=STABLE_TOPOLOGY_UPDATE, sender=dev213, joinInfo=null, topologyId=36, rebalanceId=17, currentCH=DefaultConsistentHash{ns=256, owners = (2)[dev214: 124+132, dev215: 132+124]}, pendingCH=null, availabilityMode=null, actualMembers=[dev214, dev215], throwable=null, viewId=12}
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:595)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastStableTopologyUpdate(ClusterTopologyManagerImpl.java:615)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.startQueuedRebalance(ClusterCacheStatus.java:633)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.endRebalance(ClusterCacheStatus.java:373)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.doConfirmRebalance(ClusterCacheStatus.java:310)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:263)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:176)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:153)
> [JBossINF] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:159)
> [JBossINF] at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:199)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: java.lang.IllegalStateException: channel is not connected
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:158)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:138)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:578)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:532)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:592)
> [JBossINF] ... 13 more
> [JBossINF] Caused by: java.lang.IllegalStateException: channel is not connected
> [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:682)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:172)
> [JBossINF] at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:325)
> [JBossINF] at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:76)
> [JBossINF] at org.jgroups.blocks.Request.execute(Request.java:67)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:373)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:386)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:407)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:133)
> [JBossINF] ... 16 more
> {code}
> *Variant 2*:
> {code}
> 15:53:27,821 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t19) ISPN000336: Finished cluster-wide rebalance for cache routing, topology id = 33
> [JBossINF] [0m[0m15:53:27,829 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> [JBossINF] [0m[0m15:53:27,830 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel web
> [JBossINF] [0m[0m15:53:27,830 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel web
> [JBossINF] [0m[0m15:53:27,821 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t14) ISPN000336: Finished cluster-wide rebalance for cache clusterbench-ee7.ear.clusterbench-ee7-web-default.war, topology id = 39
> [JBossINF] [0m[0m15:53:27,834 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [JBossINF] [0m[33m15:53:27,832 WARN [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread--p6-t14) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=REBALANCE_CONFIRM, sender=dev215, joinInfo=null, topologyId=39, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=12}: org.infinispan.commons.CacheException: Failed to broadcast asynchronous command: CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=CH_UPDATE, sender=dev213, joinInfo=null, topologyId=40, rebalanceId=21, currentCH=DefaultConsistentHash{ns=256, owners = (2)[dev214: 129+127, dev215: 127+129]}, pendingCH=null, availabilityMode=null, actualMembers=[dev214, dev215], throwable=null, viewId=12}
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:595)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastTopologyUpdate(ClusterTopologyManagerImpl.java:606)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.endRebalance(ClusterCacheStatus.java:369)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.doConfirmRebalance(ClusterCacheStatus.java:310)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:263)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:176)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:153)
> [JBossINF] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:159)
> [JBossINF] at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:199)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: java.lang.NullPointerException
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:611)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:532)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:592)
> [JBossINF] ... 12 more
> {code}
> Link to server log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months
[JBoss JIRA] (ISPN-10379) JCache tck-runner test failures are ignored in Jenkins
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-10379?page=com.atlassian.jira.plugin... ]
Tristan Tarrant updated ISPN-10379:
-----------------------------------
Fix Version/s: 10.0.0.Beta5
(was: 10.0.0.Beta4)
> JCache tck-runner test failures are ignored in Jenkins
> ------------------------------------------------------
>
> Key: ISPN-10379
> URL: https://issues.jboss.org/browse/ISPN-10379
> Project: Infinispan
> Issue Type: Bug
> Components: Build, Test Suite - Core
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta5
>
>
> This build has 0 failures in Jenkins, but the console log says otherwise:
> https://ci.infinispan.org/job/Infinispan/job/PR-7110/1/
> {noformat}
> [2019-07-02T18:45:17.550Z] 18:45:16.738 [INFO] --- maven-failsafe-plugin:3.0.0-M3:integration-test (tck-embedded) @ infinispan-jcache-tck-runner ---
> [2019-07-02T18:45:17.550Z] 18:45:16.750 [INFO] Failsafe report directory: /home/infinispan/workspace/Infinispan_PR-7110/jcache/tck-runner/target/failsafe-reports-embedded
> ...
> [2019-07-02T18:45:53.348Z] 18:45:53.137 [ERROR] CacheExpiryTest.invokeAllReadThroughEnabledGetOnNonExistentEntry:1202
> [2019-07-02T18:45:53.348Z] Expected: is <0>
> [2019-07-02T18:45:53.348Z] but: was <5>
> [2019-07-02T18:45:53.348Z] 18:45:53.137 [ERROR] CacheExpiryTest.invokeGetValueWithReadThroughForNonExistentEntryShouldCallGetExpiryForCreatedEntry:1110
> [2019-07-02T18:45:53.348Z] Expected: is <0>
> [2019-07-02T18:45:53.348Z] but: was <1>
> [2019-07-02T18:45:53.348Z] 18:45:53.138 [ERROR] CacheWriterTest.shouldWriteThoughUsingPutAll_partialSuccess:688 expected:<2> but was:<3>
> [2019-07-02T18:45:53.348Z] 18:45:53.138 [INFO]
> [2019-07-02T18:45:53.348Z] 18:45:53.138 [ERROR] Tests run: 491, Failures: 3, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months
[JBoss JIRA] (ISPN-10369) InvalidatedNearCacheTest random failures
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-10369?page=com.atlassian.jira.plugin... ]
Tristan Tarrant updated ISPN-10369:
-----------------------------------
Fix Version/s: 10.0.0.Beta5
(was: 10.0.0.Beta4)
> InvalidatedNearCacheTest random failures
> ----------------------------------------
>
> Key: ISPN-10369
> URL: https://issues.jboss.org/browse/ISPN-10369
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Server
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta5
>
>
> {{InvalidatedNearCacheTest.testGetNearCacheAfterConnect}} has 2 clients, performing some operations and waiting for the corresponding near cache events on client 1 and then resetting the events on client 2.
> This is not correct, because client 2 may not have received the events yet, in which case {{resetEvents()}} doesn't do anything.
> {noformat}
> 18:48:47,813 INFO (testng-InvalidatedNearCacheTest:[]) [TestSuiteProgress] Test succeeded: org.infinispan.client.hotrod.near.InvalidatedNearCacheTest.testGetNearCacheAfterConnect
> 18:48:47,814 INFO (testng-InvalidatedNearCacheTest:[]) [TestSuiteProgress] Test starting: org.infinispan.client.hotrod.near.InvalidatedNearCacheTest.testGetUpdatesNearCache
> 18:48:47,821 ERROR (testng-InvalidatedNearCacheTest:[]) [TestSuiteProgress] Test failed: org.infinispan.client.hotrod.near.InvalidatedNearCacheTest.testGetUpdatesNearCache
> 18:48:47,821 ERROR (testng-InvalidatedNearCacheTest:[]) [TestSuiteProgress] Test failed: org.infinispan.client.hotrod.near.InvalidatedNearCacheTest.testGetUpdatesNearCache
> java.lang.AssertionError: [org.infinispan.client.hotrod.near.MockNearCacheService$MockRemoveEvent@514f3f0c, org.infinispan.client.hotrod.near.MockNearCacheService$MockRemoveEvent@76bf7455] expected:<0> but was:<2>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59) ~[testng-6.14.3.jar:?]
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364) ~[testng-6.14.3.jar:?]
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80) ~[testng-6.14.3.jar:?]
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245) ~[testng-6.14.3.jar:?]
> at org.infinispan.client.hotrod.near.AssertsNearCache.expectNoNearEvents(AssertsNearCache.java:123) ~[test-classes/:?]
> at org.infinispan.client.hotrod.near.InvalidatedNearCacheTest.testGetUpdatesNearCache(InvalidatedNearCacheTest.java:153) ~[test-classes/:?]
> {noformat}
> https://ci.infinispan.org/job/Infinispan/job/master/1240/
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months
[JBoss JIRA] (ISPN-10366) ScatteredStateConsumerImpl sets segment state to OWNED before applying values
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-10366?page=com.atlassian.jira.plugin... ]
Tristan Tarrant updated ISPN-10366:
-----------------------------------
Fix Version/s: 10.0.0.Beta5
(was: 10.0.0.Beta4)
> ScatteredStateConsumerImpl sets segment state to OWNED before applying values
> -----------------------------------------------------------------------------
>
> Key: ISPN-10366
> URL: https://issues.jboss.org/browse/ISPN-10366
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 10.0.0.Beta3, 9.4.15.Final
> Reporter: Dan Berindei
> Assignee: Radim Vansa
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta5
>
> Attachments: ISPN-10363_LazyInitializingExecutorService_94x_20190627-2010_PrefetchTest-infinispan-core.log.gz
>
>
> {{ScatteredStateConsumerImpl}} uses {{InboundTransferTask}} only to request keys, then after it received all the keys of a segment it changes the segment state to {{VALUE_TRANSFER}} and starts an asynchronous request to fetch the values and replace the {{RemoteMetadata}} entries with real entries.
> {{ScatteredStateConsumerImpl.chunkCounter}} is supposed to delay the state transfer end and the segment state change to {{OWNED}}, but in rare occasions this doesn't happen.
> This happened in {{PrefetchTest.testPrefetch12}} while running the test suite with {{taskset -c 1-2}}:
> {noformat}
> 21:54:43,304 TRACE (transport-thread-Test-NodeC-p69907-t5:[Topology-___defaultcache]) [StateConsumerImpl] Received new topology for cache ___defaultcache, isRebalance = true, isMember = true, topology = CacheTopology{id=9, phase=TRANSITORY, rebalanceId=5, currentCH=PartitionerConsistentHash:ScatteredConsistentHash{ns=1, rebalanced=false, owners = (2)[Test-NodeA-39104: 1, Test-NodeC-3746: 0]}, pendingCH=PartitionerConsistentHash:ScatteredConsistentHash{ns=1, rebalanced=true, owners = (2)[Test-NodeA-39104: 0, Test-NodeC-3746: 1]}, unionCH=PartitionerConsistentHash:ScatteredConsistentHash{ns=1, rebalanced=false, owners = (2)[Test-NodeA-39104: 0, Test-NodeC-3746: 1]}, actualMembers=[Test-NodeA-39104, Test-NodeC-3746], persistentUUIDs=[f58e0a9a-dd4e-429a-8464-da64bf001d4e, 1471096f-c59a-4dc9-8f4d-31fbf399a2aa]}
> 21:54:43,305 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[StateRequest-___defaultcache]) [ScatteredStateConsumerImpl] Requesting keys for segments {0} from Test-NodeA-39104
> 21:54:43,313 TRACE (transport-thread-Test-NodeC-p69907-t5:[Topology-___defaultcache]) [StateConsumerImpl] Topology update processed, stateTransferTopologyId = 9, startRebalance = true, pending CH = PartitionerConsistentHash:ScatteredConsistentHash{ns=1, rebalanced=true, owners = (2)[Test-NodeA-39104: 0, Test-NodeC-3746: 1]}
> 21:54:43,313 TRACE (transport-thread-Test-NodeC-p69907-t5:[Topology-___defaultcache]) [StateTransferLockImpl] Signalling transaction data received for topology 9
> 21:54:43,313 TRACE (remote-thread-Test-NodeC-p69905-t2:[]) [TrianglePerCacheInboundInvocationHandler] Calling perform() on StateResponseCommand{cache=___defaultcache, pushTransfer=false, stateChunks=[StateChunk{segmentId=0, cacheEntries=1, isLastChunk=true}], origin=Test-NodeA-39104, topologyId=9, applyState=true}
> 21:54:43,313 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [StateConsumerImpl] Applying new state chunk for segment 0 of cache ___defaultcache from node Test-NodeA-39104: received 1 cache entries
> 21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredVersionManagerImpl] Finished transfer for segment 0 = KEY_TRANSFER -> VALUE_TRANSFER
> 21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredVersionManagerImpl] Node Test-NodeC-3746, segment 0 has all keys in, expects value transfer
> 21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredStateConsumerImpl] Requesting values from segments {0}, for in-memory keys
> 21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredStateConsumerImpl] Retrieving values, chunk counter is 1
> 21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [JGroupsTransport] Test-NodeC-3746 sending request 11 to Test-NodeA-39104: ClusteredGetAllCommand{keys=[key], flags=[SKIP_OWNERSHIP_CHECK], topologyId=9}
> 21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredStateConsumerImpl] Invalidating versions on Test-NodeC-3746, chunk counter incremented to 2
> 21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredStateConsumerImpl] Versions invalidated on Test-NodeC-3746, chunk counter decremented to 1
> 21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [StateConsumerImpl] Removing inbound transfers from node {0} for segments Test-NodeA-39104
> 21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [ScatteredStateConsumerImpl] Inbound transfer removed, chunk counter is 1
> 21:54:43,314 TRACE (stateTransferExecutor-thread-Test-NodeC-p69908-t6:[]) [StateConsumerImpl] Latch 0
> 21:54:43,315 TRACE (jgroups-7,Test-NodeC-3746:[]) [JGroupsTransport] Test-NodeC-3746 received response for request 11 from Test-NodeA-39104: SuccessfulResponse([MetadataImmortalCacheValue {value=v0, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=SimpleClusteredVersion{topologyId=7, version=1}}}])
> 21:54:43,316 TRACE (jgroups-7,Test-NodeC-3746:[]) [BlockingInterceptor] Command blocking before completion of PutKeyValueCommand{key=key, value=v0, flags=[CACHE_MODE_LOCAL, SKIP_REMOTE_LOOKUP, PUT_FOR_STATE_TRANSFER, SKIP_SHARED_CACHE_STORE, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], commandInvocationId=CommandInvocation:Test-NodeC-3746:121294, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=InternalMetadataImpl{actual=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=SimpleClusteredVersion{topologyId=7, version=1}}, created=-1, lastUsed=-1}, successful=true, topologyId=-1}
> 21:54:43,316 TRACE (remote-thread-Test-NodeC-p69905-t2:[___defaultcache]) [StateConsumerImpl] After applying the received state the data container of cache ___defaultcache has 1 keys
> 21:54:43,316 TRACE (remote-thread-Test-NodeC-p69905-t2:[___defaultcache]) [StateConsumerImpl] Segments not received yet for cache ___defaultcache: {}
> 21:54:43,316 DEBUG (transport-thread-Test-NodeC-p69907-t5:[Topology-___defaultcache]) [StateConsumerImpl] Finished receiving of segments for cache ___defaultcache for topology 9.
> 21:54:43,316 DEBUG (transport-thread-Test-NodeC-p69907-t5:[Topology-___defaultcache]) [ScatteredVersionManagerImpl] Node Test-NodeC-3746 received values for all segments in topology 9
> 21:54:43,316 TRACE (transport-thread-Test-NodeC-p69907-t5:[Topology-___defaultcache]) [StateConsumerImpl] Stop keeping track of changed keys for state transfer in topology 9
> {noformat}
> The test then starts a put operation and expects it to prefetch the previous value, but because the segment is {{OWNED}}, the {{RemoteMetadata}} is ignored:
> {noformat}
> 21:54:43,316 TRACE (ForkThread-1,Test:[]) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=key, value=v1, flags=[], commandInvocationId=CommandInvocation:Test-NodeC-3746:121295, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=-1} and InvocationContext [SingleKeyNonTxInvocationContext{isLocked=false, key=null, cacheEntry=null, origin=null, lockOwner=CommandInvocation:Test-NodeC-3746:121295}]
> 21:54:43,316 TRACE (ForkThread-1,Test:[]) [EntryFactoryImpl] Retrieved from container MetadataImmortalCacheEntry{key=key, value=null, metadata=RemoteMetadata{address=Test-NodeA-39104, version=1}}
> 21:54:43,316 TRACE (ForkThread-1,Test:[]) [ScatteredDistributionInterceptor] Committing entry RepeatableReadEntry(108d175b){key=key, value=v1, isCreated=false, isChanged=true, isRemoved=false, isExpired=false, skipLookup=true, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=SimpleClusteredVersion{topologyId=9, version=1}}}, replaced MetadataImmortalCacheEntry{key=key, value=null, metadata=RemoteMetadata{address=Test-NodeA-39104, version=1}}
> 21:54:53,316 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.scattered.statetransfer.PrefetchTest.testPrefetch12
> org.infinispan.test.TestException: java.util.concurrent.TimeoutException
> at org.infinispan.util.ControlledRpcManager.uncheckedGet(ControlledRpcManager.java:259) ~[test-classes/:?]
> at org.infinispan.util.ControlledRpcManager.expectCommand(ControlledRpcManager.java:124) ~[test-classes/:?]
> at org.infinispan.scattered.statetransfer.PrefetchTest.testPrefetch(PrefetchTest.java:110) ~[test-classes/:?]
> at org.infinispan.scattered.statetransfer.PrefetchTest.testPrefetch12(PrefetchTest.java:67) ~[test-classes/:?]
> {noformat}
> On a related note, {{StateConsumerImpl.applyState(pushTransfer=true)}} initializes a {{CountDownLatch(stateChunks.size())}}, but doesn't actually count down if {{stateChunk.getCacheEntries() == null}}, potentially hanging state transfer until it times out.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months
[JBoss JIRA] (ISPN-10365) PreferAvailabilityStrategy assertion failure
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-10365?page=com.atlassian.jira.plugin... ]
Tristan Tarrant updated ISPN-10365:
-----------------------------------
Fix Version/s: 10.0.0.Beta5
(was: 10.0.0.Beta4)
> PreferAvailabilityStrategy assertion failure
> --------------------------------------------
>
> Key: ISPN-10365
> URL: https://issues.jboss.org/browse/ISPN-10365
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 10.0.0.Beta3, 9.4.15.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.Beta5, 9.4.16.Final
>
>
> This scenario happens unintentionally in {{RebalancePolicyJmxTest}}, because the test waits for the default cache to finish rebalancing before killing the coordinator but doesn't care about the {{CONFIG}} cache:
> * A and B are running, rebalancing is disabled, then C and D join
> * Re-enable rebalance, but stop B and A before the rebalance is done
> * C sees the finished rebalance, D sees the READ_OLD phase
> * C becomes coordinator and should recover with C's topology, but instead has an assertion failure and doesn't install a stable topology
> {noformat}
> 16:48:48,454 TRACE (stateTransferExecutor-thread-Test-NodeC-p49651-t6:[Merge-5]) [PreferAvailabilityStrategy] Cache org.infinispan.CONFIG keeping partition from [Test-NodeC-27509(rack-id=r2)]: CacheTopology{id=9, phase=NO_REBALANCE, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeC-27509(rack-id=r2): 125, Test-NodeD-62603(rack-id=r2): 131]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeC-27509(rack-id=r2), Test-NodeD-62603(rack-id=r2)], persistentUUIDs=[59d0898c-f166-4129-9165-a22aca475286, 05d7dd0b-7cd8-464d-8adb-41fac100e8bf]}
> 16:48:48,454 TRACE (stateTransferExecutor-thread-Test-NodeC-p49651-t6:[Merge-5]) [PreferAvailabilityStrategy] Cache org.infinispan.CONFIG keeping partition from [Test-NodeD-62603(rack-id=r2)]: CacheTopology{id=6, phase=READ_OLD_WRITE_ALL, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeA-4515(rack-id=r1): 127, Test-NodeB-42590(rack-id=r1): 129]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (4)[Test-NodeA-4515(rack-id=r1): 63, Test-NodeB-42590(rack-id=r1): 62, Test-NodeC-27509(rack-id=r2): 64, Test-NodeD-62603(rack-id=r2): 67]}, unionCH=null, actualMembers=[Test-NodeA-4515(rack-id=r1), Test-NodeB-42590(rack-id=r1), Test-NodeC-27509(rack-id=r2), Test-NodeD-62603(rack-id=r2)], persistentUUIDs=[e9dcc3da-07a2-4159-a8b1-94e6428011c4, 3f27ddaa-1146-483e-8473-d79a5ba347f5, 59d0898c-f166-4129-9165-a22aca475286, 05d7dd0b-7cd8-464d-8adb-41fac100e8bf]}
> 16:48:48,454 TRACE (stateTransferExecutor-thread-Test-NodeC-p49651-t6:[Merge-5]) [PreferAvailabilityStrategy] Cache org.infinispan.CONFIG, resolveConflicts=false, newMembers=[Test-NodeC-27509(rack-id=r2), Test-NodeD-62603(rack-id=r2)], possibleOwners=[Test-NodeD-62603(rack-id=r2), Test-NodeC-27509(rack-id=r2)], preferredTopology=CacheTopology{id=6, phase=READ_OLD_WRITE_ALL, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeA-4515(rack-id=r1): 127, Test-NodeB-42590(rack-id=r1): 129]}, pendingCH=ReplicatedConsistentHash{ns = 256, owners = (4)[Test-NodeA-4515(rack-id=r1): 63, Test-NodeB-42590(rack-id=r1): 62, Test-NodeC-27509(rack-id=r2): 64, Test-NodeD-62603(rack-id=r2): 67]}, unionCH=null, actualMembers=[Test-NodeA-4515(rack-id=r1), Test-NodeB-42590(rack-id=r1), Test-NodeC-27509(rack-id=r2), Test-NodeD-62603(rack-id=r2)], persistentUUIDs=[e9dcc3da-07a2-4159-a8b1-94e6428011c4, 3f27ddaa-1146-483e-8473-d79a5ba347f5, 59d0898c-f166-4129-9165-a22aca475286, 05d7dd0b-7cd8-464d-8adb-41fac100e8bf]}, mergeTopologyId=10
> 16:48:48,454 WARN (stateTransferExecutor-thread-Test-NodeC-p49651-t6:[Merge-5]) [PreferAvailabilityStrategy] ISPN000517: Ignoring cache topology from [Test-NodeC-27509(rack-id=r2)] during merge: CacheTopology{id=9, phase=NO_REBALANCE, rebalanceId=3, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeC-27509(rack-id=r2): 125, Test-NodeD-62603(rack-id=r2): 131]}, pendingCH=null, unionCH=null, actualMembers=[Test-NodeC-27509(rack-id=r2), Test-NodeD-62603(rack-id=r2)], persistentUUIDs=[59d0898c-f166-4129-9165-a22aca475286, 05d7dd0b-7cd8-464d-8adb-41fac100e8bf]}
> 16:48:48,454 DEBUG (stateTransferExecutor-thread-Test-NodeC-p49651-t6:[Merge-5]) [CLUSTER] ISPN000521: Cache org.infinispan.CONFIG recovered after merge with topology = CacheTopology{id=10, phase=NO_REBALANCE, rebalanceId=4, currentCH=ReplicatedConsistentHash{ns = 256, owners = (2)[Test-NodeA-4515(rack-id=r1): 127, Test-NodeB-42590(rack-id=r1): 129]}, pendingCH=null, unionCH=null, actualMembers=[], persistentUUIDs=[]}, availability mode null
> 16:48:48,454 FATAL (stateTransferExecutor-thread-Test-NodeC-p49651-t6:[Merge-5]) [CLUSTER] [Context=org.infinispan.CONFIG] ISPN000313: Lost data because of abrupt leavers [Test-NodeA-4515(rack-id=r1), Test-NodeB-42590(rack-id=r1), Test-NodeC-27509(rack-id=r2), Test-NodeD-62603(rack-id=r2)]
> 16:48:48,455 ERROR (stateTransferExecutor-thread-Test-NodeC-p49651-t6:[Merge-5]) [LimitedExecutor] Exception in task
> java.lang.AssertionError: null
> at org.infinispan.partitionhandling.impl.PreferAvailabilityStrategy.onPartitionMerge(PreferAvailabilityStrategy.java:217) ~[classes/:?]
> at org.infinispan.topology.ClusterCacheStatus.doMergePartitions(ClusterCacheStatus.java:647) ~[classes/:?]
> at org.infinispan.topology.ClusterTopologyManagerImpl.lambda$recoverClusterStatus$4(ClusterTopologyManagerImpl.java:500) ~[classes/:?]
> at org.infinispan.executors.LimitedExecutor.runTasks(LimitedExecutor.java:175) [classes/:?]
> {noformat}
> Eventually the missing stable topology makes the test fail:
> {noformat}
> 16:48:49,349 DEBUG (testng-Test:[null]) [ClusterCacheStatus] ISPN000519: Updating stable topology for cache org.infinispan.CONFIG, topology null
> 16:48:49,349 WARN (testng-Test:[null]) [CacheTopologyControlCommand] ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=null, type=POLICY_ENABLE, sender=Test-NodeC-27509(rack-id=r2), joinInfo=null, topologyId=0, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, phase=null, actualMembers=null, throwable=null, viewId=5}
> java.lang.NullPointerException: null
> at org.infinispan.topology.CacheTopologyControlCommand.<init>(CacheTopologyControlCommand.java:147) ~[classes/:?]
> at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastStableTopologyUpdate(ClusterTopologyManagerImpl.java:659) ~[classes/:?]
> at org.infinispan.topology.ClusterCacheStatus.startQueuedRebalance(ClusterCacheStatus.java:806) ~[classes/:?]
> at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4772) ~[?:?]
> at org.infinispan.topology.ClusterTopologyManagerImpl.setRebalancingEnabled(ClusterTopologyManagerImpl.java:702) ~[classes/:?]
> at org.infinispan.topology.ClusterTopologyManagerImpl.setRebalancingEnabled(ClusterTopologyManagerImpl.java:682) ~[classes/:?]
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:215) ~[classes/:?]
> at org.infinispan.topology.CacheTopologyControlCommand.invokeAsync(CacheTopologyControlCommand.java:163) [classes/:?]
> at org.infinispan.commands.ReplicableCommand.invoke(ReplicableCommand.java:44) [classes/:?]
> at org.infinispan.topology.LocalTopologyManagerImpl.executeOnClusterSync(LocalTopologyManagerImpl.java:752) [classes/:?]
> at org.infinispan.topology.LocalTopologyManagerImpl.setCacheRebalancingEnabled(LocalTopologyManagerImpl.java:623) [classes/:?]
> at org.infinispan.topology.LocalTopologyManagerImpl.setRebalancingEnabled(LocalTopologyManagerImpl.java:581) [classes/:?]
> 16:48:49,355 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.statetransfer.RebalancePolicyJmxTest.testJoinAndLeaveWithRebalanceSuspendedAwaitingInitialTransfer[DIST_SYNC]
> javax.management.MBeanException: Error invoking setter for attribute rebalancingEnabled
> at org.infinispan.jmx.ResourceDMBean.setNamedAttribute(ResourceDMBean.java:358) ~[classes/:?]
> at org.infinispan.jmx.ResourceDMBean.setAttribute(ResourceDMBean.java:216) ~[classes/:?]
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.setAttribute(DefaultMBeanServerInterceptor.java:736) ~[?:?]
> at com.sun.jmx.mbeanserver.JmxMBeanServer.setAttribute(JmxMBeanServer.java:739) ~[?:?]
> at org.infinispan.statetransfer.RebalancePolicyJmxTest.doTest(RebalancePolicyJmxTest.java:163) ~[test-classes/:?]
> at org.infinispan.statetransfer.RebalancePolicyJmxTest.testJoinAndLeaveWithRebalanceSuspendedAwaitingInitialTransfer(RebalancePolicyJmxTest.java:44) ~[test-classes/:?]
> Caused by: java.lang.reflect.InvocationTargetException
> at jdk.internal.reflect.GeneratedMethodAccessor495.invoke(Unknown Source) ~[?:?]
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at org.infinispan.jmx.ResourceDMBean$InvokableSetterBasedMBeanAttributeInfo.invoke(ResourceDMBean.java:422) ~[classes/:?]
> at org.infinispan.jmx.ResourceDMBean.setNamedAttribute(ResourceDMBean.java:355) ~[classes/:?]
> ... 28 more
> Caused by: org.infinispan.commons.CacheException: Unsuccessful local response
> at org.infinispan.topology.LocalTopologyManagerImpl.executeOnClusterSync(LocalTopologyManagerImpl.java:757) ~[classes/:?]
> at org.infinispan.topology.LocalTopologyManagerImpl.setCacheRebalancingEnabled(LocalTopologyManagerImpl.java:623) ~[classes/:?]
> at org.infinispan.topology.LocalTopologyManagerImpl.setRebalancingEnabled(LocalTopologyManagerImpl.java:581) ~[classes/:?]
> at jdk.internal.reflect.GeneratedMethodAccessor495.invoke(Unknown Source) ~[?:?]
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at org.infinispan.jmx.ResourceDMBean$InvokableSetterBasedMBeanAttributeInfo.invoke(ResourceDMBean.java:422) ~[classes/:?]
> at org.infinispan.jmx.ResourceDMBean.setNamedAttribute(ResourceDMBean.java:355) ~[classes/:?]
> ... 28 more
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months
[JBoss JIRA] (ISPN-10362) Unify remove command initialization and invocation
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-10362?page=com.atlassian.jira.plugin... ]
Tristan Tarrant updated ISPN-10362:
-----------------------------------
Fix Version/s: 10.0.0.Beta5
(was: 10.0.0.Beta4)
> Unify remove command initialization and invocation
> --------------------------------------------------
>
> Key: ISPN-10362
> URL: https://issues.jboss.org/browse/ISPN-10362
> Project: Infinispan
> Issue Type: Enhancement
> Components: Core
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.Beta5, 10.0.0.Final
>
>
> ISPN-10322 unified command initialization with {{InitializableCommand}}, but we should go further and unify initialization with invocation.
> We can replace the current {{ReplicableCommand.invokeAsync}} and {{InitializableCommand.init(ComponentRegistry()}} methods with a method {{CacheRpcCommand.invokeAsync(ComponentRegistry)}} (or maybe {{execute}}?).
> For global commands we can create a {{GlobalRpcCommand}} interface with a method {{invokeAsync(GlobalComponentRegistry)}}.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months
[JBoss JIRA] (ISPN-10361) Distribution zips are not properly installed
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-10361?page=com.atlassian.jira.plugin... ]
Tristan Tarrant updated ISPN-10361:
-----------------------------------
Fix Version/s: 10.0.0.Beta5
(was: 10.0.0.Beta4)
> Distribution zips are not properly installed
> --------------------------------------------
>
> Key: ISPN-10361
> URL: https://issues.jboss.org/browse/ISPN-10361
> Project: Infinispan
> Issue Type: Bug
> Components: Build
> Affects Versions: 10.0.0.Beta3
> Reporter: Pedro Ruivo
> Assignee: Pedro Ruivo
> Priority: Major
> Fix For: 10.0.0.Beta5
>
>
> The {{maven-assembly-plugin}} uses the {{assemblyId}} as a classifier when installing/deploying the zip files.
> Since it was disabled for {{server}}, {{wildfly-server}}, and {{modules}}, maven installs all 3 zips with the same name.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months
[JBoss JIRA] (ISPN-10301) Deprecate server media type application/x-java-object
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-10301?page=com.atlassian.jira.plugin... ]
Tristan Tarrant updated ISPN-10301:
-----------------------------------
Fix Version/s: 10.0.0.Beta5
(was: 10.0.0.Beta4)
> Deprecate server media type application/x-java-object
> -----------------------------------------------------
>
> Key: ISPN-10301
> URL: https://issues.jboss.org/browse/ISPN-10301
> Project: Infinispan
> Issue Type: Task
> Components: Server
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.Beta5
>
>
> `application/x-java-object` needs to have the application classes deployed on the server in order to do anything useful with it, and the server must also be able to do reflection on those application classes.
> We should steer users towards always using `application/x-protostream` instead, because the protobuf schemas are much easier to deploy to the server. The first step would be to make protostream the default marshalling mechanism in the client.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months