ScatteredStreamIteratorTest.waitUntilProcessingResults random failures ---------------------------------------------------------------------- Key: ISPN-8002 URL: https://issues.jboss.org/browse/ISPN-8002 Project: Infinispan Issue Type: Bug Components: Core Affects Versions: 9.1.0.CR1 Reporter: Dan Berindei Assignee: Radim Vansa Priority: Critical Labels: testsuite_stability Fix For: 9.1.0.Final Attachments: ScatteredStreamIteratorTest_ISPN-7998_ConsistentHash.getNumOwners_20170629.log.gz {noformat} 20:38:39,442 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.scattered.stream.ScatteredStreamIteratorTest.waitUntilProcessingResults[SCATTERED_SYNC, tx=false] java.lang.NullPointerException: null at org.infinispan.stream.DistributedStreamIteratorTest.waitUntilProcessingResults(DistributedStreamIteratorTest.java:207) ~[test-classes/:?] {noformat} Looks like segments {{ScatteredVersionManager}} blocks access to segments 10 and 12, and they are never suspected or retried: {noformat} 20:38:38,209 TRACE (remote-thread-Test-NodeBF-p13315-t6:[]) [LocalStreamManagerImpl] Request Test-NodeBD-1062-0 completed segments [6, 7, 13, 14, 15] with [] suspected segments 20:38:38,248 TRACE (async-thread-Test-NodeBD-p13302-t1:[]) [DistributionBulkInterceptor$BackingEntrySet$1] Found [0, 1, 2, 3, 4, 5, 8, 9, 10, 11, 12] lost segments for Test-NodeBD-1062-0 20:38:38,249 TRACE (transport-thread-Test-NodeBD-p13301-t4:[Topology-org.infinispan.scattered.stream.Test]) [ScatteredVersionManager] Node Test-NodeBD-1062 blocks access to segment 10 20:38:38,253 TRACE (async-thread-Test-NodeBD-p13302-t1:[]) [ClusterStreamManagerImpl] Performing remote rehash key aware operations {Test-NodeBF-40292={8-9 11}} for id Test-NodeBD-1062-1 20:38:39,209 TRACE (remote-thread-Test-NodeBF-p13315-t6:[]) [LocalStreamManagerImpl] Request Test-NodeBD-1062-1 completed segments [] with [8, 9, 11] suspected segments 20:38:39,302 TRACE (async-thread-Test-NodeBD-p13302-t1:[]) [DistributionBulkInterceptor$BackingEntrySet$1] Found [0, 1, 2, 3, 4, 5, 8, 9, 11] lost segments for Test-NodeBD-1062-1 20:38:39,420 TRACE (remote-thread-Test-NodeBF-p13315-t6:[]) [LocalStreamManagerImpl] Request Test-NodeBD-1062-2 completed segments [8, 9, 11] with [] suspected segments 20:38:39,436 TRACE (async-thread-Test-NodeBD-p13302-t1:[]) [DistributionBulkInterceptor$BackingEntrySet$1] Completing segments [0, 1, 2, 3, 4, 5] 20:38:39,436 TRACE (async-thread-Test-NodeBD-p13302-t1:[]) {noformat} Please also add a {{toString()}} method in the stream commands, seeing the segments in the requests and responses would make it easier to follow the logs.