ClusterCacheLoader NPE when ClusteredGetCommand response value is null ---------------------------------------------------------------------- Key: ISPN-3404 URL: https://issues.jboss.org/browse/ISPN-3404 Project: Infinispan Issue Type: Bug Affects Versions: 6.0.0.Alpha2 Reporter: Tomas Sykora Assignee: Pedro Ruivo Labels: 620 Fix For: 6.0.0.CR2, 6.0.0.Final Attachments: clusterCacheLoaderNullPointerResponseValueError.txt, server1DR2.log, server2DR2.log We found this issue in DR2 (Alpha2). In Alpha1 this was ok. Our internal test: {code:title=Bar.java|borderStyle=solid} @Test public void testLazyLoadingWhenStateTransferDisabled() throws Exception { controller.start(CONTAINER1); mc = new MemcachedHelper(server1.getMemcachedEndpoint().getInetAddress().getHostName(), server1.getMemcachedEndpoint() .getPort()); mc.set("k1", "v1"); mc.set("k2", "v2"); assertEquals("v1", mc.get("k1")); assertEquals("v2", mc.get("k2")); assertEquals(2, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries()); controller.start(CONTAINER2); mc2 = new MemcachedHelper(server2.getMemcachedEndpoint().getInetAddress().getHostName(), server2.getMemcachedEndpoint() .getPort()); assertEquals(2, server2.getDefaultCacheManager().getClusterSize()); //state-transfer = false -> no entries in the newly joined node assertEquals(0, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries()); assertEquals("v1", mc2.get("k1")); //lazily load the entries assertEquals("v2", mc2.get("k2")); assertEquals(2, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries()); mc2.set("k3", "v3"); // THIS IS FAILING HERE assertEquals(3, server2.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries()); assertEquals(3, server1.getDefaultCacheManager().getCache(CACHE_NAME).getNumberOfEntries()); controller.kill(CONTAINER2); controller.kill(CONTAINER1); } {code} *mc2.set("k3", "v3"); = put into memcached cache, is throwing an exception: see in attachment.* You can see MemcachedHelper class here: https://code.engineering.redhat.com/gerrit/gitweb?p=jdg-functional-tests.... I also include trace logs for both memcached jdg servers if they can be useful. --------------- I was able to replicate the same error in Infinispan test suite for Rest endpoint. It was enough to start 2 REST nodes and issue 1 put resulting into the same error. {code:title=Bar.java|borderStyle=solid} ConfigurationBuilder cfgBuilder = AbstractCacheTest.getDefaultClusteredCacheConfig(CacheMode.REPL_SYNC, false); cfgBuilder.clustering().hash().numOwners(1); cfgBuilder.clustering().stateTransfer().fetchInMemoryState(false); cfgBuilder.clustering().stateTransfer().timeout(20000); cfgBuilder.loaders().addClusterCacheLoader().remoteCallTimeout(60000); RestServerConfigurationBuilder restCfgBuilder = new RestServerConfigurationBuilder(); addServer("1", 8890, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build()); addServer("2", 8891, TestCacheManagerFactory.createClusteredCacheManager(cfgBuilder), restCfgBuilder.build()); startServers(); ...... PutMethod put = new PutMethod(PATH1 + "key1"); put.setRequestEntity(new StringRequestEntity("value1", "application/text", null)); call(put); assertEquals(put.getStatusCode(), HttpServletResponse.SC_OK); put.releaseConnection(); {code} This scenario is different from aforementioned JDG internal test suite one but I expect this should be ok and should be performed without any problems.