When underlying store becomes available, the queued modifications are not been written -------------------------------------------------------------------------------------- Key: ISPN-9455 URL: https://issues.jboss.org/browse/ISPN-9455 Project: Infinispan Issue Type: Bug Reporter: Diego Lovison Assignee: Ryan Emerson Priority: Critical In order to reproduce this issue, you will need to do the following. 1) Start a remote Infinispan server using: ./standalone.sh -c clustered.xml 2) Add a breakpoint in AsynCacheWriter line 423. The line is: "retryWork(configuration.connectionAttempts());" into the "run" method 3) Run the application 4) When the breakpoint stops in AsynCacheWriter::423, kill the remote server and then resume. 5) Wait for the message in the log: "ISPN000053: Unable to process some async modifications after 5 retries!" 6) After the message, start again the remote server. 7) After 120 seconds we will double check in the remote store if the data is available. It will fail. There is no data in the remote store. Based on https://issues.jboss.org/browse/JDG-1051 "Once the underlying store becomes available, the queued modifications should be written and then the cache should become available." Application {code:java} public class ConfigurationJdbcStoreFileCacheExample { public static void main(String[] args) throws InterruptedException, IOException, SQLException { EmbeddedCacheManager cacheManager = new DefaultCacheManager("cache-store.xml"); ExecutorService executorService = Executors.newFixedThreadPool(10); try { Cache<String, String> cache = cacheManager.getCache("myCache"); List<Future<String>> keys = new ArrayList<>(); for (int i = 0; i < 10; i++) { keys.add(executorService.submit(() -> { String key = UUID.randomUUID().toString(); try { cache.put(key, key); System.out.println("Done: " + key); } catch (Throwable e) { e.printStackTrace(); } return key; })); } TimeUnit.SECONDS.sleep(120); RemoteCacheManager remoteCacheManager = new RemoteCacheManager(new ConfigurationBuilder() .addServer() .host("localhost") .port(11222) .build()); RemoteCache<String, String> remoteCache = remoteCacheManager.administration().getOrCreateCache("default", (String) null); keys.forEach(f -> { try { String key = f.get(); Assert.assertNotNull(remoteCache.get(key)); } catch (InterruptedException e) { throw new IllegalStateException(e); } catch (ExecutionException e) { throw new IllegalStateException(e); } }); } finally { cacheManager.stop(); executorService.shutdownNow(); } } } {code} Configuration {code:xml} <?xml version="1.0" ?> <infinispan> <jgroups> <stack-file name="external-file" path="default-jgroups-tcp.xml"/> </jgroups> <cache-container name="myCacheContainer" default-cache="myCache" statistics="true"> <transport cluster="WeatherApp" stack="external-file" /> <distributed-cache owners="2" mode="SYNC" remote-timeout="15000" name="myCache"> <transaction mode="NONE"/> <persistence availability-interval="1000" connection-attempts="5" connection-interval="1000"> <remote-store cache="default" raw-values="true"> <remote-server host="localhost" port="11222" /> <write-behind modification-queue-size="100" thread-pool-size="20" fail-silently="true"/> </remote-store> </persistence> </distributed-cache> </cache-container> </infinispan> {code}