[infinispan-dev] Embedded mode: how-to get all caches started on all nodes?

Hi, This email follows up on my testing of the Infinispan Cluster Manager for Vert.x on Kubernetes. In one of the tests, we want to make sure that, after a rolling update of the application, the data submitted to Vert.x' AsyncMap is still present. And I found that when the underlying cache is predefined in infinispan.xml, the data is present, otherwise it's not. I pushed a simple reproducer on GitHub: https://github.com/tsegismont/cachedataloss The code does this: - a first node is started, and creates data - new nodes are started, but they don't invoke cacheManager.getCache - the initial member is killed - a "testing" member is started, printing out the data in the console Here are my findings. 1/ Even when caches are declared in infinispan.xml, the data is lost after the initial member goes away. A little digging showed that the caches are really distributed only after you invoke cacheManager.getCache 2/ Checking cluster status "starts" triggers distribution I was wondering why the behavior was not the same as with my Vert.x testing on Openshift. And then realized the only difference was the cluster readiness check, which reads the cluster health. So I updated the reproducer code to add such a check (still without invoking cacheManager.getCache). Then the caches defined in infinispan.xml have their data distributed. So, 1/ How can I make sure caches are distributed on all nodes, even if some nodes never try to get a reference with cacheManager.getCache, or don't check cluster health? 2/ Are we doing something wrong with our way to declare the default configuration for caches [1][2]? Thanks, Thomas [1] https://github.com/tsegismont/cachedataloss/blob/master/src/main/resource... [2] https://github.com/tsegismont/cachedataloss/blob/master/src/main/resource...