SingleFileStore.start() often fails on File.mkdirs() during concurrent cache startup ------------------------------------------------------------------------------------ Key: ISPN-3629 URL: https://issues.jboss.org/browse/ISPN-3629 Project: Infinispan Issue Type: Bug Components: Loaders and Stores Affects Versions: 6.0.0.CR1 Reporter: Paul Ferraro Assignee: Paul Ferraro Priority: Critical If multiple caches are started concurrently, SingleFileStore.start() often fails during File.mkdirs(). File.mkdirs() will fail if multiple threads attempt to create the same directories. To improve concurrent reliability of concurrently started caches, this operation should be performed in a synchronized block. e.g. [0m[31m17:34:26,994 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 17) MSC000001: Failed to start service jboss.infinispan.web.repl: org.jboss.msc.service.StartException in service jboss.infinispan.web.repl: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869) at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216) at org.infinispan.CacheImpl.start(CacheImpl.java:668) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:539) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:502) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:395) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:409) at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:103) at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:94) at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78) at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82) ... 4 more Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:154) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183) ... 18 more Caused by: org.infinispan.persistence.CacheLoaderException: org.infinispan.commons.CacheConfigurationException: ISPN000238: Directory /home/paul/wildfly/ejb/wildfly/testsuite/integration/clust/target/jbossas-clustering-SYNC-tcp-1/standalone/data/infinispan/web does not exist and cannot be created! at org.infinispan.persistence.file.SingleFileStore.start(SingleFileStore.java:118) at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:121) ... 23 more Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000238: Directory /home/paul/wildfly/ejb/wildfly/testsuite/integration/clust/target/jbossas-clustering-SYNC-tcp-1/standalone/data/infinispan/web does not exist and cannot be created! at org.infinispan.persistence.file.SingleFileStore.start(SingleFileStore.java:102) ... 24 more