CacheStoreFactoryRegistry incorrect synchronization --------------------------------------------------- Key: ISPN-10397 URL: https://issues.jboss.org/browse/ISPN-10397 Project: Infinispan Issue Type: Bug Components: Core Affects Versions: 10.0.0.Beta3, 9.4.15.Final Reporter: Dan Berindei Assignee: Dan Berindei Priority: Major Fix For: 10.0.0.Beta4, 9.4.16.Final Starting multiple caches (configured with cache stores) simultaneously results in threading issues, which results in the failed startup of the cache (for example, [1]). It seems that org.infinispan.persistence.factory.CacheStoreFactoryRegistry does not properly synchronize access to its "factories" member. It uses a synchronized collection, but that synchronization does not include the iteration which two methods in this class use. {noformat} Caused by: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) at java.util.ArrayList$Itr.next(ArrayList.java:859) at org.infinispan.persistence.factory.CacheStoreFactoryRegistry.createInstance(CacheStoreFactoryRegistry.java:38) at org.infinispan.persistence.manager.PersistenceManagerImpl.createLoadersAndWriters(PersistenceManagerImpl.java:661) at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:152) {noformat} Configuring caches to start lazily seems to be a workaround. {noformat} ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.datagrid-infinispan.clustered.SubscriptionIndexesCache: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan.clustered.SubscriptionIndexesCache: Failed to start service at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1978) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:878) at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:643) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:630) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:232) at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:891) at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:676) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:620) at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:506) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:492) at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:150) at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:136) at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:133) at org.infinispan.security.Security.doPrivileged(Security.java:76) at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:69) at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:141) at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:91) at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:2032) at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1955) ... 3 more Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:199) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168) ... 23 more Caused by: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) at java.util.ArrayList$Itr.next(ArrayList.java:859) at org.infinispan.persistence.factory.CacheStoreFactoryRegistry.createInstance(CacheStoreFactoryRegistry.java:38) at org.infinispan.persistence.manager.PersistenceManagerImpl.createLoadersAndWriters(PersistenceManagerImpl.java:661) at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:152) ... 28 more {noformat}