[infinispan-issues] [JBoss JIRA] (ISPN-10397) CacheStoreFactoryRegistry incorrect synchronization

[Dan Berindei (Jira)]

     [ https://issues.jboss.org/browse/ISPN-10397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dan Berindei updated ISPN-10397:
--------------------------------
    Description: 
Starting multiple caches (configured with cache stores) simultaneously results in threading issues, which results in the failed startup of the cache (for example, [1]).

It seems that org.infinispan.persistence.factory.CacheStoreFactoryRegistry does not properly synchronize access to its "factories" member. It uses a synchronized collection, but that synchronization does not include the iteration which two methods in this class use.

{noformat}
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
at java.util.ArrayList$Itr.next(ArrayList.java:859)
at org.infinispan.persistence.factory.CacheStoreFactoryRegistry.createInstance(CacheStoreFactoryRegistry.java:38)
at org.infinispan.persistence.manager.PersistenceManagerImpl.createLoadersAndWriters(PersistenceManagerImpl.java:661)
at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:152)
{noformat}

Configuring caches to start lazily seems to be a workaround.

{noformat}
ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.datagrid-infinispan.clustered.SubscriptionIndexesCache: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan.clustered.SubscriptionIndexesCache: Failed to start service
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1978)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:878)
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:643)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:630)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:232)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:891)
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:676)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:620)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:506)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:492)
at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:150)
at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:136)
at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:133)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:69)
at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:141)
at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:91)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:2032)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1955)
... 3 more
Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders
at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
... 23 more
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
at java.util.ArrayList$Itr.next(ArrayList.java:859)
at org.infinispan.persistence.factory.CacheStoreFactoryRegistry.createInstance(CacheStoreFactoryRegistry.java:38)
at org.infinispan.persistence.manager.PersistenceManagerImpl.createLoadersAndWriters(PersistenceManagerImpl.java:661)
at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:152)
... 28 more
{noformat}

  was:


Starting multiple caches (configured with cache stores) simultaneously results in threading issues, which results in the failed startup of the cache (for example, [1]).

It seems that org.infinispan.persistence.factory.CacheStoreFactoryRegistry does not properly synchronize access to its "factories" member. It uses a synchronized collection, but that synchronization does not include the iteration which two methods in this class use.

Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
at java.util.ArrayList$Itr.next(ArrayList.java:859)
at org.infinispan.persistence.factory.CacheStoreFactoryRegistry.createInstance(CacheStoreFactoryRegistry.java:38)
at org.infinispan.persistence.manager.PersistenceManagerImpl.createLoadersAndWriters(PersistenceManagerImpl.java:661)
at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:152)

Configuring caches to start lazily seems to be a workaround.

[1]
ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.datagrid-infinispan.clustered.SubscriptionIndexesCache: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan.clustered.SubscriptionIndexesCache: Failed to start service
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1978)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:878)
at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:643)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:630)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:232)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:891)
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:676)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:620)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:506)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:492)
at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:150)
at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:136)
at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:133)
at org.infinispan.security.Security.doPrivileged(Security.java:76)
at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:69)
at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:141)
at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:91)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:2032)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1955)
... 3 more
Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders
at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
... 23 more
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
at java.util.ArrayList$Itr.next(ArrayList.java:859)
at org.infinispan.persistence.factory.CacheStoreFactoryRegistry.createInstance(CacheStoreFactoryRegistry.java:38)
at org.infinispan.persistence.manager.PersistenceManagerImpl.createLoadersAndWriters(PersistenceManagerImpl.java:661)
at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:152)
... 28 more




> CacheStoreFactoryRegistry incorrect synchronization
> ---------------------------------------------------
>
>                 Key: ISPN-10397
>                 URL: https://issues.jboss.org/browse/ISPN-10397
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 10.0.0.Beta3, 9.4.15.Final
>            Reporter: Dan Berindei
>            Assignee: Dan Berindei
>            Priority: Major
>             Fix For: 10.0.0.Beta4, 9.4.16.Final
>
>
> Starting multiple caches (configured with cache stores) simultaneously results in threading issues, which results in the failed startup of the cache (for example, [1]).
> It seems that org.infinispan.persistence.factory.CacheStoreFactoryRegistry does not properly synchronize access to its "factories" member. It uses a synchronized collection, but that synchronization does not include the iteration which two methods in this class use.
> {noformat}
> Caused by: java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
> at java.util.ArrayList$Itr.next(ArrayList.java:859)
> at org.infinispan.persistence.factory.CacheStoreFactoryRegistry.createInstance(CacheStoreFactoryRegistry.java:38)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.createLoadersAndWriters(PersistenceManagerImpl.java:661)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:152)
> {noformat}
> Configuring caches to start lazily seems to be a workaround.
> {noformat}
> ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.datagrid-infinispan.clustered.SubscriptionIndexesCache: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan.clustered.SubscriptionIndexesCache: Failed to start service
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1978)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:878)
> at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:643)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:630)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:232)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:891)
> at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:676)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:620)
> at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:506)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:492)
> at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:150)
> at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:136)
> at org.infinispan.server.infinispan.SecurityActions$5.run(SecurityActions.java:133)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:69)
> at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:141)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:91)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:2032)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1955)
> ... 3 more
> Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders
> at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:199)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> ... 23 more
> Caused by: java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
> at java.util.ArrayList$Itr.next(ArrayList.java:859)
> at org.infinispan.persistence.factory.CacheStoreFactoryRegistry.createInstance(CacheStoreFactoryRegistry.java:38)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.createLoadersAndWriters(PersistenceManagerImpl.java:661)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:152)
> ... 28 more
> {noformat}



--
This message was sent by Atlassian Jira
(v7.12.1#712002)