]
Radoslav Husar updated ISPN-9146:
---------------------------------
Affects Version/s: 9.2.2.Final
[pretesting] JDG throws OverlappingFileLockException when creating
distributed cache with transactions in session offloading scenario with hotrod store
configured
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key: ISPN-9146
URL:
https://issues.jboss.org/browse/ISPN-9146
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 9.2.2.Final
Reporter: Radoslav Husar
Priority: Critical
We hit an issue when running tests for RFE EAP7-942.
Wildfly was built from
https://github.com/rhusar/wildfly/tree/WFLY-9294
Scenario:
2-node Wildfly cluster + 2-node JDG 7.2.0.CR2 cluster.
Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
Each Wildfly node is gracefully shut down and then started up again.
Wildfly configuration:
http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
JDG configuration:
http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-r...
Important configuration change on JDG: default cache has transactions configured
{code:xml}
<distributed-cache name="default">
<transaction mode="NON_DURABLE_XA" locking="PESSIMISTIC"/>
</distributed-cache>
{code}
With this change, the offloading scenario fails to start. When clusterbench is deployed
on EAP and the distributed cache is created on JDG, JDG throws
{code}
[JBossINF] [0m[31m04:32:27,869 ERROR
[org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p2-t16)
ISPN000136: Error executing command PutKeyValueCommand, writing keys
[ClusterConfigurationScope{scope='cache',
name='clusterbench-ee7.ear.cb.war'}]:
org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception
[java.nio.channels.OverlappingFileLockException] while invoking method [public void
org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)]
on listener instance:
org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
[JBossINF] at
org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:401)
[JBossINF] at
org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
[JBossINF] at
org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:419)
[JBossINF] at
org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1565)
[JBossINF] at
org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1561)
[JBossINF] at
org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1556)
[JBossINF] at
org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1535)
[JBossINF] at
org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:368)
[JBossINF] at
org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:261)
[JBossINF] at
org.infinispan.interceptors.locking.ClusteringDependentLogic$ReplicationLogic.commitSingleEntry(ClusteringDependentLogic.java:493)
[JBossINF] at
org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:186)
[JBossINF] at
org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:578)
[JBossINF] at
org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:746)
[JBossINF] at
org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:555)
[JBossINF] at
org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyChanges(EntryWrappingInterceptor.java:611)
[JBossINF] at
org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$setSkipRemoteGetsAndInvokeNextForDataCommand$8(EntryWrappingInterceptor.java:667)
[JBossINF] at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:108)
[JBossINF] at
org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:664)
[JBossINF] at
org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:304)
[JBossINF] at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
[JBossINF] at
org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:107)
[JBossINF] at
org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:39)
[JBossINF] at
org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:77)
[JBossINF] at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
[JBossINF] at
org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:330)
[JBossINF] at
org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:268)
[JBossINF] at
org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:118)
[JBossINF] at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
[JBossINF] at
org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:171)
[JBossINF] at
org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
[JBossINF] at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:126)
[JBossINF] at
org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:92)
[JBossINF] at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
[JBossINF] at
org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
[JBossINF] at
org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:59)
[JBossINF] at
org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] at
org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49)
[JBossINF] at
org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:267)
[JBossINF] at
org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommandAsync(BaseRpcInvokingCommand.java:71)
[JBossINF] at
org.infinispan.commands.remote.SingleRpcCommand.invokeAsync(SingleRpcCommand.java:58)
[JBossINF] at
org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:95)
[JBossINF] at
org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:91)
[JBossINF] at
org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:69)
[JBossINF] at
org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:41)
[JBossINF] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: java.nio.channels.OverlappingFileLockException
[JBossINF] at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
[JBossINF] at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
[JBossINF] at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
[JBossINF] at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
[JBossINF] at
org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.storeAll(OverlayLocalConfigurationStorage.java:92)
[JBossINF] at
org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.createCache(OverlayLocalConfigurationStorage.java:47)
[JBossINF] at
org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.createCacheLocally(GlobalConfigurationManagerImpl.java:159)
[JBossINF] at
org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(GlobalConfigurationStateListener.java:28)
[JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JBossINF] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[JBossINF] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
[JBossINF] at
org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:396)
[JBossINF] ... 50 more
{code}
JDG log:
https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...
This results in EAP logging
{code}
[JBossINF] [0m[33m04:32:27,966 WARN [org.infinispan.client.hotrod.impl.protocol.Codec21]
(HotRod-client-async-pool-5) ISPN004005: Error received from the server:
org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf20, see
cause for remote stack trace
[JBossINF] org.infinispan.remoting.RemoteException: ISPN000217: Received exception from
perf21, see cause for remote stack trace
[JBossINF] org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception
[java.nio.channels.OverlappingFileLockException] while invoking method [public void
org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)]
on listener instance:
org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
[JBossINF] java.nio.channels.OverlappingFileLockException
[JBossINF] [0m[31m04:32:27,970 ERROR [org.jboss.msc.service.fail] (ServerService Thread
Pool -- 74) MSC000001: Failed to start service
org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war:
org.jboss.msc.service.StartException in service
org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war:
org.infinispan.commons.CacheException: Unable to invoke method public void
org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type
PersistenceManagerImpl
[JBossINF] at
org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:70)
[JBossINF] at
org.wildfly.clustering.service.AsynchronousServiceBuilder.lambda$start$0(AsynchronousServiceBuilder.java:99)
[JBossINF] at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] at
org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:485)
[JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to invoke method
public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of
type PersistenceManagerImpl
[JBossINF] at
org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
[JBossINF] at
org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
[JBossINF] at
org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
[JBossINF] at
org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
[JBossINF] at
org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
[JBossINF] at
org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
[JBossINF] at
org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
[JBossINF] at org.infinispan.security.Security.doPrivileged(Security.java:76)
[JBossINF] at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
[JBossINF] at
org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
[JBossINF] at
org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
[JBossINF] at
org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
[JBossINF] at
org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:229)
[JBossINF] at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1020)
[JBossINF] at
org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
[JBossINF] at
org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644)
[JBossINF] at
org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
[JBossINF] at
org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475)
[JBossINF] at
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
[JBossINF] at
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447)
[JBossINF] at
org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
[JBossINF] at
org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:71)
[JBossINF] at
org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:54)
[JBossINF] at
org.wildfly.clustering.service.SuppliedValueService.lambda$new$0(SuppliedValueService.java:42)
[JBossINF] at
org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:68)
[JBossINF] ... 7 more
[JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to start cache
loaders
[JBossINF] at
org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:179)
[JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JBossINF] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[JBossINF] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
[JBossINF] at
org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
[JBossINF] ... 31 more
{code}
EAP log:
https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-...