[infinispan-issues] [JBoss JIRA] (ISPN-9146) [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
Radoslav Husar (JIRA)
issues at jboss.org
Fri May 11 15:41:01 EDT 2018
[ https://issues.jboss.org/browse/ISPN-9146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radoslav Husar updated ISPN-9146:
---------------------------------
Affects Version/s: 9.2.2.Final
> [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-9146
> URL: https://issues.jboss.org/browse/ISPN-9146
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.2.2.Final
> Reporter: Radoslav Husar
> Priority: Critical
>
> We hit an issue when running tests for RFE EAP7-942.
> Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294
> Scenario:
> 2-node Wildfly cluster + 2-node JDG 7.2.0.CR2 cluster.
> Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
> Each Wildfly node is gracefully shut down and then started up again.
> Wildfly configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-remote-jdg-session-shutdown-invalidation-sync-hotrod/45/artifact/report/config/jboss-perf18/standalone-ha.xml
> JDG configuration:
> http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-remote-jdg-session-shutdown-invalidation-sync-hotrod/45/artifact/report/config/jboss-perf20/clustered.xml
> Important configuration change on JDG: default cache has transactions configured
> {code:xml}
> <distributed-cache name="default">
> <transaction mode="NON_DURABLE_XA" locking="PESSIMISTIC"/>
> </distributed-cache>
> {code}
> With this change, the offloading scenario fails to start. When clusterbench is deployed on EAP and the distributed cache is created on JDG, JDG throws
> {code}
> [JBossINF] [0m[31m04:32:27,869 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p2-t16) ISPN000136: Error executing command PutKeyValueCommand, writing keys [ClusterConfigurationScope{scope='cache', name='clusterbench-ee7.ear.cb.war'}]: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener at 24563360
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:401)
> [JBossINF] at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:419)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1565)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1561)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1556)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1535)
> [JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:368)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:261)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$ReplicationLogic.commitSingleEntry(ClusteringDependentLogic.java:493)
> [JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:186)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:578)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:746)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:555)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyChanges(EntryWrappingInterceptor.java:611)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$setSkipRemoteGetsAndInvokeNextForDataCommand$8(EntryWrappingInterceptor.java:667)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:108)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:664)
> [JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:304)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:107)
> [JBossINF] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:39)
> [JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:77)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:330)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:268)
> [JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:118)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:171)
> [JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:126)
> [JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:92)
> [JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:59)
> [JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
> [JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49)
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:267)
> [JBossINF] at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommandAsync(BaseRpcInvokingCommand.java:71)
> [JBossINF] at org.infinispan.commands.remote.SingleRpcCommand.invokeAsync(SingleRpcCommand.java:58)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:95)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:91)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:69)
> [JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:41)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: java.nio.channels.OverlappingFileLockException
> [JBossINF] at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> [JBossINF] at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> [JBossINF] at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
> [JBossINF] at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.storeAll(OverlayLocalConfigurationStorage.java:92)
> [JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.createCache(OverlayLocalConfigurationStorage.java:47)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.createCacheLocally(GlobalConfigurationManagerImpl.java:159)
> [JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(GlobalConfigurationStateListener.java:28)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:396)
> [JBossINF] ... 50 more
> {code}
> JDG log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-remote-jdg-session-shutdown-invalidation-sync-hotrod/45/console-perf21/
> This results in EAP logging
> {code}
> [JBossINF] [0m[33m04:32:27,966 WARN [org.infinispan.client.hotrod.impl.protocol.Codec21] (HotRod-client-async-pool-5) ISPN004005: Error received from the server: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf20, see cause for remote stack trace
> [JBossINF] org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf21, see cause for remote stack trace
> [JBossINF] org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener at 24563360
> [JBossINF] java.nio.channels.OverlappingFileLockException
> [JBossINF] [0m[31m04:32:27,970 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 74) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:70)
> [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder.lambda$start$0(AsynchronousServiceBuilder.java:99)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
> [JBossINF] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
> [JBossINF] at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
> [JBossINF] at org.infinispan.security.Security.doPrivileged(Security.java:76)
> [JBossINF] at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
> [JBossINF] at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:229)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1020)
> [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447)
> [JBossINF] at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:71)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:54)
> [JBossINF] at org.wildfly.clustering.service.SuppliedValueService.lambda$new$0(SuppliedValueService.java:42)
> [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:68)
> [JBossINF] ... 7 more
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders
> [JBossINF] at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:179)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
> [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
> [JBossINF] ... 31 more
> {code}
> EAP log:
> https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-remote-jdg-session-shutdown-invalidation-sync-hotrod/45/console-perf19/
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
More information about the infinispan-issues
mailing list