[JBoss JIRA] (ISPN-9836) RemoteCacheImpl.ping may hang when the server is stopped
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-9836?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9836:
-------------------------------
Attachment: threaddump-org_infinispan_persistence_remote_RemoteStoreWrapperTest_tearDown-2018-12-17-25284.log
RemoteStoreWrapperTest_infinispan-infinispan-cachestore-remote.log_20181218.log
> RemoteCacheImpl.ping may hang when the server is stopped
> --------------------------------------------------------
>
> Key: ISPN-9836
> URL: https://issues.jboss.org/browse/ISPN-9836
> Project: Infinispan
> Issue Type: Bug
> Components: Server, Test Suite - Server
> Affects Versions: 10.0.0.Alpha2, 9.4.4.Final
> Reporter: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.Beta1
>
> Attachments: RemoteStoreWrapperTest_infinispan-infinispan-cachestore-remote.log_20181218.log, threaddump-org_infinispan_persistence_remote_RemoteStoreWrapperTest_tearDown-2018-12-17-25284.log
>
>
> {{RemoteStoreWrapperTest.tearDown()}} first stops the server, then it stops the remote store connected to the server. Sometimes the store cannot be stopped because it is blocked in an availability check for > 5 min:
> {noformat}
> "testng-RemoteStoreWrapperTest" #14 prio=5 os_prio=0 tid=0x00007f1c74a1e000 nid=0x62e0 waiting on condition [0x00007f1c3cde4000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c00028f8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.stop(PersistenceManagerImpl.java:214)
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:181)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.performStop(BasicComponentRegistryImpl.java:599)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stopWrapper(BasicComponentRegistryImpl.java:588)
> at org.infinispan.factories.impl.BasicComponentRegistryImpl.stop(BasicComponentRegistryImpl.java:461)
> at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:441)
> at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:376)
> at org.infinispan.cache.impl.CacheImpl.performImmediateShutdown(CacheImpl.java:1159)
> at org.infinispan.cache.impl.CacheImpl.stop(CacheImpl.java:1124)
> at org.infinispan.cache.impl.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:520)
> at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:734)
> at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:786)
> at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:762)
> at org.infinispan.test.TestingUtil.killCacheManagers(TestingUtil.java:839)
> at org.infinispan.test.TestingUtil.killCacheManagers(TestingUtil.java:830)
> at org.infinispan.persistence.remote.RemoteStoreWrapperTest.tearDown(RemoteStoreWrapperTest.java:86)
> "persistence-thread-RemoteStoreWrapperTest-NodeB-p120-t1" #215 daemon prio=5 os_prio=0 tid=0x00007f1be128f000 nid=0x63b7 waiting on condition [0x00007f1bdd3d4000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c40059a8> (a java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695)
> at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> at org.infinispan.client.hotrod.impl.Util.await(Util.java:21)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:542)
> at org.infinispan.persistence.remote.RemoteStore.isAvailable(RemoteStore.java:128)
> at org.infinispan.persistence.manager.PersistenceManagerImpl$StoreStatus.availabilityChanged(PersistenceManagerImpl.java:1252)
> - locked <0x00000000c00027e8> (a org.infinispan.persistence.manager.PersistenceManagerImpl$StoreStatus)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.pollStoreAvailability(PersistenceManagerImpl.java:194)
> at org.infinispan.persistence.manager.PersistenceManagerImpl$$Lambda$383/1386794082.run(Unknown Source)
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months
[JBoss JIRA] (ISPN-9836) RemoteCacheImpl.ping may hang when the server is stopped
by Dan Berindei (Jira)
Dan Berindei created ISPN-9836:
----------------------------------
Summary: RemoteCacheImpl.ping may hang when the server is stopped
Key: ISPN-9836
URL: https://issues.jboss.org/browse/ISPN-9836
Project: Infinispan
Issue Type: Bug
Components: Server, Test Suite - Server
Affects Versions: 9.4.4.Final, 10.0.0.Alpha2
Reporter: Dan Berindei
Fix For: 10.0.0.Beta1
{{RemoteStoreWrapperTest.tearDown()}} first stops the server, then it stops the remote store connected to the server. Sometimes the store cannot be stopped because it is blocked in an availability check for > 5 min:
{noformat}
"testng-RemoteStoreWrapperTest" #14 prio=5 os_prio=0 tid=0x00007f1c74a1e000 nid=0x62e0 waiting on condition [0x00007f1c3cde4000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c00028f8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
at org.infinispan.persistence.manager.PersistenceManagerImpl.stop(PersistenceManagerImpl.java:214)
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:181)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.performStop(BasicComponentRegistryImpl.java:599)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.stopWrapper(BasicComponentRegistryImpl.java:588)
at org.infinispan.factories.impl.BasicComponentRegistryImpl.stop(BasicComponentRegistryImpl.java:461)
at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:441)
at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:376)
at org.infinispan.cache.impl.CacheImpl.performImmediateShutdown(CacheImpl.java:1159)
at org.infinispan.cache.impl.CacheImpl.stop(CacheImpl.java:1124)
at org.infinispan.cache.impl.AbstractDelegatingCache.stop(AbstractDelegatingCache.java:520)
at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:734)
at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:786)
at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:762)
at org.infinispan.test.TestingUtil.killCacheManagers(TestingUtil.java:839)
at org.infinispan.test.TestingUtil.killCacheManagers(TestingUtil.java:830)
at org.infinispan.persistence.remote.RemoteStoreWrapperTest.tearDown(RemoteStoreWrapperTest.java:86)
"persistence-thread-RemoteStoreWrapperTest-NodeB-p120-t1" #215 daemon prio=5 os_prio=0 tid=0x00007f1be128f000 nid=0x63b7 waiting on condition [0x00007f1bdd3d4000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c40059a8> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at org.infinispan.client.hotrod.impl.Util.await(Util.java:21)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:542)
at org.infinispan.persistence.remote.RemoteStore.isAvailable(RemoteStore.java:128)
at org.infinispan.persistence.manager.PersistenceManagerImpl$StoreStatus.availabilityChanged(PersistenceManagerImpl.java:1252)
- locked <0x00000000c00027e8> (a org.infinispan.persistence.manager.PersistenceManagerImpl$StoreStatus)
at org.infinispan.persistence.manager.PersistenceManagerImpl.pollStoreAvailability(PersistenceManagerImpl.java:194)
at org.infinispan.persistence.manager.PersistenceManagerImpl$$Lambda$383/1386794082.run(Unknown Source)
{noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months
[JBoss JIRA] (ISPN-3766) "containsKey" and "get" methods are not working properly with "putForExternalRead" during active transaction. WriteSkewException appears.
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-3766?page=com.atlassian.jira.plugin.... ]
Dan Berindei resolved ISPN-3766.
--------------------------------
Resolution: Explained
> "containsKey" and "get" methods are not working properly with "putForExternalRead" during active transaction. WriteSkewException appears.
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-3766
> URL: https://issues.jboss.org/browse/ISPN-3766
> Project: Infinispan
> Issue Type: Bug
> Components: Transactions
> Affects Versions: 5.3.0.Final, 6.0.0.CR1, 6.0.0.Final
> Reporter: V L
> Priority: Major
> Attachments: InfinispanTest.java
>
>
> I'm not new with Infinispan, we use it from 5.3 version. Now I'm working on adding WriteSkew functional to our project, but unfortunately it's not working as we expected.
> We use infinispan like a cache for mongoDb data source, but this situation doesn't depends on data source vendor.
> The problem is that WriteSkewException is available in specific situation, here are steps for reproducing:
> 1) begin tx -> create an new element -> commit tx.
> 2) wait until cache expires. cache is empty now
> 3) being tx -> try to get element from cache, but it's not found -> load element from data source -> put it to the cache via putForExternalRead for read access outside of current transaction.
> 4) make some modification with element -> put element to cache -> commit tx -> WriteSkewException
> here is my log:
> TX BEGIN tx_status=6 thread=1
> PUT id=key1, tx_status=0, thread=1 cache_size=1, cache_ contains=key:key1, value:value
> TX COMMIT tx_status=0 thread=1
> TX BEGIN. tx_status=6, thread=1
> SIZE id=key1 tx_status=0, thread=1 cache_size=1 contains=key:key1, value:null
> PUT_FOR_EXTERNAL_READ id=key1, tx_status=0 thread=1 cache_size=1 cache_contains=key:key1, value:null
> PUT id=key1 tx_status=0 thread=1 cache_size=1 contains=key:key1, value:value
> TX COMMIT status=0 thread=1
> ISPN000005: Detected write skew on key [1]. Another process has changed the entry since we last read it! Unable to copy entry for update.
> Exception executing call org.infinispan.transaction.WriteSkewException: Detected write skew.
> I think the problem is in "get" method. When you invoke "get" method with key that not available in cache, infinispan create an "null" element in cache. This element won't be updated by putForExternalRead method (look at log above). If you skip "get" method invocation everithing will work fine. By the way, everithing is working fine if you skip "step 2" of the scenario.
> I use workaround for this situation. I check if element exist in the cache not by "containsKey" or "get" method but via cache.keySet().contains(key)
> This works ok
> I'm looking forward for you comments, maybe I'm doing something wrong.
> Thanks
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months
[JBoss JIRA] (ISPN-5285) Logging for Partition Handling and Key availablility too verbose
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-5285?page=com.atlassian.jira.plugin.... ]
Dan Berindei reassigned ISPN-5285:
----------------------------------
Assignee: Dan Berindei
> Logging for Partition Handling and Key availablility too verbose
> ----------------------------------------------------------------
>
> Key: ISPN-5285
> URL: https://issues.jboss.org/browse/ISPN-5285
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Wolf-Dieter Fink
> Assignee: Dan Berindei
> Priority: Major
>
> If, during a cluster spit, the partitions are in DEGRADED mode, access to a key is logged in TRACE and ERROR, but the ERROR seems too verbose.
> TRACE Checking availablility for key=...., status DEGRADED_MODE
> TRACE Part. in DEGRADED_MODE, access is not allowed for key ....
> ERROR ISPN0000306, Key '...' not available. Not all owners .....
> and a full stacktrace
> Logging the information twice in DEBUG and ERROR seems useless.
> But having the full stacktrace might blow up the log file during DEGRADED_MODE. The error should only one line, no stacktrace as this information is not very useful at this point.
> Maybe a TRACE message or lower should include the stacktrace if there is a need for this information.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months
[JBoss JIRA] (ISPN-9835) Jenkinsfile snapshot deployment fails
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-9835?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9835:
-------------------------------
Status: Open (was: New)
> Jenkinsfile snapshot deployment fails
> -------------------------------------
>
> Key: ISPN-9835
> URL: https://issues.jboss.org/browse/ISPN-9835
> Project: Infinispan
> Issue Type: Bug
> Components: Build
> Affects Versions: 10.0.0.Alpha2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.Beta1
>
>
> {noformat}
> 11:16:21.413 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-cli) on project infinispan-checkstyle: The packaging for this project did not assign a file to the build artifact -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-cli) on project infinispan-checkstyle: The packaging for this project did not assign a file to the build artifact
> {noformat}
> https://ci.infinispan.org/job/Infinispan/job/master/1005
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months
[JBoss JIRA] (ISPN-9835) Jenkinsfile snapshot deployment fails
by Dan Berindei (Jira)
Dan Berindei created ISPN-9835:
----------------------------------
Summary: Jenkinsfile snapshot deployment fails
Key: ISPN-9835
URL: https://issues.jboss.org/browse/ISPN-9835
Project: Infinispan
Issue Type: Bug
Components: Build
Affects Versions: 10.0.0.Alpha2
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 10.0.0.Beta1
{noformat}
11:16:21.413 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-cli) on project infinispan-checkstyle: The packaging for this project did not assign a file to the build artifact -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-cli) on project infinispan-checkstyle: The packaging for this project did not assign a file to the build artifact
{noformat}
https://ci.infinispan.org/job/Infinispan/job/master/1005
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
6 years, 9 months