[JBoss JIRA] (ISPN-3938) AdvancedAsyncCacheLoader.process() concurrency issues
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3938?page=com.atlassian.jira.plugin.... ]
Dan Berindei reassigned ISPN-3938:
----------------------------------
Assignee: Dan Berindei (was: William Burns)
> AdvancedAsyncCacheLoader.process() concurrency issues
> -----------------------------------------------------
>
> Key: ISPN-3938
> URL: https://issues.jboss.org/browse/ISPN-3938
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Loaders and Stores
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 8.0.0.Final
>
>
> {{AdvancedAsyncCacheLoader.process()}} calls {{advancedLoader().process()}} to collect all the keys in the store, but the HashSet used to collect the keys it not thread-safe. This can cause problems, e.g. during state transfer:
> {noformat}
> WARN cheTopologyControlCommand | ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=sessions, type=CH_UPDATE, sender=alfie-lt-46127, joinInfo=null, topologyId=3, currentCH=DefaultConsistentHash{numSegments=60, numOwners=1, members=[alfie-lt-46127]}, pendingCH=null, throwable=null, viewId=1}java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$KeyIterator.next(HashMap.java:960)
> at org.infinispan.persistence.async.AdvancedAsyncCacheLoader.process(AdvancedAsyncCacheLoader.java:80)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:414)
> at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:910)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:393)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:100)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleConsistentHashUpdate(LocalTopologyManagerImpl.java:191)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:152)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
> at org.infinispan.topology.ClusterTopologyManagerImpl$3.run(ClusterTopologyManagerImpl.java:606)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5555) Custom externalizer for Cache Store (jdbc)
by Veli Cris (JIRA)
[ https://issues.jboss.org/browse/ISPN-5555?page=com.atlassian.jira.plugin.... ]
Veli Cris closed ISPN-5555.
---------------------------
Resolution: Done
> Custom externalizer for Cache Store (jdbc)
> ------------------------------------------
>
> Key: ISPN-5555
> URL: https://issues.jboss.org/browse/ISPN-5555
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.2.Final
> Environment: WildFly 8.2.0 / Linux
> Reporter: Veli Cris
>
> For a POJO object I created a class that implements Externalizer interface. I used @SerializeWith annotation for POJO class and I printed out debugging information in writeObject() method defined in Externalizer interface and I can see the log it's ok.
> The problem is that in my database table (defined as Datasource), in data column I see some hexa code and not what I overrided in writeObject() method.
> Thank you,
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5470) Remote-executor threads should not block to acquire locks
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-5470?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-5470:
--------------------------------
Fix Version/s: 8.0.0.Beta1
(was: 8.0.0.Alpha2)
> Remote-executor threads should not block to acquire locks
> ---------------------------------------------------------
>
> Key: ISPN-5470
> URL: https://issues.jboss.org/browse/ISPN-5470
> Project: Infinispan
> Issue Type: Task
> Components: Core
> Affects Versions: 7.2.1.Final
> Reporter: Dan Berindei
> Fix For: 8.0.0.Beta1
>
>
> Currently, enabling the queue on the remote-executor thread pool can cause deadlocks, because a CommitCommand/1PCPrepareCommand could end up in the queue while a remote-executor thread is busy waiting to acquire the same lock that this commit would release.
> If trying to acquire a lock would free the thread until the key has been acquired, we could enable the queue on the remote-executor/OOB thread pools, and we would need a lot less threads for the same load.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5454) XSite: RetryMechanismTest random failures
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-5454?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-5454:
--------------------------------
Fix Version/s: 8.0.0.Beta1
(was: 8.0.0.Alpha2)
> XSite: RetryMechanismTest random failures
> -----------------------------------------
>
> Key: ISPN-5454
> URL: https://issues.jboss.org/browse/ISPN-5454
> Project: Infinispan
> Issue Type: Bug
> Components: Cross-Site Replication
> Affects Versions: 7.2.1.Final
> Reporter: Dan Berindei
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 8.0.0.Beta1
>
>
> {{ClusteredCacheBackupReceiver.awaitRemoteTask()}} doesn't respect the state push command's timeout, at least when it's smaller than the sync replication timeout in the target cache. When that happens, the state provider will resend the state, and there will be 2 state push commands executing at the same time.
> RetryMechanismTest changes the state push timeout to 2 seconds, but the sync replication timeout stays at 15 seconds. This causes failures in {{testRetryLocally}} and {{testFailRetryLocally}}, if it takes more than 2 seconds to suspect the killed node.
> {noformat}
> 10:02:13,007 TRACE (asyncTransportThread-8,NodeN:) [RetryOnFailureXSiteCommand] Sending XSiteStatePushCommand{cacheName=___defaultcache, timeout=2000 (1 keys)} to [NYC (sync, timeout=2000)]
> 10:02:16,008 TRACE (asyncTransportThread-8,NodeN:) [RetryOnFailureXSiteCommand] Sending XSiteStatePushCommand{cacheName=___defaultcache, timeout=2000 (1 keys)} to [NYC (sync, timeout=2000)]
> 10:02:16,040 TRACE (asyncTransportThread-4,NodeP:) [RpcManagerImpl] replication exception:
> org.infinispan.remoting.transport.jgroups.SuspectException: Node NodeQ-56809 was suspected
> 10:02:16,040 TRACE (asyncTransportThread-0,NodeP:) [RpcManagerImpl] replication exception:
> org.infinispan.remoting.transport.jgroups.SuspectException: Node NodeQ-56809 was suspected
> 10:02:19,147 ERROR (testng-RetryMechanismTest:) [UnitTestTestNGListener] Test testFailRetryLocally(org.infinispan.xsite.statetransfer.failures.RetryMechanismTest) failed.
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:245)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:252)
> at org.infinispan.xsite.statetransfer.failures.RetryMechanismTest.testFailRetryLocally(RetryMechanismTest.java:227)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5403) NullPointerException on starting CacheStore
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-5403?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-5403:
--------------------------------
Fix Version/s: 8.0.0.Beta1
(was: 8.0.0.Alpha2)
> NullPointerException on starting CacheStore
> -------------------------------------------
>
> Key: ISPN-5403
> URL: https://issues.jboss.org/browse/ISPN-5403
> Project: Infinispan
> Issue Type: Bug
> Reporter: Sebastian Łaskawiec
> Assignee: Sebastian Łaskawiec
> Priority: Minor
> Fix For: 8.0.0.Beta1, 7.2.3.Final
>
>
> Relevant stacktrace:
> {code}
> boss.infinispan.clustered.default: Failed to start service
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1936) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_31]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_31]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31]
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
> at org.infinispan.CacheImpl.start(CacheImpl.java:779)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:587)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:542)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:420)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:434)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:104)
> at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:101)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:48)
> at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:109)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:85)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
> ... 3 more
> Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders
> at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:156)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_31]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_31]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_31]
> at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> ... 23 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.persistence.factory.CacheStoreFactoryRegistry.processStoreConfiguration(CacheStoreFactoryRegistry.java:48)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.createLoadersAndWriters(PersistenceManagerImpl.java:529)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:114)
> ... 28 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5506) Dist/ReplEmbeddedRestHotRodTest failing randomly with "java.net.SocketException: Socket closed"
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-5506?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-5506:
--------------------------------
Fix Version/s: 8.0.0.Beta1
(was: 8.0.0.Alpha2)
> Dist/ReplEmbeddedRestHotRodTest failing randomly with "java.net.SocketException: Socket closed"
> -----------------------------------------------------------------------------------------------
>
> Key: ISPN-5506
> URL: https://issues.jboss.org/browse/ISPN-5506
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Protocols
> Affects Versions: 7.2.1.Final, 8.0.0.Alpha1
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Fix For: 8.0.0.Beta1
>
> Attachments: infinispan_5506.tgz
>
>
> {code}
> testEmbeddedPutRestHotRodGet(org.infinispan.it.compatibility.DistEmbeddedRestHotRodTest) Time elapsed: 0.002 sec <<< FAILURE!
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at java.net.Socket.connect(Socket.java:538)
> at java.net.Socket.<init>(Socket.java:434)
> at java.net.Socket.<init>(Socket.java:286)
> at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
> at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
> at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
> at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
> at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> at org.infinispan.it.compatibility.ReplEmbeddedRestHotRodTest.testEmbeddedPutRestHotRodGet(ReplEmbeddedRestHotRodTest.java:80)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5505) DistTopologyChangeUnderLoadTest randomly fails with IllegalLifecycleStateException
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-5505?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-5505:
--------------------------------
Fix Version/s: 8.0.0.Beta1
(was: 8.0.0.Alpha2)
> DistTopologyChangeUnderLoadTest randomly fails with IllegalLifecycleStateException
> ----------------------------------------------------------------------------------
>
> Key: ISPN-5505
> URL: https://issues.jboss.org/browse/ISPN-5505
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Protocols
> Affects Versions: 7.2.1.Final, 8.0.0.Alpha1
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Labels: testsuite_stability
> Fix For: 8.0.0.Beta1
>
>
> {code}
> testPutsSucceedWhileTopologyChanges(org.infinispan.client.hotrod.DistTopologyChangeUnderLoadTest) Time elapsed: 0.334 sec <<< FAILURE!
> java.util.concurrent.ExecutionException: org.infinispan.client.hotrod.exceptions.RemoteIllegalLifecycleStateException:Request for message id[1099] returned server error (status=0x88): org.infinispan.IllegalLifecycleStateException: ISPN000324: Default cache is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.infinispan.client.hotrod.DistTopologyChangeUnderLoadTest.testPutsSucceedWhileTopologyChanges(DistTopologyChangeUnderLoadTest.java:67)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.client.hotrod.exceptions.RemoteIllegalLifecycleStateException:Request for message id[1099] returned server error (status=0x88): org.infinispan.IllegalLifecycleStateException: ISPN000324: Default cache is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:340)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:126)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:112)
> at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
> at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:57)
> at org.infinispan.client.hotrod.impl.operations.PutOperation.executeOperation(PutOperation.java:31)
> at org.infinispan.client.hotrod.impl.operations.PutOperation.executeOperation(PutOperation.java:20)
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:52)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:247)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79)
> at org.infinispan.client.hotrod.DistTopologyChangeUnderLoadTest$PutHammer.call(DistTopologyChangeUnderLoadTest.java:80)
> at org.infinispan.client.hotrod.DistTopologyChangeUnderLoadTest$PutHammer.call(DistTopologyChangeUnderLoadTest.java:72)
> at org.infinispan.test.AbstractInfinispanTest$LoggingCallable.call(AbstractInfinispanTest.java:435)
> ... 4 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years