[JBoss JIRA] (ISPN-2965) L1 and early invalidation leaves inconsistent state
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-2965?page=com.atlassian.jira.plugin.... ]
Work on ISPN-2965 stopped by William Burns.
> L1 and early invalidation leaves inconsistent state
> ---------------------------------------------------
>
> Key: ISPN-2965
> URL: https://issues.jboss.org/browse/ISPN-2965
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache, Transactions
> Affects Versions: 5.2.1.Final
> Reporter: Sebastian Tusk
> Assignee: William Burns
> Labels: 5.2.x
> Fix For: 6.0.0.Alpha3
>
>
> In a distributed transactional cache with L1 enabled I can observe the following.
> Prepare cache by adding an entry with Cache.put( k, v1 ).
> 1. Node B starts with adding a changed value. Cache.put( k, v2 )
> 2. Node B TxDistributionInterceptor.visitPrepareCommand flushL1Caches sends invalidations.
> 3. Node A calls Cache.get( k ) retrieves v1 and stores this value in L1.
> 4. Node B proceeds with transaction.
> The result is that Node A answers subsequent Cache.get(k) with v1 and Node B answers with v2.
> It seems the invalidation is either send to early or must be synchronized in some way with the transaction.
> Cache config:
> <namedCache name="entity">
> <jmxStatistics enabled="true" />
> <clustering mode="dist">
> <stateTransfer fetchInMemoryState="false" timeout="20000" />
> <async />
> <l1 enabled="true" />
> <hash numOwners="1"/>
> </clustering>
> <locking isolationLevel="READ_COMMITTED"
> lockAcquisitionTimeout="15000" useLockStriping="false" />
> <eviction maxEntries="10000" strategy="LRU" />
> <expiration maxIdle="100000" wakeUpInterval="5000"/>
> <storeAsBinary storeKeysAsBinary="true" storeValuesAsBinary="false" enabled="false" />
> <transaction transactionMode="TRANSACTIONAL" autoCommit="false" lockingMode="OPTIMISTIC"/>
> </namedCache>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-3399) Infinispan Server Configuration doesn't work properly for loader or store elements
by William Burns (JIRA)
William Burns created ISPN-3399:
-----------------------------------
Summary: Infinispan Server Configuration doesn't work properly for loader or store elements
Key: ISPN-3399
URL: https://issues.jboss.org/browse/ISPN-3399
Project: Infinispan
Issue Type: Bug
Components: Loaders and Stores, Server
Affects Versions: 6.0.0.Alpha2
Reporter: William Burns
Assignee: Tristan Tarrant
While working on ISPN-3350, I tried using a store and loader configuration value for my cache.
{code}
<local-cache name="local">
<store class="org.infinispan.loaders.file.SingleFileCacheStore"/>
</local-cache>
{code}
However when starting the server I am met with the following error and the cache is not started properly:
{quote}
08:23:25,112 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC00001: Failed to start service jboss.endpoint.hotrod.hotrod-connector: org.jboss.msc.service.StartException in service jboss.endpoint.hotrod.hotrod-connector: JDGS010004: Failed to start HotRodServer
at org.infinispan.server.endpoint.subsystem.ProtocolServerService.start(ProtocolServerService.java:106)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.loaders.manager.CacheLoaderManagerImpl.start() on object of type CacheLoaderManagerImpl
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:204)
at org.infinispan.CacheImpl.start(CacheImpl.java:664)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:666)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:629)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:522)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:536)
at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:118)
at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:109)
at org.infinispan.server.hotrod.HotRodServer$$anonfun$preStartCaches$1.apply(HotRodServer.scala:78)
at org.infinispan.server.hotrod.HotRodServer$$anonfun$preStartCaches$1.apply(HotRodServer.scala:76)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1156)
at org.infinispan.server.hotrod.HotRodServer.preStartCaches(HotRodServer.scala:76)
at org.infinispan.server.hotrod.HotRodServer.startTransport(HotRodServer.scala:65)
at org.infinispan.server.core.AbstractProtocolServer.start(AbstractProtocolServer.scala:39)
at org.infinispan.server.hotrod.HotRodServer.start(HotRodServer.scala:51)
at org.infinispan.server.hotrod.HotRodServer.start(HotRodServer.scala:27)
at org.infinispan.server.endpoint.subsystem.ProtocolServerService.startProtocolServer(ProtocolServerService.java:123)
at org.infinispan.server.endpoint.subsystem.ProtocolServerService.start(ProtocolServerService.java:100)
... 5 more
Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders
at org.infinispan.loaders.manager.CacheLoaderManagerImpl.start(CacheLoaderManagerImpl.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)
... 28 more
Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000249: The cache loader configuration org.infinispan.configuration.cache.LegacyStoreConfiguration does not specify the loader class using @ConfigurationFor
at org.infinispan.loaders.manager.CacheLoaderManagerImpl.createCacheLoader(CacheLoaderManagerImpl.java:328)
at org.infinispan.loaders.manager.CacheLoaderManagerImpl.createCacheLoader(CacheLoaderManagerImpl.java:316)
at org.infinispan.loaders.manager.CacheLoaderManagerImpl.start(CacheLoaderManagerImpl.java:129)
... 33 more
{quote}
The error stems from the fact that the LegacyStoreConfiguration is used for caches that only provide the classname and not configuration. I am not certain if this is a fix solely in the server or would need some additional changes in Infinispan core. Also maybe there is some other configuration needed? If so the schema should have a requirement for the field or have it more clearly shown in exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-2895) org.infinispan.lucene.InfinispanDirectoryIOTest.testReadWholeFile fails randomly
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2895?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2895:
-----------------------------------------------
Richard Janík <rjanik(a)redhat.com> made a comment on [bug 963166|https://bugzilla.redhat.com/show_bug.cgi?id=963166]
Also seen in 6.1.1.ER4 with Infinispan 5.2.7.Final. Bumping to 6.2.0 ?.
> org.infinispan.lucene.InfinispanDirectoryIOTest.testReadWholeFile fails randomly
> --------------------------------------------------------------------------------
>
> Key: ISPN-2895
> URL: https://issues.jboss.org/browse/ISPN-2895
> Project: Infinispan
> Issue Type: Bug
> Components: Lucene Directory
> Affects Versions: 5.2.2.Final
> Reporter: Anna Manukyan
> Assignee: Sanne Grinovero
> Labels: stable_embedded_query, testsuite_stability
> Fix For: 6.0.0.Alpha3, 6.0.0.Final
>
>
> The test fails randomly and the error message is:
> {code}
> java.lang.NullPointerException
> at org.infinispan.lucene.DirectoryIntegrityCheck.verifyDirectoryStructure(DirectoryIntegrityCheck.java:76)
> at org.infinispan.lucene.DirectoryIntegrityCheck.verifyDirectoryStructure(DirectoryIntegrityCheck.java:56)
> at org.infinispan.lucene.InfinispanDirectoryIOTest.verifyOnBuffer(InfinispanDirectoryIOTest.java:184)
> at org.infinispan.lucene.InfinispanDirectoryIOTest.testReadWholeFile(InfinispanDirectoryIOTest.java:153)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-2903) Manual eviction should not delete entry from cache store
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2903?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2903:
-----------------------------------------------
Radoslav Husar <rhusar(a)redhat.com> made a comment on [bug 900549|https://bugzilla.redhat.com/show_bug.cgi?id=900549]
Thanks Aleks for confirming on EC2 too.
> Manual eviction should not delete entry from cache store
> --------------------------------------------------------
>
> Key: ISPN-2903
> URL: https://issues.jboss.org/browse/ISPN-2903
> Project: Infinispan
> Issue Type: Bug
> Components: Eviction
> Affects Versions: 5.2.3.Final
> Reporter: Paul Ferraro
> Assignee: Galder Zamarreño
> Priority: Critical
> Labels: 5.2.x, jdg6
> Fix For: 5.2.5.Final, 5.3.0.Alpha1, 5.3.0.Final
>
> Attachments: AtomicMapServlet.java, AtomicMapTestCase.java, server.log, server.log
>
>
> Here's the scenario:
> Given 2 nodes with REPL_SYNC cache with passivating cache store (e.g. default web cache in AS7).
> 1. Create cache entry containing atomic map with 2 map entries on node1.
> 2. Passivate that cache entry on node2 via manual evict.
> 3. Modify 1 of the atomic map entries within the cache entry on node1.
> 4. Lookup atomic map on node2. It only contains 1 map entry - the map entry modified in step 3. The other map entry is lost.
> It's a side effect of ISPN-2384, where some changes were made to tighten the passivation/activation scenarios, but it did not cover manual eviction calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-2903) Manual eviction should not delete entry from cache store
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2903?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2903:
-----------------------------------------------
Aleksandar Kostadinov <akostadi(a)redhat.com> made a comment on [bug 900549|https://bugzilla.redhat.com/show_bug.cgi?id=900549]
I'm also positive the problem is finally fixed. I run the tests several times with ER4 in the EC2 where the issue was very easy to reproduce. Then increased the load and ran for a day. There was no single NPE.
Cheers!
> Manual eviction should not delete entry from cache store
> --------------------------------------------------------
>
> Key: ISPN-2903
> URL: https://issues.jboss.org/browse/ISPN-2903
> Project: Infinispan
> Issue Type: Bug
> Components: Eviction
> Affects Versions: 5.2.3.Final
> Reporter: Paul Ferraro
> Assignee: Galder Zamarreño
> Priority: Critical
> Labels: 5.2.x, jdg6
> Fix For: 5.2.5.Final, 5.3.0.Alpha1, 5.3.0.Final
>
> Attachments: AtomicMapServlet.java, AtomicMapTestCase.java, server.log, server.log
>
>
> Here's the scenario:
> Given 2 nodes with REPL_SYNC cache with passivating cache store (e.g. default web cache in AS7).
> 1. Create cache entry containing atomic map with 2 map entries on node1.
> 2. Passivate that cache entry on node2 via manual evict.
> 3. Modify 1 of the atomic map entries within the cache entry on node1.
> 4. Lookup atomic map on node2. It only contains 1 map entry - the map entry modified in step 3. The other map entry is lost.
> It's a side effect of ISPN-2384, where some changes were made to tighten the passivation/activation scenarios, but it did not cover manual eviction calls.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-3366) Data loss when entry forwarding to primary owner and primary owner shutdown
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3366?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-3366:
------------------------------------
Checking the topology id before invoking the command on the primary owner wouldn't be enough: the primary owner may have already installed a new topology in which it is no longer the primary owner, and it may not forward the command to the other owners (or it may not do anything, if it's no longer an owner). This could very well be the cause of issue ISPN-3357...
I had another idea, to let the command execute and then check in StateTransferInterceptor whether the topology id changed. If so, we would retry the command on the new primary owner. The problem is that the check yields too many false positives, and if we retry a put command after it was successfully executed then we won't get the correct return value. This is even more important for conditional commands.
> Data loss when entry forwarding to primary owner and primary owner shutdown
> ---------------------------------------------------------------------------
>
> Key: ISPN-3366
> URL: https://issues.jboss.org/browse/ISPN-3366
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache
> Affects Versions: 5.2.4.Final, 6.0.0.Alpha1
> Reporter: Takayoshi Kimura
> Assignee: Dan Berindei
> Priority: Critical
> Fix For: 5.2.8.Final, 6.0.0.Alpha3, 6.0.0.Final
>
> Attachments: ISPN-3366-full-logs-3rd.zip, ISPN-3366-logs.zip
>
>
> Looks like a problem in entry forwarding.
> Here is test scenario:
> * DIST numOwners=2, start with 4 nodes cluster then normal shutdown 1 node during load
> * HotRod putIfAbsent accesses from 40 threads (1 process, 1 remote cache instance), 40000 entries total
> After the test run, the numberOfEntries on each node are:
> * node1: 26608
> * node2: 26622
> * node3: 26746
> * node4: 0
> Total is 79976 and HotRod client received 11 errors, so 79976 + (11 * 2) = 79998. It means 1 entry is completely missing.
> Let's take a look at the missing entry, hash(thread16key59) = 574ff563.
> Current CH: owners(574ff563) are [node4, node1]
> The events sequence is:
> * hotrod -> node1
> * node1 forwarding it to primary owner node4
> * node4 doesn't process the forwarded entry, shutdown
> Result owners(7c29bccb) is [] empty. This entry is completely lost without any errors.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months