[JBoss JIRA] (ISPN-2103) Concurrent access after removal of an AtomicMap should NOT result in an IllegalStateException when accessed by other threads
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-2103?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant reassigned ISPN-2103:
-------------------------------------
Assignee: Pedro Ruivo (was: Tristan Tarrant)
> Concurrent access after removal of an AtomicMap should NOT result in an IllegalStateException when accessed by other threads
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2103
> URL: https://issues.jboss.org/browse/ISPN-2103
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.1.5.FINAL
> Reporter: Adrian Nistor
> Assignee: Pedro Ruivo
>
> ISPN-1121 introduces an IllegalStateException that is being thrown from AtomicMap methods once the map handle has become stale (ie. removed from cache). In case of concurrent access, the exception is thrown to all threads not just to the thread that performed the removal. This was fine-ish in older versions of Infinispan before introduction of optimistic and pessimistic locking but it should be reconsidered now because:
> 1. It interferes/overlaps with transaction isolation. We should rely on the selected locking scheme (OL/PL) to detect conflicts between transactions and report the problem accordingly. Especially if the stale map is used just for reading - this should be allowed to work rather than stop it.
> 2. This exception is pretty disruptive and awkward to handle. All methods of an AtomicMap can result in this exception and the current thread has to be prepared for handling it if other threads remove the map. Not much transaction isolation.
> 3. Since the TreeCache is backed by AtomicMap nearly all Tree API can throw this.
> The proposed fix consists of:
> 1. removing AtomicHashMap.removed flag and AtomicHashMap.markRemoved() method.
> 2. revising AtomicHashMapProxy.assertValid() method to check only if the map is null (ie. removed) but no longer use the removed flag.
> 3. revising ReadCommittedEntry.commit() method to no longer call markRemoved() method.
> The consequences of these changes are:
> 1. Any further access to a stale map results in IllegalStateException ONLY in the thread that performed the removal. This thread 'knows' the map is stale so it is fine to punish it. Other threads remain unaffected until lock acquisition or commit is performed (depending on locking model).
> 2. Other threads can continue to use the previously obtained map handle for reads without danger of getting an exception.
> 3. If a write operation is done on the map, the results depend on the locking model:
> 3.1 optimistic locking + write skew check: a WriteSkewException will stop the commit during prepare
> 3.2 optimistic locking, no write skew check: the write is committed and the work of the transaction that removed the map is overwritten. The map is effectively revived.
> 3.3 pessimistic locking: same as 3.2
> Please note 3.2 and 3.3 work the same as for normal values (not atomic maps).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 3 months
[JBoss JIRA] (ISPN-2103) Concurrent access after removal of an AtomicMap should NOT result in an IllegalStateException when accessed by other threads
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2103?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2103:
-----------------------------------------------
Brad Maxwell <bmaxwell(a)redhat.com> changed the Status of [bug 922699|https://bugzilla.redhat.com/show_bug.cgi?id=922699] from ASSIGNED to NEW
> Concurrent access after removal of an AtomicMap should NOT result in an IllegalStateException when accessed by other threads
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2103
> URL: https://issues.jboss.org/browse/ISPN-2103
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.1.5.FINAL
> Reporter: Adrian Nistor
> Assignee: Tristan Tarrant
>
> ISPN-1121 introduces an IllegalStateException that is being thrown from AtomicMap methods once the map handle has become stale (ie. removed from cache). In case of concurrent access, the exception is thrown to all threads not just to the thread that performed the removal. This was fine-ish in older versions of Infinispan before introduction of optimistic and pessimistic locking but it should be reconsidered now because:
> 1. It interferes/overlaps with transaction isolation. We should rely on the selected locking scheme (OL/PL) to detect conflicts between transactions and report the problem accordingly. Especially if the stale map is used just for reading - this should be allowed to work rather than stop it.
> 2. This exception is pretty disruptive and awkward to handle. All methods of an AtomicMap can result in this exception and the current thread has to be prepared for handling it if other threads remove the map. Not much transaction isolation.
> 3. Since the TreeCache is backed by AtomicMap nearly all Tree API can throw this.
> The proposed fix consists of:
> 1. removing AtomicHashMap.removed flag and AtomicHashMap.markRemoved() method.
> 2. revising AtomicHashMapProxy.assertValid() method to check only if the map is null (ie. removed) but no longer use the removed flag.
> 3. revising ReadCommittedEntry.commit() method to no longer call markRemoved() method.
> The consequences of these changes are:
> 1. Any further access to a stale map results in IllegalStateException ONLY in the thread that performed the removal. This thread 'knows' the map is stale so it is fine to punish it. Other threads remain unaffected until lock acquisition or commit is performed (depending on locking model).
> 2. Other threads can continue to use the previously obtained map handle for reads without danger of getting an exception.
> 3. If a write operation is done on the map, the results depend on the locking model:
> 3.1 optimistic locking + write skew check: a WriteSkewException will stop the commit during prepare
> 3.2 optimistic locking, no write skew check: the write is committed and the work of the transaction that removed the map is overwritten. The map is effectively revived.
> 3.3 pessimistic locking: same as 3.2
> Please note 3.2 and 3.3 work the same as for normal values (not atomic maps).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 3 months
[JBoss JIRA] (ISPN-7008) IllegalLifeCycleStateException from BlockingRunnable.isReady() escapes to originator
by Dan Berindei (JIRA)
Dan Berindei created ISPN-7008:
----------------------------------
Summary: IllegalLifeCycleStateException from BlockingRunnable.isReady() escapes to originator
Key: ISPN-7008
URL: https://issues.jboss.org/browse/ISPN-7008
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 8.2.4.Final, 9.0.0.Alpha4
Reporter: Dan Berindei
Fix For: 9.0.0.Beta1
The ISPN-5539 fix caught {{IllegalLifecycleStateException}} in {{BlockingRunnable.run()}} and sent a {{CacheNotFoundResponse}} to the originator instead. However, exceptions thrown from {{BlockingRunnable.isReady()}} are not caught, and are instead sent back to the originator. An example stack trace is available in the WFLY-7029 description.
In this case, the exception was thrown directly when the inbound invocation handler tried to submit the command to the remote commands executor. But I see a related problem in {{BlockingTaskAwareExecutorServiceImpl.ControllerThread.run()}}, which stops with the first exception from {{isReady()}}. For {{IllegalLifecycleStateException}} it's probably ok, but for other exceptions it should continue processing new tasks.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 3 months
[JBoss JIRA] (ISPN-7007) Refactor the AS modules into one package
by Tristan Tarrant (JIRA)
Tristan Tarrant created ISPN-7007:
-------------------------------------
Summary: Refactor the AS modules into one package
Key: ISPN-7007
URL: https://issues.jboss.org/browse/ISPN-7007
Project: Infinispan
Issue Type: Task
Components: WildFly modules
Reporter: Tristan Tarrant
Fix For: 9.0.0.Final
- remove as-modules/client
- move as-modules/embedded to become as-modules
- for symmetry with the uberjars, have an org.infinispan.embedded and org.infinispan.remote modules which re-export the appropriate APIs.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 3 months
[JBoss JIRA] (ISPN-7006) Module sets infinispan-as-embedded-modules and infinispan-as-client-modules are not compatible
by Sanne Grinovero (JIRA)
Sanne Grinovero created ISPN-7006:
-------------------------------------
Summary: Module sets infinispan-as-embedded-modules and infinispan-as-client-modules are not compatible
Key: ISPN-7006
URL: https://issues.jboss.org/browse/ISPN-7006
Project: Infinispan
Issue Type: Bug
Components: WildFly modules
Affects Versions: 8.2.4.Final
Reporter: Sanne Grinovero
If one downloads both packages {{infinispan-as-client-modules}} and {{infinispan-as-embedded-modules}} and unzips them both on the same appserver, Infinispan fails to initialize because of LinkageError exceptions.
The {{infinispan-as-embedded-modules}} seems to include all modules of the remote client as well, however the two structures are not fully consistent.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 3 months
[JBoss JIRA] (ISPN-5193) infinispan-embedded and infinispan-cli-interpreter don't work together
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-5193?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec updated ISPN-5193:
--------------------------------------
Priority: Blocker (was: Major)
> infinispan-embedded and infinispan-cli-interpreter don't work together
> ----------------------------------------------------------------------
>
> Key: ISPN-5193
> URL: https://issues.jboss.org/browse/ISPN-5193
> Project: Infinispan
> Issue Type: Bug
> Components: CLI
> Affects Versions: 7.1.0.CR2, 8.0.0.Beta2
> Reporter: Jakub Markos
> Assignee: Sebastian Łaskawiec
> Priority: Blocker
>
> Creating a java application (no container) with both infinispan-embedded and infinispan-cli-interpreter dependencies results in this error when starting a cache manager:
> {code}Exception in thread "main" java.util.ServiceConfigurationError: org.infinispan.lifecycle.ModuleLifecycle: Provider org.infinispan.cli.interpreter.LifecycleCallbacks could not be instantiated: java.lang.ExceptionInInitializerError
> at java.util.ServiceLoader.fail(ServiceLoader.java:224)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
> at org.infinispan.commons.util.ServiceFinder.addServices(ServiceFinder.java:60)
> at org.infinispan.commons.util.ServiceFinder.load(ServiceFinder.java:42)
> at org.infinispan.util.ModuleProperties.resolveModuleLifecycles(ModuleProperties.java:41)
> at org.infinispan.factories.GlobalComponentRegistry.<init>(GlobalComponentRegistry.java:94)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:292)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:271)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:244)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:231)
> at Main64.main(Main64.java:17)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
> Caused by: java.lang.ExceptionInInitializerError
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at java.lang.Class.newInstance(Class.java:374)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
> ... 15 more
> Caused by: java.lang.IllegalArgumentException: Logger implementation class org.infinispan.cli.interpreter.logging.Log_$logger has no matching constructor
> at infinispan.org.jboss.logging.Logger.getMessageLogger(Logger.java:2255)
> at infinispan.org.jboss.logging.Logger.getMessageLogger(Logger.java:2211)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:21)
> at org.infinispan.cli.interpreter.LifecycleCallbacks.<clinit>(LifecycleCallbacks.java:20)
> ... 21 more
> {code}
> Tried with 7.1.0.CR2, config:
> {code}
> <?xml version="1.0" encoding="UTF-8"?>
> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns="urn:infinispan:config:7.1">
> <cache-container default-cache="localcache">
> <local-cache name="localcache"/>
> </cache-container>
> </infinispan>
> {code}
> application:
> {code}
> public static void main(String[] args) throws Exception {
> EmbeddedCacheManager manager = new DefaultCacheManager("config.xml");
> Cache defaultCache = manager.getCache("localcache");
> for (int i = 0; i < 10; i++) {
> defaultCache.put("key"+i, "value"+i);
> }
> Thread.sleep(5000000);
> }
> {code}
> Using infinispan-core dependency instead of infinispan-embedded works.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 3 months
[JBoss JIRA] (ISPN-5193) infinispan-embedded and infinispan-cli-interpreter don't work together
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-5193?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec commented on ISPN-5193:
-------------------------------------------
Unfortunately there's no perfect solution in this scenario...
The main problem here are duplicated classes (class X is available in the Uber Jar and in the module or other Jar). Even if we decided not to relocate JBoss Logging we would probably get a {{ClassCastException}} at some point (because the root problem remains the same). The situation gets even worse when you consider CDI as the spec clearly says that in case of duplicated classes, non portable behavior might occur.
You are also right about the workaround. The suggested solution is to use small jars instead.
I will look into this but I'm afraid there's not much we can do (apart from telling our users that they should not mix Uber Jars with Small Jars).
> infinispan-embedded and infinispan-cli-interpreter don't work together
> ----------------------------------------------------------------------
>
> Key: ISPN-5193
> URL: https://issues.jboss.org/browse/ISPN-5193
> Project: Infinispan
> Issue Type: Bug
> Components: CLI
> Affects Versions: 7.1.0.CR2, 8.0.0.Beta2
> Reporter: Jakub Markos
> Assignee: Sebastian Łaskawiec
>
> Creating a java application (no container) with both infinispan-embedded and infinispan-cli-interpreter dependencies results in this error when starting a cache manager:
> {code}Exception in thread "main" java.util.ServiceConfigurationError: org.infinispan.lifecycle.ModuleLifecycle: Provider org.infinispan.cli.interpreter.LifecycleCallbacks could not be instantiated: java.lang.ExceptionInInitializerError
> at java.util.ServiceLoader.fail(ServiceLoader.java:224)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
> at org.infinispan.commons.util.ServiceFinder.addServices(ServiceFinder.java:60)
> at org.infinispan.commons.util.ServiceFinder.load(ServiceFinder.java:42)
> at org.infinispan.util.ModuleProperties.resolveModuleLifecycles(ModuleProperties.java:41)
> at org.infinispan.factories.GlobalComponentRegistry.<init>(GlobalComponentRegistry.java:94)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:292)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:271)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:244)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:231)
> at Main64.main(Main64.java:17)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
> Caused by: java.lang.ExceptionInInitializerError
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at java.lang.Class.newInstance(Class.java:374)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
> ... 15 more
> Caused by: java.lang.IllegalArgumentException: Logger implementation class org.infinispan.cli.interpreter.logging.Log_$logger has no matching constructor
> at infinispan.org.jboss.logging.Logger.getMessageLogger(Logger.java:2255)
> at infinispan.org.jboss.logging.Logger.getMessageLogger(Logger.java:2211)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:21)
> at org.infinispan.cli.interpreter.LifecycleCallbacks.<clinit>(LifecycleCallbacks.java:20)
> ... 21 more
> {code}
> Tried with 7.1.0.CR2, config:
> {code}
> <?xml version="1.0" encoding="UTF-8"?>
> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns="urn:infinispan:config:7.1">
> <cache-container default-cache="localcache">
> <local-cache name="localcache"/>
> </cache-container>
> </infinispan>
> {code}
> application:
> {code}
> public static void main(String[] args) throws Exception {
> EmbeddedCacheManager manager = new DefaultCacheManager("config.xml");
> Cache defaultCache = manager.getCache("localcache");
> for (int i = 0; i < 10; i++) {
> defaultCache.put("key"+i, "value"+i);
> }
> Thread.sleep(5000000);
> }
> {code}
> Using infinispan-core dependency instead of infinispan-embedded works.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 3 months
[JBoss JIRA] (ISPN-5193) infinispan-embedded and infinispan-cli-interpreter don't work together
by Michal Karm Babacek (JIRA)
[ https://issues.jboss.org/browse/ISPN-5193?page=com.atlassian.jira.plugin.... ]
Michal Karm Babacek reassigned ISPN-5193:
-----------------------------------------
Assignee: Sebastian Łaskawiec
> infinispan-embedded and infinispan-cli-interpreter don't work together
> ----------------------------------------------------------------------
>
> Key: ISPN-5193
> URL: https://issues.jboss.org/browse/ISPN-5193
> Project: Infinispan
> Issue Type: Bug
> Components: CLI
> Affects Versions: 7.1.0.CR2, 8.0.0.Beta2
> Reporter: Jakub Markos
> Assignee: Sebastian Łaskawiec
>
> Creating a java application (no container) with both infinispan-embedded and infinispan-cli-interpreter dependencies results in this error when starting a cache manager:
> {code}Exception in thread "main" java.util.ServiceConfigurationError: org.infinispan.lifecycle.ModuleLifecycle: Provider org.infinispan.cli.interpreter.LifecycleCallbacks could not be instantiated: java.lang.ExceptionInInitializerError
> at java.util.ServiceLoader.fail(ServiceLoader.java:224)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
> at org.infinispan.commons.util.ServiceFinder.addServices(ServiceFinder.java:60)
> at org.infinispan.commons.util.ServiceFinder.load(ServiceFinder.java:42)
> at org.infinispan.util.ModuleProperties.resolveModuleLifecycles(ModuleProperties.java:41)
> at org.infinispan.factories.GlobalComponentRegistry.<init>(GlobalComponentRegistry.java:94)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:292)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:271)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:244)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:231)
> at Main64.main(Main64.java:17)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
> Caused by: java.lang.ExceptionInInitializerError
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at java.lang.Class.newInstance(Class.java:374)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
> ... 15 more
> Caused by: java.lang.IllegalArgumentException: Logger implementation class org.infinispan.cli.interpreter.logging.Log_$logger has no matching constructor
> at infinispan.org.jboss.logging.Logger.getMessageLogger(Logger.java:2255)
> at infinispan.org.jboss.logging.Logger.getMessageLogger(Logger.java:2211)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:21)
> at org.infinispan.cli.interpreter.LifecycleCallbacks.<clinit>(LifecycleCallbacks.java:20)
> ... 21 more
> {code}
> Tried with 7.1.0.CR2, config:
> {code}
> <?xml version="1.0" encoding="UTF-8"?>
> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns="urn:infinispan:config:7.1">
> <cache-container default-cache="localcache">
> <local-cache name="localcache"/>
> </cache-container>
> </infinispan>
> {code}
> application:
> {code}
> public static void main(String[] args) throws Exception {
> EmbeddedCacheManager manager = new DefaultCacheManager("config.xml");
> Cache defaultCache = manager.getCache("localcache");
> for (int i = 0; i < 10; i++) {
> defaultCache.put("key"+i, "value"+i);
> }
> Thread.sleep(5000000);
> }
> {code}
> Using infinispan-core dependency instead of infinispan-embedded works.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 3 months
[JBoss JIRA] (ISPN-6981) AffinityIndexManager fails to index documents in async mode
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-6981?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes commented on ISPN-6981:
-----------------------------------------
Digging a bit more on the {{This lock is no longer being held}} error, it seems a visibility issue with the LockCache:
1) Index {{entity.235}} is started on {{NodeB}}, it acquires and release the lock, common practice when initializing the Infinispan Directory:
{code}
21:04:15,598 DEBUG (pool-3-thread-2:[]) [WorkspaceHolder] HSEARCH000236: Backend for index 'entity.235' started: using asynchronous backend with periodic commits.
21:04:15,608 TRACE (pool-3-thread-2:[]) [BaseLuceneLock] Lock: write.lock acquired for index: entity.235 from AffinityTopologyChangeAsyncTest-NodeB-33534
21:04:15,628 TRACE (pool-3-thread-2:[]) [BaseLuceneLock] Lock: write.lock removed for index: entity.235 from AffinityTopologyChangeAsyncTest-NodeB-33534 (was AffinityTopologyChangeAsyncTest-NodeB-33534)
{code}
2) An index work arrives on {{NodeB}}, and it re-acquires the lock
{code}
21:04:15,632 TRACE (Hibernate Search: Index updates queue processor for index entity.235-1:[]) [BaseLuceneLock] Lock: write.lock acquired for index: entity.235 from AffinityTopologyChangeAsyncTest-NodeB-33534
21:04:15,635 TRACE (Hibernate Search: Index updates queue processor for index entity.235-1:[]) [InfinispanIndexOutput] Opened new IndexOutput for file:_0.fdt in index: entity.235
{code}
3) Due to topology changes, {{NodeB}} looses ownership of segment 235, and thus from index {{entity.235}}. It starts the process of flushing the index as part of the close:
{code}
21:04:16,479 DEBUG (transport-thread-AffinityTopologyChangeAsyncTest-NodeB-p12-t3:[]) [AffinityIndexManager] Topology changed notification for entity.235
21:04:16,480 DEBUG (transport-thread-AffinityTopologyChangeAsyncTest-NodeB-p12-t3:[]) [AffinityIndexManager] Ownership lost to 'AffinityTopologyChangeAsyncTest-NodeD-30445', closing index manager
21:04:16,480 DEBUG (transport-thread-AffinityTopologyChangeAsyncTest-NodeB-p12-t3:[]) [AffinityIndexManager] Flushing directory provider
21:04:16,489 DEBUG (transport-thread-AffinityTopologyChangeAsyncTest-NodeB-p12-t3:[]) [AbstractWorkspaceImpl] HSEARCH000304: Closing index writer for IndexManager 'entity.235'
21:04:17,542 DEBUG (transport-thread-AffinityTopologyChangeAsyncTest-NodeB-p12-t3:[]) [DirectoryImplementor] Removed file: segments_1 from index: entity.235 from AffinityTopologyChangeAsyncTest-NodeB-33534
21:04:17,543 TRACE (transport-thread-AffinityTopologyChangeAsyncTest-NodeB-p12-t3:[]) [InfinispanIndexOutput] Refreshed file listing view
{code}
4) While the flushing is happening, the new owner of segment 235, {{NodeD}}, *for an unknown reason, acquires the lock* for {{entity.235}}, when it shouldn't, since in theory {{NodeB}} still holds the lock.
{code}
21:04:17,835 DEBUG (remote-thread-AffinityTopologyChangeAsyncTest-NodeD-p26-t5:[]) [AffinityIndexManager] *Lock holder for 235 is null*
21:04:17,843 TRACE (Hibernate Search: Index updates queue processor for index entity.235-1:[]) [BaseLuceneLock] Lock: write.lock acquired for index: entity.235 from AffinityTopologyChangeAsyncTest-NodeD-30445
21:04:17,849 TRACE (Hibernate Search: Index updates queue processor for index entity.235-1:[]) [InfinispanIndexOutput] Opened new IndexOutput for file:_1.fdt in index: entity.235
{code}
5) When {{NodeB}} finishes flushing the index started on 3), it removes the lock from the index, that was strangely held by {{NodeD}}
{code}
21:04:18,491 TRACE (transport-thread-AffinityTopologyChangeAsyncTest-NodeB-p12-t3:[]) [BaseLuceneLock] Lock: write.lock removed for index: entity.235 from AffinityTopologyChangeAsyncTest-NodeB-33534 (was AffinityTopologyChangeAsyncTest-NodeD-30445)
21:04:18,491 TRACE (transport-thread-AffinityTopologyChangeAsyncTest-NodeB-p12-t3:[]) [IndexWriterHolder] IndexWriter closed
{code}
6) The Comit scheduler on {{NodeD}} starts to fail since it does not lock the lock anymore.
> AffinityIndexManager fails to index documents in async mode
> -----------------------------------------------------------
>
> Key: ISPN-6981
> URL: https://issues.jboss.org/browse/ISPN-6981
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying
> Affects Versions: 9.0.0.Alpha4
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> During topology changes in async mode ("<index>.worker.execution" set to "async"), the {{AfinityIndexManager}} sometimes fails to index entries. Some of errors thrown:
> {noformat}
> 19:06:04,638 ERROR (Hibernate Search: Index updates queue processor for index entity.5-1) [LogErrorHandler] HSEARCH000058: Exception occurred
> org.apache.lucene.store.LockObtainFailedException: lock instance already assigned
> {noformat}
> {noformat}
> 18:53:59,553 ERROR (Hibernate Search: Index updates queue processor for index entity.2-1) [LuceneBackendQueueTask] HSEARCH000073: Error in
> backend
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:720)
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:734)
> {noformat}
> {noformat}
> 18:55:31,328 ERROR (Hibernate Search: Commit Scheduler for index entity.143-1) [AffinityErrorHandler] HSEARCH000117: IOException
> on the IndexWriter
> java.io.IOException: This lock is no longer being held
> at org.infinispan.lucene.impl.BaseLuceneLock.ensureValid(BaseLuceneLock.java:89)
> at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
> at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
> at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.<init>(Lucene50PostingsWriter.java:105)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 3 months