[JBoss JIRA] (ISPN-6425) FileNotFoundException with async indexing backend
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-6425?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-6425:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 9.0.0.Final
Resolution: Done
> FileNotFoundException with async indexing backend
> -------------------------------------------------
>
> Key: ISPN-6425
> URL: https://issues.jboss.org/browse/ISPN-6425
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying, Lucene Directory
> Affects Versions: 8.2.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Fix For: 9.0.0.Final, 9.0.0.Alpha1, 8.2.1.Final
>
>
> The Infinispan directory defaults to {{write_metadata_async=true}} when the indexing backend is configured as async, i.e. {{default.worker.execution}} is {{true}}.
> The {{write_metadata_async=true}} will use {{cache.putAsync}} to write the index file metadata, while still deleting and creating files syncronously. This can lead to
> a stale metadata causing FileNotFoundExceptions when executing queries:
> Suppose a lucene directory contains files \[segments_4, _4.si\]. During normal regime, apart from the user thread, there could be other 2 threads that could be changing the index, the periodic commit thread (since backend is async) and the async deletion of files.
> The following race can happen:
> ||Time||Thread||work type||work||
> |T1|Hibernate Search: Commit Scheduler for index| SYNC | write files segments_5 and _5.si to the index
> |T2|Hibernate Search: Commit Scheduler for index| ASYNC | write the new file list containing \[segments_4, _4.si, segments_5,_5.si\]
> |T3|Hibernate Search: Commit Scheduler for index| ASYNC | enqueue a deletion task for files segments_4 and _4.si
> |T4|Hibernate Search: async deletion of index| SYNC | dequeue deletion task for files segments_4 and _4.si
> |T5|Hibernate Search: async deletion of index| SYNC | delete files segments_4 and _4.si from the index
> |T6|Hibernate Search: async deletion of index| ASYNC | write the new file list containing \[segments_5,_5.si\]
> |T7|User-thread| |open index reader, file list is \[segments_4, _4.si\], highest segment number is 4 (file list is not updated yet)
> |T8|User-thread| |open segments_4
> |T9|User-thread| |FileNotFoundException!
> |T10|remote-thread-User| | new file list received \[segments_4, _4.si, segments_5,_5.si\]
> |T11|remote-thread-User| | new file list received \[segments_5,_5.si\]
> This race can be observed in {{MassIndexerAsyncBackendTest#testMassIndexOnAsync}} that fails intermittently with the exception:
> {noformat}
> Caused by: java.io.FileNotFoundException: Error loading metadata for index file: M|segments_4|commonIndex|-1
> at org.infinispan.lucene.impl.DirectoryImplementor.openInput(DirectoryImplementor.java:138) ~[infinispan-lucene-directory-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT]
> at org.infinispan.lucene.impl.DirectoryLucene.openInput(DirectoryLucene.java:102) ~[infinispan-lucene-directory-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT]
> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:109) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:294) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:493) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:490) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:490) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.StandardDirectoryReader.isCurrent(StandardDirectoryReader.java:344) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:300) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:263) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:251) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> {noformat}
> We should not enable {{write_metadata_async=true}} for async backends. The file list is already {{DeltaAware}}, so writing should not pose a meaningfull overhead when done synchronously.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6356) SecurityException during HotRod server stop
by Vojtech Juranek (JIRA)
[ https://issues.jboss.org/browse/ISPN-6356?page=com.atlassian.jira.plugin.... ]
Vojtech Juranek reassigned ISPN-6356:
-------------------------------------
Assignee: Vojtech Juranek
> SecurityException during HotRod server stop
> -------------------------------------------
>
> Key: ISPN-6356
> URL: https://issues.jboss.org/browse/ISPN-6356
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 8.2.0.Final
> Reporter: Dan Berindei
> Assignee: Vojtech Juranek
>
> The HotRod server should use a SecurityAction to remove the `CrashedMemberDetectorListener` listener (and probably to remove the `ReAddMyAddressListener` listener as well).
> {noformat}
> 20:41:29,066 WARN [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10010: Failed to stop connector HotRodServer: java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'null' lacks 'LISTEN' permission
> at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:86)
> at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:56)
> at org.infinispan.manager.DefaultCacheManager.removeListener(DefaultCacheManager.java:727)
> at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.removeListener(AbstractDelegatingEmbeddedCacheManager.java:193)
> at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.scala:310)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:199)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:189)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6356) SecurityException during HotRod server stop
by Vojtech Juranek (JIRA)
[ https://issues.jboss.org/browse/ISPN-6356?page=com.atlassian.jira.plugin.... ]
Vojtech Juranek updated ISPN-6356:
----------------------------------
Status: Open (was: New)
> SecurityException during HotRod server stop
> -------------------------------------------
>
> Key: ISPN-6356
> URL: https://issues.jboss.org/browse/ISPN-6356
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 8.2.0.Final
> Reporter: Dan Berindei
> Assignee: Vojtech Juranek
>
> The HotRod server should use a SecurityAction to remove the `CrashedMemberDetectorListener` listener (and probably to remove the `ReAddMyAddressListener` listener as well).
> {noformat}
> 20:41:29,066 WARN [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10010: Failed to stop connector HotRodServer: java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'null' lacks 'LISTEN' permission
> at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:86)
> at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:56)
> at org.infinispan.manager.DefaultCacheManager.removeListener(DefaultCacheManager.java:727)
> at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.removeListener(AbstractDelegatingEmbeddedCacheManager.java:193)
> at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.scala:310)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:199)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:189)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6356) SecurityException during HotRod server stop
by Vojtech Juranek (JIRA)
[ https://issues.jboss.org/browse/ISPN-6356?page=com.atlassian.jira.plugin.... ]
Vojtech Juranek updated ISPN-6356:
----------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/4165
> SecurityException during HotRod server stop
> -------------------------------------------
>
> Key: ISPN-6356
> URL: https://issues.jboss.org/browse/ISPN-6356
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 8.2.0.Final
> Reporter: Dan Berindei
> Assignee: Vojtech Juranek
>
> The HotRod server should use a SecurityAction to remove the `CrashedMemberDetectorListener` listener (and probably to remove the `ReAddMyAddressListener` listener as well).
> {noformat}
> 20:41:29,066 WARN [org.infinispan.server.endpoint] (MSC service thread 1-7) DGENDPT10010: Failed to stop connector HotRodServer: java.lang.SecurityException: ISPN000287: Unauthorized access: subject 'null' lacks 'LISTEN' permission
> at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:86)
> at org.infinispan.security.impl.AuthorizationHelper.checkPermission(AuthorizationHelper.java:56)
> at org.infinispan.manager.DefaultCacheManager.removeListener(DefaultCacheManager.java:727)
> at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.removeListener(AbstractDelegatingEmbeddedCacheManager.java:193)
> at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.scala:310)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:199)
> at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:189)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6421) Instantiation of cache based on template should not modify template
by Vladimir Blagojevic (JIRA)
[ https://issues.jboss.org/browse/ISPN-6421?page=com.atlassian.jira.plugin.... ]
Vladimir Blagojevic commented on ISPN-6421:
-------------------------------------------
[~NadirX] The server is not allowing me to create a configuration with configuration set to some template and template to false. Here is what I did:
[domain@localhost:9990 distributed-cache-configuration] ./testconfig:add(configuration=memory-bounded,template=false)
And I get an error from all servers that I need to include MODE. However, as soon as I set mode, I get:
"failure-description" => {"domain-failure-description" => "WFLYCTL0105: configuration is invalid in combination with mode"},
Please confirm and advise on further steps.
> Instantiation of cache based on template should not modify template
> -------------------------------------------------------------------
>
> Key: ISPN-6421
> URL: https://issues.jboss.org/browse/ISPN-6421
> Project: Infinispan
> Issue Type: Bug
> Components: Console
> Affects Versions: 8.2.0.Final
> Reporter: Tristan Tarrant
> Assignee: Vladimir Blagojevic
> Fix For: 9.0.0.Final, 9.0.0.Alpha1, 8.2.1.Final
>
>
> In cache add, instiating a cache A based on template T should not modify T but should create a configuration A under configurations=CONFIGURATIONS with the "template" attribute set to false and the "configuration" attribute set to T and then a cache A with "configuration" A.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6425) FileNotFoundException with async indexing backend
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-6425?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-6425:
------------------------------------
Fix Version/s: 9.0.0.Alpha1
(was: 9.0.0.Final)
> FileNotFoundException with async indexing backend
> -------------------------------------------------
>
> Key: ISPN-6425
> URL: https://issues.jboss.org/browse/ISPN-6425
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying, Lucene Directory
> Affects Versions: 8.2.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Fix For: 9.0.0.Alpha1, 8.2.1.Final
>
>
> The Infinispan directory defaults to {{write_metadata_async=true}} when the indexing backend is configured as async, i.e. {{default.worker.execution}} is {{true}}.
> The {{write_metadata_async=true}} will use {{cache.putAsync}} to write the index file metadata, while still deleting and creating files syncronously. This can lead to
> a stale metadata causing FileNotFoundExceptions when executing queries:
> Suppose a lucene directory contains files \[segments_4, _4.si\]. During normal regime, apart from the user thread, there could be other 2 threads that could be changing the index, the periodic commit thread (since backend is async) and the async deletion of files.
> The following race can happen:
> ||Time||Thread||work type||work||
> |T1|Hibernate Search: Commit Scheduler for index| SYNC | write files segments_5 and _5.si to the index
> |T2|Hibernate Search: Commit Scheduler for index| ASYNC | write the new file list containing \[segments_4, _4.si, segments_5,_5.si\]
> |T3|Hibernate Search: Commit Scheduler for index| ASYNC | enqueue a deletion task for files segments_4 and _4.si
> |T4|Hibernate Search: async deletion of index| SYNC | dequeue deletion task for files segments_4 and _4.si
> |T5|Hibernate Search: async deletion of index| SYNC | delete files segments_4 and _4.si from the index
> |T6|Hibernate Search: async deletion of index| ASYNC | write the new file list containing \[segments_5,_5.si\]
> |T7|User-thread| |open index reader, file list is \[segments_4, _4.si\], highest segment number is 4 (file list is not updated yet)
> |T8|User-thread| |open segments_4
> |T9|User-thread| |FileNotFoundException!
> |T10|remote-thread-User| | new file list received \[segments_4, _4.si, segments_5,_5.si\]
> |T11|remote-thread-User| | new file list received \[segments_5,_5.si\]
> This race can be observed in {{MassIndexerAsyncBackendTest#testMassIndexOnAsync}} that fails intermittently with the exception:
> {noformat}
> Caused by: java.io.FileNotFoundException: Error loading metadata for index file: M|segments_4|commonIndex|-1
> at org.infinispan.lucene.impl.DirectoryImplementor.openInput(DirectoryImplementor.java:138) ~[infinispan-lucene-directory-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT]
> at org.infinispan.lucene.impl.DirectoryLucene.openInput(DirectoryLucene.java:102) ~[infinispan-lucene-directory-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT]
> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:109) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:294) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:493) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:490) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:490) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.StandardDirectoryReader.isCurrent(StandardDirectoryReader.java:344) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:300) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:263) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:251) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> {noformat}
> We should not enable {{write_metadata_async=true}} for async backends. The file list is already {{DeltaAware}}, so writing should not pose a meaningfull overhead when done synchronously.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6337) MassIndexerAsyncBackendTest.testMassIndexOnAsync random failures
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-6337?page=com.atlassian.jira.plugin.... ]
Work on ISPN-6337 stopped by Gustavo Fernandes.
-----------------------------------------------
> MassIndexerAsyncBackendTest.testMassIndexOnAsync random failures
> ----------------------------------------------------------------
>
> Key: ISPN-6337
> URL: https://issues.jboss.org/browse/ISPN-6337
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Query
> Affects Versions: 8.2.0.CR1
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> {code}
> java.lang.RuntimeException: Unexpected!
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:166)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:146)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:122)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:456)
> at org.infinispan.query.distributed.MassIndexerAsyncBackendTest.assertAllIndexed(MassIndexerAsyncBackendTest.java:59)
> at org.infinispan.query.distributed.MassIndexerAsyncBackendTest.testMassIndexOnAsync(MassIndexerAsyncBackendTest.java:53)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
> at org.testng.internal.MethodInvocationHelper$1.runTestMethod(MethodInvocationHelper.java:200)
> at org.jboss.byteman.contrib.bmunit.BMNGAbstractRunner.run(BMNGAbstractRunner.java:58)
> at sun.reflect.GeneratedMethodAccessor169.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.testng.internal.MethodInvocationHelper.invokeHookable(MethodInvocationHelper.java:212)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:707)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.hibernate.search.exception.SearchException: Unable to reopen IndexReader
> at org.hibernate.search.indexes.impl.SharingBufferReaderProvider$PerDirectoryLatestReader.refreshAndGet(SharingBufferReaderProvider.java:242)
> at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.openIndexReader(SharingBufferReaderProvider.java:73)
> at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.openIndexReader(SharingBufferReaderProvider.java:35)
> at org.hibernate.search.reader.impl.ManagedMultiReader.createInstance(ManagedMultiReader.java:68)
> at org.hibernate.search.reader.impl.MultiReaderFactory.openReader(MultiReaderFactory.java:42)
> at org.hibernate.search.query.engine.impl.LuceneHSQuery.buildSearcher(LuceneHSQuery.java:477)
> at org.hibernate.search.query.engine.impl.LuceneHSQuery.buildSearcher(LuceneHSQuery.java:368)
> at org.hibernate.search.query.engine.impl.LuceneHSQuery.queryEntityInfos(LuceneHSQuery.java:131)
> at org.infinispan.query.impl.CacheQueryImpl.list(CacheQueryImpl.java:161)
> at org.infinispan.query.distributed.MassIndexerAsyncBackendTest$1.isSatisfied(MassIndexerAsyncBackendTest.java:62)
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:161)
> ... 31 more
> Caused by: java.io.FileNotFoundException: Error loading metadata for index file: M|segments_4|commonIndex|-1
> at org.infinispan.lucene.impl.DirectoryImplementor.openInput(DirectoryImplementor.java:138)
> at org.infinispan.lucene.impl.DirectoryLucene.openInput(DirectoryLucene.java:102)
> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:109)
> at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:294)
> at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:493)
> at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:490)
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731)
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683)
> at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:490)
> at org.apache.lucene.index.StandardDirectoryReader.isCurrent(StandardDirectoryReader.java:344)
> at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:300)
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:263)
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:251)
> at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:137)
> at org.hibernate.search.indexes.impl.SharingBufferReaderProvider$PerDirectoryLatestReader.refreshAndGet(SharingBufferReaderProvider.java:239)
> ... 41 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6425) FileNotFoundException with async indexing backend
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-6425?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-6425:
------------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/4164
> FileNotFoundException with async indexing backend
> -------------------------------------------------
>
> Key: ISPN-6425
> URL: https://issues.jboss.org/browse/ISPN-6425
> Project: Infinispan
> Issue Type: Bug
> Components: Embedded Querying, Lucene Directory
> Affects Versions: 8.2.0.Final
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Fix For: 9.0.0.Alpha1, 8.2.1.Final
>
>
> The Infinispan directory defaults to {{write_metadata_async=true}} when the indexing backend is configured as async, i.e. {{default.worker.execution}} is {{true}}.
> The {{write_metadata_async=true}} will use {{cache.putAsync}} to write the index file metadata, while still deleting and creating files syncronously. This can lead to
> a stale metadata causing FileNotFoundExceptions when executing queries:
> Suppose a lucene directory contains files \[segments_4, _4.si\]. During normal regime, apart from the user thread, there could be other 2 threads that could be changing the index, the periodic commit thread (since backend is async) and the async deletion of files.
> The following race can happen:
> ||Time||Thread||work type||work||
> |T1|Hibernate Search: Commit Scheduler for index| SYNC | write files segments_5 and _5.si to the index
> |T2|Hibernate Search: Commit Scheduler for index| ASYNC | write the new file list containing \[segments_4, _4.si, segments_5,_5.si\]
> |T3|Hibernate Search: Commit Scheduler for index| ASYNC | enqueue a deletion task for files segments_4 and _4.si
> |T4|Hibernate Search: async deletion of index| SYNC | dequeue deletion task for files segments_4 and _4.si
> |T5|Hibernate Search: async deletion of index| SYNC | delete files segments_4 and _4.si from the index
> |T6|Hibernate Search: async deletion of index| ASYNC | write the new file list containing \[segments_5,_5.si\]
> |T7|User-thread| |open index reader, file list is \[segments_4, _4.si\], highest segment number is 4 (file list is not updated yet)
> |T8|User-thread| |open segments_4
> |T9|User-thread| |FileNotFoundException!
> |T10|remote-thread-User| | new file list received \[segments_4, _4.si, segments_5,_5.si\]
> |T11|remote-thread-User| | new file list received \[segments_5,_5.si\]
> This race can be observed in {{MassIndexerAsyncBackendTest#testMassIndexOnAsync}} that fails intermittently with the exception:
> {noformat}
> Caused by: java.io.FileNotFoundException: Error loading metadata for index file: M|segments_4|commonIndex|-1
> at org.infinispan.lucene.impl.DirectoryImplementor.openInput(DirectoryImplementor.java:138) ~[infinispan-lucene-directory-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT]
> at org.infinispan.lucene.impl.DirectoryLucene.openInput(DirectoryLucene.java:102) ~[infinispan-lucene-directory-9.0.0-SNAPSHOT.jar:9.0.0-SNAPSHOT]
> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:109) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:294) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:493) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:490) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:731) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:490) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.StandardDirectoryReader.isCurrent(StandardDirectoryReader.java:344) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:300) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:263) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:251) ~[lucene-core-5.5.0.jar:5.5.0 2a228b3920a07f930f7afb6a42d0d20e184a943c - mike - 2016-02-16 15:18:34]
> {noformat}
> We should not enable {{write_metadata_async=true}} for async backends. The file list is already {{DeltaAware}}, so writing should not pose a meaningfull overhead when done synchronously.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years