[JBoss JIRA] (ISPN-8980) High concurrency : Infinispan Directory Provider: Lucene : Error loading metadata for index file
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8980?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes commented on ISPN-8980:
-----------------------------------------
Answering all the questions:
>> The same change has been applied to a clustered env and it is working there too (when backend_execution is set as sync).
You should also change your neutrino-hibernatesearch-infinispan.xml to use a <replicated-cache> cache for {{LuceneIndexesLocking}} otherwise you'll likely see index corruption since the lock is only visible to a local node
>> For this version (6.0.2) too, will infinispan handle the locking itself?
Yes, the infinispan directory always used a locking cache since its inception.
On a side note, I'd strongly encourage you to upgrade as this version is no longer maintained and besides that, newer versions Infinispan have orders of magnitude performance improvement, which is many cases you can even avoid using async indexing since the sync performs very well.
>> When we set the backend_execution as async, the exception starts coming again.
Could you re-try changing the LuceneIndexesLocking cache as suggested above?
> High concurrency : Infinispan Directory Provider: Lucene : Error loading metadata for index file
> ------------------------------------------------------------------------------------------------
>
> Key: ISPN-8980
> URL: https://issues.jboss.org/browse/ISPN-8980
> Project: Infinispan
> Issue Type: Bug
> Components: Lucene Directory
> Affects Versions: 8.2.5.Final
> Reporter: Debashish Bharali
> Assignee: Gustavo Fernandes
> Priority: Critical
> Attachments: SysOutLogs.txt, neutrino-hibernate-search-worker-jgroups.xml, neutrino-hibernatesearch-infinispan.xml
>
>
> During high concurrency of action, we are getting *{color:red}'Error loading metadata for index file'{color}* even in *{color:red}Non-Clustered{color}* env.
> *Hibernate Search Indexes (Lucene Indexes) - 5.7.0.Final*
> *Infinispan - 8.2.5.Final*
> *infinispan-directory-provider-8.2.5.Final*
> *jgroups-3.6.7.Final*
> *Worker Backend : JGroups*
> *Worker Execution: Sync*
> *write_metadata_async: false (implicitly)*
> *Note:* Currently we are on Non-Clustered env. We are moving to Clustered Env within few days.
> On analyzing the code, and putting some additional SYSOUT loggers into FileListOperations and DirectoryImplementor classes, we have established the following points:
> # This is happening during high concurrency on non-clustered env.
> # One thread *'T1'* is deleting a segment and segment name *'SEG1'* from the *'FileListCacheKey'* list* stored in MetaDatacache*.
> # Concurrently, at the same time, another thread *'T2'* is looping through the FileList ['copy list' from MetadataCache - for -FileListCacheKey - provided by toArray method of *FileListOperations* (changes also being done in the corresponding original list by T1 thread) ].
> # *'T2'* is calling open input method on each segment name - getting corresponding Metadata segment from *MetadataCache*.
> # However, for *'T2'*, the *'copy list'* still contains the name of segment *'SEG1'*.
> # So while looping through the list, *'T2'* tries to get Segment from MetadataCache for segment name *'SEG1'*.
> # But at this instant, *segment* corresponding to segment name *'SEG1'*, has been already removed from *MetadataCache* by *'T1'*.
> # This results in *'java.io.FileNotFoundException: Error loading metadata for index file'* for segment name *'SEG1'*
> # As mentioned earlier, this happens more often during high concurrency.
> *{color:red}On a standalone server (non-clustered), we are getting below error intermittently:{color}*
> Full Stack trace:
> 2018-03-19 17:29:11,938 ERROR [Hibernate Search sync consumer thread for index com.nucleus.integration.ws.server.globalcustomer.entity.GlobalCustomer] o.h.s.e.i.LogErrorHandler [LogErrorHandler.java:69]
> *{color:red}HSEARCH000058: Exception occurred java.io.FileNotFoundException: Error loading metadata for index file{color}*: M|segments_w6|com.nucleus.integration.ws.server.globalcustomer.entity.GlobalCustomer|-1
> Primary Failure:
> Entity com.nucleus.integration.ws.server.globalcustomer.entity.GlobalCustomer Id 1649990024999813056 Work Type org.hibernate.search.backend.AddLuceneWork
> java.io.FileNotFoundException: Error loading metadata for index file: M|segments_w6|com.nucleus.integration.ws.server.globalcustomer.entity.GlobalCustomer|-1
> at org.infinispan.lucene.impl.DirectoryImplementor.openInput(DirectoryImplementor.java:138) ~[infinispan-lucene-directory-8.2.5.Final.jar:8.2.5.Final]
> at org.infinispan.lucene.impl.DirectoryLucene.openInput(DirectoryLucene.java:102) ~[infinispan-lucene-directory-8.2.5.Final.jar:8.2.5.Final]
> at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:109) ~[lucene-core-5.5.4.jar:5.5.4 31012120ebbd93744753eb37f1dbc5e654628291 - jpountz - 2017-02-08 19:08:03]
> at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:294) ~[lucene-core-5.5.4.jar:5.5.4 31012120ebbd93744753eb37f1dbc5e654628291 - jpountz - 2017-02-08 19:08:03]
> at org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:171) ~[lucene-core-5.5.4.jar:5.5.4 31012120ebbd93744753eb37f1dbc5e654628291 - jpountz - 2017-02-08 19:08:03]
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:949) ~[lucene-core-5.5.4.jar:5.5.4 31012120ebbd93744753eb37f1dbc5e654628291 - jpountz - 2017-02-08 19:08:03]
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:126) ~[hibernate-search-engine-5.7.0.Final.jar:5.7.0.Final]
> at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:92) ~[hibernate-search-engine-5.7.0.Final.jar:5.7.0.Final]
> at org.hibernate.search.backend.impl.lucene.AbstractCommitPolicy.getIndexWriter(AbstractCommitPolicy.java:33) ~[hibernate-search-engine-5.7.0.Final.jar:5.7.0.Final]
> at org.hibernate.search.backend.impl.lucene.SharedIndexCommitPolicy.getIndexWriter(SharedIndexCommitPolicy.java:77) ~[hibernate-search-engine-5.7.0.Final.jar:5.7.0.Final]
> at org.hibernate.search.backend.impl.lucene.SharedIndexWorkspaceImpl.getIndexWriter(SharedIndexWorkspaceImpl.java:36) ~[hibernate-search-engine-5.7.0.Final.jar:5.7.0.Final]
> at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriterDelegate(AbstractWorkspaceImpl.java:203) [hibernate-search-engine-5.7.0.Final.jar:5.7.0.Final]
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:81) [hibernate-search-engine-5.7.0.Final.jar:5.7.0.Final]
> at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:46) [hibernate-search-engine-5.7.0.Final.jar:5.7.0.Final]
> at org.hibernate.search.backend.impl.lucene.SyncWorkProcessor$Consumer.applyChangesets(SyncWorkProcessor.java:165) [hibernate-search-engine-5.7.0.Final.jar:5.7.0.Final]
> at org.hibernate.search.backend.impl.lucene.SyncWorkProcessor$Consumer.run(SyncWorkProcessor.java:151) [hibernate-search-engine-5.7.0.Final.jar:5.7.0.Final]
> at java.lang.Thread.run(Thread.java:785) [na:1.8.0-internal]
> *As per our understanding, this issue should not come in {color:red}'non-clustered'{color} env. Also it should not arise when worker execution is {color:red}'sync'{color}.*
> *We have debugged the code, and confirmed that the value for {color:red}'write_metadata_async'{color} is coming as 'false' only (as expected).*
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8988) CacheLoader not works after expiration in JCache
by Andrei Arkaev (JIRA)
Andrei Arkaev created ISPN-8988:
-----------------------------------
Summary: CacheLoader not works after expiration in JCache
Key: ISPN-8988
URL: https://issues.jboss.org/browse/ISPN-8988
Project: Infinispan
Issue Type: Bug
Components: JCache
Affects Versions: 9.2.0.Final
Reporter: Andrei Arkaev
Priority: Blocker
Attachments: InfinispanCacheTest.java
In configuration "JCache + ExirationPolicy + CacheLoader" cache returns null after cache key expiration (CacheLoader not called after expiration)
8.2.x versions not affected
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8935) AbstractFileLookup.lookupFileStrict does not work for jar scheme
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-8935?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-8935:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> AbstractFileLookup.lookupFileStrict does not work for jar scheme
> ----------------------------------------------------------------
>
> Key: ISPN-8935
> URL: https://issues.jboss.org/browse/ISPN-8935
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, JCache
> Affects Versions: 9.1.6.Final, 9.2.0.Final
> Environment: Spring Boot 1.5.10, Infinispan 9.1.6
> Reporter: Damir Murat
> Assignee: Tristan Tarrant
> Fix For: 9.2.1.Final, 9.1.8.Final
>
>
> As documented [here|https://docs.oracle.com/javase/7/docs/api/java/net/JarURLConnection....], JAR URL has following syntax {{jar:<url>!/\{entry\}}}. Current implementation of {{AbstractFileLookup.lookupFileStrict}} does not rely on this syntax, and fails when correctly specified JAR URI comes in.
> I'm using Infinispan JCache implementation from Spring Boot. This works when run from gradle/maven/ide since in these cases {{file}} scheme is used. However, I'm targeting running Spring Boot app inside of Docker container where natural way for using Spring Boot apps is to deploy them as boot packaged fat jar. In that case, Spring Boot passes correct JAR URI into {{AbstractFileLookup.lookupFileStrict}}, which unfortunately fails silently and JCache reverts to default configuration.
> I believe that {{AbstractFileLookup.lookupFileStrict}} can be fixed by replacing current jar scheme handling
> {code}
> case "jar":
> String uriAsString = uri.toString();
> String fileName = uriAsString.substring(uriAsString.lastIndexOf(":") + 1);
> return new FileInputStream(new File(fileName));
> {code}
> with something like this
> {code}
> case "jar": {
> // Invalid code commented out.
> // String uriAsString = uri.toString();
> // String fileName = uriAsString.substring(uriAsString.lastIndexOf(":") + 1);
> // return new FileInputStream(new File(fileName));
> // ===== fix - start =====
> String uriAsString = uri.toString();
> String insideJarFilePath = uriAsString.substring(uriAsString.lastIndexOf("!") + 1);
> InputStream streamToBeReturned = getAsInputStreamFromClassLoader(insideJarFilePath, cl);
> if (streamToBeReturned == null) {
> throw log.unableToLoadFileUsingScheme(scheme);
> }
> return streamToBeReturned;
> // ===== fix - end =====
> }
> {code}
> I'm using Infinispan 9.1.6, but I think that 9.2.0 has same problem. I will try to submit pull request.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8935) AbstractFileLookup.lookupFileStrict does not work for jar scheme
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-8935?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-8935:
----------------------------------
Status: Open (was: New)
> AbstractFileLookup.lookupFileStrict does not work for jar scheme
> ----------------------------------------------------------------
>
> Key: ISPN-8935
> URL: https://issues.jboss.org/browse/ISPN-8935
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, JCache
> Affects Versions: 9.1.6.Final, 9.2.0.Final
> Environment: Spring Boot 1.5.10, Infinispan 9.1.6
> Reporter: Damir Murat
> Fix For: 9.2.1.Final, 9.1.8.Final
>
>
> As documented [here|https://docs.oracle.com/javase/7/docs/api/java/net/JarURLConnection....], JAR URL has following syntax {{jar:<url>!/\{entry\}}}. Current implementation of {{AbstractFileLookup.lookupFileStrict}} does not rely on this syntax, and fails when correctly specified JAR URI comes in.
> I'm using Infinispan JCache implementation from Spring Boot. This works when run from gradle/maven/ide since in these cases {{file}} scheme is used. However, I'm targeting running Spring Boot app inside of Docker container where natural way for using Spring Boot apps is to deploy them as boot packaged fat jar. In that case, Spring Boot passes correct JAR URI into {{AbstractFileLookup.lookupFileStrict}}, which unfortunately fails silently and JCache reverts to default configuration.
> I believe that {{AbstractFileLookup.lookupFileStrict}} can be fixed by replacing current jar scheme handling
> {code}
> case "jar":
> String uriAsString = uri.toString();
> String fileName = uriAsString.substring(uriAsString.lastIndexOf(":") + 1);
> return new FileInputStream(new File(fileName));
> {code}
> with something like this
> {code}
> case "jar": {
> // Invalid code commented out.
> // String uriAsString = uri.toString();
> // String fileName = uriAsString.substring(uriAsString.lastIndexOf(":") + 1);
> // return new FileInputStream(new File(fileName));
> // ===== fix - start =====
> String uriAsString = uri.toString();
> String insideJarFilePath = uriAsString.substring(uriAsString.lastIndexOf("!") + 1);
> InputStream streamToBeReturned = getAsInputStreamFromClassLoader(insideJarFilePath, cl);
> if (streamToBeReturned == null) {
> throw log.unableToLoadFileUsingScheme(scheme);
> }
> return streamToBeReturned;
> // ===== fix - end =====
> }
> {code}
> I'm using Infinispan 9.1.6, but I think that 9.2.0 has same problem. I will try to submit pull request.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8935) AbstractFileLookup.lookupFileStrict does not work for jar scheme
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-8935?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-8935:
----------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/5835
> AbstractFileLookup.lookupFileStrict does not work for jar scheme
> ----------------------------------------------------------------
>
> Key: ISPN-8935
> URL: https://issues.jboss.org/browse/ISPN-8935
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, JCache
> Affects Versions: 9.1.6.Final, 9.2.0.Final
> Environment: Spring Boot 1.5.10, Infinispan 9.1.6
> Reporter: Damir Murat
> Fix For: 9.2.1.Final, 9.1.8.Final
>
>
> As documented [here|https://docs.oracle.com/javase/7/docs/api/java/net/JarURLConnection....], JAR URL has following syntax {{jar:<url>!/\{entry\}}}. Current implementation of {{AbstractFileLookup.lookupFileStrict}} does not rely on this syntax, and fails when correctly specified JAR URI comes in.
> I'm using Infinispan JCache implementation from Spring Boot. This works when run from gradle/maven/ide since in these cases {{file}} scheme is used. However, I'm targeting running Spring Boot app inside of Docker container where natural way for using Spring Boot apps is to deploy them as boot packaged fat jar. In that case, Spring Boot passes correct JAR URI into {{AbstractFileLookup.lookupFileStrict}}, which unfortunately fails silently and JCache reverts to default configuration.
> I believe that {{AbstractFileLookup.lookupFileStrict}} can be fixed by replacing current jar scheme handling
> {code}
> case "jar":
> String uriAsString = uri.toString();
> String fileName = uriAsString.substring(uriAsString.lastIndexOf(":") + 1);
> return new FileInputStream(new File(fileName));
> {code}
> with something like this
> {code}
> case "jar": {
> // Invalid code commented out.
> // String uriAsString = uri.toString();
> // String fileName = uriAsString.substring(uriAsString.lastIndexOf(":") + 1);
> // return new FileInputStream(new File(fileName));
> // ===== fix - start =====
> String uriAsString = uri.toString();
> String insideJarFilePath = uriAsString.substring(uriAsString.lastIndexOf("!") + 1);
> InputStream streamToBeReturned = getAsInputStreamFromClassLoader(insideJarFilePath, cl);
> if (streamToBeReturned == null) {
> throw log.unableToLoadFileUsingScheme(scheme);
> }
> return streamToBeReturned;
> // ===== fix - end =====
> }
> {code}
> I'm using Infinispan 9.1.6, but I think that 9.2.0 has same problem. I will try to submit pull request.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8935) AbstractFileLookup.lookupFileStrict does not work for jar scheme
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-8935?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-8935:
----------------------------------
Fix Version/s: 9.2.1.Final
9.1.8.Final
> AbstractFileLookup.lookupFileStrict does not work for jar scheme
> ----------------------------------------------------------------
>
> Key: ISPN-8935
> URL: https://issues.jboss.org/browse/ISPN-8935
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, JCache
> Affects Versions: 9.1.6.Final, 9.2.0.Final
> Environment: Spring Boot 1.5.10, Infinispan 9.1.6
> Reporter: Damir Murat
> Fix For: 9.2.1.Final, 9.1.8.Final
>
>
> As documented [here|https://docs.oracle.com/javase/7/docs/api/java/net/JarURLConnection....], JAR URL has following syntax {{jar:<url>!/\{entry\}}}. Current implementation of {{AbstractFileLookup.lookupFileStrict}} does not rely on this syntax, and fails when correctly specified JAR URI comes in.
> I'm using Infinispan JCache implementation from Spring Boot. This works when run from gradle/maven/ide since in these cases {{file}} scheme is used. However, I'm targeting running Spring Boot app inside of Docker container where natural way for using Spring Boot apps is to deploy them as boot packaged fat jar. In that case, Spring Boot passes correct JAR URI into {{AbstractFileLookup.lookupFileStrict}}, which unfortunately fails silently and JCache reverts to default configuration.
> I believe that {{AbstractFileLookup.lookupFileStrict}} can be fixed by replacing current jar scheme handling
> {code}
> case "jar":
> String uriAsString = uri.toString();
> String fileName = uriAsString.substring(uriAsString.lastIndexOf(":") + 1);
> return new FileInputStream(new File(fileName));
> {code}
> with something like this
> {code}
> case "jar": {
> // Invalid code commented out.
> // String uriAsString = uri.toString();
> // String fileName = uriAsString.substring(uriAsString.lastIndexOf(":") + 1);
> // return new FileInputStream(new File(fileName));
> // ===== fix - start =====
> String uriAsString = uri.toString();
> String insideJarFilePath = uriAsString.substring(uriAsString.lastIndexOf("!") + 1);
> InputStream streamToBeReturned = getAsInputStreamFromClassLoader(insideJarFilePath, cl);
> if (streamToBeReturned == null) {
> throw log.unableToLoadFileUsingScheme(scheme);
> }
> return streamToBeReturned;
> // ===== fix - end =====
> }
> {code}
> I'm using Infinispan 9.1.6, but I think that 9.2.0 has same problem. I will try to submit pull request.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8935) AbstractFileLookup.lookupFileStrict does not work for jar scheme
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-8935?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant reassigned ISPN-8935:
-------------------------------------
Assignee: Tristan Tarrant
> AbstractFileLookup.lookupFileStrict does not work for jar scheme
> ----------------------------------------------------------------
>
> Key: ISPN-8935
> URL: https://issues.jboss.org/browse/ISPN-8935
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, JCache
> Affects Versions: 9.1.6.Final, 9.2.0.Final
> Environment: Spring Boot 1.5.10, Infinispan 9.1.6
> Reporter: Damir Murat
> Assignee: Tristan Tarrant
> Fix For: 9.2.1.Final, 9.1.8.Final
>
>
> As documented [here|https://docs.oracle.com/javase/7/docs/api/java/net/JarURLConnection....], JAR URL has following syntax {{jar:<url>!/\{entry\}}}. Current implementation of {{AbstractFileLookup.lookupFileStrict}} does not rely on this syntax, and fails when correctly specified JAR URI comes in.
> I'm using Infinispan JCache implementation from Spring Boot. This works when run from gradle/maven/ide since in these cases {{file}} scheme is used. However, I'm targeting running Spring Boot app inside of Docker container where natural way for using Spring Boot apps is to deploy them as boot packaged fat jar. In that case, Spring Boot passes correct JAR URI into {{AbstractFileLookup.lookupFileStrict}}, which unfortunately fails silently and JCache reverts to default configuration.
> I believe that {{AbstractFileLookup.lookupFileStrict}} can be fixed by replacing current jar scheme handling
> {code}
> case "jar":
> String uriAsString = uri.toString();
> String fileName = uriAsString.substring(uriAsString.lastIndexOf(":") + 1);
> return new FileInputStream(new File(fileName));
> {code}
> with something like this
> {code}
> case "jar": {
> // Invalid code commented out.
> // String uriAsString = uri.toString();
> // String fileName = uriAsString.substring(uriAsString.lastIndexOf(":") + 1);
> // return new FileInputStream(new File(fileName));
> // ===== fix - start =====
> String uriAsString = uri.toString();
> String insideJarFilePath = uriAsString.substring(uriAsString.lastIndexOf("!") + 1);
> InputStream streamToBeReturned = getAsInputStreamFromClassLoader(insideJarFilePath, cl);
> if (streamToBeReturned == null) {
> throw log.unableToLoadFileUsingScheme(scheme);
> }
> return streamToBeReturned;
> // ===== fix - end =====
> }
> {code}
> I'm using Infinispan 9.1.6, but I think that 9.2.0 has same problem. I will try to submit pull request.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years