[JBoss JIRA] Created: (ISPN-689) Read past EOF caused in Lucene Directory when Lucene flushes but doesn't close the segment
2:54 p.m.
Read past EOF caused in Lucene Directory when Lucene flushes but doesn't close the
segment
------------------------------------------------------------------------------------------
Key: ISPN-689
URL: https://jira.jboss.org/browse/ISPN-689
Project: Infinispan
Issue Type: Bug
Components: Lucene Directory
Affects Versions: 4.2.0.ALPHA2, 4.1.0.Final
Reporter: Sanne Grinovero
Assignee: Sanne Grinovero
Fix For: 4.2.0.BETA1, 4.2.0.Final, 5.0.0.Final
While this is not the default access scenario performed by Lucene it's possible in
some branches to flush the segment and be able to read it before the close.
I could reproduce the following stacktrace in high load / huge sized index but not in a
unit test; disabling the batching started between .flush() and .close() seems to resolve
the issue; also this batch seems totally useless as I couldn't find any chance in
performance when disabling it.
java.io.IOException: Read past EOF: Chunk value could not be found for key
_ni.fdt|4|issues
at
org.infinispan.lucene.InfinispanIndexInput.setBufferToCurrentChunk(InfinispanIndexInput.java:138)
at org.infinispan.lucene.InfinispanIndexInput.nextChunk(InfinispanIndexInput.java:131)
at org.infinispan.lucene.InfinispanIndexInput.readBytes(InfinispanIndexInput.java:96)
at org.apache.lucene.store.IndexInput.readBytes(IndexInput.java:61)
at org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:228)
at org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)
at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:2342)
at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4359)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:17 p.m.
New subject: [JBoss JIRA] Resolved: (ISPN-689) Read past EOF caused in Lucene Directory when Lucene flushes but doesn't close the segment
[
https://jira.jboss.org/browse/ISPN-689?page=com.atlassian.jira.plugin.sys...
]
Sanne Grinovero resolved ISPN-689.
----------------------------------
Fix Version/s: 5.0.0.BETA1
Resolution: Done
Read past EOF caused in Lucene Directory when Lucene flushes but
doesn't close the segment
------------------------------------------------------------------------------------------
Key: ISPN-689
URL: https://jira.jboss.org/browse/ISPN-689
Project: Infinispan
Issue Type: Bug
Components: Lucene Directory
Affects Versions: 4.1.0.Final, 4.2.0.ALPHA2
Reporter: Sanne Grinovero
Assignee: Sanne Grinovero
Fix For: 4.2.0.BETA1, 4.2.0.Final, 5.0.0.BETA1, 5.0.0.Final
While this is not the default access scenario performed by Lucene it's possible in
some branches to flush the segment and be able to read it before the close.
I could reproduce the following stacktrace in high load / huge sized index but not in a
unit test; disabling the batching started between .flush() and .close() seems to resolve
the issue; also this batch seems totally useless as I couldn't find any chance in
performance when disabling it.
java.io.IOException: Read past EOF: Chunk value could not be found for key
_ni.fdt|4|issues
at
org.infinispan.lucene.InfinispanIndexInput.setBufferToCurrentChunk(InfinispanIndexInput.java:138)
at org.infinispan.lucene.InfinispanIndexInput.nextChunk(InfinispanIndexInput.java:131)
at org.infinispan.lucene.InfinispanIndexInput.readBytes(InfinispanIndexInput.java:96)
at org.apache.lucene.store.IndexInput.readBytes(IndexInput.java:61)
at org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:228)
at org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)
at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:2342)
at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4359)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:05 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-689) Read past EOF caused in Lucene Directory when Lucene flushes but doesn't close the segment
[
https://jira.jboss.org/browse/ISPN-689?page=com.atlassian.jira.plugin.sys...
]
Mircea Markus updated ISPN-689:
-------------------------------
Fix Version/s: 4.2.0.ALPHA3
(was: 4.2.0.BETA1)
(was: 5.0.0.Final)
(was: 5.0.0.BETA1)
Affects Version/s: (was: 4.2.0.ALPHA2)
Read past EOF caused in Lucene Directory when Lucene flushes but
doesn't close the segment
------------------------------------------------------------------------------------------
Key: ISPN-689
URL: https://jira.jboss.org/browse/ISPN-689
Project: Infinispan
Issue Type: Bug
Components: Lucene Directory
Affects Versions: 4.1.0.Final
Reporter: Sanne Grinovero
Assignee: Sanne Grinovero
Fix For: 4.2.0.ALPHA3, 4.2.0.Final
While this is not the default access scenario performed by Lucene it's possible in
some branches to flush the segment and be able to read it before the close.
I could reproduce the following stacktrace in high load / huge sized index but not in a
unit test; disabling the batching started between .flush() and .close() seems to resolve
the issue; also this batch seems totally useless as I couldn't find any chance in
performance when disabling it.
java.io.IOException: Read past EOF: Chunk value could not be found for key
_ni.fdt|4|issues
at
org.infinispan.lucene.InfinispanIndexInput.setBufferToCurrentChunk(InfinispanIndexInput.java:138)
at org.infinispan.lucene.InfinispanIndexInput.nextChunk(InfinispanIndexInput.java:131)
at org.infinispan.lucene.InfinispanIndexInput.readBytes(InfinispanIndexInput.java:96)
at org.apache.lucene.store.IndexInput.readBytes(IndexInput.java:61)
at org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:228)
at org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)
at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:2342)
at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4359)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:38 p.m.
New subject: [JBoss JIRA] (ISPN-689) Read past EOF caused in Lucene Directory when Lucene flushes but doesn't close the segment
[
https://issues.jboss.org/browse/ISPN-689?page=com.atlassian.jira.plugin.s...
]
Horacio Vico commented on ISPN-689:
-----------------------------------
I am getting a similar exception using Infinispan 'Pagoa' 5.0.1.FINAL.
I have a two node cache, which share a MySQL JDBC CacheStore to store Lucene indexes. This
exceptions appear randomly when I restart jboss at the nodes, and the only solution I
found after getting this error is to reindex all my entities from scratch.
JAVA Stack trace:
Caused by: java.io.IOException: Read past EOF: Chunk value could not be found for key
_0.tis|0|some.package.entity.SomeEntity
at
org.infinispan.lucene.InfinispanIndexInput.setBufferToCurrentChunk(InfinispanIndexInput.java:138)
at org.infinispan.lucene.InfinispanIndexInput.nextChunk(InfinispanIndexInput.java:131)
at org.infinispan.lucene.InfinispanIndexInput.readByte(InfinispanIndexInput.java:78)
at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:71)
at org.apache.lucene.index.SegmentTermEnum.<init>(SegmentTermEnum.java:50)
at org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:102)
at
org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:121)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:578)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:556)
at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:113)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:29)
at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:736)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:428)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:288)
at
org.hibernate.search.reader.SharingBufferReaderProvider.readerFactory(SharingBufferReaderProvider.java:169)
at
org.hibernate.search.reader.SharingBufferReaderProvider$PerDirectoryLatestReader.<init>(SharingBufferReaderProvider.java:243)
at
org.hibernate.search.reader.SharingBufferReaderProvider.createReader(SharingBufferReaderProvider.java:113)
... 164 more
Read past EOF caused in Lucene Directory when Lucene flushes but
doesn't close the segment
------------------------------------------------------------------------------------------
Key: ISPN-689
URL: https://issues.jboss.org/browse/ISPN-689
Project: Infinispan
Issue Type: Bug
Components: Lucene Directory
Affects Versions: 4.1.0.Final
Reporter: Sanne Grinovero
Assignee: Sanne Grinovero
Fix For: 4.2.0.ALPHA3, 4.2.0.Final
While this is not the default access scenario performed by Lucene it's possible in
some branches to flush the segment and be able to read it before the close.
I could reproduce the following stacktrace in high load / huge sized index but not in a
unit test; disabling the batching started between .flush() and .close() seems to resolve
the issue; also this batch seems totally useless as I couldn't find any chance in
performance when disabling it.
java.io.IOException: Read past EOF: Chunk value could not be found for key
_ni.fdt|4|issues
at
org.infinispan.lucene.InfinispanIndexInput.setBufferToCurrentChunk(InfinispanIndexInput.java:138)
at org.infinispan.lucene.InfinispanIndexInput.nextChunk(InfinispanIndexInput.java:131)
at org.infinispan.lucene.InfinispanIndexInput.readBytes(InfinispanIndexInput.java:96)
at org.apache.lucene.store.IndexInput.readBytes(IndexInput.java:61)
at org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:228)
at org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)
at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:2342)
at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4359)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:25 a.m.
New subject: [JBoss JIRA] (ISPN-689) Read past EOF caused in Lucene Directory when Lucene flushes but doesn't close the segment
[
https://issues.jboss.org/browse/ISPN-689?page=com.atlassian.jira.plugin.s...
]
Sanne Grinovero commented on ISPN-689:
--------------------------------------
Hi,
could you post your Infinispan configuration? Especially the cacheloader; I'm
wondering if your problem could be related to ISPN-1174.
Read past EOF caused in Lucene Directory when Lucene flushes but
doesn't close the segment
------------------------------------------------------------------------------------------
Key: ISPN-689
URL: https://issues.jboss.org/browse/ISPN-689
Project: Infinispan
Issue Type: Bug
Components: Lucene Directory
Affects Versions: 4.1.0.Final
Reporter: Sanne Grinovero
Assignee: Sanne Grinovero
Fix For: 4.2.0.ALPHA3, 4.2.0.Final
While this is not the default access scenario performed by Lucene it's possible in
some branches to flush the segment and be able to read it before the close.
I could reproduce the following stacktrace in high load / huge sized index but not in a
unit test; disabling the batching started between .flush() and .close() seems to resolve
the issue; also this batch seems totally useless as I couldn't find any chance in
performance when disabling it.
java.io.IOException: Read past EOF: Chunk value could not be found for key
_ni.fdt|4|issues
at
org.infinispan.lucene.InfinispanIndexInput.setBufferToCurrentChunk(InfinispanIndexInput.java:138)
at org.infinispan.lucene.InfinispanIndexInput.nextChunk(InfinispanIndexInput.java:131)
at org.infinispan.lucene.InfinispanIndexInput.readBytes(InfinispanIndexInput.java:96)
at org.apache.lucene.store.IndexInput.readBytes(IndexInput.java:61)
at org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:228)
at org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)
at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:2342)
at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4359)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:46 a.m.
New subject: [JBoss JIRA] (ISPN-689) Read past EOF caused in Lucene Directory when Lucene flushes but doesn't close the segment
[
https://issues.jboss.org/browse/ISPN-689?page=com.atlassian.jira.plugin.s...
]
Horacio Vico commented on ISPN-689:
-----------------------------------
Sure! Here is my infinispan.xml config:
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:4.2
http://www.infinispan.org/schemas/infinispan-config-4.2.xsd"
xmlns="urn:infinispan:config:4.2">
<global>
<!-- Duplicate domains are allowed so that multiple deployments with default
configuration
of Hibernate Search applications work - if possible it would be better to use
JNDI to share
the CacheManager across applications -->
<globalJmxStatistics
enabled="true"
cacheManagerName="HibernateSearch"
allowDuplicateDomains="true" />
<!-- If the transport is omitted, there is no way to create distributed or
clustered
caches. There is no added cost to defining a transport but not creating a
cache that uses one,
since the transport is created and initialized lazily. -->
<transport
clusterName="HibernateSearch-Infinispan-cluster"
distributedSyncTimeout="50000">
<!-- Note that the JGroups transport uses sensible defaults if no
configuration
property is defined. See the JGroupsTransport javadocs for more flags
-->
<properties>
<property name="configurationFile"
value="jgroups.xml"/>
</properties>
</transport>
<!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER,
DONT_REGISTER.
Hibernate Search takes care to stop the CacheManager so registering is not
needed -->
<shutdown
hookBehavior="DONT_REGISTER" />
</global>
<default>
<!--<locking
lockAcquisitionTimeout="2000"
writeSkewCheck="false"
concurrencyLevel="500"
useLockStriping="false" />-->
<locking isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="15000"
writeSkewCheck="false"
concurrencyLevel="5000" useLockStriping="false"
/>
<!-- Invocation batching is required for use with the Lucene Directory -->
<invocationBatching
enabled="true" />
<!-- This element specifies that the cache is clustered. modes supported:
distribution
(d), replication (r) or invalidation (i). Don't use invalidation to store
Lucene indexes (as
with Hibernate Search DirectoryProvider). Replication is recommended for best
performance of
Lucene indexes, but make sure you have enough memory to store the index in
your heap.
Also distribution scales much better than replication on high number of nodes
in the cluster. -->
<jmxStatistics
enabled="false" />
<eviction
maxEntries="-1"
strategy="NONE" />
<expiration
maxIdle="-1" />
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="false"
logFlushTimeout="60000"/>
<sync replTimeout="50000"/>
</clustering>
</default>
<namedCache name="LuceneIndexesLocking">
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="false"
logFlushTimeout="60000"/>
<sync replTimeout="50000"/>
</clustering>
<invocationBatching enabled="true" />
</namedCache>
<namedCache name="LuceneIndexesMetadata">
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="false" />
</clustering>
<invocationBatching enabled="true" />
<jmxStatistics enabled="false" />
<loaders passivation="false" shared="true"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
ignoreModifications="false"
purgeOnStartup="false">
<!-- See the documentation for more configuration examples and flags.
-->
<properties>
<property name="key2StringMapperClass"
value="org.infinispan.lucene.LuceneKey2StringMapper" />
<property name="createTableOnStart" value="true"
/>
<!-- Settings for MySQL: -->
<property name="datasourceJndiLocation"
value="java:/BDZWEB2Datasource" />
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"
/>
<property name="dataColumnType" value="BLOB" />
<property name="idColumnType" value="VARCHAR(256)"
/>
<property name="idColumnName" value="idCol" />
<property name="dataColumnName" value="dataCol"
/>
<property name="stringsTableNamePrefix"
value="LuceneIndexesMetadata" />
<property name="timestampColumnName"
value="timestampCol" />
<property name="timestampColumnType" value="BIGINT"
/>
</properties>
<async enabled="true" flushLockTimeout="2500"
shutdownTimeout="7200" threadPoolSize="5" />
</loader>
</loaders>
<eviction maxEntries="-1" strategy="NONE" />
<expiration maxIdle="-1" />
</namedCache>
<namedCache name="LuceneIndexesData">
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="false" />
</clustering>
<invocationBatching enabled="true" />
<jmxStatistics enabled="false" />
<loaders passivation="false" shared="true"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
ignoreModifications="false"
purgeOnStartup="false">
<!-- See the documentation for more configuration examples and flags.
-->
<properties>
<property name="key2StringMapperClass"
value="org.infinispan.lucene.LuceneKey2StringMapper" />
<property name="createTableOnStart" value="true"
/>
<!-- Settings for MySQL: -->
<property name="datasourceJndiLocation"
value="java:/MyDatasource" />
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"
/>
<property name="dataColumnType" value="BLOB" />
<property name="idColumnType" value="VARCHAR(256)"
/>
<property name="idColumnName" value="idCol" />
<property name="dataColumnName" value="dataCol"
/>
<property name="stringsTableNamePrefix"
value="LuceneIndexesData" />
<property name="timestampColumnName"
value="timestampCol" />
<property name="timestampColumnType" value="BIGINT"
/>
</properties>
<async enabled="true" flushLockTimeout="2500"
shutdownTimeout="7200" threadPoolSize="5" />
</loader>
</loaders>
<eviction maxEntries="-1" strategy="NONE" />
<expiration maxIdle="-1" />
</namedCache>
</infinispan>
And here it is my jgroups.xml
<!--
Default flush stack using IP multicasting.
-->
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups
http://www.jgroups.org/schema/JGroups-2.12.xsd">
<UDP
mcast_addr="224.0.0.1"
mcast_port="${jgroups.udp.mcast_port:45588}"
tos="8"
ucast_recv_buf_size="20000000"
ucast_send_buf_size="640000"
mcast_recv_buf_size="25000000"
mcast_send_buf_size="640000"
loopback="true"
discard_incompatible_packets="true"
max_bundle_size="64000"
max_bundle_timeout="30"
ip_ttl="${jgroups.udp.ip_ttl:2}"
enable_bundling="true"
enable_diagnostics="true"
thread_naming_pattern="pl"
thread_pool.enabled="true"
thread_pool.min_threads="2"
thread_pool.max_threads="8"
thread_pool.keep_alive_time="5000"
thread_pool.queue_enabled="false"
thread_pool.queue_max_size="100"
thread_pool.rejection_policy="Run"
oob_thread_pool.enabled="true"
oob_thread_pool.min_threads="1"
oob_thread_pool.max_threads="8"
oob_thread_pool.keep_alive_time="5000"
oob_thread_pool.queue_enabled="false"
oob_thread_pool.queue_max_size="100"
oob_thread_pool.rejection_policy="Run"/>
<PING timeout="10000" num_initial_members="2"/>
<MERGE2 max_interval="300000" min_interval="10000"/>
<FD_SOCK/>
<FD_ALL/>
<VERIFY_SUSPECT timeout="1500"/>
<pbcast.NAKACK use_stats_for_retransmission="false"
exponential_backoff="150"
use_mcast_xmit="true" gc_lag="0"
retransmit_timeout="300,600,1200"
discard_delivered_msgs="false"/>
<UNICAST timeout="300,600,1200"/>
<pbcast.STABLE stability_delay="1000"
desired_avg_gossip="50000"
max_bytes="4m"/>
<pbcast.GMS print_local_addr="true" join_timeout="30000"
view_bundling="true" />
<UFC max_credits="2M"
min_threshold="0.4"/>
<MFC max_credits="2M"
min_threshold="0.4"/>
<FRAG2 frag_size="60000"/>
<pbcast.STREAMING_STATE_TRANSFER />
<!-- <pbcast.STATE_TRANSFER/> -->
<pbcast.FLUSH timeout="0"/>
</config>
Thanks,
Read past EOF caused in Lucene Directory when Lucene flushes but
doesn't close the segment
------------------------------------------------------------------------------------------
Key: ISPN-689
URL: https://issues.jboss.org/browse/ISPN-689
Project: Infinispan
Issue Type: Bug
Components: Lucene Directory
Affects Versions: 4.1.0.Final
Reporter: Sanne Grinovero
Assignee: Sanne Grinovero
Fix For: 4.2.0.ALPHA3, 4.2.0.Final
While this is not the default access scenario performed by Lucene it's possible in
some branches to flush the segment and be able to read it before the close.
I could reproduce the following stacktrace in high load / huge sized index but not in a
unit test; disabling the batching started between .flush() and .close() seems to resolve
the issue; also this batch seems totally useless as I couldn't find any chance in
performance when disabling it.
java.io.IOException: Read past EOF: Chunk value could not be found for key
_ni.fdt|4|issues
at
org.infinispan.lucene.InfinispanIndexInput.setBufferToCurrentChunk(InfinispanIndexInput.java:138)
at org.infinispan.lucene.InfinispanIndexInput.nextChunk(InfinispanIndexInput.java:131)
at org.infinispan.lucene.InfinispanIndexInput.readBytes(InfinispanIndexInput.java:96)
at org.apache.lucene.store.IndexInput.readBytes(IndexInput.java:61)
at org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:228)
at org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)
at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:2342)
at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4359)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
8:48 a.m.
New subject: [JBoss JIRA] (ISPN-689) Read past EOF caused in Lucene Directory when Lucene flushes but doesn't close the segment
[
https://issues.jboss.org/browse/ISPN-689?page=com.atlassian.jira.plugin.s...
]
Horacio Vico edited comment on ISPN-689 at 11/14/11 9:47 AM:
-------------------------------------------------------------
Sure! Here is my infinispan.xml config:
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:4.2
http://www.infinispan.org/schemas/infinispan-config-4.2.xsd"
xmlns="urn:infinispan:config:4.2">
<global>
<!-- Duplicate domains are allowed so that multiple deployments with default
configuration
of Hibernate Search applications work - if possible it would be better to use
JNDI to share
the CacheManager across applications -->
<globalJmxStatistics
enabled="true"
cacheManagerName="HibernateSearch"
allowDuplicateDomains="true" />
<!-- If the transport is omitted, there is no way to create distributed or
clustered
caches. There is no added cost to defining a transport but not creating a
cache that uses one,
since the transport is created and initialized lazily. -->
<transport
clusterName="HibernateSearch-Infinispan-cluster"
distributedSyncTimeout="50000">
<!-- Note that the JGroups transport uses sensible defaults if no
configuration
property is defined. See the JGroupsTransport javadocs for more flags
-->
<properties>
<property name="configurationFile"
value="jgroups.xml"/>
</properties>
</transport>
<!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER,
DONT_REGISTER.
Hibernate Search takes care to stop the CacheManager so registering is not
needed -->
<shutdown
hookBehavior="DONT_REGISTER" />
</global>
<default>
<!--<locking
lockAcquisitionTimeout="2000"
writeSkewCheck="false"
concurrencyLevel="500"
useLockStriping="false" />-->
<locking isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="15000"
writeSkewCheck="false"
concurrencyLevel="5000" useLockStriping="false"
/>
<!-- Invocation batching is required for use with the Lucene Directory -->
<invocationBatching
enabled="true" />
<!-- This element specifies that the cache is clustered. modes supported:
distribution
(d), replication (r) or invalidation (i). Don't use invalidation to store
Lucene indexes (as
with Hibernate Search DirectoryProvider). Replication is recommended for best
performance of
Lucene indexes, but make sure you have enough memory to store the index in
your heap.
Also distribution scales much better than replication on high number of nodes
in the cluster. -->
<jmxStatistics
enabled="false" />
<eviction
maxEntries="-1"
strategy="NONE" />
<expiration
maxIdle="-1" />
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="false"
logFlushTimeout="60000"/>
<sync replTimeout="50000"/>
</clustering>
</default>
<namedCache name="LuceneIndexesLocking">
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="false"
logFlushTimeout="60000"/>
<sync replTimeout="50000"/>
</clustering>
<invocationBatching enabled="true" />
</namedCache>
<namedCache name="LuceneIndexesMetadata">
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="false" />
</clustering>
<invocationBatching enabled="true" />
<jmxStatistics enabled="false" />
<loaders passivation="false" shared="true"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
ignoreModifications="false"
purgeOnStartup="false">
<!-- See the documentation for more configuration examples and flags.
-->
<properties>
<property name="key2StringMapperClass"
value="org.infinispan.lucene.LuceneKey2StringMapper" />
<property name="createTableOnStart" value="true"
/>
<!-- Settings for MySQL: -->
<property name="datasourceJndiLocation"
value="java:/MyDatasource" />
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"
/>
<property name="dataColumnType" value="BLOB" />
<property name="idColumnType" value="VARCHAR(256)"
/>
<property name="idColumnName" value="idCol" />
<property name="dataColumnName" value="dataCol"
/>
<property name="stringsTableNamePrefix"
value="LuceneIndexesMetadata" />
<property name="timestampColumnName"
value="timestampCol" />
<property name="timestampColumnType" value="BIGINT"
/>
</properties>
<async enabled="true" flushLockTimeout="2500"
shutdownTimeout="7200" threadPoolSize="5" />
</loader>
</loaders>
<eviction maxEntries="-1" strategy="NONE" />
<expiration maxIdle="-1" />
</namedCache>
<namedCache name="LuceneIndexesData">
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="false" />
</clustering>
<invocationBatching enabled="true" />
<jmxStatistics enabled="false" />
<loaders passivation="false" shared="true"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
ignoreModifications="false"
purgeOnStartup="false">
<!-- See the documentation for more configuration examples and flags.
-->
<properties>
<property name="key2StringMapperClass"
value="org.infinispan.lucene.LuceneKey2StringMapper" />
<property name="createTableOnStart" value="true"
/>
<!-- Settings for MySQL: -->
<property name="datasourceJndiLocation"
value="java:/MyDatasource" />
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"
/>
<property name="dataColumnType" value="BLOB" />
<property name="idColumnType" value="VARCHAR(256)"
/>
<property name="idColumnName" value="idCol" />
<property name="dataColumnName" value="dataCol"
/>
<property name="stringsTableNamePrefix"
value="LuceneIndexesData" />
<property name="timestampColumnName"
value="timestampCol" />
<property name="timestampColumnType" value="BIGINT"
/>
</properties>
<async enabled="true" flushLockTimeout="2500"
shutdownTimeout="7200" threadPoolSize="5" />
</loader>
</loaders>
<eviction maxEntries="-1" strategy="NONE" />
<expiration maxIdle="-1" />
</namedCache>
</infinispan>
And here it is my jgroups.xml
<!--
Default flush stack using IP multicasting.
-->
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups
http://www.jgroups.org/schema/JGroups-2.12.xsd">
<UDP
mcast_addr="224.0.0.1"
mcast_port="${jgroups.udp.mcast_port:45588}"
tos="8"
ucast_recv_buf_size="20000000"
ucast_send_buf_size="640000"
mcast_recv_buf_size="25000000"
mcast_send_buf_size="640000"
loopback="true"
discard_incompatible_packets="true"
max_bundle_size="64000"
max_bundle_timeout="30"
ip_ttl="${jgroups.udp.ip_ttl:2}"
enable_bundling="true"
enable_diagnostics="true"
thread_naming_pattern="pl"
thread_pool.enabled="true"
thread_pool.min_threads="2"
thread_pool.max_threads="8"
thread_pool.keep_alive_time="5000"
thread_pool.queue_enabled="false"
thread_pool.queue_max_size="100"
thread_pool.rejection_policy="Run"
oob_thread_pool.enabled="true"
oob_thread_pool.min_threads="1"
oob_thread_pool.max_threads="8"
oob_thread_pool.keep_alive_time="5000"
oob_thread_pool.queue_enabled="false"
oob_thread_pool.queue_max_size="100"
oob_thread_pool.rejection_policy="Run"/>
<PING timeout="10000" num_initial_members="2"/>
<MERGE2 max_interval="300000" min_interval="10000"/>
<FD_SOCK/>
<FD_ALL/>
<VERIFY_SUSPECT timeout="1500"/>
<pbcast.NAKACK use_stats_for_retransmission="false"
exponential_backoff="150"
use_mcast_xmit="true" gc_lag="0"
retransmit_timeout="300,600,1200"
discard_delivered_msgs="false"/>
<UNICAST timeout="300,600,1200"/>
<pbcast.STABLE stability_delay="1000"
desired_avg_gossip="50000"
max_bytes="4m"/>
<pbcast.GMS print_local_addr="true" join_timeout="30000"
view_bundling="true" />
<UFC max_credits="2M"
min_threshold="0.4"/>
<MFC max_credits="2M"
min_threshold="0.4"/>
<FRAG2 frag_size="60000"/>
<pbcast.STREAMING_STATE_TRANSFER />
<!-- <pbcast.STATE_TRANSFER/> -->
<pbcast.FLUSH timeout="0"/>
</config>
Thanks,
was (Author: hvico):
Sure! Here is my infinispan.xml config:
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:4.2
http://www.infinispan.org/schemas/infinispan-config-4.2.xsd"
xmlns="urn:infinispan:config:4.2">
<global>
<!-- Duplicate domains are allowed so that multiple deployments with default
configuration
of Hibernate Search applications work - if possible it would be better to use
JNDI to share
the CacheManager across applications -->
<globalJmxStatistics
enabled="true"
cacheManagerName="HibernateSearch"
allowDuplicateDomains="true" />
<!-- If the transport is omitted, there is no way to create distributed or
clustered
caches. There is no added cost to defining a transport but not creating a
cache that uses one,
since the transport is created and initialized lazily. -->
<transport
clusterName="HibernateSearch-Infinispan-cluster"
distributedSyncTimeout="50000">
<!-- Note that the JGroups transport uses sensible defaults if no
configuration
property is defined. See the JGroupsTransport javadocs for more flags
-->
<properties>
<property name="configurationFile"
value="jgroups.xml"/>
</properties>
</transport>
<!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER,
DONT_REGISTER.
Hibernate Search takes care to stop the CacheManager so registering is not
needed -->
<shutdown
hookBehavior="DONT_REGISTER" />
</global>
<default>
<!--<locking
lockAcquisitionTimeout="2000"
writeSkewCheck="false"
concurrencyLevel="500"
useLockStriping="false" />-->
<locking isolationLevel="READ_COMMITTED"
lockAcquisitionTimeout="15000"
writeSkewCheck="false"
concurrencyLevel="5000" useLockStriping="false"
/>
<!-- Invocation batching is required for use with the Lucene Directory -->
<invocationBatching
enabled="true" />
<!-- This element specifies that the cache is clustered. modes supported:
distribution
(d), replication (r) or invalidation (i). Don't use invalidation to store
Lucene indexes (as
with Hibernate Search DirectoryProvider). Replication is recommended for best
performance of
Lucene indexes, but make sure you have enough memory to store the index in
your heap.
Also distribution scales much better than replication on high number of nodes
in the cluster. -->
<jmxStatistics
enabled="false" />
<eviction
maxEntries="-1"
strategy="NONE" />
<expiration
maxIdle="-1" />
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="false"
logFlushTimeout="60000"/>
<sync replTimeout="50000"/>
</clustering>
</default>
<namedCache name="LuceneIndexesLocking">
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="false"
logFlushTimeout="60000"/>
<sync replTimeout="50000"/>
</clustering>
<invocationBatching enabled="true" />
</namedCache>
<namedCache name="LuceneIndexesMetadata">
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="false" />
</clustering>
<invocationBatching enabled="true" />
<jmxStatistics enabled="false" />
<loaders passivation="false" shared="true"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
ignoreModifications="false"
purgeOnStartup="false">
<!-- See the documentation for more configuration examples and flags.
-->
<properties>
<property name="key2StringMapperClass"
value="org.infinispan.lucene.LuceneKey2StringMapper" />
<property name="createTableOnStart" value="true"
/>
<!-- Settings for MySQL: -->
<property name="datasourceJndiLocation"
value="java:/BDZWEB2Datasource" />
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"
/>
<property name="dataColumnType" value="BLOB" />
<property name="idColumnType" value="VARCHAR(256)"
/>
<property name="idColumnName" value="idCol" />
<property name="dataColumnName" value="dataCol"
/>
<property name="stringsTableNamePrefix"
value="LuceneIndexesMetadata" />
<property name="timestampColumnName"
value="timestampCol" />
<property name="timestampColumnType" value="BIGINT"
/>
</properties>
<async enabled="true" flushLockTimeout="2500"
shutdownTimeout="7200" threadPoolSize="5" />
</loader>
</loaders>
<eviction maxEntries="-1" strategy="NONE" />
<expiration maxIdle="-1" />
</namedCache>
<namedCache name="LuceneIndexesData">
<clustering mode="replication">
<stateRetrieval fetchInMemoryState="false" />
</clustering>
<invocationBatching enabled="true" />
<jmxStatistics enabled="false" />
<loaders passivation="false" shared="true"
preload="true">
<loader
class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
fetchPersistentState="true"
ignoreModifications="false"
purgeOnStartup="false">
<!-- See the documentation for more configuration examples and flags.
-->
<properties>
<property name="key2StringMapperClass"
value="org.infinispan.lucene.LuceneKey2StringMapper" />
<property name="createTableOnStart" value="true"
/>
<!-- Settings for MySQL: -->
<property name="datasourceJndiLocation"
value="java:/MyDatasource" />
<property name="connectionFactoryClass"
value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory"
/>
<property name="dataColumnType" value="BLOB" />
<property name="idColumnType" value="VARCHAR(256)"
/>
<property name="idColumnName" value="idCol" />
<property name="dataColumnName" value="dataCol"
/>
<property name="stringsTableNamePrefix"
value="LuceneIndexesData" />
<property name="timestampColumnName"
value="timestampCol" />
<property name="timestampColumnType" value="BIGINT"
/>
</properties>
<async enabled="true" flushLockTimeout="2500"
shutdownTimeout="7200" threadPoolSize="5" />
</loader>
</loaders>
<eviction maxEntries="-1" strategy="NONE" />
<expiration maxIdle="-1" />
</namedCache>
</infinispan>
And here it is my jgroups.xml
<!--
Default flush stack using IP multicasting.
-->
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups
http://www.jgroups.org/schema/JGroups-2.12.xsd">
<UDP
mcast_addr="224.0.0.1"
mcast_port="${jgroups.udp.mcast_port:45588}"
tos="8"
ucast_recv_buf_size="20000000"
ucast_send_buf_size="640000"
mcast_recv_buf_size="25000000"
mcast_send_buf_size="640000"
loopback="true"
discard_incompatible_packets="true"
max_bundle_size="64000"
max_bundle_timeout="30"
ip_ttl="${jgroups.udp.ip_ttl:2}"
enable_bundling="true"
enable_diagnostics="true"
thread_naming_pattern="pl"
thread_pool.enabled="true"
thread_pool.min_threads="2"
thread_pool.max_threads="8"
thread_pool.keep_alive_time="5000"
thread_pool.queue_enabled="false"
thread_pool.queue_max_size="100"
thread_pool.rejection_policy="Run"
oob_thread_pool.enabled="true"
oob_thread_pool.min_threads="1"
oob_thread_pool.max_threads="8"
oob_thread_pool.keep_alive_time="5000"
oob_thread_pool.queue_enabled="false"
oob_thread_pool.queue_max_size="100"
oob_thread_pool.rejection_policy="Run"/>
<PING timeout="10000" num_initial_members="2"/>
<MERGE2 max_interval="300000" min_interval="10000"/>
<FD_SOCK/>
<FD_ALL/>
<VERIFY_SUSPECT timeout="1500"/>
<pbcast.NAKACK use_stats_for_retransmission="false"
exponential_backoff="150"
use_mcast_xmit="true" gc_lag="0"
retransmit_timeout="300,600,1200"
discard_delivered_msgs="false"/>
<UNICAST timeout="300,600,1200"/>
<pbcast.STABLE stability_delay="1000"
desired_avg_gossip="50000"
max_bytes="4m"/>
<pbcast.GMS print_local_addr="true" join_timeout="30000"
view_bundling="true" />
<UFC max_credits="2M"
min_threshold="0.4"/>
<MFC max_credits="2M"
min_threshold="0.4"/>
<FRAG2 frag_size="60000"/>
<pbcast.STREAMING_STATE_TRANSFER />
<!-- <pbcast.STATE_TRANSFER/> -->
<pbcast.FLUSH timeout="0"/>
</config>
Thanks,
Read past EOF caused in Lucene Directory when Lucene flushes but
doesn't close the segment
------------------------------------------------------------------------------------------
Key: ISPN-689
URL: https://issues.jboss.org/browse/ISPN-689
Project: Infinispan
Issue Type: Bug
Components: Lucene Directory
Affects Versions: 4.1.0.Final
Reporter: Sanne Grinovero
Assignee: Sanne Grinovero
Fix For: 4.2.0.ALPHA3, 4.2.0.Final
While this is not the default access scenario performed by Lucene it's possible in
some branches to flush the segment and be able to read it before the close.
I could reproduce the following stacktrace in high load / huge sized index but not in a
unit test; disabling the batching started between .flush() and .close() seems to resolve
the issue; also this batch seems totally useless as I couldn't find any chance in
performance when disabling it.
java.io.IOException: Read past EOF: Chunk value could not be found for key
_ni.fdt|4|issues
at
org.infinispan.lucene.InfinispanIndexInput.setBufferToCurrentChunk(InfinispanIndexInput.java:138)
at org.infinispan.lucene.InfinispanIndexInput.nextChunk(InfinispanIndexInput.java:131)
at org.infinispan.lucene.InfinispanIndexInput.readBytes(InfinispanIndexInput.java:96)
at org.apache.lucene.store.IndexInput.readBytes(IndexInput.java:61)
at org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:228)
at org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)
at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:2342)
at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4359)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4795
days inactive
5202
days old
infinispan-issues@lists.jboss.org
6 comments
5 participants
participants (5)
-
Horacio Vico (Commented) (JIRA) -
Horacio Vico (Issue Comment Edited) (JIRA)
-
Mircea Markus (JIRA)
-
Sanne Grinovero (Commented) (JIRA)
-
Sanne Grinovero (JIRA)