[infinispan-issues] [JBoss JIRA] (ISPN-689) Read past EOF caused in Lucene Directory when Lucene flushes but doesn't close the segment

Mon Nov 14 09:48:40 EST 2011

    [ https://issues.jboss.org/browse/ISPN-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642547#comment-12642547 ] 

Horacio Vico edited comment on ISPN-689 at 11/14/11 9:47 AM:
-------------------------------------------------------------

Sure! Here is my infinispan.xml config:

<?xml version="1.0" encoding="UTF-8"?>

<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:4.2 http://www.infinispan.org/schemas/infinispan-config-4.2.xsd"
    xmlns="urn:infinispan:config:4.2">
  <global>

        <!-- Duplicate domains are allowed so that multiple deployments with default configuration
            of Hibernate Search applications work - if possible it would be better to use JNDI to share
            the CacheManager across applications -->
        <globalJmxStatistics
            enabled="true"
            cacheManagerName="HibernateSearch"
            allowDuplicateDomains="true" />

        <!-- If the transport is omitted, there is no way to create distributed or clustered
            caches. There is no added cost to defining a transport but not creating a cache that uses one,
            since the transport is created and initialized lazily. -->
        <transport
            clusterName="HibernateSearch-Infinispan-cluster"
            distributedSyncTimeout="50000">
            <!-- Note that the JGroups transport uses sensible defaults if no configuration
                property is defined. See the JGroupsTransport javadocs for more flags -->

            <properties>
                <property name="configurationFile" value="jgroups.xml"/>
            </properties>

        </transport>

        <!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.
            Hibernate Search takes care to stop the CacheManager so registering is not needed -->
        <shutdown
            hookBehavior="DONT_REGISTER" />

    </global>

  <default>

        <!--<locking
            lockAcquisitionTimeout="2000"
            writeSkewCheck="false"
            concurrencyLevel="500"
            useLockStriping="false" />-->
		<locking isolationLevel="READ_COMMITTED"

                    lockAcquisitionTimeout="15000" writeSkewCheck="false"

                    concurrencyLevel="5000" useLockStriping="false" />
        <!-- Invocation batching is required for use with the Lucene Directory -->
        <invocationBatching
            enabled="true" />

        <!-- This element specifies that the cache is clustered. modes supported: distribution
            (d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as
            with Hibernate Search DirectoryProvider). Replication is recommended for best performance of
            Lucene indexes, but make sure you have enough memory to store the index in your heap.
            Also distribution scales much better than replication on high number of nodes in the cluster. -->

        <jmxStatistics
            enabled="false" />

        <eviction
            maxEntries="-1"
            strategy="NONE" />

        <expiration
            maxIdle="-1" />
		 <clustering mode="replication">
                <stateRetrieval fetchInMemoryState="false" logFlushTimeout="60000"/>
                <sync replTimeout="50000"/>
         </clustering>
    </default>

   <namedCache name="LuceneIndexesLocking">
        <clustering mode="replication">
                <stateRetrieval fetchInMemoryState="false" logFlushTimeout="60000"/>
                <sync replTimeout="50000"/>
         </clustering>
         <invocationBatching enabled="true" />      
   </namedCache>

   <namedCache name="LuceneIndexesMetadata">
      <clustering mode="replication">
         <stateRetrieval fetchInMemoryState="false" />

      </clustering>
      <invocationBatching enabled="true" />
      <jmxStatistics enabled="false" />
      <loaders passivation="false" shared="true" preload="true">
         <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
                fetchPersistentState="true"
                ignoreModifications="false"
                purgeOnStartup="false">

            <!-- See the documentation for more configuration examples and flags. -->
            <properties>
               <property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />
               <property name="createTableOnStart" value="true" />

               <!-- Settings for MySQL: -->
               <property name="datasourceJndiLocation" value="java:/MyDatasource" />
               <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory" />
               <property name="dataColumnType" value="BLOB" />

               <property name="idColumnType" value="VARCHAR(256)" />
               <property name="idColumnName" value="idCol" />
               <property name="dataColumnName" value="dataCol" />
               <property name="stringsTableNamePrefix" value="LuceneIndexesMetadata" />

               <property name="timestampColumnName" value="timestampCol" />
               <property name="timestampColumnType" value="BIGINT" />
            </properties>
            <async enabled="true" flushLockTimeout="2500" shutdownTimeout="7200" threadPoolSize="5" />
         </loader>
      </loaders>
      <eviction maxEntries="-1" strategy="NONE" />
      <expiration maxIdle="-1" />
   </namedCache>

   <namedCache name="LuceneIndexesData">
      <clustering mode="replication">
         <stateRetrieval fetchInMemoryState="false" />

      </clustering>
      <invocationBatching enabled="true" />
      <jmxStatistics enabled="false" />
      <loaders passivation="false" shared="true" preload="true">
         <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
                fetchPersistentState="true"
                ignoreModifications="false"
                purgeOnStartup="false">

            <!-- See the documentation for more configuration examples and flags. -->
            <properties>
               <property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />
               <property name="createTableOnStart" value="true" />

               <!-- Settings for MySQL: -->
               <property name="datasourceJndiLocation" value="java:/MyDatasource" />
               <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory" />
               <property name="dataColumnType" value="BLOB" />

               <property name="idColumnType" value="VARCHAR(256)" />
               <property name="idColumnName" value="idCol" />
               <property name="dataColumnName" value="dataCol" />
               <property name="stringsTableNamePrefix" value="LuceneIndexesData" />

               <property name="timestampColumnName" value="timestampCol" />
               <property name="timestampColumnType" value="BIGINT" />
            </properties>
            <async enabled="true" flushLockTimeout="2500" shutdownTimeout="7200" threadPoolSize="5" />
         </loader>
      </loaders>
      <eviction maxEntries="-1" strategy="NONE" />
      <expiration maxIdle="-1" />
   </namedCache>
</infinispan>

And here it is my jgroups.xml

<!--
  Default flush stack using IP multicasting.
-->

<config xmlns="urn:org:jgroups"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/JGroups-2.12.xsd">
    <UDP
         mcast_addr="224.0.0.1"
         mcast_port="${jgroups.udp.mcast_port:45588}"
         tos="8"
         ucast_recv_buf_size="20000000"
         ucast_send_buf_size="640000"
         mcast_recv_buf_size="25000000"
         mcast_send_buf_size="640000"
         loopback="true"
         discard_incompatible_packets="true"
         max_bundle_size="64000"
         max_bundle_timeout="30"
         ip_ttl="${jgroups.udp.ip_ttl:2}"
         enable_bundling="true"
         enable_diagnostics="true"

         thread_naming_pattern="pl"

         thread_pool.enabled="true"
         thread_pool.min_threads="2"
         thread_pool.max_threads="8"
         thread_pool.keep_alive_time="5000"
         thread_pool.queue_enabled="false"
         thread_pool.queue_max_size="100"
         thread_pool.rejection_policy="Run"

         oob_thread_pool.enabled="true"
         oob_thread_pool.min_threads="1"
         oob_thread_pool.max_threads="8"
         oob_thread_pool.keep_alive_time="5000"
         oob_thread_pool.queue_enabled="false"
         oob_thread_pool.queue_max_size="100"
         oob_thread_pool.rejection_policy="Run"/>

    <PING timeout="10000" num_initial_members="2"/>
    <MERGE2 max_interval="300000" min_interval="10000"/>
    <FD_SOCK/>
    <FD_ALL/>
    <VERIFY_SUSPECT timeout="1500"/>
    <pbcast.NAKACK use_stats_for_retransmission="false"
                   exponential_backoff="150"
                   use_mcast_xmit="true" gc_lag="0"
                   retransmit_timeout="300,600,1200"
                   discard_delivered_msgs="false"/>
    <UNICAST timeout="300,600,1200"/>
    <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
                   max_bytes="4m"/>   
    <pbcast.GMS print_local_addr="true" join_timeout="30000" view_bundling="true" />
    <UFC max_credits="2M"
         min_threshold="0.4"/>
    <MFC max_credits="2M"
         min_threshold="0.4"/>
    <FRAG2 frag_size="60000"/>
    <pbcast.STREAMING_STATE_TRANSFER />
    <!-- <pbcast.STATE_TRANSFER/> -->
    <pbcast.FLUSH timeout="0"/>
</config>

Thanks,

      was (Author: hvico):
    Sure! Here is my infinispan.xml config:

<?xml version="1.0" encoding="UTF-8"?>

<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:4.2 http://www.infinispan.org/schemas/infinispan-config-4.2.xsd"
    xmlns="urn:infinispan:config:4.2">
  <global>

        <!-- Duplicate domains are allowed so that multiple deployments with default configuration
            of Hibernate Search applications work - if possible it would be better to use JNDI to share
            the CacheManager across applications -->
        <globalJmxStatistics
            enabled="true"
            cacheManagerName="HibernateSearch"
            allowDuplicateDomains="true" />

        <!-- If the transport is omitted, there is no way to create distributed or clustered
            caches. There is no added cost to defining a transport but not creating a cache that uses one,
            since the transport is created and initialized lazily. -->
        <transport
            clusterName="HibernateSearch-Infinispan-cluster"
            distributedSyncTimeout="50000">
            <!-- Note that the JGroups transport uses sensible defaults if no configuration
                property is defined. See the JGroupsTransport javadocs for more flags -->

            <properties>
                <property name="configurationFile" value="jgroups.xml"/>
            </properties>

        </transport>

        <!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.
            Hibernate Search takes care to stop the CacheManager so registering is not needed -->
        <shutdown
            hookBehavior="DONT_REGISTER" />

    </global>

  <default>

        <!--<locking
            lockAcquisitionTimeout="2000"
            writeSkewCheck="false"
            concurrencyLevel="500"
            useLockStriping="false" />-->
		<locking isolationLevel="READ_COMMITTED"

                    lockAcquisitionTimeout="15000" writeSkewCheck="false"

                    concurrencyLevel="5000" useLockStriping="false" />
        <!-- Invocation batching is required for use with the Lucene Directory -->
        <invocationBatching
            enabled="true" />

        <!-- This element specifies that the cache is clustered. modes supported: distribution
            (d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as
            with Hibernate Search DirectoryProvider). Replication is recommended for best performance of
            Lucene indexes, but make sure you have enough memory to store the index in your heap.
            Also distribution scales much better than replication on high number of nodes in the cluster. -->

        <jmxStatistics
            enabled="false" />

        <eviction
            maxEntries="-1"
            strategy="NONE" />

        <expiration
            maxIdle="-1" />
		 <clustering mode="replication">
                <stateRetrieval fetchInMemoryState="false" logFlushTimeout="60000"/>
                <sync replTimeout="50000"/>
         </clustering>
    </default>

   <namedCache name="LuceneIndexesLocking">
        <clustering mode="replication">
                <stateRetrieval fetchInMemoryState="false" logFlushTimeout="60000"/>
                <sync replTimeout="50000"/>
         </clustering>
         <invocationBatching enabled="true" />      
   </namedCache>

   <namedCache name="LuceneIndexesMetadata">
      <clustering mode="replication">
         <stateRetrieval fetchInMemoryState="false" />

      </clustering>
      <invocationBatching enabled="true" />
      <jmxStatistics enabled="false" />
      <loaders passivation="false" shared="true" preload="true">
         <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
                fetchPersistentState="true"
                ignoreModifications="false"
                purgeOnStartup="false">

            <!-- See the documentation for more configuration examples and flags. -->
            <properties>
               <property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />
               <property name="createTableOnStart" value="true" />

               <!-- Settings for MySQL: -->
               <property name="datasourceJndiLocation" value="java:/BDZWEB2Datasource" />
               <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory" />
               <property name="dataColumnType" value="BLOB" />

               <property name="idColumnType" value="VARCHAR(256)" />
               <property name="idColumnName" value="idCol" />
               <property name="dataColumnName" value="dataCol" />
               <property name="stringsTableNamePrefix" value="LuceneIndexesMetadata" />

               <property name="timestampColumnName" value="timestampCol" />
               <property name="timestampColumnType" value="BIGINT" />
            </properties>
            <async enabled="true" flushLockTimeout="2500" shutdownTimeout="7200" threadPoolSize="5" />
         </loader>
      </loaders>
      <eviction maxEntries="-1" strategy="NONE" />
      <expiration maxIdle="-1" />
   </namedCache>

   <namedCache name="LuceneIndexesData">
      <clustering mode="replication">
         <stateRetrieval fetchInMemoryState="false" />

      </clustering>
      <invocationBatching enabled="true" />
      <jmxStatistics enabled="false" />
      <loaders passivation="false" shared="true" preload="true">
         <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore"
                fetchPersistentState="true"
                ignoreModifications="false"
                purgeOnStartup="false">

            <!-- See the documentation for more configuration examples and flags. -->
            <properties>
               <property name="key2StringMapperClass" value="org.infinispan.lucene.LuceneKey2StringMapper" />
               <property name="createTableOnStart" value="true" />

               <!-- Settings for MySQL: -->
               <property name="datasourceJndiLocation" value="java:/MyDatasource" />
               <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory" />
               <property name="dataColumnType" value="BLOB" />

               <property name="idColumnType" value="VARCHAR(256)" />
               <property name="idColumnName" value="idCol" />
               <property name="dataColumnName" value="dataCol" />
               <property name="stringsTableNamePrefix" value="LuceneIndexesData" />

               <property name="timestampColumnName" value="timestampCol" />
               <property name="timestampColumnType" value="BIGINT" />
            </properties>
            <async enabled="true" flushLockTimeout="2500" shutdownTimeout="7200" threadPoolSize="5" />
         </loader>
      </loaders>
      <eviction maxEntries="-1" strategy="NONE" />
      <expiration maxIdle="-1" />
   </namedCache>
</infinispan>

And here it is my jgroups.xml

<!--
  Default flush stack using IP multicasting.
-->

<config xmlns="urn:org:jgroups"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/JGroups-2.12.xsd">
    <UDP
         mcast_addr="224.0.0.1"
         mcast_port="${jgroups.udp.mcast_port:45588}"
         tos="8"
         ucast_recv_buf_size="20000000"
         ucast_send_buf_size="640000"
         mcast_recv_buf_size="25000000"
         mcast_send_buf_size="640000"
         loopback="true"
         discard_incompatible_packets="true"
         max_bundle_size="64000"
         max_bundle_timeout="30"
         ip_ttl="${jgroups.udp.ip_ttl:2}"
         enable_bundling="true"
         enable_diagnostics="true"

         thread_naming_pattern="pl"

         thread_pool.enabled="true"
         thread_pool.min_threads="2"
         thread_pool.max_threads="8"
         thread_pool.keep_alive_time="5000"
         thread_pool.queue_enabled="false"
         thread_pool.queue_max_size="100"
         thread_pool.rejection_policy="Run"

         oob_thread_pool.enabled="true"
         oob_thread_pool.min_threads="1"
         oob_thread_pool.max_threads="8"
         oob_thread_pool.keep_alive_time="5000"
         oob_thread_pool.queue_enabled="false"
         oob_thread_pool.queue_max_size="100"
         oob_thread_pool.rejection_policy="Run"/>

    <PING timeout="10000" num_initial_members="2"/>
    <MERGE2 max_interval="300000" min_interval="10000"/>
    <FD_SOCK/>
    <FD_ALL/>
    <VERIFY_SUSPECT timeout="1500"/>
    <pbcast.NAKACK use_stats_for_retransmission="false"
                   exponential_backoff="150"
                   use_mcast_xmit="true" gc_lag="0"
                   retransmit_timeout="300,600,1200"
                   discard_delivered_msgs="false"/>
    <UNICAST timeout="300,600,1200"/>
    <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
                   max_bytes="4m"/>   
    <pbcast.GMS print_local_addr="true" join_timeout="30000" view_bundling="true" />
    <UFC max_credits="2M"
         min_threshold="0.4"/>
    <MFC max_credits="2M"
         min_threshold="0.4"/>
    <FRAG2 frag_size="60000"/>
    <pbcast.STREAMING_STATE_TRANSFER />
    <!-- <pbcast.STATE_TRANSFER/> -->
    <pbcast.FLUSH timeout="0"/>
</config>

Thanks,

> Read past EOF caused in Lucene Directory when Lucene flushes but doesn't close the segment
> ------------------------------------------------------------------------------------------
>
>                 Key: ISPN-689
>                 URL: https://issues.jboss.org/browse/ISPN-689
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Lucene Directory
>    Affects Versions: 4.1.0.Final
>            Reporter: Sanne Grinovero
>            Assignee: Sanne Grinovero
>             Fix For: 4.2.0.ALPHA3, 4.2.0.Final
>
>
> While this is not the default access scenario performed by Lucene it's possible in some branches to flush the segment and be able to read it before the close.
> I could reproduce the following stacktrace in high load / huge sized index but not in a unit test; disabling the batching started between .flush() and .close() seems to resolve the issue; also this batch seems totally useless as I couldn't find any chance in performance when disabling it.
> java.io.IOException: Read past EOF: Chunk value could not be found for key _ni.fdt|4|issues
> 	at org.infinispan.lucene.InfinispanIndexInput.setBufferToCurrentChunk(InfinispanIndexInput.java:138)
> 	at org.infinispan.lucene.InfinispanIndexInput.nextChunk(InfinispanIndexInput.java:131)
> 	at org.infinispan.lucene.InfinispanIndexInput.readBytes(InfinispanIndexInput.java:96)
> 	at org.apache.lucene.store.IndexInput.readBytes(IndexInput.java:61)
> 	at org.apache.lucene.index.CompoundFileWriter.copyFile(CompoundFileWriter.java:228)
> 	at org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:184)
> 	at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:2342)
> 	at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4359)
> 	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264)
> 	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255)
> 	at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:4133)
> 	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4206)
> 	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:4179)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira