[infinispan-issues] [JBoss JIRA] Commented: (ISPN-1362) FileCacheStore should limit the number of files kept open at the same time

FileCacheStore should limit the number of files kept open at the same time -------------------------------------------------------------------------- Key: ISPN-1362 URL: https://issues.jboss.org/browse/ISPN-1362 Project: Infinispan Issue Type: Bug Components: Loaders and Stores, Lucene Directory Affects Versions: 5.0.0.FINAL Environment: OS: Mac OSX 10.6 IDE: IntelliJ IDEA 10 Java: 10.6.0_26 Hibernate Search: 3.4.0.Final Lucene: 3.1.0 jGroups: 2.12.1.3.Final Reporter: Todd Underwood Assignee: Manik Surtani Labels: hibernate-search, persistence When trying to initalize my index for Hibernate search with persistance I get the following exception after several hours of indexing: [2011-08-29 11:30:53,425] ERROR FileCacheStore.java:317 Hibernate Search: indexwriter-154 ) ISPN000063: Exception while saving bucket Bucket{entries={_4o.fdt|M|cnwk.foreman.model.SoftwareDownload=ImmortalCacheEntry{key=_4o.fdt|M|cnwk.foreman.model.SoftwareDownload, value=ImmortalCacheValue{value=FileMetadata{lastModified=1314642653425, size=32768}}}}, bucketId='1509281792'} java.io.FileNotFoundException: /var/opt/fullTextStore/LuceneIndexesMetadata/1509281792 (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216) at org.infinispan.loaders.file.FileCacheStore$BufferedFileSync.createChannel(FileCacheStore.java:494) at org.infinispan.loaders.file.FileCacheStore$BufferedFileSync.write(FileCacheStore.java:472) at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:315) at org.infinispan.loaders.bucket.BucketBasedCacheStore.insertBucket(BucketBasedCacheStore.java:137) at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:94) at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49) at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:195) at org.infinispan.interceptors.CacheStoreInterceptor.visitPutKeyValueCommand(CacheStoreInterceptor.java:210) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:82) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:214) at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:162) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:114) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119) at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:77) at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60) at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:274) at org.infinispan.CacheImpl.put(CacheImpl.java:515) at org.infinispan.CacheSupport.put(CacheSupport.java:51) at org.infinispan.lucene.InfinispanIndexOutput.close(InfinispanIndexOutput.java:206) at org.apache.lucene.util.IOUtils.closeSafely(IOUtils.java:80) at org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:111) at org.apache.lucene.index.FieldsWriter.abort(FieldsWriter.java:121) at org.apache.lucene.index.StoredFieldsWriter.abort(StoredFieldsWriter.java:90) at org.apache.lucene.index.DocFieldProcessor.abort(DocFieldProcessor.java:71) at org.apache.lucene.index.DocumentsWriter.abort(DocumentsWriter.java:421) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:729) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2042) at org.hibernate.search.backend.impl.lucene.works.AddWorkDelegate.performWork(AddWorkDelegate.java:76) at org.hibernate.search.backend.impl.batchlucene.DirectoryProviderWorkspace.doWorkInSync(DirectoryProviderWorkspace.java:96) at org.hibernate.search.backend.impl.batchlucene.DirectoryProviderWorkspace$AsyncIndexRunnable.run(DirectoryProviderWorkspace.java:144) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) The open file limit on my machine has already been increased to try to fix the issue. This is the configuration used when the exception is thrown: ]<?xml version="1.0" encoding="UTF-8"?> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:infinispan:config:4.2 http://www.infinispan.org/schemas/infinispan-config-4.2.xsd" xmlns="urn:infinispan:config:4.2"> <!-- *************************** --> <!-- System-wide global settings --> <!-- *************************** --> <global> <!-- Duplicate domains are allowed so that multiple deployments with default configuration of Hibernate Search applications work - if possible it would be better to use JNDI to share the CacheManager across applications --> <globalJmxStatistics enabled="true" cacheManagerName="HibernateSearch" allowDuplicateDomains="true"/> <!-- If the transport is omitted, there is no way to create distributed or clustered caches. There is no added cost to defining a transport but not creating a cache that uses one, since the transport is created and initialized lazily. --> <transport clusterName="HibernateSearch-Infinispan-cluster" distributedSyncTimeout="50000"> <!-- Note that the JGroups transport uses sensible defaults if no configuration property is defined. See the JGroupsTransport javadocs for more flags --> </transport> <!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER. Hibernate Search takes care to stop the CacheManager so registering is not needed --> <shutdown hookBehavior="DONT_REGISTER"/> </global> <!-- *************************** --> <!-- Default "template" settings --> <!-- *************************** --> <default> <locking lockAcquisitionTimeout="20000" writeSkewCheck="false" concurrencyLevel="500" useLockStriping="false"/> <lazyDeserialization enabled="false"/> <!-- Invocation batching is required for use with the Lucene Directory --> <invocationBatching enabled="true"/> <!-- This element specifies that the cache is clustered. modes supported: distribution (d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as with Hibernate Search DirectoryProvider). Replication is recommended for best performance of Lucene indexes, but make sure you have enough memory to store the index in your heap. Also distribution scales much better than replication on high number of nodes in the cluster. --> <clustering mode="replication"> <!-- Prefer loading all data at startup than later --> <stateRetrieval timeout="60000" logFlushTimeout="30000" fetchInMemoryState="true" alwaysProvideInMemoryState="true"/> <!-- Network calls are synchronous by default --> <sync replTimeout="20000"/> </clustering> <jmxStatistics enabled="true"/> <eviction maxEntries="-1" strategy="NONE"/> <expiration maxIdle="-1"/> </default> <!-- ******************************************************************************* --> <!-- Individually configured "named" caches. --> <!-- --> <!-- While default configuration happens to be fine with similar settings across the --> <!-- three caches, they should generally be different in a production environment. --> <!-- --> <!-- Current settings could easily lead to OutOfMemory exception as a CacheStore --> <!-- should be enabled, and maybe distribution is desired. --> <!-- ******************************************************************************* --> <!-- *************************************** --> <!-- Cache to store Lucene's file metadata --> <!-- *************************************** --> <namedCache name="LuceneIndexesMetadata"> <clustering mode="replication"> <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000"/> <sync replTimeout="25000"/> </clustering> <loaders preload="true"> <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true"> <properties> <property name="location" value="/var/opt/fullTextStore"/> </properties> </loader> </loaders> </namedCache> <!-- **************************** --> <!-- Cache to store Lucene data --> <!-- **************************** --> <namedCache name="LuceneIndexesData"> <clustering mode="replication"> <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000"/> <sync replTimeout="25000"/> </clustering> <loaders> <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true"> <properties> <property name="location" value="/var/opt/fullTextStore"/> </properties> </loader> </loaders> </namedCache> <!-- ***************************** --> <!-- Cache to store Lucene locks --> <!-- ***************************** --> <namedCache name="LuceneIndexesLocking"> <clustering mode="replication"> <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000"/> <sync replTimeout="25000"/> </clustering> </namedCache> </infinispan> There are 10160 open files in the cache store when the exception is thrown and a total of 10178 files visible in the cache store. Submitting this so the issue can be tracked after being suggested to do so on Hibernate Search Forums.