| --On production Env-- Ours is a J2EE application (clustered across 2 nodes) internally using Hibernate Core 4.3.6, Spring 4.3.1 and Hibernate Search 4.5.1. We are using hibernate search indexes in our application heavily. These are stored on FILESYSTEM - ZFS (Extreme Flash on Oracle ExaStack), and shared between both the nodes. Application runs fine, but frequent locking exception comes in logs. (Cause: Unable to acquire lock) We are concerned that, these exceptions might cause inconsistency in data. However application seems to be working fine - in terms of user experience. But if we now restart the applications (both the nodes), application startup fails with exception: java.io.EOFException: read past EOF: MMapIndexInput (path="/.../.../_39d.cfs") Note:The file name '_39d.cfs' is not fixed, and it varies in different occurences. As this is a production env, it is impacting adversely. We have created a trigger at server startup, which based on a flag, manually creates all the hibernate indexes at startup. (Takes around 12-14 hours) So as a workaround to this issue, we delete all the corrupted hibernate indexes from the filesystem, and recreate the indexes manually using this startup trigger. Stack Trace: <Error> <Deployer> <PRODCAS1-172-16-5-29> <CAS-SERVER-OHS-1> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <8118a889-fb3b-41f7-9951-b3129fe5130a-00000044> <1515523387207> <BEA-149265> <Failure occurred in the execution of deployment request with ID "17254792326560163" for task "23". Error is: "weblogic.application.ModuleException: java.io.EOFException: read past EOF: MMapIndexInput(path="/artifacts_dumps/hibernate-search_index-caslms/casTCFSL/com.nucleus.contact.PhoneNumber/_39d.cfs")" weblogic.application.ModuleException: java.io.EOFException: read past EOF: MMapIndexInput(path="/artifacts_dumps/hibernate-search_index-caslms/casTCFSL/com.nucleus.contact.PhoneNumber/_39d.cfs") at weblogic.application.internal.ExtensibleModuleWrapper.start(ExtensibleModuleWrapper.java:140) at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:124) at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:216) at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:211) at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42) at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:73) at weblogic.application.internal.flow.StartModulesFlow.activate(StartModulesFlow.java:24) at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:729) at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42) at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:258) at weblogic.application.internal.SingleModuleDeployment.activate(SingleModuleDeployment.java:48) at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:165) at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:80) at weblogic.deploy.internal.targetserver.operations.AbstractOperation.activate(AbstractOperation.java:587) at weblogic.deploy.internal.targetserver.operations.ActivateOperation.activateDeployment(ActivateOperation.java:150) at weblogic.deploy.internal.targetserver.operations.ActivateOperation.doCommit(ActivateOperation.java:116) at weblogic.deploy.internal.targetserver.operations.StartOperation.doCommit(StartOperation.java:151) at weblogic.deploy.internal.targetserver.operations.AbstractOperation.commit(AbstractOperation.java:339) at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentCommit(DeploymentManager.java:846) at weblogic.deploy.internal.targetserver.DeploymentManager.activateDeploymentList(DeploymentManager.java:1275) at weblogic.deploy.internal.targetserver.DeploymentManager.handleCommit(DeploymentManager.java:442) at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.commit(DeploymentServiceDispatcher.java:176) at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doCommitCallback(DeploymentReceiverCallbackDeliverer.java:195) at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$100(DeploymentReceiverCallbackDeliverer.java:13) at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$2.run(DeploymentReceiverCallbackDeliverer.java:68) at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:548) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311) at weblogic.work.ExecuteThread.run(ExecuteThread.java:263) Caused By: java.io.EOFException: read past EOF: MMapIndexInput(path="/artifacts_dumps/hibernate-search_index-caslms/casTCFSL/com.nucleus.contact.PhoneNumber/_39d.cfs") at org.apache.lucene.store.MMapDirectory$MMapIndexInput.readByte(MMapDirectory.java:284) at org.apache.lucene.store.DataInput.readVInt(DataInput.java:107) at org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:69) at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:75) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94) at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105) at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:27) at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72) at org.apache.lucene.index.IndexReader.open(IndexReader.java:256) at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.readerFactory(SharingBufferReaderProvider.java:143) at org.hibernate.search.indexes.impl.SharingBufferReaderProvider$PerDirectoryLatestReader.<init>(SharingBufferReaderProvider.java:218) at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.createReader(SharingBufferReaderProvider.java:120) at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.initialize(SharingBufferReaderProvider.java:103) at org.hibernate.search.indexes.impl.PropertiesParseHelper.createDirectoryBasedReaderProvider(PropertiesParseHelper.java:158) at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.createIndexReader(DirectoryBasedIndexManager.java:208) at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.initialize(DirectoryBasedIndexManager.java:104) at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:261) at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:528) at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManagers(IndexManagerHolder.java:495) at org.hibernate.search.indexes.impl.IndexManagerHolder.buildEntityIndexBinding(IndexManagerHolder.java:104) at org.hibernate.search.spi.SearchFactoryBuilder.initDocumentBuilders(SearchFactoryBuilder.java:363) at org.hibernate.search.spi.SearchFactoryBuilder.buildNewSearchFactory(SearchFactoryBuilder.java:219) at org.hibernate.search.spi.SearchFactoryBuilder.buildSearchFactory(SearchFactoryBuilder.java:143) at org.hibernate.search.hcore.impl.HibernateSearchSessionFactoryObserver.sessionFactoryCreated(HibernateSearchSessionFactoryObserver.java:74) at org.hibernate.internal.SessionFactoryObserverChain.sessionFactoryCreated(SessionFactoryObserverChain.java:52) at org.hibernate.internal.SessionFactoryImpl.<init>(SessionFactoryImpl.java:588) at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1857) at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl$4.perform(EntityManagerFactoryBuilderImpl.java:850) at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl$4.perform(EntityManagerFactoryBuilderImpl.java:843) at org.hibernate.boot.registry.classloading.internal.ClassLoaderServiceImpl.withTccl(ClassLoaderServiceImpl.java:398) at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:842) at org.springframework.orm.jpa.vendor.SpringHibernateJpaPersistenceProvider.createContainerEntityManagerFactory(SpringHibernateJpaPersistenceProvider.java:60) at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:338) at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:373) at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:362) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1637) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:545) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor.findDefaultEntityManagerFactory(PersistenceAnnotationBeanPostProcessor.java:580) at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor.findEntityManagerFactory(PersistenceAnnotationBeanPostProcessor.java:546) at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor$PersistenceElement.resolveEntityManager(PersistenceAnnotationBeanPostProcessor.java:712) at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor$PersistenceElement.getResourceToInject(PersistenceAnnotationBeanPostProcessor.java:685) at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:169) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88) at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor.postProcessPropertyValues(PersistenceAnnotationBeanPostProcessor.java:354) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:187) at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1213) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1053) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1018) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:566) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:349) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:187) at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1213) at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1053) at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1018) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:566) at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:349) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:240) at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:697) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:526) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107) at weblogic.servlet.internal.EventsManager$FireContextListenerAction.run(EventsManager.java:678) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57) at weblogic.servlet.internal.EventsManager.executeContextListener(EventsManager.java:243) at weblogic.servlet.internal.EventsManager.notifyContextCreatedEvent(EventsManager.java:200) at weblogic.servlet.internal.EventsManager.notifyContextCreatedEvent(EventsManager.java:185) at weblogic.servlet.internal.WebAppServletContext.preloadResources(WebAppServletContext.java:1838) at weblogic.servlet.internal.WebAppServletContext.start(WebAppServletContext.java:2876) at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:1661) at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:823) at weblogic.application.internal.ExtensibleModuleWrapper$StartStateChange.next(ExtensibleModuleWrapper.java:360) at weblogic.application.internal.ExtensibleModuleWrapper$StartStateChange.next(ExtensibleModuleWrapper.java:356) at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42) at weblogic.application.internal.ExtensibleModuleWrapper.start(ExtensibleModuleWrapper.java:138) at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:124) at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:216) at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:211) at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42) at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:73) at weblogic.application.internal.flow.StartModulesFlow.activate(StartModulesFlow.java:24) at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:729) at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:42) at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:258) at weblogic.application.internal.SingleModuleDeployment.activate(SingleModuleDeployment.java:48) at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:165) at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:80) at weblogic.deploy.internal.targetserver.operations.AbstractOperation.activate(AbstractOperation.java:587) at weblogic.deploy.internal.targetserver.operations.ActivateOperation.activateDeployment(ActivateOperation.java:150) at weblogic.deploy.internal.targetserver.operations.ActivateOperation.doCommit(ActivateOperation.java:116) at weblogic.deploy.internal.targetserver.operations.StartOperation.doCommit(StartOperation.java:151) at weblogic.deploy.internal.targetserver.operations.AbstractOperation.commit(AbstractOperation.java:339) at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentCommit(DeploymentManager.java:846) at weblogic.deploy.internal.targetserver.DeploymentManager.activateDeploymentList(DeploymentManager.java:1275) at weblogic.deploy.internal.targetserver.DeploymentManager.handleCommit(DeploymentManager.java:442) at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.commit(DeploymentServiceDispatcher.java:176) at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doCommitCallback(DeploymentReceiverCallbackDeliverer.java:195) at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$100(DeploymentReceiverCallbackDeliverer.java:13) at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$2.run(DeploymentReceiverCallbackDeliverer.java:68) at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:548) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311) at weblogic.work.ExecuteThread.run(ExecuteThread.java:263) Note: Before moving to clustered env (when onlt 1 node was there), neither we face any locking issue, nor we face this corruption of indexes issue. |