[infinispan-issues] [JBoss JIRA] (ISPN-8503) Error executing MassIndexer using off-heap

Error executing MassIndexer using off-heap ------------------------------------------ Key: ISPN-8503 URL: https://issues.jboss.org/browse/ISPN-8503 Project: Infinispan Issue Type: Bug Reporter: Lucas Ponce Assignee: Gustavo Fernandes Invoking a re-index operation on startup from the API {code} SearchManager searchResourceManager = Search.getSearchManager(resource); CompletableFuture<Void> reindexResource = searchResourceManager.getMassIndexer().startAsync(); SearchManager searchResourceTypeManager = Search.getSearchManager(resourceType); CompletableFuture<Void> reindexResourceType = searchResourceTypeManager.getMassIndexer().startAsync(); CompletableFuture.allOf(reindexResource, reindexResourceType).get(); {code} Throws the following Exception {code} 14:01:23,537 ERROR [org.hawkular.inventory.service.InventoryConfig] (ServerService Thread Pool -- 62) HAWKINV100005: Error reindexing caches: java.util.concurrent.ExecutionException: org.infinispan.commons.CacheException: ISPN014018: Error executing MassIndexer at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.hawkular.inventory.service.InventoryConfig.init(InventoryConfig.java:111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:114) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:103) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:53) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:100) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275) at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349) at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161) at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134) at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88) at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:124) at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:138) at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320) Caused by: org.infinispan.commons.CacheException: ISPN014018: Error executing MassIndexer at org.infinispan.query.impl.massindex.DistributedExecutorMassIndexer.lambda$null$0(DistributedExecutorMassIndexer.java:73) at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870) at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) at org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart.lambda$execute$1(DefaultExecutorService.java:1086) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: Indexing only works with entries keyed on Strings, primitives and classes that have the @Transformable annotation - you passed in a class org.infinispan.commons.marshall.WrappedByteArray. Alternatively, see org.infinispan.query.SearchManager#registerKeyTransformer at org.infinispan.query.backend.KeyTransformationHandler.keyToString(KeyTransformationHandler.java:154) at org.infinispan.query.impl.massindex.IndexUpdater.updateIndex(IndexUpdater.java:67) at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:101) at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:34) at org.infinispan.commands.read.DistributedExecuteCommand.invokeAsync(DistributedExecuteCommand.java:99) at org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart.lambda$execute$1(DefaultExecutorService.java:1074) ... 3 more {code} Infinispan 9.2.0.Alpha1 is running in library mode embedded in a .war application with the following configuration: {code} <cache-container name="hawkular-inventory"> <jmx duplicate-domains="true" /> <local-cache name="resource" statistics="true"> <transaction mode="NON_XA"/> <persistence> <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory"> <write-behind thread-pool-size="5" modification-queue-size="10000" /> </file-store> </persistence> <memory> <!-- <binary eviction="COUNT" size="10000" /> --> <!-- <object size="5000" /> --> <off-heap eviction="COUNT" size="100000" /> <!-- <off-heap eviction="COUNT" size="100000" /> --> </memory> <indexing index="LOCAL"> <indexed-entities> <indexed-entity>org.hawkular.inventory.service.ispn.IspnResource</indexed-entity> </indexed-entities> <property name="default.indexmanager">near-real-time</property> <property name="default.directory_provider">infinispan</property> <property name="default.chunk_size">128000</property> <property name="default.locking_cachename">resource_indexes_locking</property> <property name="default.data_cachename">resource_indexes_data</property> <property name="default.metadata_cachename">resource_indexes_metadata</property> <!-- The default is 10, but we don't want to waste many cycles in merging (tune for writes at cost of reader fragmentation) --> <property name="default.indexwriter.merge_factor">30</property> <!-- Never create segments larger than 1GB --> <property name="default.indexwriter.merge_max_size">1024</property> <!-- IndexWriter flush buffer size in MB --> <property name="default.indexwriter.ram_buffer_size">64</property> <!-- Enable sharding on writers --> <property name="default.sharding_strategy.nbr_of_shards">6</property> <property name="lucene_version">LUCENE_CURRENT</property> </indexing> </local-cache> <local-cache name="resource_indexes_metadata"> <persistence passivation="false"> <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory"> <write-behind thread-pool-size="5" /> </file-store> </persistence> <indexing index="NONE"/> </local-cache> <local-cache name="resource_indexes_data"> <persistence passivation="false"> <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory"> <write-behind thread-pool-size="5" /> </file-store> </persistence> <indexing index="NONE" /> </local-cache> <local-cache name="resource_indexes_locking"> <persistence passivation="false"> <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory"> <write-behind thread-pool-size="5" /> </file-store> </persistence> <indexing index="NONE" /> </local-cache> <local-cache name="resource_type" statistics="true"> <transaction mode="NON_XA"/> <persistence> <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory"> <write-behind thread-pool-size="5" modification-queue-size="10000" /> </file-store> </persistence> <indexing index="LOCAL"> <indexed-entities> <indexed-entity>org.hawkular.inventory.service.ispn.IspnResourceType</indexed-entity> </indexed-entities> <property name="default.indexmanager">near-real-time</property> <property name="default.directory_provider">infinispan</property> <property name="default.chunk_size">128000</property> <property name="default.locking_cachename">resource_type_indexes_locking</property> <property name="default.data_cachename">resource_type_indexes_data</property> <property name="default.metadata_cachename">resource_type_indexes_metadata</property> <!-- The default is 10, but we don't want to waste many cycles in merging (tune for writes at cost of reader fragmentation) --> <property name="default.indexwriter.merge_factor">30</property> <!-- Never create segments larger than 1GB --> <property name="default.indexwriter.merge_max_size">1024</property> <!-- IndexWriter flush buffer size in MB --> <property name="default.indexwriter.ram_buffer_size">64</property> <!-- Enable sharding on writers --> <property name="default.sharding_strategy.nbr_of_shards">6</property> <property name="lucene_version">LUCENE_CURRENT</property> </indexing> </local-cache> <local-cache name="resource_type_indexes_metadata"> <persistence passivation="false"> <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory"> <write-behind thread-pool-size="5" /> </file-store> </persistence> <indexing index="NONE"/> </local-cache> <local-cache name="resource_type_indexes_data"> <persistence passivation="false"> <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory"> <write-behind thread-pool-size="5" /> </file-store> </persistence> <indexing index="NONE" /> </local-cache> <local-cache name="resource_type_indexes_locking"> <persistence passivation="false"> <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory"> <write-behind thread-pool-size="5" /> </file-store> </persistence> <indexing index="NONE" /> </local-cache> </cache-container> {code} At initialization, it tries to re-index existing resources*.dat files for data and lucene indexes.