CacheWriterInterceptor invokes batch persistence operations with potentially empty entry sets --------------------------------------------------------------------------------------------- Key: ISPN-9139 URL: https://issues.jboss.org/browse/ISPN-9139 Project: Infinispan Issue Type: Bug Components: Loaders and Stores Affects Versions: 9.2.2.Final Reporter: Moritz Becker Assignee: Ryan Emerson In {{org.infinispan.interceptors.impl.CacheWriterInterceptor#store}}, you have the following: {code:java} persistenceManager.writeBatchToAllNonTxStores(sharedMods.getMarshalledEntries(), BOTH, 0); persistenceManager.writeBatchToAllNonTxStores(nonSharedMods.getMarshalledEntries(), PRIVATE, 0); persistenceManager.deleteBatchFromAllNonTxStores(sharedMods.getKeysToRemove(), BOTH, 0); persistenceManager.deleteBatchFromAllNonTxStores(nonSharedMods.getKeysToRemove(), PRIVATE, 0); {code} The batch deletion causes an error with JPAStore in case of an empty key set because the an SQL query like {{where id in ()}}, i.e. the comparison set for the in clause is empty. At least with postgres, this causes the following exception: {code} ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (EE-ManagedExecutorService-default-Thread-2) ISPN000136: Error executing command CommitCommand, writing keys []: org.infinispan.persistence.jpa.JpaStoreException: Exception caught in deleteBatch() at org.infinispan.persistence.jpa.JpaStore.deleteBatch(JpaStore.java:350) at org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$deleteBatchFromAllNonTxStores$14(PersistenceManagerImpl.java:562) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at org.infinispan.persistence.manager.PersistenceManagerImpl.deleteBatchFromAllNonTxStores(PersistenceManagerImpl.java:562) at org.infinispan.interceptors.impl.CacheWriterInterceptor.store(CacheWriterInterceptor.java:406) at org.infinispan.interceptors.impl.CacheWriterInterceptor.commitCommand(CacheWriterInterceptor.java:123) at org.infinispan.interceptors.impl.CacheWriterInterceptor.visitCommitCommand(CacheWriterInterceptor.java:102) at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) at org.infinispan.interceptors.impl.TransactionalStoreInterceptor.visitCommitCommand(TransactionalStoreInterceptor.java:53) at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) at org.infinispan.interceptors.DDAsyncInterceptor.visitCommitCommand(DDAsyncInterceptor.java:142) at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:150) at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitCommitCommand(EntryWrappingInterceptor.java:183) at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:98) at org.infinispan.interceptors.impl.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:46) at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:150) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:51) at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:98) at org.infinispan.interceptors.impl.TxInterceptor.finishCommit(TxInterceptor.java:189) at org.infinispan.interceptors.impl.TxInterceptor.visitCommitCommand(TxInterceptor.java:183) at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54) at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) at org.infinispan.interceptors.DDAsyncInterceptor.visitCommitCommand(DDAsyncInterceptor.java:142) at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:38) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123) at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137) at org.infinispan.transaction.impl.TransactionCoordinator.commitInternal(TransactionCoordinator.java:219) at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:161) at org.infinispan.transaction.impl.TransactionTable.afterCompletion(TransactionTable.java:851) at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:33) at org.wildfly.transaction.client.AbstractTransaction.performConsumer(AbstractTransaction.java:196) at org.wildfly.transaction.client.AbstractTransaction$AssociatingSynchronization.afterCompletion(AbstractTransaction.java:279) at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:542) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:101) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1289) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89) at org.wildfly.transaction.client.LocalTransaction.commitAndDissociate(LocalTransaction.java:73) at org.wildfly.transaction.client.ContextTransactionManager.commit(ContextTransactionManager.java:71) at org.infinispan.persistence.manager.PersistenceManagerImpl.commitIfNeeded(PersistenceManagerImpl.java:874) at org.infinispan.persistence.manager.PersistenceManagerImpl.preloadKey(PersistenceManagerImpl.java:830) at org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$preload$2(PersistenceManagerImpl.java:279) at org.infinispan.persistence.jpa.JpaStore$LoadingProcessTask.call(JpaStore.java:915) at org.infinispan.persistence.jpa.JpaStore$LoadingProcessTask.call(JpaStore.java:843) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20) at java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:181) at org.infinispan.executors.ExecutorAllCompletionService.submit(ExecutorAllCompletionService.java:31) at org.infinispan.persistence.jpa.JpaStore.process(JpaStore.java:644) at org.infinispan.persistence.jpa.JpaStore.process(JpaStore.java:583) at org.infinispan.persistence.manager.PersistenceManagerImpl.preload(PersistenceManagerImpl.java:271) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) at org.infinispan.security.Security.doPrivileged(Security.java:76) at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:229) at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1020) at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) at com.ordami.geo.service.impl.service.MatrixUpdateServiceImpl.updateMatrixEntry(MatrixUpdateServiceImpl.java:157) at com.ordami.geo.service.impl.service.MatrixUpdateServiceImpl$Proxy$_$$_WeldSubclass.updateMatrixEntry$$super(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:49) at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:77) at com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:120) at com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorRequired.doIntercept(TransactionalInterceptorRequired.java:53) at com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.intercept(TransactionalInterceptorBase.java:79) at com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorRequired.intercept(TransactionalInterceptorRequired.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) at com.ordami.geo.service.impl.service.MatrixUpdateServiceImpl$Proxy$_$$_WeldSubclass.updateMatrixEntry(Unknown Source) at com.ordami.geo.service.impl.service.MatrixUpdateServiceImpl$Proxy$_$$_WeldClientProxy.updateMatrixEntry(Unknown Source) at com.ordami.geo.service.impl.service.MatrixUpdateServiceImpl$MatrixEntryUpdateTask.run(MatrixUpdateServiceImpl.java:207) at org.jboss.as.ee.concurrent.ControlPointUtils$ControlledRunnable.run(ControlPointUtils.java:105) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at org.glassfish.enterprise.concurrent.internal.ManagedFutureTask.run(ManagedFutureTask.java:141) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78) Caused by: javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not execute statement at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:149) at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:157) at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1514) at org.infinispan.persistence.jpa.JpaStore.deleteBatch(JpaStore.java:334) ... 133 more Caused by: org.hibernate.exception.SQLGrammarException: could not execute statement at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:106) at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:178) at org.hibernate.hql.internal.ast.exec.BasicExecutor.doExecute(BasicExecutor.java:100) at org.hibernate.hql.internal.ast.exec.BasicExecutor.execute(BasicExecutor.java:59) at org.hibernate.hql.internal.ast.exec.DeleteExecutor.execute(DeleteExecutor.java:110) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.executeUpdate(QueryTranslatorImpl.java:450) at org.hibernate.engine.query.spi.HQLQueryPlan.performExecuteUpdate(HQLQueryPlan.java:374) at org.hibernate.internal.SessionImpl.executeUpdate(SessionImpl.java:1514) at org.hibernate.query.internal.AbstractProducedQuery.doExecuteUpdate(AbstractProducedQuery.java:1526) at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1504) ... 134 more Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near ")" Position: 105 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2477) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2190) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:300) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:169) at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:136) at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:537) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:175) ... 142 more {code} Since you are doing expensive locking operations in all batch operations, I propose to change is code part to: {code:java} if (!sharedMods.getMarshalledEntries().isEmpty()) { persistenceManager.writeBatchToAllNonTxStores(sharedMods.getMarshalledEntries(), BOTH, 0); } if (!nonSharedMods.getMarshalledEntries().isEmpty()) { persistenceManager.writeBatchToAllNonTxStores(nonSharedMods.getMarshalledEntries(), PRIVATE, 0); } if (!sharedMods.getKeysToRemove().isEmpty()) { persistenceManager.deleteBatchFromAllNonTxStores(sharedMods.getKeysToRemove(), BOTH, 0); } if (!nonSharedMods.getKeysToRemove().isEmpty()) { persistenceManager.deleteBatchFromAllNonTxStores(nonSharedMods.getKeysToRemove(), PRIVATE, 0); } {code}