Database Outage causes Session Persistence to break ---------------------------------------------------- Key: DROOLS-1093 URL: https://issues.jboss.org/browse/DROOLS-1093 Project: Drools Issue Type: Bug Components: core engine Affects Versions: 6.3.0.Final Environment: Mac OS 10.10.5, Eclipse Mars Release, Java 1.8, Drools 6.3.0.Final Docs QE Status: NEW Reporter: Artur Kronenberg Assignee: Mario Fusco Attachments: test-standalone.zip, test-standalone.zip Hi, I have ran into this a number of times and I am still working on the reproducer. I decided to raise this issue in case someone has seen something similar and/or I am on the wrong track and I'd rather know now before spending even more days on chasing this down. We had a DB outage 2 nights ago, and I have a timer task inserting 1 object into my rules every minute. The first time this happened with the DB outage, the result was: {code:java} java.lang.RuntimeException: Unable to commit transaction at org.drools.persistence.jta.JtaTransactionManager.commit(JtaTransactionManager.java:239) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final] at org.drools.persistence.SingleSessionCommandService$TransactionInterceptor.execute(SingleSessionCommandService.java:587) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final] at org.drools.persistence.SingleSessionCommandService.execute(SingleSessionCommandService.java:377) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final] at org.drools.core.command.impl.CommandBasedStatefulKnowledgeSession.insert(CommandBasedStatefulKnowledgeSession.java:410) ~[drools-core-6.3.0.Final.jar:6.3.0.Final] at io.shureview.drools.model.impl.SessionHolder.insert(SessionHolder.java:62) ~[shureview-drools-engine-1.0.jar:na] at io.shureview.drools.tasks.ExpiryTimerTask.lambda$run$18(ExpiryTimerTask.java:60) ~[shureview-drools-engine-1.0.jar:na] at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[na:1.8.0_72] at io.shureview.drools.tasks.ExpiryTimerTask.run(ExpiryTimerTask.java:55) ~[shureview-drools-engine-1.0.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] Caused by: bitronix.tm.internal.BitronixRollbackException: RuntimeException thrown during beforeCompletion cycle caused transaction rollback at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:241) ~[btm-2.1.4.jar:2.1.4] at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:143) ~[btm-2.1.4.jar:2.1.4] at org.drools.persistence.jta.JtaTransactionManager.commit(JtaTransactionManager.java:236) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final] ... 14 common frames omitted Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not execute statement at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1316) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.ejb.AbstractEntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(AbstractEntityManagerImpl.java:1510) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:114) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:53) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at bitronix.tm.BitronixTransaction.fireBeforeCompletionEvent(BitronixTransaction.java:532) ~[btm-2.1.4.jar:2.1.4] at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:235) ~[btm-2.1.4.jar:2.1.4] ... 16 common frames omitted Caused by: org.hibernate.exception.JDBCConnectionException: could not execute statement at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:132) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:124) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:109) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:189) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3240) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:3138) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3468) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:140) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:395) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:387) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:304) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:349) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1195) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:404) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:109) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] ... 19 common frames omitted Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1884) ~[sqljdbc4-4.2.jar:na] at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1868) ~[sqljdbc4-4.2.jar:na] at com.microsoft.sqlserver.jdbc.TDSChannel.write(IOBuffer.java:1855) ~[sqljdbc4-4.2.jar:na] at com.microsoft.sqlserver.jdbc.TDSWriter.flush(IOBuffer.java:4215) ~[sqljdbc4-4.2.jar:na] at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(IOBuffer.java:4116) ~[sqljdbc4-4.2.jar:na] at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(IOBuffer.java:3122) ~[sqljdbc4-4.2.jar:na] at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:7127) ~[sqljdbc4-4.2.jar:na] at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:444) ~[sqljdbc4-4.2.jar:na] at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:383) ~[sqljdbc4-4.2.jar:na] at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:6703) ~[sqljdbc4-4.2.jar:na] at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1929) ~[sqljdbc4-4.2.jar:na] at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:184) ~[sqljdbc4-4.2.jar:na] at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:159) ~[sqljdbc4-4.2.jar:na] at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:326) ~[sqljdbc4-4.2.jar:na] at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:186) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final] ... 31 common frames omitted {code} This is then followed by: {code:java} java.lang.RuntimeException: Unable to load session snapshot at org.drools.persistence.SessionMarshallingHelper.loadSnapshot(SessionMarshallingHelper.java:103) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final] at org.drools.persistence.SingleSessionCommandService.initExistingKnowledgeSession(SingleSessionCommandService.java:242) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final] at org.drools.persistence.SingleSessionCommandService$TransactionInterceptor.execute(SingleSessionCommandService.java:557) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final] at org.drools.persistence.SingleSessionCommandService.execute(SingleSessionCommandService.java:377) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final] at org.drools.core.command.impl.CommandBasedStatefulKnowledgeSession.insert(CommandBasedStatefulKnowledgeSession.java:410) ~[drools-core-6.3.0.Final.jar:6.3.0.Final] at io.shureview.drools.model.impl.SessionHolder.insert(SessionHolder.java:62) ~[shureview-drools-engine-1.0.jar:na] at io.shureview.drools.tasks.ExpiryTimerTask.lambda$run$18(ExpiryTimerTask.java:60) ~[shureview-drools-engine-1.0.jar:na] at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[na:1.8.0_72] at io.shureview.drools.tasks.ExpiryTimerTask.run(ExpiryTimerTask.java:55) ~[shureview-drools-engine-1.0.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72] Caused by: java.lang.NullPointerException: null at org.drools.core.marshalling.impl.ProtobufInputMarshaller.readTruthMaintenanceSystem(ProtobufInputMarshaller.java:586) ~[drools-core-6.3.0.Final.jar:6.3.0.Final] at org.drools.core.marshalling.impl.ProtobufInputMarshaller.readSession(ProtobufInputMarshaller.java:247) ~[drools-core-6.3.0.Final.jar:6.3.0.Final] at org.drools.core.marshalling.impl.ProtobufInputMarshaller.readSession(ProtobufInputMarshaller.java:118) ~[drools-core-6.3.0.Final.jar:6.3.0.Final] at org.drools.core.marshalling.impl.ProtobufMarshaller.unmarshall(ProtobufMarshaller.java:131) ~[drools-core-6.3.0.Final.jar:6.3.0.Final] at org.drools.persistence.SessionMarshallingHelper.loadSnapshot(SessionMarshallingHelper.java:95) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final] ... 15 common frames omitted {code} My rules update and retract facts each time the rules are run. I wonder if somehow an object is removed from the working memory, then the persistence fails to update the table, the rollback does not rollback the working memory, and the persistence is broken. I have NOT found a solution for this so far. I was wondering though, is there a way for me to run a replace-all flush operation to simply serialise the working state of my session and throw away all other data. This could be a temporary workaround. Thanks and let me know if there is something else you need. I will try and add a reproducer as soon as I can. P.S.: Not sure if this may be Bitronix at fault here. I am running without a container, so I provided this standalone implementation for JTA. Thanks, Artur