[jboss-jira] [JBoss JIRA] (DROOLS-1093) Database Outage causes Session Persistence to break
Artur Kronenberg (JIRA)
issues at jboss.org
Thu Mar 17 06:33:00 EDT 2016
[ https://issues.jboss.org/browse/DROOLS-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Artur Kronenberg updated DROOLS-1093:
-------------------------------------
Attachment: test-standalone.zip
I played around some more and managed to reproduce this bug in a unit test. This might be easier to debug/work with than the mess I produced in the previous step.
Note: The test currently passes if it throws the necessary NPE at the very end.
Class: ExceptionTest
I created a Datasource that can be configured to throw SQL exceptions. As before, the setup requires MySql to be running. There are methods for configuring the setup.
Let me know if there's questions
> Database Outage causes Session Persistence to break
> ----------------------------------------------------
>
> Key: DROOLS-1093
> URL: https://issues.jboss.org/browse/DROOLS-1093
> Project: Drools
> Issue Type: Bug
> Components: core engine
> Affects Versions: 6.3.0.Final
> Environment: Mac OS 10.10.5, Eclipse Mars Release, Java 1.8, Drools 6.3.0.Final
> Docs QE Status: NEW
> Reporter: Artur Kronenberg
> Assignee: Mario Fusco
> Attachments: test-standalone.zip, test-standalone.zip
>
>
> Hi,
> I have ran into this a number of times and I am still working on the reproducer. I decided to raise this issue in case someone has seen something similar and/or I am on the wrong track and I'd rather know now before spending even more days on chasing this down.
> We had a DB outage 2 nights ago, and I have a timer task inserting 1 object into my rules every minute. The first time this happened with the DB outage, the result was:
> {code:java}
> java.lang.RuntimeException: Unable to commit transaction
> at org.drools.persistence.jta.JtaTransactionManager.commit(JtaTransactionManager.java:239) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.SingleSessionCommandService$TransactionInterceptor.execute(SingleSessionCommandService.java:587) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.SingleSessionCommandService.execute(SingleSessionCommandService.java:377) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.core.command.impl.CommandBasedStatefulKnowledgeSession.insert(CommandBasedStatefulKnowledgeSession.java:410) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
> at io.shureview.drools.model.impl.SessionHolder.insert(SessionHolder.java:62) ~[shureview-drools-engine-1.0.jar:na]
> at io.shureview.drools.tasks.ExpiryTimerTask.lambda$run$18(ExpiryTimerTask.java:60) ~[shureview-drools-engine-1.0.jar:na]
> at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[na:1.8.0_72]
> at io.shureview.drools.tasks.ExpiryTimerTask.run(ExpiryTimerTask.java:55) ~[shureview-drools-engine-1.0.jar:na]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
> Caused by: bitronix.tm.internal.BitronixRollbackException: RuntimeException thrown during beforeCompletion cycle caused transaction rollback
> at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:241) ~[btm-2.1.4.jar:2.1.4]
> at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:143) ~[btm-2.1.4.jar:2.1.4]
> at org.drools.persistence.jta.JtaTransactionManager.commit(JtaTransactionManager.java:236) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> ... 14 common frames omitted
> Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not execute statement
> at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1316) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.ejb.AbstractEntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(AbstractEntityManagerImpl.java:1510) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:114) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:53) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at bitronix.tm.BitronixTransaction.fireBeforeCompletionEvent(BitronixTransaction.java:532) ~[btm-2.1.4.jar:2.1.4]
> at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:235) ~[btm-2.1.4.jar:2.1.4]
> ... 16 common frames omitted
> Caused by: org.hibernate.exception.JDBCConnectionException: could not execute statement
> at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:132) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:124) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:109) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:189) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3240) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:3138) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3468) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:140) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:395) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:387) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:304) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:349) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1195) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:404) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:109) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> ... 19 common frames omitted
> Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1884) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1868) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.TDSChannel.write(IOBuffer.java:1855) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.TDSWriter.flush(IOBuffer.java:4215) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(IOBuffer.java:4116) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(IOBuffer.java:3122) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:7127) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:444) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:383) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:6703) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1929) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:184) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:159) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:326) ~[sqljdbc4-4.2.jar:na]
> at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:186) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> ... 31 common frames omitted
> {code}
> This is then followed by:
> {code:java}
> java.lang.RuntimeException: Unable to load session snapshot
> at org.drools.persistence.SessionMarshallingHelper.loadSnapshot(SessionMarshallingHelper.java:103) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.SingleSessionCommandService.initExistingKnowledgeSession(SingleSessionCommandService.java:242) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.SingleSessionCommandService$TransactionInterceptor.execute(SingleSessionCommandService.java:557) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.SingleSessionCommandService.execute(SingleSessionCommandService.java:377) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.core.command.impl.CommandBasedStatefulKnowledgeSession.insert(CommandBasedStatefulKnowledgeSession.java:410) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
> at io.shureview.drools.model.impl.SessionHolder.insert(SessionHolder.java:62) ~[shureview-drools-engine-1.0.jar:na]
> at io.shureview.drools.tasks.ExpiryTimerTask.lambda$run$18(ExpiryTimerTask.java:60) ~[shureview-drools-engine-1.0.jar:na]
> at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[na:1.8.0_72]
> at io.shureview.drools.tasks.ExpiryTimerTask.run(ExpiryTimerTask.java:55) ~[shureview-drools-engine-1.0.jar:na]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
> Caused by: java.lang.NullPointerException: null
> at org.drools.core.marshalling.impl.ProtobufInputMarshaller.readTruthMaintenanceSystem(ProtobufInputMarshaller.java:586) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.core.marshalling.impl.ProtobufInputMarshaller.readSession(ProtobufInputMarshaller.java:247) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.core.marshalling.impl.ProtobufInputMarshaller.readSession(ProtobufInputMarshaller.java:118) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.core.marshalling.impl.ProtobufMarshaller.unmarshall(ProtobufMarshaller.java:131) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.SessionMarshallingHelper.loadSnapshot(SessionMarshallingHelper.java:95) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> ... 15 common frames omitted
> {code}
> My rules update and retract facts each time the rules are run.
> I wonder if somehow an object is removed from the working memory, then the persistence fails to update the table, the rollback does not rollback the working memory, and the persistence is broken.
> I have NOT found a solution for this so far. I was wondering though, is there a way for me to run a replace-all flush operation to simply serialise the working state of my session and throw away all other data. This could be a temporary workaround.
> Thanks and let me know if there is something else you need. I will try and add a reproducer as soon as I can.
> P.S.: Not sure if this may be Bitronix at fault here. I am running without a container, so I provided this standalone implementation for JTA.
> Thanks,
> Artur
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
More information about the jboss-jira
mailing list