[jboss-jira] [JBoss JIRA] (DROOLS-1093) Database Outage causes Session Persistence to break
Artur Kronenberg (JIRA)
issues at jboss.org
Wed Mar 16 12:22:00 EDT 2016
[ https://issues.jboss.org/browse/DROOLS-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Artur Kronenberg updated DROOLS-1093:
-------------------------------------
Attachment: test-standalone.zip
Standalone Eclipse project with gradle build file that helps to reproduce this issue.
> Database Outage causes Session Persistence to break
> ----------------------------------------------------
>
> Key: DROOLS-1093
> URL: https://issues.jboss.org/browse/DROOLS-1093
> Project: Drools
> Issue Type: Bug
> Components: core engine
> Affects Versions: 6.3.0.Final
> Environment: Mac OS 10.10.5, Eclipse Mars Release, Java 1.8, Drools 6.3.0.Final
> Docs QE Status: NEW
> Reporter: Artur Kronenberg
> Assignee: Mario Fusco
> Attachments: test-standalone.zip
>
>
> Hi,
> I have ran into this a number of times and I am still working on the reproducer. I decided to raise this issue in case someone has seen something similar and/or I am on the wrong track and I'd rather know now before spending even more days on chasing this down.
> We had a DB outage 2 nights ago, and I have a timer task inserting 1 object into my rules every minute. The first time this happened with the DB outage, the result was:
> {code:java}
> java.lang.RuntimeException: Unable to commit transaction
> at org.drools.persistence.jta.JtaTransactionManager.commit(JtaTransactionManager.java:239) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.SingleSessionCommandService$TransactionInterceptor.execute(SingleSessionCommandService.java:587) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.SingleSessionCommandService.execute(SingleSessionCommandService.java:377) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.core.command.impl.CommandBasedStatefulKnowledgeSession.insert(CommandBasedStatefulKnowledgeSession.java:410) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
> at io.shureview.drools.model.impl.SessionHolder.insert(SessionHolder.java:62) ~[shureview-drools-engine-1.0.jar:na]
> at io.shureview.drools.tasks.ExpiryTimerTask.lambda$run$18(ExpiryTimerTask.java:60) ~[shureview-drools-engine-1.0.jar:na]
> at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[na:1.8.0_72]
> at io.shureview.drools.tasks.ExpiryTimerTask.run(ExpiryTimerTask.java:55) ~[shureview-drools-engine-1.0.jar:na]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
> Caused by: bitronix.tm.internal.BitronixRollbackException: RuntimeException thrown during beforeCompletion cycle caused transaction rollback
> at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:241) ~[btm-2.1.4.jar:2.1.4]
> at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:143) ~[btm-2.1.4.jar:2.1.4]
> at org.drools.persistence.jta.JtaTransactionManager.commit(JtaTransactionManager.java:236) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> ... 14 common frames omitted
> Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not execute statement
> at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1316) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.ejb.AbstractEntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(AbstractEntityManagerImpl.java:1510) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:114) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:53) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at bitronix.tm.BitronixTransaction.fireBeforeCompletionEvent(BitronixTransaction.java:532) ~[btm-2.1.4.jar:2.1.4]
> at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:235) ~[btm-2.1.4.jar:2.1.4]
> ... 16 common frames omitted
> Caused by: org.hibernate.exception.JDBCConnectionException: could not execute statement
> at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:132) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:124) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:109) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:189) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3240) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:3138) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3468) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:140) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:395) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:387) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:304) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:349) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1195) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:404) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:109) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> ... 19 common frames omitted
> Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1884) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1868) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.TDSChannel.write(IOBuffer.java:1855) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.TDSWriter.flush(IOBuffer.java:4215) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(IOBuffer.java:4116) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(IOBuffer.java:3122) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:7127) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:444) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:383) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:6703) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1929) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:184) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:159) ~[sqljdbc4-4.2.jar:na]
> at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:326) ~[sqljdbc4-4.2.jar:na]
> at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:186) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
> ... 31 common frames omitted
> {code}
> This is then followed by:
> {code:java}
> java.lang.RuntimeException: Unable to load session snapshot
> at org.drools.persistence.SessionMarshallingHelper.loadSnapshot(SessionMarshallingHelper.java:103) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.SingleSessionCommandService.initExistingKnowledgeSession(SingleSessionCommandService.java:242) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.SingleSessionCommandService$TransactionInterceptor.execute(SingleSessionCommandService.java:557) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.SingleSessionCommandService.execute(SingleSessionCommandService.java:377) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.core.command.impl.CommandBasedStatefulKnowledgeSession.insert(CommandBasedStatefulKnowledgeSession.java:410) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
> at io.shureview.drools.model.impl.SessionHolder.insert(SessionHolder.java:62) ~[shureview-drools-engine-1.0.jar:na]
> at io.shureview.drools.tasks.ExpiryTimerTask.lambda$run$18(ExpiryTimerTask.java:60) ~[shureview-drools-engine-1.0.jar:na]
> at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[na:1.8.0_72]
> at io.shureview.drools.tasks.ExpiryTimerTask.run(ExpiryTimerTask.java:55) ~[shureview-drools-engine-1.0.jar:na]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
> Caused by: java.lang.NullPointerException: null
> at org.drools.core.marshalling.impl.ProtobufInputMarshaller.readTruthMaintenanceSystem(ProtobufInputMarshaller.java:586) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.core.marshalling.impl.ProtobufInputMarshaller.readSession(ProtobufInputMarshaller.java:247) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.core.marshalling.impl.ProtobufInputMarshaller.readSession(ProtobufInputMarshaller.java:118) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.core.marshalling.impl.ProtobufMarshaller.unmarshall(ProtobufMarshaller.java:131) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
> at org.drools.persistence.SessionMarshallingHelper.loadSnapshot(SessionMarshallingHelper.java:95) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
> ... 15 common frames omitted
> {code}
> My rules update and retract facts each time the rules are run.
> I wonder if somehow an object is removed from the working memory, then the persistence fails to update the table, the rollback does not rollback the working memory, and the persistence is broken.
> I have NOT found a solution for this so far. I was wondering though, is there a way for me to run a replace-all flush operation to simply serialise the working state of my session and throw away all other data. This could be a temporary workaround.
> Thanks and let me know if there is something else you need. I will try and add a reproducer as soon as I can.
> P.S.: Not sure if this may be Bitronix at fault here. I am running without a container, so I provided this standalone implementation for JTA.
> Thanks,
> Artur
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
More information about the jboss-jira
mailing list