[jboss-jira] [JBoss JIRA] (DROOLS-1093) Database Outage causes Session Persistence to break
Artur Kronenberg (JIRA)
issues at jboss.org
Wed Mar 16 07:20:00 EDT 2016
Artur Kronenberg created DROOLS-1093:
----------------------------------------
Summary: Database Outage causes Session Persistence to break
Key: DROOLS-1093
URL: https://issues.jboss.org/browse/DROOLS-1093
Project: Drools
Issue Type: Bug
Components: core engine
Affects Versions: 6.3.0.Final
Environment: Mac OS 10.10.5, Eclipse Mars Release, Java 1.8, Drools 6.3.0.Final
Docs QE Status: NEW
Reporter: Artur Kronenberg
Assignee: Mario Fusco
Hi,
I have ran into this a number of times and I am still working on the reproducer. I decided to raise this issue in case someone has seen something similar and/or I am on the wrong track and I'd rather know now before spending even more days on chasing this down.
We had a DB outage 2 nights ago, and I have a timer task inserting 1 object into my rules every minute. The first time this happened with the DB outage, the result was:
{code:java}
java.lang.RuntimeException: Unable to commit transaction
at org.drools.persistence.jta.JtaTransactionManager.commit(JtaTransactionManager.java:239) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
at org.drools.persistence.SingleSessionCommandService$TransactionInterceptor.execute(SingleSessionCommandService.java:587) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
at org.drools.persistence.SingleSessionCommandService.execute(SingleSessionCommandService.java:377) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
at org.drools.core.command.impl.CommandBasedStatefulKnowledgeSession.insert(CommandBasedStatefulKnowledgeSession.java:410) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
at io.shureview.drools.model.impl.SessionHolder.insert(SessionHolder.java:62) ~[shureview-drools-engine-1.0.jar:na]
at io.shureview.drools.tasks.ExpiryTimerTask.lambda$run$18(ExpiryTimerTask.java:60) ~[shureview-drools-engine-1.0.jar:na]
at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[na:1.8.0_72]
at io.shureview.drools.tasks.ExpiryTimerTask.run(ExpiryTimerTask.java:55) ~[shureview-drools-engine-1.0.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: bitronix.tm.internal.BitronixRollbackException: RuntimeException thrown during beforeCompletion cycle caused transaction rollback
at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:241) ~[btm-2.1.4.jar:2.1.4]
at bitronix.tm.BitronixTransactionManager.commit(BitronixTransactionManager.java:143) ~[btm-2.1.4.jar:2.1.4]
at org.drools.persistence.jta.JtaTransactionManager.commit(JtaTransactionManager.java:236) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
... 14 common frames omitted
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: could not execute statement
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1316) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.ejb.AbstractEntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(AbstractEntityManagerImpl.java:1510) ~[hibernate-entitymanager-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:114) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:53) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at bitronix.tm.BitronixTransaction.fireBeforeCompletionEvent(BitronixTransaction.java:532) ~[btm-2.1.4.jar:2.1.4]
at bitronix.tm.BitronixTransaction.commit(BitronixTransaction.java:235) ~[btm-2.1.4.jar:2.1.4]
... 16 common frames omitted
Caused by: org.hibernate.exception.JDBCConnectionException: could not execute statement
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:132) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:124) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:109) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:189) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3240) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:3138) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3468) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:140) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:395) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:387) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:304) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:349) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1195) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:404) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:109) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
... 19 common frames omitted
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1884) ~[sqljdbc4-4.2.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:1868) ~[sqljdbc4-4.2.jar:na]
at com.microsoft.sqlserver.jdbc.TDSChannel.write(IOBuffer.java:1855) ~[sqljdbc4-4.2.jar:na]
at com.microsoft.sqlserver.jdbc.TDSWriter.flush(IOBuffer.java:4215) ~[sqljdbc4-4.2.jar:na]
at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(IOBuffer.java:4116) ~[sqljdbc4-4.2.jar:na]
at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(IOBuffer.java:3122) ~[sqljdbc4-4.2.jar:na]
at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:7127) ~[sqljdbc4-4.2.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:444) ~[sqljdbc4-4.2.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:383) ~[sqljdbc4-4.2.jar:na]
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:6703) ~[sqljdbc4-4.2.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1929) ~[sqljdbc4-4.2.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:184) ~[sqljdbc4-4.2.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:159) ~[sqljdbc4-4.2.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:326) ~[sqljdbc4-4.2.jar:na]
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:186) ~[hibernate-core-4.2.21.Final.jar:4.2.21.Final]
... 31 common frames omitted
{code}
This is then followed by:
{code:java}
java.lang.RuntimeException: Unable to load session snapshot
at org.drools.persistence.SessionMarshallingHelper.loadSnapshot(SessionMarshallingHelper.java:103) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
at org.drools.persistence.SingleSessionCommandService.initExistingKnowledgeSession(SingleSessionCommandService.java:242) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
at org.drools.persistence.SingleSessionCommandService$TransactionInterceptor.execute(SingleSessionCommandService.java:557) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
at org.drools.persistence.SingleSessionCommandService.execute(SingleSessionCommandService.java:377) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
at org.drools.core.command.impl.CommandBasedStatefulKnowledgeSession.insert(CommandBasedStatefulKnowledgeSession.java:410) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
at io.shureview.drools.model.impl.SessionHolder.insert(SessionHolder.java:62) ~[shureview-drools-engine-1.0.jar:na]
at io.shureview.drools.tasks.ExpiryTimerTask.lambda$run$18(ExpiryTimerTask.java:60) ~[shureview-drools-engine-1.0.jar:na]
at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[na:1.8.0_72]
at io.shureview.drools.tasks.ExpiryTimerTask.run(ExpiryTimerTask.java:55) ~[shureview-drools-engine-1.0.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.NullPointerException: null
at org.drools.core.marshalling.impl.ProtobufInputMarshaller.readTruthMaintenanceSystem(ProtobufInputMarshaller.java:586) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
at org.drools.core.marshalling.impl.ProtobufInputMarshaller.readSession(ProtobufInputMarshaller.java:247) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
at org.drools.core.marshalling.impl.ProtobufInputMarshaller.readSession(ProtobufInputMarshaller.java:118) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
at org.drools.core.marshalling.impl.ProtobufMarshaller.unmarshall(ProtobufMarshaller.java:131) ~[drools-core-6.3.0.Final.jar:6.3.0.Final]
at org.drools.persistence.SessionMarshallingHelper.loadSnapshot(SessionMarshallingHelper.java:95) ~[drools-persistence-jpa-6.3.0.Final.jar:6.3.0.Final]
... 15 common frames omitted
{code}
My rules update and retract facts each time the rules are run.
I wonder if somehow an object is removed from the working memory, then the persistence fails to update the table, the rollback does not rollback the working memory, and the persistence is broken.
I have NOT found a solution for this so far. I was wondering though, is there a way for me to run a replace-all flush operation to simply serialise the working state of my session and throw away all other data. This could be a temporary workaround.
Thanks and let me know if there is something else you need. I will try and add a reproducer as soon as I can.
P.S.: Not sure if this may be Bitronix at fault here. I am running without a container, so I provided this standalone implementation for JTA.
Thanks,
Artur
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
More information about the jboss-jira
mailing list