[JBoss JIRA] (JBTM-2741) Losing message during inflow transaction processing
by Tom Jenkinson (JIRA)
[ https://issues.jboss.org/browse/JBTM-2741?page=com.atlassian.jira.plugin.... ]
Tom Jenkinson commented on JBTM-2741:
-------------------------------------
Please can you attach a log? What makes you think Narayana was involved? You look to be sending messages directly to your RAR?
> Losing message during inflow transaction processing
> ---------------------------------------------------
>
> Key: JBTM-2741
> URL: https://issues.jboss.org/browse/JBTM-2741
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: JCA, JTA, JTS
> Affects Versions: 5.3.3.Final
> Reporter: Ondra Chaloupka
> Assignee: Tom Jenkinson
>
> I experience losing messages when they are received with the same xid when messages are received in parallel. This means case that prior message is not yet fully processed when meanwhile new message is promoted for being processed.
> This is the scenario which behaves wrong by my view
> * EIS passes a message with xid1 to RAR to be processed
> * first message is passed as work to be process (stays in progress)
> * EIS passes a second message with xid1 to RAR to be processed
> * the second message is forgotten. It will never reach a {{MessageListner}}
> ** no error is returned or shown in log
> Compared following scenario passes without a problem.
> * EIS passes a message with xid1 to RAR to be processed
> * first message is fully processed with {{MessageListner}} (it reaches the end of the _onMessage_ method)
> * EIS passes a second message with xid1 to RAR to be processed
> * second message is processed by {{MessageListener}}
> By reading jca spec and description of JBTM-2164 I do understand that several messages could be passed with the same xid in parrarel. If my interpretation or scenario setup is wrong, please, let me know.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (JBTM-2741) Losing message during inflow transaction processing
by Tom Jenkinson (JIRA)
[ https://issues.jboss.org/browse/JBTM-2741?page=com.atlassian.jira.plugin.... ]
Tom Jenkinson updated JBTM-2741:
--------------------------------
Component/s: JCA
> Losing message during inflow transaction processing
> ---------------------------------------------------
>
> Key: JBTM-2741
> URL: https://issues.jboss.org/browse/JBTM-2741
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: JCA, JTA, JTS
> Affects Versions: 5.3.3.Final
> Reporter: Ondra Chaloupka
> Assignee: Tom Jenkinson
>
> I experience losing messages when they are received with the same xid when messages are received in parallel. This means case that prior message is not yet fully processed when meanwhile new message is promoted for being processed.
> This is the scenario which behaves wrong by my view
> * EIS passes a message with xid1 to RAR to be processed
> * first message is passed as work to be process (stays in progress)
> * EIS passes a second message with xid1 to RAR to be processed
> * the second message is forgotten. It will never reach a {{MessageListner}}
> ** no error is returned or shown in log
> Compared following scenario passes without a problem.
> * EIS passes a message with xid1 to RAR to be processed
> * first message is fully processed with {{MessageListner}} (it reaches the end of the _onMessage_ method)
> * EIS passes a second message with xid1 to RAR to be processed
> * second message is processed by {{MessageListener}}
> By reading jca spec and description of JBTM-2164 I do understand that several messages could be passed with the same xid in parrarel. If my interpretation or scenario setup is wrong, please, let me know.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (JBTM-2741) Losing message during inflow transaction processing
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-2741?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka updated JBTM-2741:
----------------------------------
Component/s: JTA
JTS
> Losing message during inflow transaction processing
> ---------------------------------------------------
>
> Key: JBTM-2741
> URL: https://issues.jboss.org/browse/JBTM-2741
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: JTA, JTS
> Affects Versions: 5.3.3.Final
> Reporter: Ondra Chaloupka
> Assignee: Tom Jenkinson
>
> I experience losing messages when they are received with the same xid when messages are received in parallel. This means case that prior message is not yet fully processed when meanwhile new message is promoted for being processed.
> This is the scenario which behaves wrong by my view
> * EIS passes a message with xid1 to RAR to be processed
> * first message is passed as work to be process (stays in progress)
> * EIS passes a second message with xid1 to RAR to be processed
> * the second message is forgotten. It will never reach a {{MessageListner}}
> ** no error is returned or shown in log
> Compared following scenario passes without a problem.
> * EIS passes a message with xid1 to RAR to be processed
> * first message is fully processed with {{MessageListner}} (it reaches the end of the _onMessage_ method)
> * EIS passes a second message with xid1 to RAR to be processed
> * second message is processed by {{MessageListener}}
> By reading jca spec and description of JBTM-2164 I do understand that several messages could be passed with the same xid in parrarel. If my interpretation or scenario setup is wrong, please, let me know.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (JBTM-2741) Losing message during inflow transaction processing
by Ondra Chaloupka (JIRA)
Ondra Chaloupka created JBTM-2741:
-------------------------------------
Summary: Losing message during inflow transaction processing
Key: JBTM-2741
URL: https://issues.jboss.org/browse/JBTM-2741
Project: JBoss Transaction Manager
Issue Type: Bug
Reporter: Ondra Chaloupka
I experience losing messages when they are received with the same xid when messages are received in parallel. This means case that prior message is not yet fully processed when meanwhile new message is promoted for being processed.
This is the scenario which behaves wrong by my view
* EIS passes a message with xid1 to RAR to be processed
* first message is passed as work to be process (stays in progress)
* EIS passes a second message with xid1 to RAR to be processed
* the second message is forgotten. It will never reach a {{MessageListner}}
** no error is returned or shown in log
Compared following scenario passes without a problem.
* EIS passes a message with xid1 to RAR to be processed
* first message is fully processed with {{MessageListner}} (it reaches the end of the _onMessage_ method)
* EIS passes a second message with xid1 to RAR to be processed
* second message is processed by {{MessageListener}}
By reading jca spec and description of JBTM-2164 I do understand that several messages could be passed with the same xid in parrarel. If my interpretation or scenario setup is wrong, please, let me know.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (JBTM-2741) Losing message during inflow transaction processing
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-2741?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka reassigned JBTM-2741:
-------------------------------------
Assignee: Tom Jenkinson
> Losing message during inflow transaction processing
> ---------------------------------------------------
>
> Key: JBTM-2741
> URL: https://issues.jboss.org/browse/JBTM-2741
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Affects Versions: 5.3.3.Final
> Reporter: Ondra Chaloupka
> Assignee: Tom Jenkinson
>
> I experience losing messages when they are received with the same xid when messages are received in parallel. This means case that prior message is not yet fully processed when meanwhile new message is promoted for being processed.
> This is the scenario which behaves wrong by my view
> * EIS passes a message with xid1 to RAR to be processed
> * first message is passed as work to be process (stays in progress)
> * EIS passes a second message with xid1 to RAR to be processed
> * the second message is forgotten. It will never reach a {{MessageListner}}
> ** no error is returned or shown in log
> Compared following scenario passes without a problem.
> * EIS passes a message with xid1 to RAR to be processed
> * first message is fully processed with {{MessageListner}} (it reaches the end of the _onMessage_ method)
> * EIS passes a second message with xid1 to RAR to be processed
> * second message is processed by {{MessageListener}}
> By reading jca spec and description of JBTM-2164 I do understand that several messages could be passed with the same xid in parrarel. If my interpretation or scenario setup is wrong, please, let me know.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (JBTM-2741) Losing message during inflow transaction processing
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-2741?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka updated JBTM-2741:
----------------------------------
Affects Version/s: 5.3.3.Final
> Losing message during inflow transaction processing
> ---------------------------------------------------
>
> Key: JBTM-2741
> URL: https://issues.jboss.org/browse/JBTM-2741
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Affects Versions: 5.3.3.Final
> Reporter: Ondra Chaloupka
> Assignee: Tom Jenkinson
>
> I experience losing messages when they are received with the same xid when messages are received in parallel. This means case that prior message is not yet fully processed when meanwhile new message is promoted for being processed.
> This is the scenario which behaves wrong by my view
> * EIS passes a message with xid1 to RAR to be processed
> * first message is passed as work to be process (stays in progress)
> * EIS passes a second message with xid1 to RAR to be processed
> * the second message is forgotten. It will never reach a {{MessageListner}}
> ** no error is returned or shown in log
> Compared following scenario passes without a problem.
> * EIS passes a message with xid1 to RAR to be processed
> * first message is fully processed with {{MessageListner}} (it reaches the end of the _onMessage_ method)
> * EIS passes a second message with xid1 to RAR to be processed
> * second message is processed by {{MessageListener}}
> By reading jca spec and description of JBTM-2164 I do understand that several messages could be passed with the same xid in parrarel. If my interpretation or scenario setup is wrong, please, let me know.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (JBTM-2734) EIS can't recover transaction when heuristic outcome happens
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-2734?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka updated JBTM-2734:
----------------------------------
Steps to Reproduce:
Crashrecovery testsuite could be used for reproducing the issue
{code}
git clone http://git.app.eng.bos.redhat.com/git/jbossqe-eap-tests-transactions.git
mvn clean verify -am -pl jbossts -DfailIfNoTests=false -fn -Djbossts.noJTS -Dtest=JcaInflowTransactionTestCase#rmerrWithRecovery
{code}
was:
Crashrecovery testsuite could be used for reproducing the issue
mvn clean verify -am -pl jbossts -DfailIfNoTests=false -fn -Djbossts.noJTS -Dtest=JcaInflowTransactionTestCase#rmerrWithRecovery
> EIS can't recover transaction when heuristic outcome happens
> ------------------------------------------------------------
>
> Key: JBTM-2734
> URL: https://issues.jboss.org/browse/JBTM-2734
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: JTA
> Affects Versions: 5.3.3.Final
> Reporter: Ondra Chaloupka
> Assignee: Tom Jenkinson
> Priority: Critical
> Fix For: 5.next
>
>
> For "a normal" transaction which is started and managed by transaction manager heuristic state is expected to be solved this way
> # prepare phase successfully finishes, participant make a heuristic decision (ie. rollbacks) which causes that TM receives a XAException with error code as e.g. XA_RMERR or XA_HEURRB.
> # transaction is marked to have heuristic result and manual intervention is necessary
> # user calls {{recover}} for participant at such state and its state is changed to _prepared_
> # replay of commit is called for the participant which causes to get commit called once again for the {{XAResource}} and participant record is removed from TM object store
> This workflow does not work for inflow transaction where EIS is expected to order the TM to commit (as TM stands in position of subordinate transaction manager). The expected procedure should be
> # EIS propagates EIS TX to Narayana
> # enlisting XAR in Narayana subordinate TX
> # committing EIS TX
> ## It commits Narayana subordinate TX
> # XAR in Narayana TX returns RMERR
> # Narayana returns XA_HEURMIX
> # calling the Narayana tooling op {{recover}} to move the heuristic back to prepared
> # EIS should not forget its TX where Narayana TX is a subordinate as we returned a heursitic so its not allowed to forget it yet
> # EIS should call Narayana via XATerminator::recover()
> ## getting back an Xid that matches to Narayana subordinate TX
> # EIS should try again to complete this Xid it sees fit according to its own recovery log, e.g. e.g. call XATerminator::commit(xid)
> (_in the real world once an XAR had rolled back it would be unlikely to ever be able to commit again so it would be a bit synthetic to allow it commit so it would be up to you how you decide to complete the TX_)
> The issue is that {{recover}} operation returns the state of participant to _prepared_ but EIS commit call (({{XATerminator.commit}}) does cause {{XAResource.commit}} being invoked and participant log being removed from the TM object store.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (JBTM-2735) EIS can't finish all participants of inflow in-doubt transaction after jvm crash
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-2735?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka updated JBTM-2735:
----------------------------------
Steps to Reproduce:
Crashrecovery testsuite could be used for reproducing the issue
{code}
git clone http://git.app.eng.bos.redhat.com/git/jbossqe-eap-tests-transactions.git
mvn clean verify -am -pl jbossts -DfailIfNoTests=false -fn -Djbossts.noJTS -Dtest=JcaInflowTransactionTestCase#rmerrWithRecovery#jvmCrashAfterPrepare
{code}
was:
Crashrecovery testsuite could be used for reproducing the issue
{code}
mvn clean verify -am -pl jbossts -DfailIfNoTests=false -fn -Djbossts.noJTS -Dtest=JcaInflowTransactionTestCase#rmerrWithRecovery#jvmCrashAfterPrepare
{code}
> EIS can't finish all participants of inflow in-doubt transaction after jvm crash
> --------------------------------------------------------------------------------
>
> Key: JBTM-2735
> URL: https://issues.jboss.org/browse/JBTM-2735
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: JTA, JTS
> Affects Versions: 5.3.3.Final
> Reporter: Ondra Chaloupka
> Assignee: Tom Jenkinson
> Priority: Critical
> Fix For: 5.next
>
>
> When EIS RAR manages inflow transaction through {{XATerminator}} calls and jvm of TM is crashed at the start of commit phase then EIS can't recover all transaction participants.
> Expected flow of the case would be
> # enlist two participants of inflow transaction - that means TM manages subordinate transaction and accepts orders from EIS RAR
> # TM receives commit order by EIS RAR call {{XATerminator.commit}}
> # TM prepares both participants
> # end phase prepares, start phase commits
> # JVM crashes
> # app server is restarted again
> # EIS system repeats commit call as subordinate txn was not finished
> # TM calls commit on both participants to finish the transaction
> This scenario has two troubles in current implementation
> If app server is restarted (after jvm crash happened) and EIS calls commit right after the start there is thrown {{XAER_RMFAIL}} when XATerminator.commit is called.
> For the commit have got a real effect on in-doubt transaciton there needs to be called recovery first. From that point in time the in-doubt inflow transaction is "available" for EIS RAR XATerminator.commit call. But there is another issue. Transaction was stopped with two participants prepared. But after commit is called from XATerminator only one XAResource is committed. The other one is left and forgotten. As the transaction itself is resolved as finished (thus removed from log store) there is no way to get to the remaining in-doubt participant - at least with help of jboss cli tooling.
> (_The reason could be that both XA resources shared the xid, as discussed in other place, TM does not rights to change xid and it needs to reuse existing one provided by EIS to all the work_)
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (JBTM-2735) EIS can't finish all participants of inflow in-doubt transaction after jvm crash
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-2735?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka commented on JBTM-2735:
---------------------------------------
The reproducer is under master branch of repo {{http://git.app.eng.bos.redhat.com/git/jbossqe-eap-tests-transactions.git}}.
> EIS can't finish all participants of inflow in-doubt transaction after jvm crash
> --------------------------------------------------------------------------------
>
> Key: JBTM-2735
> URL: https://issues.jboss.org/browse/JBTM-2735
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: JTA, JTS
> Affects Versions: 5.3.3.Final
> Reporter: Ondra Chaloupka
> Assignee: Tom Jenkinson
> Priority: Critical
> Fix For: 5.next
>
>
> When EIS RAR manages inflow transaction through {{XATerminator}} calls and jvm of TM is crashed at the start of commit phase then EIS can't recover all transaction participants.
> Expected flow of the case would be
> # enlist two participants of inflow transaction - that means TM manages subordinate transaction and accepts orders from EIS RAR
> # TM receives commit order by EIS RAR call {{XATerminator.commit}}
> # TM prepares both participants
> # end phase prepares, start phase commits
> # JVM crashes
> # app server is restarted again
> # EIS system repeats commit call as subordinate txn was not finished
> # TM calls commit on both participants to finish the transaction
> This scenario has two troubles in current implementation
> If app server is restarted (after jvm crash happened) and EIS calls commit right after the start there is thrown {{XAER_RMFAIL}} when XATerminator.commit is called.
> For the commit have got a real effect on in-doubt transaciton there needs to be called recovery first. From that point in time the in-doubt inflow transaction is "available" for EIS RAR XATerminator.commit call. But there is another issue. Transaction was stopped with two participants prepared. But after commit is called from XATerminator only one XAResource is committed. The other one is left and forgotten. As the transaction itself is resolved as finished (thus removed from log store) there is no way to get to the remaining in-doubt participant - at least with help of jboss cli tooling.
> (_The reason could be that both XA resources shared the xid, as discussed in other place, TM does not rights to change xid and it needs to reuse existing one provided by EIS to all the work_)
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (JBTM-2734) EIS can't recover transaction when heuristic outcome happens
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-2734?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka commented on JBTM-2734:
---------------------------------------
The reproducer is under master branch of repo {{http://git.app.eng.bos.redhat.com/git/jbossqe-eap-tests-transactions.git}}.
> EIS can't recover transaction when heuristic outcome happens
> ------------------------------------------------------------
>
> Key: JBTM-2734
> URL: https://issues.jboss.org/browse/JBTM-2734
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: JTA
> Affects Versions: 5.3.3.Final
> Reporter: Ondra Chaloupka
> Assignee: Tom Jenkinson
> Priority: Critical
> Fix For: 5.next
>
>
> For "a normal" transaction which is started and managed by transaction manager heuristic state is expected to be solved this way
> # prepare phase successfully finishes, participant make a heuristic decision (ie. rollbacks) which causes that TM receives a XAException with error code as e.g. XA_RMERR or XA_HEURRB.
> # transaction is marked to have heuristic result and manual intervention is necessary
> # user calls {{recover}} for participant at such state and its state is changed to _prepared_
> # replay of commit is called for the participant which causes to get commit called once again for the {{XAResource}} and participant record is removed from TM object store
> This workflow does not work for inflow transaction where EIS is expected to order the TM to commit (as TM stands in position of subordinate transaction manager). The expected procedure should be
> # EIS propagates EIS TX to Narayana
> # enlisting XAR in Narayana subordinate TX
> # committing EIS TX
> ## It commits Narayana subordinate TX
> # XAR in Narayana TX returns RMERR
> # Narayana returns XA_HEURMIX
> # calling the Narayana tooling op {{recover}} to move the heuristic back to prepared
> # EIS should not forget its TX where Narayana TX is a subordinate as we returned a heursitic so its not allowed to forget it yet
> # EIS should call Narayana via XATerminator::recover()
> ## getting back an Xid that matches to Narayana subordinate TX
> # EIS should try again to complete this Xid it sees fit according to its own recovery log, e.g. e.g. call XATerminator::commit(xid)
> (_in the real world once an XAR had rolled back it would be unlikely to ever be able to commit again so it would be a bit synthetic to allow it commit so it would be up to you how you decide to complete the TX_)
> The issue is that {{recover}} operation returns the state of participant to _prepared_ but EIS commit call (({{XATerminator.commit}}) does cause {{XAResource.commit}} being invoked and participant log being removed from the TM object store.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months