[JBoss JIRA] (JBTM-2920) Investigate the use of transactions within a reactive microservices environment
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-2920?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka resolved JBTM-2920.
-----------------------------------
Resolution: Done
Link to the thesis: [http://liver.englab.brq.redhat.com/redirect/?url=https://is.muni.cz/th/gv...]
The thesis was successfully defended and awarded with the dean award.
> Investigate the use of transactions within a reactive microservices environment
> -------------------------------------------------------------------------------
>
> Key: JBTM-2920
> URL: https://issues.jboss.org/browse/JBTM-2920
> Project: JBoss Transaction Manager
> Issue Type: Task
> Environment: *
> Reporter: Ondra Chaloupka
> Assignee: Martin Stefanko
> Labels: student
>
> Investigate the use of a transaction manager when used within a reactive microservices environment[1][2].
> h2. Motivation
> Reactive microservice architecture is a trending architectural pattern. The reactive architecture could simply be defined as a system where messages are sent to a jms queue or a specialized event storage and other services are registered to, for being notified when an event is generated to be able to react on it. Synchronous calls from a service to a service are not meant to be a good practice in such system. Here all the communication between services is done with message passing through the event storage.
> Such a system is expected to decouple data [3] where each microservice manages data, which needs for working, on its own. Still there are cases where some data coordination is needed. As the all communication is asynchronous it may be that we need to modify the transaction manager away from synchronous operations. One possibly good match seems to be the Saga transactions[4] and eventual consistency[5][6] but handling of them is fitting to the message driven system. The way of understanding Saga in microservice world could be overviewed at [15].
> Narayana transaction manager is good in managing Saga transactions where having long history experience from running it for WS[7] or managing consistency in NoSQL databases[8]. But Narayana is designed to manage transactions in synchronous way. The transaction context is normally passed along a RPC call where the transaction is finished after such synchronous call is returned back to the caller.
> h2. Steps to go
> The point here is to investigate on the topic of usage Saga in reactive architecture, gather working solutions - especially in Java world (some references eg. [9][10][11][12][13][14]), get experience in how they are working and on top of this experience considering implementation of the reactive approach with Narayana. It may be that Narayana requires some modification to execute optimally for such task and then it’s up to reconsideration if some greenfield implementation should be started. As the last step there should be created some sample project (ie. quickstart) where the usage of the implementation would be shown.
> With that we expect usage of projects in Red Hat portfolio. The sample project would be probably implemented with usage of the Vert.x, meybe there could be considered usage of Infinispan or AMQ7 for handling events.
> h2. Expected output
> The expected output of this effort is
> * A research document with state of the art.
> ** Overview of what are the problem of the synchronous/blocking approach.
> ** What are the other approaches that people have to this problem,
> * A proof-of-concept implementation
> ** Preferably with use of Narayana transaction manager prepare a service capable to manage transactions in world of reactive microservices
> * An example/quickstart showing this in practical terms
> ** A proof that a transaction manager can be made to work in an asynchronous environment. For example if you have a Vert.x event and it has to commit a transaction. The handler for that message currently does sync ops with transaction manager and that blocks. One of the goals for the project is to prevent blocking of threads in an async environment.
> h2. Resources
> [1] [https://www.youtube.com/watch?v=STKCRSUsyP0|https://www.youtube.com/watch... 2017, The Many Meanings of Event-Driven Architecture, Martin Fowler)
> [2] [http://www.oreilly.com/programming/free/reactive-microservices-architectu...] (Reactive Microservices Architecture)
> [3] [http://blog.christianposta.com/microservices/the-hardest-part-about-micro...] (The Hardest Part About Microservices: Your Data)
> [4] [https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf] (Sagas paper)
> [5] [http://queue.acm.org/detail.cfm?id=2462076] (Eventual Consistency Today: Limitations, Extensions, and Beyond)
> [6] [http://queue.acm.org/detail.cfm?id=3025012] (Life Beyond Distributed Transactions)
> [7] [http://narayana.io//docs/project/index.html#d0e14874] (Narayana doc, WS-Business Activity)
> [8] [http://jbossts.blogspot.cz/2014/05/bringing-transactional-guarantees-to.html] (Bringing Transactional Guarantees to MongoDB)
> [9] [https://www.youtube.com/watch?v=0UTOLRTwOX0] (Distributed Sagas: A Protocol for Coordinating Microservices - Caitie McCaffrey - JOTB17)
> [10] [https://www.youtube.com/watch?v=YPbGW3Fnmbc] (Using sagas to maintain data consistency in a microservice architecture by Chris Richardson)
> [11] [http://www.axonframework.org]
> [12] [http://eventuate.io]
> [13] [https://docs.particular.net/nservicebus/sagas] (.NET NServiceBus Saga)
> [14] [https://www.youtube.com/watch?v=Rm8n-H6zI1k] (Dr. Roland Kuhn: Reactive Design Patterns, Akka)
> [15] [http://microservices.io/patterns/data/saga.html] (Pattern: Saga)
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (JBTM-3047) Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-3047?page=com.atlassian.jira.plugin.... ]
Issue was automatically transitioned when Ondra Chaloupka created pull request #1353 in GitHub
----------------------------------------------------------------------------------------------
Status: Pull Request Sent (was: Open)
> Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
> ------------------------------------------------------------------------------------------
>
> Key: JBTM-3047
> URL: https://issues.jboss.org/browse/JBTM-3047
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: Tooling
> Affects Versions: 5.9.0.Final
> Reporter: Ondra Chaloupka
> Assignee: Ondra Chaloupka
>
> If {{PeriodicMonitor}} has active unfinished request to start recovery scan
> {code}
> INFO [com.arjuna.ats.arjuna] (Server.Connection:127.0.0.1:36326) ARJUNA012340: RecoveryManager scan scheduled to begin.
> {code}
> and meanwhile [the RecoveryManager is suspended|https://github.com/jbosstm/narayana/blob/5.9.0.Final/ArjunaCore...] then the WorkerService is waiting in deadlock forever and JVM can't be cleanly stopped.
> The whole stopping EAP server stacktrace could be seen at https://issues.jboss.org/browse/CLOUD-2767 the related Narayana related threads are[10].
> It seems to me as the recovery was put to suspended[1] the processing thread waits for the state change[2]. The state change to set to terminated could be done by periodic recovery shutdown[3] but it's stuck on waiting for listners to be stopped[4]. Stopping the listeners could be done by the thread waiting for the state change[5].
> [1] https://github.com/wildfly/wildfly/blob/master/transactions/src/main/java...
> [2] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [3] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [4] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [5] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [10]
> {code}
> "Transaction Reaper Worker 0" #121 daemon prio=5 os_prio=0 tid=0x000000000197a000 nid=0x2b6 in Object.wait() [0x00007f3661d2b000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c310> (a java.util.LinkedList)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:328)
> - locked <0x00000000b289c310> (a java.util.LinkedList)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
> Locked ownable synchronizers:
> - None
> "Transaction Reaper" #120 daemon prio=5 os_prio=0 tid=0x0000000001b4b800 nid=0x2b5 in Object.wait() [0x00007f3661e2c000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:90)
> - locked <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46200" #335 daemon prio=5 os_prio=0 tid=0x0000000003853800 nid=0x449 in Object.wait() [0x00007f365ec2c000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46090" #225 daemon prio=5 os_prio=0 tid=0x0000000003a4c800 nid=0x3ca in Object.wait() [0x00007f365c2ef000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "MSC service thread 1-5" #19 prio=5 os_prio=0 tid=0x0000000001903000 nid=0x251 in Object.wait() [0x00007f366aebe000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.Listener.stopListener(Listener.java:222)
> - locked <0x00000000b29161d0> (a com.arjuna.ats.internal.arjuna.recovery.Listener)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.shutdown(PeriodicRecovery.java:169)
> at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:179)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:210)
> - locked <0x00000000b0badbb8> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:194)
> at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:73)
> at org.jboss.as.txn.service.ArjunaRecoveryManagerService.stop(ArjunaRecoveryManagerService.java:167)
> - locked <0x00000000b313ddb0> (a org.jboss.as.txn.service.ArjunaRecoveryManagerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000b351b500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "Periodic Recovery" #116 prio=5 os_prio=0 tid=0x0000000002357000 nid=0x2b4 in Object.wait() [0x00007f3661f2d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doSuspendedWait(PeriodicRecovery.java:709)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:346)
> - locked <0x00000000b289c898> (a java.lang.Object)
> Locked ownable synchronizers:
> - None
> {code}
> NOTE: there is interesting point there are two {{WorkerService}} instances in the stack trace. It's causes two sequential requests hits the RecoveryManager while the periodic recovery was in progress before the server was shutdown. The code invoking the two calls and then immediately running the shutdown was https://github.com/jboss-openshift/cct_module/blob/7f891cca882f86f983b298...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (JBTM-3047) Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-3047?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka updated JBTM-3047:
----------------------------------
Git Pull Request: https://github.com/jbosstm/narayana/pull/1353
> Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
> ------------------------------------------------------------------------------------------
>
> Key: JBTM-3047
> URL: https://issues.jboss.org/browse/JBTM-3047
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: Tooling
> Affects Versions: 5.9.0.Final
> Reporter: Ondra Chaloupka
> Assignee: Ondra Chaloupka
>
> If {{PeriodicMonitor}} has active unfinished request to start recovery scan
> {code}
> INFO [com.arjuna.ats.arjuna] (Server.Connection:127.0.0.1:36326) ARJUNA012340: RecoveryManager scan scheduled to begin.
> {code}
> and meanwhile [the RecoveryManager is suspended|https://github.com/jbosstm/narayana/blob/5.9.0.Final/ArjunaCore...] then the WorkerService is waiting in deadlock forever and JVM can't be cleanly stopped.
> The whole stopping EAP server stacktrace could be seen at https://issues.jboss.org/browse/CLOUD-2767 the related Narayana related threads are[10].
> It seems to me as the recovery was put to suspended[1] the processing thread waits for the state change[2]. The state change to set to terminated could be done by periodic recovery shutdown[3] but it's stuck on waiting for listners to be stopped[4]. Stopping the listeners could be done by the thread waiting for the state change[5].
> [1] https://github.com/wildfly/wildfly/blob/master/transactions/src/main/java...
> [2] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [3] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [4] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [5] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [10]
> {code}
> "Transaction Reaper Worker 0" #121 daemon prio=5 os_prio=0 tid=0x000000000197a000 nid=0x2b6 in Object.wait() [0x00007f3661d2b000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c310> (a java.util.LinkedList)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:328)
> - locked <0x00000000b289c310> (a java.util.LinkedList)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
> Locked ownable synchronizers:
> - None
> "Transaction Reaper" #120 daemon prio=5 os_prio=0 tid=0x0000000001b4b800 nid=0x2b5 in Object.wait() [0x00007f3661e2c000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:90)
> - locked <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46200" #335 daemon prio=5 os_prio=0 tid=0x0000000003853800 nid=0x449 in Object.wait() [0x00007f365ec2c000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46090" #225 daemon prio=5 os_prio=0 tid=0x0000000003a4c800 nid=0x3ca in Object.wait() [0x00007f365c2ef000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "MSC service thread 1-5" #19 prio=5 os_prio=0 tid=0x0000000001903000 nid=0x251 in Object.wait() [0x00007f366aebe000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.Listener.stopListener(Listener.java:222)
> - locked <0x00000000b29161d0> (a com.arjuna.ats.internal.arjuna.recovery.Listener)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.shutdown(PeriodicRecovery.java:169)
> at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:179)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:210)
> - locked <0x00000000b0badbb8> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:194)
> at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:73)
> at org.jboss.as.txn.service.ArjunaRecoveryManagerService.stop(ArjunaRecoveryManagerService.java:167)
> - locked <0x00000000b313ddb0> (a org.jboss.as.txn.service.ArjunaRecoveryManagerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000b351b500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "Periodic Recovery" #116 prio=5 os_prio=0 tid=0x0000000002357000 nid=0x2b4 in Object.wait() [0x00007f3661f2d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doSuspendedWait(PeriodicRecovery.java:709)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:346)
> - locked <0x00000000b289c898> (a java.lang.Object)
> Locked ownable synchronizers:
> - None
> {code}
> NOTE: there is interesting point there are two {{WorkerService}} instances in the stack trace. It's causes two sequential requests hits the RecoveryManager while the periodic recovery was in progress before the server was shutdown. The code invoking the two calls and then immediately running the shutdown was https://github.com/jboss-openshift/cct_module/blob/7f891cca882f86f983b298...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (JBTM-3047) Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-3047?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka commented on JBTM-3047:
---------------------------------------
[~mmusgrov] what do you think about this? https://github.com/jbosstm/narayana/compare/master...ochaloup:JBTM-3047-p...
> Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
> ------------------------------------------------------------------------------------------
>
> Key: JBTM-3047
> URL: https://issues.jboss.org/browse/JBTM-3047
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: Tooling
> Affects Versions: 5.9.0.Final
> Reporter: Ondra Chaloupka
> Assignee: Ondra Chaloupka
>
> If {{PeriodicMonitor}} has active unfinished request to start recovery scan
> {code}
> INFO [com.arjuna.ats.arjuna] (Server.Connection:127.0.0.1:36326) ARJUNA012340: RecoveryManager scan scheduled to begin.
> {code}
> and meanwhile [the RecoveryManager is suspended|https://github.com/jbosstm/narayana/blob/5.9.0.Final/ArjunaCore...] then the WorkerService is waiting in deadlock forever and JVM can't be cleanly stopped.
> The whole stopping EAP server stacktrace could be seen at https://issues.jboss.org/browse/CLOUD-2767 the related Narayana related threads are[10].
> It seems to me as the recovery was put to suspended[1] the processing thread waits for the state change[2]. The state change to set to terminated could be done by periodic recovery shutdown[3] but it's stuck on waiting for listners to be stopped[4]. Stopping the listeners could be done by the thread waiting for the state change[5].
> [1] https://github.com/wildfly/wildfly/blob/master/transactions/src/main/java...
> [2] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [3] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [4] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [5] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [10]
> {code}
> "Transaction Reaper Worker 0" #121 daemon prio=5 os_prio=0 tid=0x000000000197a000 nid=0x2b6 in Object.wait() [0x00007f3661d2b000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c310> (a java.util.LinkedList)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:328)
> - locked <0x00000000b289c310> (a java.util.LinkedList)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
> Locked ownable synchronizers:
> - None
> "Transaction Reaper" #120 daemon prio=5 os_prio=0 tid=0x0000000001b4b800 nid=0x2b5 in Object.wait() [0x00007f3661e2c000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:90)
> - locked <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46200" #335 daemon prio=5 os_prio=0 tid=0x0000000003853800 nid=0x449 in Object.wait() [0x00007f365ec2c000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46090" #225 daemon prio=5 os_prio=0 tid=0x0000000003a4c800 nid=0x3ca in Object.wait() [0x00007f365c2ef000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "MSC service thread 1-5" #19 prio=5 os_prio=0 tid=0x0000000001903000 nid=0x251 in Object.wait() [0x00007f366aebe000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.Listener.stopListener(Listener.java:222)
> - locked <0x00000000b29161d0> (a com.arjuna.ats.internal.arjuna.recovery.Listener)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.shutdown(PeriodicRecovery.java:169)
> at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:179)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:210)
> - locked <0x00000000b0badbb8> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:194)
> at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:73)
> at org.jboss.as.txn.service.ArjunaRecoveryManagerService.stop(ArjunaRecoveryManagerService.java:167)
> - locked <0x00000000b313ddb0> (a org.jboss.as.txn.service.ArjunaRecoveryManagerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000b351b500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "Periodic Recovery" #116 prio=5 os_prio=0 tid=0x0000000002357000 nid=0x2b4 in Object.wait() [0x00007f3661f2d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doSuspendedWait(PeriodicRecovery.java:709)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:346)
> - locked <0x00000000b289c898> (a java.lang.Object)
> Locked ownable synchronizers:
> - None
> {code}
> NOTE: there is interesting point there are two {{WorkerService}} instances in the stack trace. It's causes two sequential requests hits the RecoveryManager while the periodic recovery was in progress before the server was shutdown. The code invoking the two calls and then immediately running the shutdown was https://github.com/jboss-openshift/cct_module/blob/7f891cca882f86f983b298...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (JBTM-3047) Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-3047?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka commented on JBTM-3047:
---------------------------------------
I'm borrowing more detailed description of the issue from [~mmusgrov]
{quote}
The suspend call is coming from the wildfly RecoverySuspendController which puts PeriodicRecovery in the SUSPENDED state. The shutdown request waits for the recovery listeners to finish but the recovery listeners are waiting for the periodic recovery wakeUp call to complete which never happens because periodic recovery is suspended.
{quote}
> Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
> ------------------------------------------------------------------------------------------
>
> Key: JBTM-3047
> URL: https://issues.jboss.org/browse/JBTM-3047
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: Tooling
> Affects Versions: 5.9.0.Final
> Reporter: Ondra Chaloupka
> Assignee: Ondra Chaloupka
>
> If {{PeriodicMonitor}} has active unfinished request to start recovery scan
> {code}
> INFO [com.arjuna.ats.arjuna] (Server.Connection:127.0.0.1:36326) ARJUNA012340: RecoveryManager scan scheduled to begin.
> {code}
> and meanwhile [the RecoveryManager is suspended|https://github.com/jbosstm/narayana/blob/5.9.0.Final/ArjunaCore...] then the WorkerService is waiting in deadlock forever and JVM can't be cleanly stopped.
> The whole stopping EAP server stacktrace could be seen at https://issues.jboss.org/browse/CLOUD-2767 the related Narayana related threads are[10].
> It seems to me as the recovery was put to suspended[1] the processing thread waits for the state change[2]. The state change to set to terminated could be done by periodic recovery shutdown[3] but it's stuck on waiting for listners to be stopped[4]. Stopping the listeners could be done by the thread waiting for the state change[5].
> [1] https://github.com/wildfly/wildfly/blob/master/transactions/src/main/java...
> [2] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [3] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [4] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [5] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [10]
> {code}
> "Transaction Reaper Worker 0" #121 daemon prio=5 os_prio=0 tid=0x000000000197a000 nid=0x2b6 in Object.wait() [0x00007f3661d2b000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c310> (a java.util.LinkedList)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:328)
> - locked <0x00000000b289c310> (a java.util.LinkedList)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
> Locked ownable synchronizers:
> - None
> "Transaction Reaper" #120 daemon prio=5 os_prio=0 tid=0x0000000001b4b800 nid=0x2b5 in Object.wait() [0x00007f3661e2c000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:90)
> - locked <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46200" #335 daemon prio=5 os_prio=0 tid=0x0000000003853800 nid=0x449 in Object.wait() [0x00007f365ec2c000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46090" #225 daemon prio=5 os_prio=0 tid=0x0000000003a4c800 nid=0x3ca in Object.wait() [0x00007f365c2ef000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "MSC service thread 1-5" #19 prio=5 os_prio=0 tid=0x0000000001903000 nid=0x251 in Object.wait() [0x00007f366aebe000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.Listener.stopListener(Listener.java:222)
> - locked <0x00000000b29161d0> (a com.arjuna.ats.internal.arjuna.recovery.Listener)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.shutdown(PeriodicRecovery.java:169)
> at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:179)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:210)
> - locked <0x00000000b0badbb8> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:194)
> at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:73)
> at org.jboss.as.txn.service.ArjunaRecoveryManagerService.stop(ArjunaRecoveryManagerService.java:167)
> - locked <0x00000000b313ddb0> (a org.jboss.as.txn.service.ArjunaRecoveryManagerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000b351b500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "Periodic Recovery" #116 prio=5 os_prio=0 tid=0x0000000002357000 nid=0x2b4 in Object.wait() [0x00007f3661f2d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doSuspendedWait(PeriodicRecovery.java:709)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:346)
> - locked <0x00000000b289c898> (a java.lang.Object)
> Locked ownable synchronizers:
> - None
> {code}
> NOTE: there is interesting point there are two {{WorkerService}} instances in the stack trace. It's causes two sequential requests hits the RecoveryManager while the periodic recovery was in progress before the server was shutdown. The code invoking the two calls and then immediately running the shutdown was https://github.com/jboss-openshift/cct_module/blob/7f891cca882f86f983b298...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (JBTM-3047) Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-3047?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka updated JBTM-3047:
----------------------------------
Description:
If {{PeriodicMonitor}} has active unfinished request to start recovery scan
{code}
INFO [com.arjuna.ats.arjuna] (Server.Connection:127.0.0.1:36326) ARJUNA012340: RecoveryManager scan scheduled to begin.
{code}
and meanwhile [the RecoveryManager is suspended|https://github.com/jbosstm/narayana/blob/5.9.0.Final/ArjunaCore...] then the WorkerService is waiting in deadlock forever and JVM can't be cleanly stopped.
The whole stopping EAP server stacktrace could be seen at https://issues.jboss.org/browse/CLOUD-2767 the related Narayana related threads are[10].
It seems to me as the recovery was put to suspended[1] the processing thread waits for the state change[2]. The state change to set to terminated could be done by periodic recovery shutdown[3] but it's stuck on waiting for listners to be stopped[4]. Stopping the listeners could be done by the thread waiting for the state change[5].
[1] https://github.com/wildfly/wildfly/blob/master/transactions/src/main/java...
[2] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[3] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[4] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[5] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[10]
{code}
"Transaction Reaper Worker 0" #121 daemon prio=5 os_prio=0 tid=0x000000000197a000 nid=0x2b6 in Object.wait() [0x00007f3661d2b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000b289c310> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:328)
- locked <0x00000000b289c310> (a java.util.LinkedList)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
Locked ownable synchronizers:
- None
"Transaction Reaper" #120 daemon prio=5 os_prio=0 tid=0x0000000001b4b800 nid=0x2b5 in Object.wait() [0x00007f3661e2c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:90)
- locked <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
Locked ownable synchronizers:
- None
"Server.Connection:10.130.0.162:46200" #335 daemon prio=5 os_prio=0 tid=0x0000000003853800 nid=0x449 in Object.wait() [0x00007f365ec2c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
- locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
Locked ownable synchronizers:
- None
"Server.Connection:10.130.0.162:46090" #225 daemon prio=5 os_prio=0 tid=0x0000000003a4c800 nid=0x3ca in Object.wait() [0x00007f365c2ef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
- locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
Locked ownable synchronizers:
- None
"MSC service thread 1-5" #19 prio=5 os_prio=0 tid=0x0000000001903000 nid=0x251 in Object.wait() [0x00007f366aebe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.Listener.stopListener(Listener.java:222)
- locked <0x00000000b29161d0> (a com.arjuna.ats.internal.arjuna.recovery.Listener)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.shutdown(PeriodicRecovery.java:169)
at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:179)
at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:210)
- locked <0x00000000b0badbb8> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)
at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:194)
at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:73)
at org.jboss.as.txn.service.ArjunaRecoveryManagerService.stop(ArjunaRecoveryManagerService.java:167)
- locked <0x00000000b313ddb0> (a org.jboss.as.txn.service.ArjunaRecoveryManagerService)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000000b351b500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Periodic Recovery" #116 prio=5 os_prio=0 tid=0x0000000002357000 nid=0x2b4 in Object.wait() [0x00007f3661f2d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doSuspendedWait(PeriodicRecovery.java:709)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:346)
- locked <0x00000000b289c898> (a java.lang.Object)
Locked ownable synchronizers:
- None
{code}
NOTE: there is interesting point there are two {{WorkerService}} instances in the stack trace. It's causes two sequential requests hits the RecoveryManager while the periodic recovery was in progress before the server was shutdown. The code invoking the two calls and then immediately running the shutdown was https://github.com/jboss-openshift/cct_module/blob/7f891cca882f86f983b298...
was:
If {{PeriodicMonitor}} has active unfinished request to start recovery scan
{code}
INFO [com.arjuna.ats.arjuna] (Server.Connection:127.0.0.1:36326) ARJUNA012340: RecoveryManager scan scheduled to begin.
{code}
and meanwhile [the RecoveryManager is suspended|https://github.com/jbosstm/narayana/blob/5.9.0.Final/ArjunaCore...] then the WorkerService is waiting in deadlock forever and JVM can't be cleanly stopped.
The whole stopping EAP server stacktrace could be seen at https://issues.jboss.org/browse/CLOUD-2767 the related Narayana related threads are[10].
It seems to me as the recovery was put to suspended[1] the processing thread waits for the state change[2]. The state change to set to terminated could be done by periodic recovery shutdown[3] but it's stuck on waiting for listners to be stopped[4]. Stopping the listeners could be done by the thread waiting for the state change[5].
[1] https://github.com/wildfly/wildfly/blob/master/transactions/src/main/java...
[2] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[3] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[4] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[5] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[10]
{code}
"Transaction Reaper Worker 0" #121 daemon prio=5 os_prio=0 tid=0x000000000197a000 nid=0x2b6 in Object.wait() [0x00007f3661d2b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000b289c310> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:328)
- locked <0x00000000b289c310> (a java.util.LinkedList)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
Locked ownable synchronizers:
- None
"Transaction Reaper" #120 daemon prio=5 os_prio=0 tid=0x0000000001b4b800 nid=0x2b5 in Object.wait() [0x00007f3661e2c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:90)
- locked <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
Locked ownable synchronizers:
- None
"Server.Connection:10.130.0.162:46200" #335 daemon prio=5 os_prio=0 tid=0x0000000003853800 nid=0x449 in Object.wait() [0x00007f365ec2c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
- locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
Locked ownable synchronizers:
- None
"Server.Connection:10.130.0.162:46090" #225 daemon prio=5 os_prio=0 tid=0x0000000003a4c800 nid=0x3ca in Object.wait() [0x00007f365c2ef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
- locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
Locked ownable synchronizers:
- None
"MSC service thread 1-5" #19 prio=5 os_prio=0 tid=0x0000000001903000 nid=0x251 in Object.wait() [0x00007f366aebe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.Listener.stopListener(Listener.java:222)
- locked <0x00000000b29161d0> (a com.arjuna.ats.internal.arjuna.recovery.Listener)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.shutdown(PeriodicRecovery.java:169)
at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:179)
at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:210)
- locked <0x00000000b0badbb8> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)
at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:194)
at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:73)
at org.jboss.as.txn.service.ArjunaRecoveryManagerService.stop(ArjunaRecoveryManagerService.java:167)
- locked <0x00000000b313ddb0> (a org.jboss.as.txn.service.ArjunaRecoveryManagerService)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000000b351b500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Periodic Recovery" #116 prio=5 os_prio=0 tid=0x0000000002357000 nid=0x2b4 in Object.wait() [0x00007f3661f2d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doSuspendedWait(PeriodicRecovery.java:709)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:346)
- locked <0x00000000b289c898> (a java.lang.Object)
Locked ownable synchronizers:
- None
{code}
> Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
> ------------------------------------------------------------------------------------------
>
> Key: JBTM-3047
> URL: https://issues.jboss.org/browse/JBTM-3047
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: Tooling
> Affects Versions: 5.9.0.Final
> Reporter: Ondra Chaloupka
> Assignee: Ondra Chaloupka
>
> If {{PeriodicMonitor}} has active unfinished request to start recovery scan
> {code}
> INFO [com.arjuna.ats.arjuna] (Server.Connection:127.0.0.1:36326) ARJUNA012340: RecoveryManager scan scheduled to begin.
> {code}
> and meanwhile [the RecoveryManager is suspended|https://github.com/jbosstm/narayana/blob/5.9.0.Final/ArjunaCore...] then the WorkerService is waiting in deadlock forever and JVM can't be cleanly stopped.
> The whole stopping EAP server stacktrace could be seen at https://issues.jboss.org/browse/CLOUD-2767 the related Narayana related threads are[10].
> It seems to me as the recovery was put to suspended[1] the processing thread waits for the state change[2]. The state change to set to terminated could be done by periodic recovery shutdown[3] but it's stuck on waiting for listners to be stopped[4]. Stopping the listeners could be done by the thread waiting for the state change[5].
> [1] https://github.com/wildfly/wildfly/blob/master/transactions/src/main/java...
> [2] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [3] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [4] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [5] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [10]
> {code}
> "Transaction Reaper Worker 0" #121 daemon prio=5 os_prio=0 tid=0x000000000197a000 nid=0x2b6 in Object.wait() [0x00007f3661d2b000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c310> (a java.util.LinkedList)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:328)
> - locked <0x00000000b289c310> (a java.util.LinkedList)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
> Locked ownable synchronizers:
> - None
> "Transaction Reaper" #120 daemon prio=5 os_prio=0 tid=0x0000000001b4b800 nid=0x2b5 in Object.wait() [0x00007f3661e2c000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:90)
> - locked <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46200" #335 daemon prio=5 os_prio=0 tid=0x0000000003853800 nid=0x449 in Object.wait() [0x00007f365ec2c000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46090" #225 daemon prio=5 os_prio=0 tid=0x0000000003a4c800 nid=0x3ca in Object.wait() [0x00007f365c2ef000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "MSC service thread 1-5" #19 prio=5 os_prio=0 tid=0x0000000001903000 nid=0x251 in Object.wait() [0x00007f366aebe000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.Listener.stopListener(Listener.java:222)
> - locked <0x00000000b29161d0> (a com.arjuna.ats.internal.arjuna.recovery.Listener)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.shutdown(PeriodicRecovery.java:169)
> at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:179)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:210)
> - locked <0x00000000b0badbb8> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:194)
> at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:73)
> at org.jboss.as.txn.service.ArjunaRecoveryManagerService.stop(ArjunaRecoveryManagerService.java:167)
> - locked <0x00000000b313ddb0> (a org.jboss.as.txn.service.ArjunaRecoveryManagerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000b351b500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "Periodic Recovery" #116 prio=5 os_prio=0 tid=0x0000000002357000 nid=0x2b4 in Object.wait() [0x00007f3661f2d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doSuspendedWait(PeriodicRecovery.java:709)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:346)
> - locked <0x00000000b289c898> (a java.lang.Object)
> Locked ownable synchronizers:
> - None
> {code}
> NOTE: there is interesting point there are two {{WorkerService}} instances in the stack trace. It's causes two sequential requests hits the RecoveryManager while the periodic recovery was in progress before the server was shutdown. The code invoking the two calls and then immediately running the shutdown was https://github.com/jboss-openshift/cct_module/blob/7f891cca882f86f983b298...
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (JBTM-3047) Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-3047?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka updated JBTM-3047:
----------------------------------
Steps to Reproduce:
This could be simulated for WFLY when it should be stopped. WildFly {{:shutdown}} command causes suspend being invoked and then JVM is about to be closed.
* Periodic recovery is active and running - for processing both phases it needs more than 20 seconds which is the default socket timeout for PeriodicMonitor call
* Client requests SCAN, the default timeout for the socket is 20 seconds[5]
{{java -cp modules/system/layers/base/org/jboss/jts/main/narayana-jts-idlj-5.9.0.Final.jar com.arjuna.ats.arjuna.tools.RecoveryMonitor -host localhost -port 4712}}
* ServiceWorker is started and waits
* 20 seconds elapsed, client gets "java.net.SocketTimeoutException: Read timed out"
* (recovery needs to be still in progress while stopping the server)
* server is asked to stop {{./bin/jboss-cli.sh --command=':shutdown()' -c}} which causes suspend is called on the recovery manager and we have the deadlock
was:
This could be simulated for WFLY when it should be stopped. WildFly {{:shutdown}} command causes suspend being invoked and then JVM is about to be closed.
* Periodic recovery is active and running - for processing both phases it needs more than 20 seconds which is the default socket timeout for PeriodicMonitor call
* Client requests SCAN, the default timeout for the socket is 20 seconds[5]
{{java -cp modules/system/layers/base/org/jboss/jts/main/narayana-jts-idlj-5.9.0.Final.jar com.arjuna.ats.arjuna.tools.RecoveryMonitor -host localhost -port 4712}}
* ServiceWorker is started and waits
* 20 seconds elapsed, client gets "java.net.SocketTimeoutException: Read timed out"
* server is asked to stop {{./bin/jboss-cli.sh --command=':shutdown()' -c}} which causes suspend is called on the recovery manager and we have the deadlock
> Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
> ------------------------------------------------------------------------------------------
>
> Key: JBTM-3047
> URL: https://issues.jboss.org/browse/JBTM-3047
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: Tooling
> Affects Versions: 5.9.0.Final
> Reporter: Ondra Chaloupka
> Assignee: Ondra Chaloupka
>
> If {{PeriodicMonitor}} has active unfinished request to start recovery scan
> {code}
> INFO [com.arjuna.ats.arjuna] (Server.Connection:127.0.0.1:36326) ARJUNA012340: RecoveryManager scan scheduled to begin.
> {code}
> and meanwhile [the RecoveryManager is suspended|https://github.com/jbosstm/narayana/blob/5.9.0.Final/ArjunaCore...] then the WorkerService is waiting in deadlock forever and JVM can't be cleanly stopped.
> The whole stopping EAP server stacktrace could be seen at https://issues.jboss.org/browse/CLOUD-2767 the related Narayana related threads are[10].
> It seems to me as the recovery was put to suspended[1] the processing thread waits for the state change[2]. The state change to set to terminated could be done by periodic recovery shutdown[3] but it's stuck on waiting for listners to be stopped[4]. Stopping the listeners could be done by the thread waiting for the state change[5].
> [1] https://github.com/wildfly/wildfly/blob/master/transactions/src/main/java...
> [2] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [3] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [4] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [5] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [10]
> {code}
> "Transaction Reaper Worker 0" #121 daemon prio=5 os_prio=0 tid=0x000000000197a000 nid=0x2b6 in Object.wait() [0x00007f3661d2b000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c310> (a java.util.LinkedList)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:328)
> - locked <0x00000000b289c310> (a java.util.LinkedList)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
> Locked ownable synchronizers:
> - None
> "Transaction Reaper" #120 daemon prio=5 os_prio=0 tid=0x0000000001b4b800 nid=0x2b5 in Object.wait() [0x00007f3661e2c000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:90)
> - locked <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46200" #335 daemon prio=5 os_prio=0 tid=0x0000000003853800 nid=0x449 in Object.wait() [0x00007f365ec2c000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46090" #225 daemon prio=5 os_prio=0 tid=0x0000000003a4c800 nid=0x3ca in Object.wait() [0x00007f365c2ef000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "MSC service thread 1-5" #19 prio=5 os_prio=0 tid=0x0000000001903000 nid=0x251 in Object.wait() [0x00007f366aebe000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.Listener.stopListener(Listener.java:222)
> - locked <0x00000000b29161d0> (a com.arjuna.ats.internal.arjuna.recovery.Listener)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.shutdown(PeriodicRecovery.java:169)
> at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:179)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:210)
> - locked <0x00000000b0badbb8> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:194)
> at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:73)
> at org.jboss.as.txn.service.ArjunaRecoveryManagerService.stop(ArjunaRecoveryManagerService.java:167)
> - locked <0x00000000b313ddb0> (a org.jboss.as.txn.service.ArjunaRecoveryManagerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000b351b500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "Periodic Recovery" #116 prio=5 os_prio=0 tid=0x0000000002357000 nid=0x2b4 in Object.wait() [0x00007f3661f2d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doSuspendedWait(PeriodicRecovery.java:709)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:346)
> - locked <0x00000000b289c898> (a java.lang.Object)
> Locked ownable synchronizers:
> - None
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (JBTM-3047) Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
by Ondra Chaloupka (JIRA)
[ https://issues.jboss.org/browse/JBTM-3047?page=com.atlassian.jira.plugin.... ]
Ondra Chaloupka updated JBTM-3047:
----------------------------------
Description:
If {{PeriodicMonitor}} has active unfinished request to start recovery scan
{code}
INFO [com.arjuna.ats.arjuna] (Server.Connection:127.0.0.1:36326) ARJUNA012340: RecoveryManager scan scheduled to begin.
{code}
and meanwhile [the RecoveryManager is suspended|https://github.com/jbosstm/narayana/blob/5.9.0.Final/ArjunaCore...] then the WorkerService is waiting in deadlock forever and JVM can't be cleanly stopped.
The whole stopping EAP server stacktrace could be seen at https://issues.jboss.org/browse/CLOUD-2767 the related Narayana related threads are[10].
It seems to me as the recovery was put to suspended[1] the processing thread waits for the state change[2]. The state change to set to terminated could be done by periodic recovery shutdown[3] but it's stuck on waiting for listners to be stopped[4]. Stopping the listeners could be done by the thread waiting for the state change[5].
[1] https://github.com/wildfly/wildfly/blob/master/transactions/src/main/java...
[2] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[3] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[4] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[5] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[10]
{code}
"Transaction Reaper Worker 0" #121 daemon prio=5 os_prio=0 tid=0x000000000197a000 nid=0x2b6 in Object.wait() [0x00007f3661d2b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000b289c310> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:328)
- locked <0x00000000b289c310> (a java.util.LinkedList)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
Locked ownable synchronizers:
- None
"Transaction Reaper" #120 daemon prio=5 os_prio=0 tid=0x0000000001b4b800 nid=0x2b5 in Object.wait() [0x00007f3661e2c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:90)
- locked <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
Locked ownable synchronizers:
- None
"Server.Connection:10.130.0.162:46200" #335 daemon prio=5 os_prio=0 tid=0x0000000003853800 nid=0x449 in Object.wait() [0x00007f365ec2c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
- locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
Locked ownable synchronizers:
- None
"Server.Connection:10.130.0.162:46090" #225 daemon prio=5 os_prio=0 tid=0x0000000003a4c800 nid=0x3ca in Object.wait() [0x00007f365c2ef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
- locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
Locked ownable synchronizers:
- None
"MSC service thread 1-5" #19 prio=5 os_prio=0 tid=0x0000000001903000 nid=0x251 in Object.wait() [0x00007f366aebe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.Listener.stopListener(Listener.java:222)
- locked <0x00000000b29161d0> (a com.arjuna.ats.internal.arjuna.recovery.Listener)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.shutdown(PeriodicRecovery.java:169)
at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:179)
at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:210)
- locked <0x00000000b0badbb8> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)
at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:194)
at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:73)
at org.jboss.as.txn.service.ArjunaRecoveryManagerService.stop(ArjunaRecoveryManagerService.java:167)
- locked <0x00000000b313ddb0> (a org.jboss.as.txn.service.ArjunaRecoveryManagerService)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000000b351b500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Periodic Recovery" #116 prio=5 os_prio=0 tid=0x0000000002357000 nid=0x2b4 in Object.wait() [0x00007f3661f2d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doSuspendedWait(PeriodicRecovery.java:709)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:346)
- locked <0x00000000b289c898> (a java.lang.Object)
Locked ownable synchronizers:
- None
{code}
was:
If {{PeriodicMonitor}} has active unfinished request to start recovery scan
{code}
INFO [com.arjuna.ats.arjuna] (Server.Connection:127.0.0.1:36326) ARJUNA012340: RecoveryManager scan scheduled to begin.
{code}
and meanwhile [the RecoveryManager is suspended|https://github.com/jbosstm/narayana/blob/5.9.0.Final/ArjunaCore...] then the WorkerService is waiting in deadlock forever and JVM can't be cleanly stopped.
The whole stopping EAP server stacktrace could be seen at https://issues.jboss.org/browse/CLOUD-2767 the related Narayana related threads are[10].
It seems to me as the recovery was put to suspended[1] the processing thread waits for the state change[2]. The state change to set to terminated could be done by periodic recovery shutdown[3] but it's stuck on waiting for listners to be stopped[4]. Stopping the listeners could be done by the thread waiting for the state change[5].
[1] https://github.com/wildfly/wildfly/blob/master/transactions/src/main/java...
[2] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[3] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[4] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[5] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[10]
{code}
"Transaction Reaper Worker 0" #121 daemon prio=5 os_prio=0 tid=0x000000000197a000 nid=0x2b6 in Object.wait() [0x00007f3661d2b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000b289c310> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:328)
- locked <0x00000000b289c310> (a java.util.LinkedList)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
Locked ownable synchronizers:
- None
"Transaction Reaper" #120 daemon prio=5 os_prio=0 tid=0x0000000001b4b800 nid=0x2b5 in Object.wait() [0x00007f3661e2c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:90)
- locked <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
Locked ownable synchronizers:
- None
"Server.Connection:10.130.0.162:46200" #335 daemon prio=5 os_prio=0 tid=0x0000000003853800 nid=0x449 in Object.wait() [0x00007f365ec2c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
- locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
Locked ownable synchronizers:
- None
"Server.Connection:10.130.0.162:46090" #225 daemon prio=5 os_prio=0 tid=0x0000000003a4c800 nid=0x3ca in Object.wait() [0x00007f365c2ef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
- locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
Locked ownable synchronizers:
- None
"MSC service thread 1-5" #19 prio=5 os_prio=0 tid=0x0000000001903000 nid=0x251 in Object.wait() [0x00007f366aebe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.Listener.stopListener(Listener.java:222)
- locked <0x00000000b29161d0> (a com.arjuna.ats.internal.arjuna.recovery.Listener)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.shutdown(PeriodicRecovery.java:169)
at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:179)
at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:210)
- locked <0x00000000b0badbb8> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)
at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:194)
at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:73)
at org.jboss.as.txn.service.ArjunaRecoveryManagerService.stop(ArjunaRecoveryManagerService.java:167)
- locked <0x00000000b313ddb0> (a org.jboss.as.txn.service.ArjunaRecoveryManagerService)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000000b351b500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Periodic Recovery" #116 prio=5 os_prio=0 tid=0x0000000002357000 nid=0x2b4 in Object.wait() [0x00007f3661f2d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doSuspendedWait(PeriodicRecovery.java:709)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:346)
- locked <0x00000000b289c898> (a java.lang.Object)
Locked ownable synchronizers:
- None
{code}
[1]
> Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
> ------------------------------------------------------------------------------------------
>
> Key: JBTM-3047
> URL: https://issues.jboss.org/browse/JBTM-3047
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: Tooling
> Affects Versions: 5.9.0.Final
> Reporter: Ondra Chaloupka
> Assignee: Ondra Chaloupka
>
> If {{PeriodicMonitor}} has active unfinished request to start recovery scan
> {code}
> INFO [com.arjuna.ats.arjuna] (Server.Connection:127.0.0.1:36326) ARJUNA012340: RecoveryManager scan scheduled to begin.
> {code}
> and meanwhile [the RecoveryManager is suspended|https://github.com/jbosstm/narayana/blob/5.9.0.Final/ArjunaCore...] then the WorkerService is waiting in deadlock forever and JVM can't be cleanly stopped.
> The whole stopping EAP server stacktrace could be seen at https://issues.jboss.org/browse/CLOUD-2767 the related Narayana related threads are[10].
> It seems to me as the recovery was put to suspended[1] the processing thread waits for the state change[2]. The state change to set to terminated could be done by periodic recovery shutdown[3] but it's stuck on waiting for listners to be stopped[4]. Stopping the listeners could be done by the thread waiting for the state change[5].
> [1] https://github.com/wildfly/wildfly/blob/master/transactions/src/main/java...
> [2] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [3] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [4] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [5] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
> [10]
> {code}
> "Transaction Reaper Worker 0" #121 daemon prio=5 os_prio=0 tid=0x000000000197a000 nid=0x2b6 in Object.wait() [0x00007f3661d2b000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c310> (a java.util.LinkedList)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:328)
> - locked <0x00000000b289c310> (a java.util.LinkedList)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
> Locked ownable synchronizers:
> - None
> "Transaction Reaper" #120 daemon prio=5 os_prio=0 tid=0x0000000001b4b800 nid=0x2b5 in Object.wait() [0x00007f3661e2c000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:90)
> - locked <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46200" #335 daemon prio=5 os_prio=0 tid=0x0000000003853800 nid=0x449 in Object.wait() [0x00007f365ec2c000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "Server.Connection:10.130.0.162:46090" #225 daemon prio=5 os_prio=0 tid=0x0000000003a4c800 nid=0x3ca in Object.wait() [0x00007f365c2ef000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
> - locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
> at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
> Locked ownable synchronizers:
> - None
> "MSC service thread 1-5" #19 prio=5 os_prio=0 tid=0x0000000001903000 nid=0x251 in Object.wait() [0x00007f366aebe000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.Listener.stopListener(Listener.java:222)
> - locked <0x00000000b29161d0> (a com.arjuna.ats.internal.arjuna.recovery.Listener)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.shutdown(PeriodicRecovery.java:169)
> at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:179)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:210)
> - locked <0x00000000b0badbb8> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)
> at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:194)
> at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:73)
> at org.jboss.as.txn.service.ArjunaRecoveryManagerService.stop(ArjunaRecoveryManagerService.java:167)
> - locked <0x00000000b313ddb0> (a org.jboss.as.txn.service.ArjunaRecoveryManagerService)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
> at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000b351b500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> "Periodic Recovery" #116 prio=5 os_prio=0 tid=0x0000000002357000 nid=0x2b4 in Object.wait() [0x00007f3661f2d000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doSuspendedWait(PeriodicRecovery.java:709)
> at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:346)
> - locked <0x00000000b289c898> (a java.lang.Object)
> Locked ownable synchronizers:
> - None
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (JBTM-3047) Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
by Ondra Chaloupka (JIRA)
Ondra Chaloupka created JBTM-3047:
-------------------------------------
Summary: Suspending recovery manager causes deadlock when acive RecoveryMonitor scan request exists
Key: JBTM-3047
URL: https://issues.jboss.org/browse/JBTM-3047
Project: JBoss Transaction Manager
Issue Type: Bug
Components: Tooling
Affects Versions: 5.9.0.Final
Reporter: Ondra Chaloupka
Assignee: Ondra Chaloupka
If {{PeriodicMonitor}} has active unfinished request to start recovery scan
{code}
INFO [com.arjuna.ats.arjuna] (Server.Connection:127.0.0.1:36326) ARJUNA012340: RecoveryManager scan scheduled to begin.
{code}
and meanwhile [the RecoveryManager is suspended|https://github.com/jbosstm/narayana/blob/5.9.0.Final/ArjunaCore...] then the WorkerService is waiting in deadlock forever and JVM can't be cleanly stopped.
The whole stopping EAP server stacktrace could be seen at https://issues.jboss.org/browse/CLOUD-2767 the related Narayana related threads are[10].
It seems to me as the recovery was put to suspended[1] the processing thread waits for the state change[2]. The state change to set to terminated could be done by periodic recovery shutdown[3] but it's stuck on waiting for listners to be stopped[4]. Stopping the listeners could be done by the thread waiting for the state change[5].
[1] https://github.com/wildfly/wildfly/blob/master/transactions/src/main/java...
[2] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[3] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[4] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[5] https://github.com/jbosstm/narayana/blob/master/ArjunaCore/arjuna/classes...
[10]
{code}
"Transaction Reaper Worker 0" #121 daemon prio=5 os_prio=0 tid=0x000000000197a000 nid=0x2b6 in Object.wait() [0x00007f3661d2b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000b289c310> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:328)
- locked <0x00000000b289c310> (a java.util.LinkedList)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:65)
Locked ownable synchronizers:
- None
"Transaction Reaper" #120 daemon prio=5 os_prio=0 tid=0x0000000001b4b800 nid=0x2b5 in Object.wait() [0x00007f3661e2c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:90)
- locked <0x00000000b289c330> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
Locked ownable synchronizers:
- None
"Server.Connection:10.130.0.162:46200" #335 daemon prio=5 os_prio=0 tid=0x0000000003853800 nid=0x449 in Object.wait() [0x00007f365ec2c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
- locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
Locked ownable synchronizers:
- None
"Server.Connection:10.130.0.162:46090" #225 daemon prio=5 os_prio=0 tid=0x0000000003a4c800 nid=0x3ca in Object.wait() [0x00007f365c2ef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.WorkerService.doWork(WorkerService.java:95)
- locked <0x00000000b2916608> (a com.arjuna.ats.internal.arjuna.recovery.WorkerService)
at com.arjuna.ats.internal.arjuna.recovery.Connection.run(Connection.java:88)
Locked ownable synchronizers:
- None
"MSC service thread 1-5" #19 prio=5 os_prio=0 tid=0x0000000001903000 nid=0x251 in Object.wait() [0x00007f366aebe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.Listener.stopListener(Listener.java:222)
- locked <0x00000000b29161d0> (a com.arjuna.ats.internal.arjuna.recovery.Listener)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.shutdown(PeriodicRecovery.java:169)
at com.arjuna.ats.internal.arjuna.recovery.RecoveryManagerImple.stop(RecoveryManagerImple.java:179)
at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:210)
- locked <0x00000000b0badbb8> (a com.arjuna.ats.arjuna.recovery.RecoveryManager)
at com.arjuna.ats.arjuna.recovery.RecoveryManager.terminate(RecoveryManager.java:194)
at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.stop(RecoveryManagerService.java:73)
at org.jboss.as.txn.service.ArjunaRecoveryManagerService.stop(ArjunaRecoveryManagerService.java:167)
- locked <0x00000000b313ddb0> (a org.jboss.as.txn.service.ArjunaRecoveryManagerService)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2150)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2101)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000000b351b500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"Periodic Recovery" #116 prio=5 os_prio=0 tid=0x0000000002357000 nid=0x2b4 in Object.wait() [0x00007f3661f2d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doSuspendedWait(PeriodicRecovery.java:709)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:346)
- locked <0x00000000b289c898> (a java.lang.Object)
Locked ownable synchronizers:
- None
{code}
[1]
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0
[JBoss JIRA] (JBTM-3045) JTA CDI regression (or Standalone JTA 1.2 Quickstart failure)
by Michael Musgrove (JIRA)
[ https://issues.jboss.org/browse/JBTM-3045?page=com.atlassian.jira.plugin.... ]
Michael Musgrove updated JBTM-3045:
-----------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> JTA CDI regression (or Standalone JTA 1.2 Quickstart failure)
> -------------------------------------------------------------
>
> Key: JBTM-3045
> URL: https://issues.jboss.org/browse/JBTM-3045
> Project: JBoss Transaction Manager
> Issue Type: Bug
> Components: Demonstrator
> Affects Versions: 5.9.0.Final
> Reporter: Michael Musgrove
> Priority: Critical
> Fix For: 5.next
>
>
> The recent changes to our JTA CDI support (https://issues.jboss.org/browse/JBTM-3044) have either caused a regression or the Standalone JTA 1.2 Quickstart needs updating. The failure is:
> {quote}
> testMandatoryTransactionWithExistingTransaction(org.jboss.narayana.quickstarts.jta.TestCase) Time elapsed: 0.536 sec <<< ERROR!
> javax.enterprise.inject.CreationException: Not a compound name: TransactionManager
> {quote}
> See for example CI run http://narayanaci1.eng.hst.ams2.redhat.com/job/narayana-quickstarts/159/c...
> If the issue does turn out to be a regression then the priority needs to be marked as Critical.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
6 years, 4 months
- 1
- 0