[JBoss JIRA] (JGRP-2360) DeadLock while acqiring a distributed lock consecutively by the same thread in a loop
by Daniel Klosinski (Jira)
[ https://issues.jboss.org/browse/JGRP-2360?page=com.atlassian.jira.plugin.... ]
Daniel Klosinski commented on JGRP-2360:
----------------------------------------
Hi [~belaban],
Thanks for resolving this issue. It works for me now using the latest version.
Would it be possible to deliver the fix also to the version 3.x.x. ?
> DeadLock while acqiring a distributed lock consecutively by the same thread in a loop
> -------------------------------------------------------------------------------------
>
> Key: JGRP-2360
> URL: https://issues.jboss.org/browse/JGRP-2360
> Project: JGroups
> Issue Type: Bug
> Affects Versions: 3.6.18, 4.1.1
> Environment: JGroups-4.1.1-Final
> Red Hat 4.4.7-23
> JDK 1.8.0_202
> Reporter: Daniel Klosinski
> Assignee: Bela Ban
> Priority: Major
> Fix For: 4.1.2
>
> Attachments: DLTest.java, DistributedLockRepoducer.zip, log.log
>
>
> Deadlock intermittently happens when trying to acquire a distributed lock by the same VM, consecutively by the same thread in a loop. Here is a code snippet for which this issue can occur :
> {code}
> for(String s : list){
> Lock lock=lock_service.getLock("test_lock_name");
> lock.lock();
> //perform bussines logic
> lock.unlock();
> }
> {code}
> Running such loop I am getting dead look after a few loop iterations. In the attached logs program hanged after 3 iterations
> During the troubleshooting, I found out that lock_id is not being incremented for the new distributed lock. In the first two loop iterations everything was fine. At the third iteration lock_id didn't get increased:
> {code}
> 2019-07-15 16:03:32 TRACE CENTRAL_LOCK:163 - svc-2-sps-34594 --> svc-1-sps-4688: GRANT_LOCK[test_lock_name, lock_id=1, owner=svc-2-sps-34594::1]
> 2019-07-15 16:03:32 TRACE CENTRAL_LOCK:163 - svc-2-sps-34594 <-- svc-1-sps-4688: LOCK_GRANTED[test_lock_name, lock_id=1, owner=svc-2-sps-34594::1, sender=svc-1-sps-4688]
> 2019-07-15 16:03:32 TRACE CENTRAL_LOCK:163 - svc-2-sps-34594 --> svc-1-sps-4688: RELEASE_LOCK[test_lock_name, lock_id=1, owner=svc-2-sps-34594::1]
> 2019-07-15 16:03:32 TRACE CENTRAL_LOCK:163 - svc-2-sps-34594 <-- svc-1-sps-4688: RELEASE_LOCK_OK[test_lock_name, lock_id=1, owner=svc-2-sps-34594::1, sender=svc-1-sps-4688]
> 2019-07-15 16:03:32 TRACE CENTRAL_LOCK:163 - svc-2-sps-34594 --> svc-1-sps-4688: GRANT_LOCK[test_lock_name, lock_id=2, owner=svc-2-sps-34594::1]
> 2019-07-15 16:03:32 TRACE CENTRAL_LOCK:163 - svc-2-sps-34594 <-- svc-1-sps-4688: LOCK_GRANTED[test_lock_name, lock_id=2, owner=svc-2-sps-34594::1, sender=svc-1-sps-4688]
> 2019-07-15 16:03:32 TRACE CENTRAL_LOCK:163 - svc-2-sps-34594 --> svc-1-sps-4688: RELEASE_LOCK[test_lock_name, lock_id=2, owner=svc-2-sps-34594::1]
> 2019-07-15 16:03:32 TRACE CENTRAL_LOCK:163 - svc-2-sps-34594 <-- svc-1-sps-4688: RELEASE_LOCK_OK[test_lock_name, lock_id=2, owner=svc-2-sps-34594::1, sender=svc-1-sps-4688]
> 2019-07-15 16:03:32 TRACE CENTRAL_LOCK:163 - svc-2-sps-34594 --> svc-1-sps-4688: GRANT_LOCK[test_lock_name, lock_id=2, owner=svc-2-sps-34594::1]
> 2019-07-15 16:03:32 TRACE CENTRAL_LOCK:163 - svc-2-sps-34594 <-- svc-1-sps-4688: CREATE_LOCK[test_lock_name, owner=svc-2-sps-34594::1, sender=svc-1-sps-4688]
> 2019-07-15 16:03:32 TRACE CENTRAL_LOCK:163 - svc-2-sps-34594 <-- svc-1-sps-4688: LOCK_GRANTED[test_lock_name, lock_id=2, owner=svc-2-sps-34594::1, sender=svc-1-sps-4688]
> {code}
> I've added few extra loggers into Jgroups-4.1.1.Final code and I realized that the second client lock was not removed from the client lock table before the creation of 3rd client lock. The issue lays in below piece of code. Owner consists of address and threadID. If the same thread, on the same VM, creates distributed lock consecutively and if there is an existing entry in the client lock table for the same owner, the new lock won't be created. The old client lock will be used to acquire a new distributed lock :
> {code}
> protected synchronized ClientLock getLock(String name, Owner owner, boolean create_if_absent) {
> Map<Owner,ClientLock> owners=table.get(name);
> if(owners == null) {
> if(!create_if_absent)
> return null;
> owners=Util.createConcurrentMap(20);
> Map<Owner,ClientLock> existing=table.putIfAbsent(name,owners);
> if(existing != null)
> owners=existing;
> }
> ClientLock lock=owners.get(owner);
> if(lock == null) {
> if(!create_if_absent)
> return null;
> lock=createLock(name, owner);
> owners.put(owner, lock);
> }
> return lock;
> }
> {code}
> I believe that this issue was introduced by the fix for JGRP-2234 and it is caused by the race condition. The logic that deletes client lock from the client lock table is now executed when the client's VM receives RELEASE_LOCK_OK message from the coordinator. Previously this deletion was executed by the thread in which unlock() method was called. Now, it is executed by the separate thread which handles RELEASE_LOCK_OK from the coordinator and this is why we have a race condition here. Here is a sequence which leads to deadlock:
> 1. Create client lock (lock_id=2)
> 2. Send GRANT_LOCK (lock_id=2) to coordinator
> 3. Receive LOCK_GRANTED (lock_id=2) from coordinator
> 4. Send RELEASE_LOCK (lock_id=2) to coordinator
> 5. Call look() method in the same thread (new client lock won't be created as there is an existing entry in the client lock table for this owner)
> 6. Receive RELEASE_LOCK_OK and delete client lock from client lock table.
> 7. Send GRANT_LOCK (lock_id=2) to coordinator
> 8. Receive LOCK_GRANTED (lock_id=2) from coordinator
> 9. No entry in the client lock table. It's not possible to get the thread which needs to be notified.
> I am attaching a simple program which can be used to reproduce and generated logs.
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 11 months
[JBoss JIRA] (WFLY-12445) Document some quickstarts to use new openshift templates
by Jean Francois Denise (Jira)
Jean Francois Denise created WFLY-12445:
-------------------------------------------
Summary: Document some quickstarts to use new openshift templates
Key: WFLY-12445
URL: https://issues.jboss.org/browse/WFLY-12445
Project: WildFly
Issue Type: Enhancement
Components: Documentation, Quickstarts
Reporter: Jean Francois Denise
Assignee: Eduardo Martins
A subset of openshift quickstarts can take benefit of chained build and usage of galleon offered by new template. This should be documented in README.
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 11 months
[JBoss JIRA] (WFCORE-4633) StandaloneScriptTestCase fails on IBM JDK8
by Richard Opalka (Jira)
Richard Opalka created WFCORE-4633:
--------------------------------------
Summary: StandaloneScriptTestCase fails on IBM JDK8
Key: WFCORE-4633
URL: https://issues.jboss.org/browse/WFCORE-4633
Project: WildFly Core
Issue Type: Bug
Components: Scripts
Environment: java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 8.0.5.36 - pxa6480sr5fp36-20190510_01(SR5 FP36))
IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20190502_415899 (JIT enabled, AOT enabled)
OpenJ9 - 46e57f9
OMR - 06a046a
IBM - 0b909bf)
JCL - 20190409_01 based on Oracle jdk8u211-b25
Reporter: Richard Opalka
Assignee: James Perkins
Fix For: 10.0.0.Beta5
[INFO] Running org.wildfly.scripts.test.StandaloneScriptTestCase
[ERROR] Tests run: 10, Failures: 1, Errors: 0, Skipped: 8, Time elapsed: 20.724 s <<< FAILURE! - in org.wildfly.scripts.test.StandaloneScriptTestCase
[ERROR] testBashScript[1](org.wildfly.scripts.test.StandaloneScriptTestCase) Time elapsed: 3.729 s <<< FAILURE!
java.lang.AssertionError: Missing gc.log.0.current file in /home/opalka/git/redhat/wildfly-core/testsuite/scripts/target/wildfly-core/standalone/log
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.wildfly.scripts.test.StandaloneScriptTestCase.testScript(StandaloneScriptTestCase.java:135)
at org.wildfly.scripts.test.ScriptTestCase.executeTests(ScriptTestCase.java:257)
at org.wildfly.scripts.test.ScriptTestCase.testBashScript(ScriptTestCase.java:170)
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 11 months
[JBoss JIRA] (WFLY-12444) EJB/JNDI over HTTP-Invoker Throws CommunicationException instead of AuthenticationException
by Tomas Hofman (Jira)
[ https://issues.jboss.org/browse/WFLY-12444?page=com.atlassian.jira.plugin... ]
Tomas Hofman moved JBEAP-17505 to WFLY-12444:
---------------------------------------------
Project: WildFly (was: JBoss Enterprise Application Platform)
Key: WFLY-12444 (was: JBEAP-17505)
Workflow: GIT Pull Request workflow (was: CDW with loose statuses v1)
Component/s: EJB
(was: EJB)
Affects Version/s: (was: 7.2.0.GA)
(was: 7.2.2.GA)
> EJB/JNDI over HTTP-Invoker Throws CommunicationException instead of AuthenticationException
> -------------------------------------------------------------------------------------------
>
> Key: WFLY-12444
> URL: https://issues.jboss.org/browse/WFLY-12444
> Project: WildFly
> Issue Type: Bug
> Components: EJB
> Environment: * Red Hat JBoss Enterprise Application Server (JBoss EAP) 7.2
> * EJB / JNDI
> * undertow http-invoker
> * Authentication Failure
> Reporter: Tomas Hofman
> Assignee: Tomas Hofman
> Priority: Major
>
> Switching from http-remoting to the http-invoker when calling an EJB, if an authentication failure occurs, the exception wrapped in the EJBException has changed.
> It was a javax.naming.AuthenticationException before, but now it's throwing a CommunicationException even though the log shows it's getting a 401 from the server:
> WFHTTP000005: Invalid response code 401 (full response ClientResponse{responseHeaders={www-authenticate=[Basic realm="MyRealm"], content-length=[77], content-type=[text/html], date=[Thu, 28 Jul 2019 12:18:43 GMT]}, responseCode=401, status='', protocol=HTTP/2.0})
> Before, AuthenticationException could be used for logic flow, but communication exception is too general.
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 11 months
[JBoss JIRA] (WFLY-12444) EJB/JNDI over HTTP-Invoker Throws CommunicationException instead of AuthenticationException
by Tomas Hofman (Jira)
[ https://issues.jboss.org/browse/WFLY-12444?page=com.atlassian.jira.plugin... ]
Tomas Hofman updated WFLY-12444:
--------------------------------
Affects Version/s: 17.0.1.Final
> EJB/JNDI over HTTP-Invoker Throws CommunicationException instead of AuthenticationException
> -------------------------------------------------------------------------------------------
>
> Key: WFLY-12444
> URL: https://issues.jboss.org/browse/WFLY-12444
> Project: WildFly
> Issue Type: Bug
> Components: EJB
> Affects Versions: 17.0.1.Final
> Environment: * Red Hat JBoss Enterprise Application Server (JBoss EAP) 7.2
> * EJB / JNDI
> * undertow http-invoker
> * Authentication Failure
> Reporter: Tomas Hofman
> Assignee: Tomas Hofman
> Priority: Major
>
> Switching from http-remoting to the http-invoker when calling an EJB, if an authentication failure occurs, the exception wrapped in the EJBException has changed.
> It was a javax.naming.AuthenticationException before, but now it's throwing a CommunicationException even though the log shows it's getting a 401 from the server:
> WFHTTP000005: Invalid response code 401 (full response ClientResponse{responseHeaders={www-authenticate=[Basic realm="MyRealm"], content-length=[77], content-type=[text/html], date=[Thu, 28 Jul 2019 12:18:43 GMT]}, responseCode=401, status='', protocol=HTTP/2.0})
> Before, AuthenticationException could be used for logic flow, but communication exception is too general.
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 11 months