[JBoss JIRA] (ISPN-10073) Nested async operations with HotRod client hang
by William Burns (Jira)
[ https://issues.jboss.org/browse/ISPN-10073?page=com.atlassian.jira.plugin... ]
William Burns commented on ISPN-10073:
--------------------------------------
This is caused due to blocking the callback thread in the get invocation. The problem is that the accept Consumer is invoked using the netty thread for the given channel, which means the putIfAbsentAsync call here is not yet sent and thus the get will always block forever. We need to isolate the callbacks to not be done on such important threads.
> Nested async operations with HotRod client hang
> -----------------------------------------------
>
> Key: ISPN-10073
> URL: https://issues.jboss.org/browse/ISPN-10073
> Project: Infinispan
> Issue Type: Bug
> Components: Hot Rod
> Affects Versions: 9.4.10.Final
> Reporter: Dejan Bosanac
> Assignee: William Burns
> Priority: Major
>
> We noticed some issued with the HotRod client, when trying to use nested async calls. Originally, this was discovered in the context of Vert.x tests, but below you can find the smallest reproducer code that I could create (without any Vert.x dependencies).
> {code:java}
> //DefaultCacheManager cm = new DefaultCacheManager();
> //Cache<String, String> cache = cm.createCache("default", new ConfigurationBuilder().build());
> RemoteCacheManager cm = new RemoteCacheManager();
> RemoteCache<String, String> cache = cm.getCache("default");
> final CompletableFuture put = new CompletableFuture();
> cache.putIfAbsentAsync("A", "A").thenAccept(resultA -> {
> final CompletableFuture nested = new CompletableFuture();
> cache.putIfAbsentAsync("B", "B").thenAccept(resultB -> {
> nested.complete("B");
> });
> try {
> nested.get();
> } catch (Exception e) {
> e.printStackTrace();
> }
> put.complete("B");
> });
> put.get();
> {code}
> This would basically hang and the nested operation would never receive response. What I noticed in the original test is that response would become available after the framework timeout the test.
> Everything works as expected with embedded cache.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months
[JBoss JIRA] (ISPN-10073) Nested async operations with HotRod client hang
by William Burns (Jira)
[ https://issues.jboss.org/browse/ISPN-10073?page=com.atlassian.jira.plugin... ]
William Burns commented on ISPN-10073:
--------------------------------------
Okay, I can reproduce this with a mainline method connecting to just an empty server. I have to investigate what the difference is that is causing this.
> Nested async operations with HotRod client hang
> -----------------------------------------------
>
> Key: ISPN-10073
> URL: https://issues.jboss.org/browse/ISPN-10073
> Project: Infinispan
> Issue Type: Bug
> Components: Hot Rod
> Affects Versions: 9.4.10.Final
> Reporter: Dejan Bosanac
> Assignee: William Burns
> Priority: Major
>
> We noticed some issued with the HotRod client, when trying to use nested async calls. Originally, this was discovered in the context of Vert.x tests, but below you can find the smallest reproducer code that I could create (without any Vert.x dependencies).
> {code:java}
> //DefaultCacheManager cm = new DefaultCacheManager();
> //Cache<String, String> cache = cm.createCache("default", new ConfigurationBuilder().build());
> RemoteCacheManager cm = new RemoteCacheManager();
> RemoteCache<String, String> cache = cm.getCache("default");
> final CompletableFuture put = new CompletableFuture();
> cache.putIfAbsentAsync("A", "A").thenAccept(resultA -> {
> final CompletableFuture nested = new CompletableFuture();
> cache.putIfAbsentAsync("B", "B").thenAccept(resultB -> {
> nested.complete("B");
> });
> try {
> nested.get();
> } catch (Exception e) {
> e.printStackTrace();
> }
> put.complete("B");
> });
> put.get();
> {code}
> This would basically hang and the nested operation would never receive response. What I noticed in the original test is that response would become available after the framework timeout the test.
> Everything works as expected with embedded cache.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months
[JBoss JIRA] (ISPN-10073) Nested async operations with HotRod client hang
by Jean-Baptiste Trystram (Jira)
[ https://issues.jboss.org/browse/ISPN-10073?page=com.atlassian.jira.plugin... ]
Jean-Baptiste Trystram commented on ISPN-10073:
-----------------------------------------------
How can I check if the call is blocked by the server ? Nothing obvious in the logs.
Dejam and I got this reproduced by simply putting this code in a plain java main app, I can't think of any other operation blocking.
> Nested async operations with HotRod client hang
> -----------------------------------------------
>
> Key: ISPN-10073
> URL: https://issues.jboss.org/browse/ISPN-10073
> Project: Infinispan
> Issue Type: Bug
> Components: Hot Rod
> Affects Versions: 9.4.10.Final
> Reporter: Dejan Bosanac
> Assignee: William Burns
> Priority: Major
>
> We noticed some issued with the HotRod client, when trying to use nested async calls. Originally, this was discovered in the context of Vert.x tests, but below you can find the smallest reproducer code that I could create (without any Vert.x dependencies).
> {code:java}
> //DefaultCacheManager cm = new DefaultCacheManager();
> //Cache<String, String> cache = cm.createCache("default", new ConfigurationBuilder().build());
> RemoteCacheManager cm = new RemoteCacheManager();
> RemoteCache<String, String> cache = cm.getCache("default");
> final CompletableFuture put = new CompletableFuture();
> cache.putIfAbsentAsync("A", "A").thenAccept(resultA -> {
> final CompletableFuture nested = new CompletableFuture();
> cache.putIfAbsentAsync("B", "B").thenAccept(resultB -> {
> nested.complete("B");
> });
> try {
> nested.get();
> } catch (Exception e) {
> e.printStackTrace();
> }
> put.complete("B");
> });
> put.get();
> {code}
> This would basically hang and the nested operation would never receive response. What I noticed in the original test is that response would become available after the framework timeout the test.
> Everything works as expected with embedded cache.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months
[JBoss JIRA] (ISPN-10073) Nested async operations with HotRod client hang
by William Burns (Jira)
[ https://issues.jboss.org/browse/ISPN-10073?page=com.atlassian.jira.plugin... ]
William Burns edited comment on ISPN-10073 at 3/22/19 9:52 AM:
---------------------------------------------------------------
I added
{code}
public void testNestedPutIfAbsentCalls() throws ExecutionException, InterruptedException {
CompletableFuture<Object> future = cache.putIfAbsentAsync("A", "A").thenCompose(resultA ->
cache.putIfAbsentAsync("B", "B"));
future.get();
}
public void testNestedPutIfAbsentCalls2() throws ExecutionException, InterruptedException {
final CompletableFuture put = new CompletableFuture();
cache.putIfAbsentAsync("A", "A").thenAccept(resultA -> {
final CompletableFuture nested = new CompletableFuture();
cache.putIfAbsentAsync("B", "B").thenAccept(resultB -> {
nested.complete("B");
});
try {
nested.get();
} catch (Exception e) {
e.printStackTrace();
}
put.complete("B");
});
put.get();
}
{code}
to the RemoteAsyncAPITest class and I am unable to reproduce the issue. I also tried with and without force return value and both passed.
Is it possible the putIfAbsent call was being blocked in the server? Maybe another operation was holding the lock?
was (Author: william.burns):
I added
{code}
public void testNestedPutIfAbsentCalls() throws ExecutionException, InterruptedException {
CompletableFuture<Object> future = cache.putIfAbsentAsync("A", "A").thenCompose(resultA ->
cache.putIfAbsentAsync("B", "B"));
future.get();
}
public void testNestedPutIfAbsentCalls2() throws ExecutionException, InterruptedException {
final CompletableFuture put = new CompletableFuture();
cache.putIfAbsentAsync("A", "A").thenAccept(resultA -> {
final CompletableFuture nested = new CompletableFuture();
cache.putIfAbsentAsync("B", "B").thenAccept(resultB -> {
nested.complete("B");
});
try {
nested.get();
} catch (Exception e) {
e.printStackTrace();
}
put.complete("B");
});
put.get();
}
{code}
to the RemoteAsyncAPITest class and I am unable to reproduce the issue. I also tried with and without force return value and it both still passed.
Is it possible the putIfAbsent call was being blocked in the server? Maybe another operation was holding the lock?
> Nested async operations with HotRod client hang
> -----------------------------------------------
>
> Key: ISPN-10073
> URL: https://issues.jboss.org/browse/ISPN-10073
> Project: Infinispan
> Issue Type: Bug
> Components: Hot Rod
> Affects Versions: 9.4.10.Final
> Reporter: Dejan Bosanac
> Assignee: William Burns
> Priority: Major
>
> We noticed some issued with the HotRod client, when trying to use nested async calls. Originally, this was discovered in the context of Vert.x tests, but below you can find the smallest reproducer code that I could create (without any Vert.x dependencies).
> {code:java}
> //DefaultCacheManager cm = new DefaultCacheManager();
> //Cache<String, String> cache = cm.createCache("default", new ConfigurationBuilder().build());
> RemoteCacheManager cm = new RemoteCacheManager();
> RemoteCache<String, String> cache = cm.getCache("default");
> final CompletableFuture put = new CompletableFuture();
> cache.putIfAbsentAsync("A", "A").thenAccept(resultA -> {
> final CompletableFuture nested = new CompletableFuture();
> cache.putIfAbsentAsync("B", "B").thenAccept(resultB -> {
> nested.complete("B");
> });
> try {
> nested.get();
> } catch (Exception e) {
> e.printStackTrace();
> }
> put.complete("B");
> });
> put.get();
> {code}
> This would basically hang and the nested operation would never receive response. What I noticed in the original test is that response would become available after the framework timeout the test.
> Everything works as expected with embedded cache.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months
[JBoss JIRA] (ISPN-10073) Nested async operations with HotRod client hang
by William Burns (Jira)
[ https://issues.jboss.org/browse/ISPN-10073?page=com.atlassian.jira.plugin... ]
William Burns commented on ISPN-10073:
--------------------------------------
I added
{code}
public void testNestedPutIfAbsentCalls() throws ExecutionException, InterruptedException {
CompletableFuture<Object> future = cache.putIfAbsentAsync("A", "A").thenCompose(resultA ->
cache.putIfAbsentAsync("B", "B"));
future.get();
}
public void testNestedPutIfAbsentCalls2() throws ExecutionException, InterruptedException {
final CompletableFuture put = new CompletableFuture();
cache.putIfAbsentAsync("A", "A").thenAccept(resultA -> {
final CompletableFuture nested = new CompletableFuture();
cache.putIfAbsentAsync("B", "B").thenAccept(resultB -> {
nested.complete("B");
});
try {
nested.get();
} catch (Exception e) {
e.printStackTrace();
}
put.complete("B");
});
put.get();
}
{code}
to the RemoteAsyncAPITest class and I am unable to reproduce the issue. I also tried with and without force return value and it both still passed.
Is it possible the putIfAbsent call was being blocked in the server? Maybe another operation was holding the lock?
> Nested async operations with HotRod client hang
> -----------------------------------------------
>
> Key: ISPN-10073
> URL: https://issues.jboss.org/browse/ISPN-10073
> Project: Infinispan
> Issue Type: Bug
> Components: Hot Rod
> Affects Versions: 9.4.10.Final
> Reporter: Dejan Bosanac
> Assignee: William Burns
> Priority: Major
>
> We noticed some issued with the HotRod client, when trying to use nested async calls. Originally, this was discovered in the context of Vert.x tests, but below you can find the smallest reproducer code that I could create (without any Vert.x dependencies).
> {code:java}
> //DefaultCacheManager cm = new DefaultCacheManager();
> //Cache<String, String> cache = cm.createCache("default", new ConfigurationBuilder().build());
> RemoteCacheManager cm = new RemoteCacheManager();
> RemoteCache<String, String> cache = cm.getCache("default");
> final CompletableFuture put = new CompletableFuture();
> cache.putIfAbsentAsync("A", "A").thenAccept(resultA -> {
> final CompletableFuture nested = new CompletableFuture();
> cache.putIfAbsentAsync("B", "B").thenAccept(resultB -> {
> nested.complete("B");
> });
> try {
> nested.get();
> } catch (Exception e) {
> e.printStackTrace();
> }
> put.complete("B");
> });
> put.get();
> {code}
> This would basically hang and the nested operation would never receive response. What I noticed in the original test is that response would become available after the framework timeout the test.
> Everything works as expected with embedded cache.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months
[JBoss JIRA] (ISPN-9714) Update CacheNotifier to return CompletionStage
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-9714?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-9714:
------------------------------------
Forgot to answer here: a listener that needs to do some blocking stuff should use its own thread pool to avoid blocking Infinispan threads. However, long running listeners will have a dramatic impact on performance, regardless of whether they're blocking or not.
I'm optimistic about users who need prodding to change their code, because blocking listeners will stand out in thread dumps.
> Update CacheNotifier to return CompletionStage
> ----------------------------------------------
>
> Key: ISPN-9714
> URL: https://issues.jboss.org/browse/ISPN-9714
> Project: Infinispan
> Issue Type: Sub-task
> Components: Core, Listeners
> Reporter: William Burns
> Assignee: William Burns
> Priority: Major
> Fix For: 10.0.0.Beta3
>
>
> We need to update CacheNotifier to return CompletionStage for all its appropriate methods. We also need to update all the internals to use these appropriately.
> Not all notification usages may provide support non blocking, but our listener internals should support non blocking listeners for all.
> The simplest way internally is to treat all current listeners as "alien" and invoke them in the notification thread pool. If it is sync we would wait for this task to complete. We would also now allow a listener to return a CompletionStage. If this is returned we will use this operate in a non blocking way.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months
[JBoss JIRA] (ISPN-9714) Update CacheNotifier to return CompletionStage
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-9714?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-9714:
-------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> Update CacheNotifier to return CompletionStage
> ----------------------------------------------
>
> Key: ISPN-9714
> URL: https://issues.jboss.org/browse/ISPN-9714
> Project: Infinispan
> Issue Type: Sub-task
> Components: Core, Listeners
> Reporter: William Burns
> Assignee: William Burns
> Priority: Major
> Fix For: 10.0.0.Beta3
>
>
> We need to update CacheNotifier to return CompletionStage for all its appropriate methods. We also need to update all the internals to use these appropriately.
> Not all notification usages may provide support non blocking, but our listener internals should support non blocking listeners for all.
> The simplest way internally is to treat all current listeners as "alien" and invoke them in the notification thread pool. If it is sync we would wait for this task to complete. We would also now allow a listener to return a CompletionStage. If this is returned we will use this operate in a non blocking way.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months
[JBoss JIRA] (ISPN-10079) Upgrade to jboss-modules 1.8.7.Final
by Dan Berindei (Jira)
Dan Berindei created ISPN-10079:
-----------------------------------
Summary: Upgrade to jboss-modules 1.8.7.Final
Key: ISPN-10079
URL: https://issues.jboss.org/browse/ISPN-10079
Project: Infinispan
Issue Type: Component Upgrade
Components: Server
Affects Versions: 9.4.10.Final, 10.0.0.Beta2
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 10.0.0.Beta3, 9.4.11.Final
Includes fix for MODULES-375
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months