]
Peter Levart commented on ISPN-8889:
------------------------------------
Hi,
I won't pretend that I understand all that. I'm new to Infinispan. I'm
migrating an application that uses Oracle Coherence to Infinispan. For pre-loading entries
into a cache, I have been using Map.putAll(chunk) with chunks of entries because that was
a quicker way to put entries into a cache on Coherence than putting individual key/value
pairs. I suspect this holds for Infinispan too, since it uses special command for that.
I don't quite understand your last sentence: "Luckily, adding synchronization on
the context for the remoteGet handling should be easy."
Are you thinking of putting-in some synchronization just for remoteGet ? Where would you
put it ? Would puting synchronization on
NonTxInvocationContext.[lookupEntry|removeLookedUpEntry|putLookedUpEntry|getLookedUpEntries]
be to strong and not needed?
Regards, Peter
Data race in NonTxInvocationContext
-----------------------------------
Key: ISPN-8889
URL:
https://issues.jboss.org/browse/ISPN-8889
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 9.2.0.CR2, 9.1.6.Final
Environment: Java 8 (Oracle JDK 8), Solaris SPARC
Reporter: Peter Levart
Labels: data-race
Attachments: DataRacer.java, DataRacer.java
Got the following exceptions starting up an Infinispan node while joining the cluster:
{noformat}
17:10:59.012 [remote-thread--p2-t8] ERROR
org.infinispan.interceptors.impl.InvocationContextInterceptor - ISPN000136: Error
executing command PutMapCommand, writing keys [11906696627, 11906696626, 11906696625,
11906696624, 11906696631, 11906696630, 11906696629, 11906696628...<9992 other
elements>]
java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to
java.util.HashMap$TreeNode
at java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1832) ~[?:1.8.0_162]
at java.util.HashMap$TreeNode.treeify(HashMap.java:1949) ~[?:1.8.0_162]
at java.util.HashMap$TreeNode.split(HashMap.java:2175) ~[?:1.8.0_162]
at java.util.HashMap.resize(HashMap.java:714) ~[?:1.8.0_162]
at java.util.HashMap.putVal(HashMap.java:663) ~[?:1.8.0_162]
at java.util.HashMap.put(HashMap.java:612) ~[?:1.8.0_162]
at
org.infinispan.context.impl.NonTxInvocationContext.putLookedUpEntry(NonTxInvocationContext.java:48)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.container.EntryFactoryImpl.wrapExternalEntry(EntryFactoryImpl.java:143)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.distribution.BaseDistributionInterceptor.wrapRemoteEntry(BaseDistributionInterceptor.java:222)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.distribution.BaseDistributionInterceptor.lambda$remoteGet$1(BaseDistributionInterceptor.java:192)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
~[?:1.8.0_162]
at
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
~[?:1.8.0_162]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_162]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
~[?:1.8.0_162]
at
org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:66)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:102)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.jgroups.StaggeredRequest.onResponse(StaggeredRequest.java:50)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:53)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1302)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1205)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1340)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at org.jgroups.JChannel.up(JChannel.java:819)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.Discovery.up(Discovery.java:267)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at
org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
{noformat}
Immediately suspected that plain HashMap is being modified from multiple threads without
proper synchronization. But since I'm new to Infinispan, I can't know whether the
usage of plain HashMap in NonTxInvocationContext is intentional and Infinispan code is
relying on some external synchronization to be performed by other parts of system that
calls
NonTxInvocationContext.[lookupEntry|removeLookedUpEntry|putLookedUpEntry|getLookedUpEntries].
In any way, if this external synchronization is taking place, it has a bug and allows
calling these methods concurrently without proper synchronization.
I did some experiments to prove that and also to obtain stack traces of at least two of
involved threads, but trivial things that I tried instrumenting those methods all involved
some kind of synchronization and the symptom disappeared. I had to be extra smart to
obtain the stack traces of two threads without inter-thread synchronization so that I
could detect the data race. Here's how I instrumented NonTxInvocationContext code:
{noformat}
Index: core/src/main/java/org/infinispan/context/impl/NonTxInvocationContext.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- core/src/main/java/org/infinispan/context/impl/NonTxInvocationContext.java (revision
6014e751c20daba1f00e23168281e02afb209905)
+++ core/src/main/java/org/infinispan/context/impl/NonTxInvocationContext.java (date
1519744397000)
@@ -22,6 +22,7 @@
private Set<Object> lockedKeys;
private Object lockOwner;
+ private final DataRacer dataRacer = new DataRacer();
public NonTxInvocationContext(int numEntries, Address origin) {
super(origin);
@@ -35,22 +36,26 @@
@Override
public CacheEntry lookupEntry(Object k) {
+ dataRacer.detect();
return lookedUpEntries.get(k);
}
@Override
public void removeLookedUpEntry(Object key) {
+ dataRacer.detectAndWrite();
lookedUpEntries.remove(key);
}
@Override
public void putLookedUpEntry(Object key, CacheEntry e) {
+ dataRacer.detectAndWrite();
lookedUpEntries.put(key, e);
}
@Override
@SuppressWarnings("unchecked")
public Map<Object, CacheEntry> getLookedUpEntries() {
+ dataRacer.detect();
return (Map<Object, CacheEntry>)
(lookedUpEntries == null ?
Collections.emptyMap() : lookedUpEntries);
{noformat}
Attached, you will find the DataRacer source. While running with such patched Infinispan,
I got some of the following:
{noformat}
java.lang.IllegalThreadStateException: Thread[remote-thread--p2-t4,5,main]: data race
detected with writer of state: 271f7814a414 - see suppressed for writer thread stack
trace
at org.infinispan.context.impl.DataRacer.detect(DataRacer.java:37)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.context.impl.NonTxInvocationContext.lookupEntry(NonTxInvocationContext.java:39)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.addRemoteGet(NonTxDistributionInterceptor.java:405)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.handleRemoteReadWriteManyCommand(NonTxDistributionInterceptor.java:385)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.handleReadWriteManyCommand(NonTxDistributionInterceptor.java:303)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutMapCommand(NonTxDistributionInterceptor.java:143)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:80)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:98)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForManyEntriesCommand(EntryWrappingInterceptor.java:614)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutMapCommand(EntryWrappingInterceptor.java:385)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:80)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.handleWriteManyCommand(NonTransactionalLockingInterceptor.java:53)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutMapCommand(AbstractLockingInterceptor.java:179)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:80)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:306)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:252)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.statetransfer.StateTransferInterceptor.visitPutMapCommand(StateTransferInterceptor.java:102)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:80)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutMapCommand(CacheMgmtInterceptor.java:154)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:80)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.DDAsyncInterceptor.visitPutMapCommand(DDAsyncInterceptor.java:90)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:80)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:234)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommandAsync(BaseRpcInvokingCommand.java:63)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.commands.remote.SingleRpcCommand.invokeAsync(SingleRpcCommand.java:57)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Suppressed: org.infinispan.context.impl.DataRacer$StackTrace:
Thread[jgroups-13,tbd2-40757,5,main], state: 271f7814a414
at org.infinispan.context.impl.DataRacer.write(DataRacer.java:59)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.context.impl.DataRacer.detectAndWrite(DataRacer.java:49)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.context.impl.NonTxInvocationContext.putLookedUpEntry(NonTxInvocationContext.java:51)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.container.EntryFactoryImpl.wrapExternalEntry(EntryFactoryImpl.java:143)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.distribution.BaseDistributionInterceptor.wrapRemoteEntry(BaseDistributionInterceptor.java:222)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.interceptors.distribution.BaseDistributionInterceptor.lambda$remoteGet$1(BaseDistributionInterceptor.java:192)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
~[?:1.8.0_162]
at
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
~[?:1.8.0_162]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_162]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
~[?:1.8.0_162]
at
org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:66)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:102)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.jgroups.StaggeredRequest.onResponse(StaggeredRequest.java:50)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:53)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1302)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1205)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$200(JGroupsTransport.java:123)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.receive(JGroupsTransport.java:1340)
~[infinispan-core-9.2.0-SNAPSHOT.jar:9.2.0-SNAPSHOT]
at org.jgroups.JChannel.up(JChannel.java:819)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.FlowControl.up(FlowControl.java:343)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:864)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1002)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:728)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:383)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:119)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:199)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:252)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.MERGE3.up(MERGE3.java:276)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.Discovery.up(Discovery.java:267)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at
org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
~[jgroups-4.0.10.Final.jar:4.0.10.Final]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_162]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
{noformat}
So here you have it. Either using plain HashMap in NonTxInvocationContext is wrong and
ConcurrentHashMap should be used instead or some external synchronization has a bug and is
inadequate. I hope this helps to fix it.