[infinispan-dev] Concurrency problem on org.infinispan.transaction.xa.TransactionTable

[Sebastiano Peluso]

Hi all!
We are Diego and Sebastiano and we are working in the Cloud-TM project; 
in particular we are focusing on designing new replication schemes in 
Infinispan. During our code inspection we have noted that Infinispan can 
suffer from a concurrency issue:  multiple threads in execution on the 
same local cache instance can access without any synchronization scheme 
on a org.infinispan.transaction.xa.TransactionTable's field named 
localTransactions; the problem is that the aforementioned 
localTransactions field is an instance of HashMap, which isn't 
thread-safe. We have noticed the problem during some intensive tests 
(executed with a version of RadarGun modified by us in order to enable 
transactions executions) which stressed heavily the cache with 
concurrent threads: in some cases all threads on some nodes blocked in 
an infinite loop on a get() operation on the aforementioned HashMap 
instance.
We attach the dump relevant to a single blocked node and you can note 
that each thread, namely a Stresser, is blocked on the get() operation.
We have noted the same concurrency problem on the field 
remoteTransactions in the same object.
To partially validate our hypothesis we have replaced the type HashMap 
with ConcurrentHashMap and everithing works fine.
What is your opinion about this?

Best regards
    Diego, Sebastiano





"Stresser-7" prio=10 tid=0x00007fb2c0295000 nid=0x5bd3 runnable 
[0x00007fb2c6bea000]
    java.lang.Thread.State: RUNNABLE
     at java.util.HashMap.get(HashMap.java:303)
     at 
org.infinispan.transaction.xa.TransactionTable.getXaCacheAdapter(TransactionTable.java:222)
     at 
org.infinispan.context.InvocationContextContainerImpl.createInvocationContext(InvocationContextContainerImpl.java:67)
     at 
org.infinispan.CacheDelegate.getInvocationContext(CacheDelegate.java:285)
     at org.infinispan.CacheDelegate.get(CacheDelegate.java:199)
     at 
org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:99)
     at 
org.radargun.stressors.PutGetStressor$Stresser.run(PutGetStressor.java:367)

"Stresser-6" prio=10 tid=0x00007fb2c0293000 nid=0x5bd2 runnable 
[0x00007fb2c6ceb000]
    java.lang.Thread.State: RUNNABLE
     at java.util.HashMap.get(HashMap.java:303)
     at 
org.infinispan.transaction.xa.TransactionTable.getXaCacheAdapter(TransactionTable.java:222)
     at 
org.infinispan.context.InvocationContextContainerImpl.createInvocationContext(InvocationContextContainerImpl.java:67)
     at 
org.infinispan.CacheDelegate.getInvocationContext(CacheDelegate.java:285)
     at org.infinispan.CacheDelegate.get(CacheDelegate.java:199)
     at 
org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:99)
     at 
org.radargun.stressors.PutGetStressor$Stresser.run(PutGetStressor.java:367)

"Stresser-5" prio=10 tid=0x00007fb2c0292000 nid=0x5bd1 runnable 
[0x00007fb2c6dec000]
    java.lang.Thread.State: RUNNABLE
     at java.util.HashMap.get(HashMap.java:303)
     at 
org.infinispan.transaction.xa.TransactionTable.getXaCacheAdapter(TransactionTable.java:222)
     at 
org.infinispan.context.InvocationContextContainerImpl.createInvocationContext(InvocationContextContainerImpl.java:67)
     at 
org.infinispan.CacheDelegate.getInvocationContext(CacheDelegate.java:285)
     at org.infinispan.CacheDelegate.get(CacheDelegate.java:199)
     at 
org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:99)
     at 
org.radargun.stressors.PutGetStressor$Stresser.run(PutGetStressor.java:367)

"Stresser-4" prio=10 tid=0x00007fb2c0291800 nid=0x5bd0 runnable 
[0x00007fb2c77f6000]
    java.lang.Thread.State: RUNNABLE
     at java.util.HashMap.get(HashMap.java:303)
     at 
org.infinispan.transaction.xa.TransactionTable.getXaCacheAdapter(TransactionTable.java:222)
     at 
org.infinispan.context.InvocationContextContainerImpl.createInvocationContext(InvocationContextContainerImpl.java:67)
     at 
org.infinispan.CacheDelegate.getInvocationContext(CacheDelegate.java:285)
     at org.infinispan.CacheDelegate.get(CacheDelegate.java:199)
     at 
org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:99)
     at 
org.radargun.stressors.PutGetStressor$Stresser.run(PutGetStressor.java:367)

"Stresser-3" prio=10 tid=0x00007fb2c0290800 nid=0x5bcf runnable 
[0x00007fb2c78f7000]
    java.lang.Thread.State: RUNNABLE
     at java.util.HashMap.get(HashMap.java:303)
     at 
org.infinispan.transaction.xa.TransactionTable.getXaCacheAdapter(TransactionTable.java:222)
     at 
org.infinispan.context.InvocationContextContainerImpl.createInvocationContext(InvocationContextContainerImpl.java:67)
     at 
org.infinispan.CacheDelegate.getInvocationContext(CacheDelegate.java:285)
     at org.infinispan.CacheDelegate.get(CacheDelegate.java:199)
     at 
org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:99)
     at 
org.radargun.stressors.PutGetStressor$Stresser.run(PutGetStressor.java:367)

"Stresser-2" prio=10 tid=0x00007fb2c0044000 nid=0x5bce runnable 
[0x00007fb2c76f5000]
    java.lang.Thread.State: RUNNABLE
     at java.util.HashMap.get(HashMap.java:303)
     at 
org.infinispan.transaction.xa.TransactionTable.getXaCacheAdapter(TransactionTable.java:222)
     at 
org.infinispan.context.InvocationContextContainerImpl.createInvocationContext(InvocationContextContainerImpl.java:67)
     at 
org.infinispan.CacheDelegate.getInvocationContext(CacheDelegate.java:285)
     at org.infinispan.CacheDelegate.get(CacheDelegate.java:199)
     at 
org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:99)
     at 
org.radargun.stressors.PutGetStressor$Stresser.run(PutGetStressor.java:367)

"Stresser-1" prio=10 tid=0x00007fb2c0043000 nid=0x5bcd runnable 
[0x00007fb2c74f3000]
    java.lang.Thread.State: RUNNABLE
     at java.util.HashMap.get(HashMap.java:303)
     at 
org.infinispan.transaction.xa.TransactionTable.getXaCacheAdapter(TransactionTable.java:222)
     at 
org.infinispan.context.InvocationContextContainerImpl.createInvocationContext(InvocationContextContainerImpl.java:67)
     at 
org.infinispan.CacheDelegate.getInvocationContext(CacheDelegate.java:285)
     at org.infinispan.CacheDelegate.get(CacheDelegate.java:199)
     at 
org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:99)
     at 
org.radargun.stressors.PutGetStressor$Stresser.run(PutGetStressor.java:367)

"Stresser-0" prio=10 tid=0x00007fb2c0048000 nid=0x5bcc runnable 
[0x00007fb2c75f4000]
    java.lang.Thread.State: RUNNABLE
     at java.util.HashMap.get(HashMap.java:303)
     at 
org.infinispan.transaction.xa.TransactionTable.getXaCacheAdapter(TransactionTable.java:222)
     at 
org.infinispan.context.InvocationContextContainerImpl.createInvocationContext(InvocationContextContainerImpl.java:67)
     at 
org.infinispan.CacheDelegate.getInvocationContext(CacheDelegate.java:285)
     at org.infinispan.CacheDelegate.get(CacheDelegate.java:199)
     at 
org.radargun.cachewrappers.InfinispanWrapper.get(InfinispanWrapper.java:99)
     at 
org.radargun.stressors.PutGetStressor$Stresser.run(PutGetStressor.java:367)