Seems that cache operations are not properly propagated from one node (1) in the cluster to another (2). Looks like node (2) executes cache operation concurrently so they can be executed in the wrong order.

Here is the test case:

There is a cluster with two nodes.
1) Parent (Customer) and child (Contact) objects are persisted on node (1).
2) Parent and child are loaded on node (2).
3) New child is persisted on node (1).
4) New child is deleted on node (1).
5) Parent and child are loaded on node (2), which results with error:

org.hibernate.ObjectNotFoundException: No row with the given identifier exists: org.hibernate.test.Contact#2

According to production logs it seems that on node (2) PutKeyValueCommand and RemoveCommand are executed in the wrong order and Contact#2 object, that was deleted, stays in the cache. Reason for this could be that Infinispan's "putForExternalRead()" method is used, "...where, concurrent modification and transactions are not a consideration..." (https://issues.jboss.org/browse/JBCACHE-848).

In the test log (attached) it's hard to tell which thread belongs to which "node" in order to see what really happens (I was not able to make "threadNamePrefix" work).

I also had problems setting up interceptors (another Infinispan bug?) in order to synchronize cache operations and make the problem 100% reproducible so you'll have to run test case several times until you reproduce it.