[infinispan-issues] [JBoss JIRA] Commented: (ISPN-691) ReplicationQueue has an out-of-order issue

[craig bomba (JIRA)]

    [ https://jira.jboss.org/browse/ISPN-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558783#action_12558783 ] 

craig bomba commented on ISPN-691:
----------------------------------

Galder, 

After further verification we believe the timer thread did not play a role.  Recall this was a high throughput test.

We believe what happened was that 2 worker threads were involved in all 3 of the cases highlighted by our diagnosis.  Looking at the 2 key scenario, we now believe what happened was the following:
- both threads enter the add method and find that they have exceeded the max elements setting (which for us is 100)
- both threads will call flush as a result
- one of the threads enters flush before the other and get to the drainTo call 
- the other thread entering flush later only sees entries made after the drainTo was done by the prior thread (which is why it only drains/flushes 1)

End result is that the synchonized is needed.  This additional diagnosis only re-confirms that.


> ReplicationQueue has an out-of-order issue
> ------------------------------------------
>
>                 Key: ISPN-691
>                 URL: https://jira.jboss.org/browse/ISPN-691
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Core API
>    Affects Versions: 4.0.0.Final, 4.1.0.Final, 4.2.0.ALPHA3
>            Reporter: craig bomba
>            Assignee: Galder Zamarreño
>             Fix For: 4.2.0.BETA1, 4.2.0.Final
>
>         Attachments: ReplicationQueueImpl.java
>
>
> The ReplicationQueue has an exposure to distributing items out of order.  The ReplicationQueue may flush items in either of 2 ways.  One way is the background thread provided by a ThreadPoolExecutor flushing on a timer.  The other is via the current thread that does a put (which calls ReplicationQueue.add).  In the case of the call to add if it hits the max size (set in the config by replQueueMaxElements) then items may get flushed to other nodes out of order.  This is not evident when a test case only includes puts (new items or updated items in a cache).  Your test must include removals to expose this concern.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira