[jboss-jira] [JBoss JIRA] (JGRP-1604) SEQUENCER problems and slowness

[Bela Ban (JIRA)]

    [ https://issues.jboss.org/browse/JGRP-1604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758360#comment-12758360 ] 

Bela Ban commented on JGRP-1604:
--------------------------------

The culprit is delivery_table and canDeliver(), which calls size() on the ConcurrentSkipListSet: size() has a cost linear to the number of elements in the set, and size() is called on *every* delivery.

Suggested SOLUTION:
# Make canDeliver() fast, e.g. use a linked hashmap with seqnos as keys for delivery_table
# (Indepenent from 1) Make addition to forward-table blocking, so we can't run out of memory when many threads are adding messages to the forward-table. The blocking should be based on the number of bytes in the forward-table (Message.getLength())
                
> SEQUENCER problems and slowness
> -------------------------------
>
>                 Key: JGRP-1604
>                 URL: https://issues.jboss.org/browse/JGRP-1604
>             Project: JGroups
>          Issue Type: Enhancement
>            Reporter: Bela Ban
>            Assignee: Bela Ban
>             Fix For: 3.3
>
>
> On high volumes of messages (e.g. MPerf), SEQUENCER runs out of memory. 
> The culprit is the forward-table, which is unbounded, and if messages are added as a faster rate than the time it takes to forward the message to the coordinator (sequencer) and for it to be received as a multicast and subsequently be removed from the forward-table, memory increases.
> This also makes SEQUENCER slow.
> SOLUTION:
> * Look into bounding the forward-table, e.g. define the max number of bytes to be in the table at any given time
> * When a message is to be added, its length is checked (Message.getLength())
> ** If the length is 0 or length + accumulated bytes is less than max: add, else block
> * A received message (sent by self) causes the entry to be removed from the forward-table and its length to be subtracted from accumulated bytes (unblocking potentially blocked threads)
> We should also look at delivery table and see if we can make it simpler, e.g. by running a gathering round when a new sequencer is chosen, determining the highest seqnos of all members

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira