[ https://issues.jboss.org/browse/JGRP-1331?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1331: -------------------------------- For marshalling purposes, we could add a flag 'options' to the serialized bytes (Digest.writeTo(),readFrom()): - NORMAL: 3 longs follow - ZERO: nothing follows (all 3 longs are 0) - COMPRESSED_SHORT_INT: 1 long, 1 short, 1 int. The first value is uncompressed, the second is a short diff to the second, the 3rd is an int diff to the first. - COMPRESSED_SHORT_SHORT: (similar to above) - COMPRESSED_INT_SHORT: - COMPRESSED_INT_INT: If we have 100000 (low), 120000 (highest_delivered) and 125000 (highest_received), we'd send the digest as: | COMPRESSED_SHORT_SHORT | 100000 (long) | 20000 (short) | 5000 (short) | As an additional optimization, we could use the special marker -1 to mean that the value is the same as the previous value, e.g. for (100000,120000,120000): | 100000 | 20000 (short) | -1 (short) | These optimizations are possible because we know that the highest_received and highest_delivered seqnos are usually very close to each other (unless there are gaps and retransmission). The low seqno is a bit further apart, depending on settings of STABLE. If we can get rid of low, then we could even further optimize the serialization of highest_received and highest_delivered seqnos ! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1331?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1331: -------------------------------- Digest actually consists of a ConcurrentHashMap of senders (Address) and (low,high_recv,high_del) entries (Entry). The ConcurrentHashMap is (1) absolutely not necessary, as Entry is immutable and (2) overkill as it adds to the size of a Digest in memory ! SOLUTION: replace ConcurrentHashMap with something more specific and less memory hungry, e.g. similar to org.jgroups.util.Headers. Even a HashMap would be less memory hungry than a ConcurrentHashMap ! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1331?page=com.atlassian.jira.plugin.... ] Bela Ban commented on JGRP-1331: -------------------------------- OK, 'low' was removed from Digest.Entry; now we have only highest_delivered and highest_received left, and can therefore apply the optimizations described above -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1331?page=com.atlassian.jira.plugin.... ] Bela Ban edited comment on JGRP-1331 at 8/23/11 7:19 AM: --------------------------------------------------------- Digest actually consists of a ConcurrentHashMap of senders (Address) and (low,high_recv,high_del) entries (Entry). The ConcurrentHashMap is (1) absolutely not necessary, as Entry is immutable and (2) overkill as it adds to the size of a Digest in memory ! SOLUTION: replace ConcurrentHashMap with something more specific and less memory hungry, e.g. similar to org.jgroups.util.Headers. Even a HashMap would be less memory hungry than a ConcurrentHashMap ! We could use 2 arrays: one for senders and one for Digests (or byte[] buffers, see below). Note that digests are usually *not* accessed randomly, but most access patterns iterate through a digest, and get all senders and their associated Digest.Entry's in turn. This favors use of 2 arrays rather than a hashmap ! was (Author: belaban): Digest actually consists of a ConcurrentHashMap of senders (Address) and (low,high_recv,high_del) entries (Entry). The ConcurrentHashMap is (1) absolutely not necessary, as Entry is immutable and (2) overkill as it adds to the size of a Digest in memory ! SOLUTION: replace ConcurrentHashMap with something more specific and less memory hungry, e.g. similar to org.jgroups.util.Headers. Even a HashMap would be less memory hungry than a ConcurrentHashMap ! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1331?page=com.atlassian.jira.plugin.... ] Bela Ban updated JGRP-1331: --------------------------- Summary: Digest: remove low seqno and compress highest_delivered and highest_received seqnos (was: Digest: remove low seqno) Priority: Major (was: Minor) Description: We don't need Digest.Entry.low_seqno for NakReceiverWindow anymore. We only need it for the Retransmitter implementations. If we can remove that last dependency, e.g. by storing the last seqno removed from a retransmitter, we can remove low_seqno from a digest. This is significant, for example if we have 100 nodes, a digest includes 300 longs:(low+highest_received+highest_delivered) * 100. If we remove low_seqno, we only have 200 longs. Also, if we could compress the highest_delivered and highest_received seqnos, we could save even more space to send digests across the network. This is important for large clusters, to reduce the size of STABLE messages and views. was: We don't need Digest.Entry.low_seqno for NakReceiverWindow anymore. We only need it for the Retransmitter implementations. If we can remove that last dependency, e.g. by storing the last seqno removed from a retransmitter, we can remove low_seqno from a digest. This is significant, for example if we have 100 nodes, a digest includes 300 longs:(low+highest_received+highest_delivered) * 100. If we remove low_seqno, we only have 200 longs. (These will still be further compressed in a different JIRA). This is important for large clusters, to reduce the size of STABLE messages and views. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JGRP-1331?page=com.atlassian.jira.plugin.... ] Bela Ban resolved JGRP-1331. ---------------------------- Resolution: Done Further optimizations are going to be done in 3.1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira