[ https://issues.jboss.org/browse/JGRP-1718?page=com.atlassian.jira.plugin.... ] Rich DiCroce commented on JGRP-1718: ------------------------------------ I spent the whole day today trying to nail this down, and I have some more information now. Unfortunately, this information only makes the issue more baffling. To answer your questions: - As part of my investigation, I added some logging of the byte arrays being processed. I have confirmed that, when an error occurs, the byte array I get in my RequestHandler does not match the byte array I passed to the Message constructor. I have also determined that when a byte array gets corrupted, I can send a second Message with the exact same byte array and request options, and the byte array in the second Message is readable. I have also made sure I can deserialize an object immediately after serializing it (before sending the Message), and I have also tried deserializing a second time after the first time failed (when receiving the Message; the second time also fails). - I tried using normal Java serialization, and that does not seem to exhibit the problem. This does not necessarily implicate Kryo or exhonerate JGroups, however. See below. - Kryo has input/output buffers, specifically, the Input and Output classes. I am not reusing instances of those classes. I am reusing instances of the Kryo class itself, which handles the actual serialization/deserialization. Kryo is not thread-safe, so I am obtaining the instances from a ThreadLocal to prevent concurrent use. I have determined that the byte arrays are being corrupted in a way that's too consistent to be random chance. It seems to be triggered by a particular bit sequence, specifically: {code}00000001{code} followed by {code}11111001{code} or {code}11100101{code} These two bytes can be anywhere in the array. The corruption that occurs is always the same: the first bit of the second byte is flipped from 1 to 0. Unfortunately, these bit sequences only seem to be part of the cause. I believe a race condition of some sort is also at work here, because I've been unable to create a simple test that reproduces the problem. Another twist that only makes the issue weirder: changing the loopback setting of the UDP transport affects the frequency of the corruption. With loopback = false, the problem occurs much more rarely, to the point that I initially thought it had eliminated the problem altogether. I only observed one corruption during my testing with loopback = false, and curiously, a third bit pattern that I have not observed with loopback = true got corrupted: {code}00000001 11000100{code} Again, the first bit of the second byte was flipped from 1 to 0. I also logged the number of nodes that were unable to read the message, and there was only one error despite the cluster having two nodes and the message being a multicast to all nodes. This implies that the corruption occurs when the message is received, not when it is sent. Having looked at the JGroups source code, it seems like there would be less concurrency with loopback = false since a multicast message would not be immediately sent back up the stack, potentially in parallel with messages being received from other nodes. Hence the speculation about a race condition. Does any of this make any sense to you? Because my brain feels like it's about to melt. :-) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira