[jbosscache-dev] Critical CacheMarshaller issue

A nasty bug, spotted by someone in the user forum (initially as a CCE) http://jira.jboss.org/jira/browse/JBCACHE-1211 Copying from the JIRA: "This is a nasty. What started life as an optimisation for certain types of objects in a marshalled stream (Fqn, GlobalTransactio, String and Serializable) has become a major limitation in that a single stream can only hold up to 32767 different (not equal()) instances of such objects. Basically the optimisation was, for example, instead of writing "hello" to a stream twice, just write it once and use a reference for all subsequent times. Unfortunately this reference was encoded as a short, hence the limitation of 32767. Fixing this will definitely break wire compatibility with JBoss Cache 2.0.0, although JBC does allow backward compatibility by specifying replication version in your configuration, thanks to the VersionAwareMarshaller. " So I guess this mandates the need for a CacheMarshaller210. The question is how do we fix this. The obvious thing is to replace the short references with integers. The 2 ^ 31 - 1 number of references this would allow should be plenty! The drawback though, is larger streams. 4-byte refs instead of 2-byte refs can be an unnecessary overhead especially if objects aren't repeated much. One thing we could do is just limit ref counting to Fqn and GlobalTransaction types, of which we know can often be repeated in a modification list, and where the optimisation can benefit, and assume it isn't worth optimising in the same way for user data (Strings and Serializables). Or maybe we should drop the ref counting altogether, save a few more bytes' worth of payload. Thoughts? -- Manik Surtani Lead, JBoss Cache manik(a)jboss.org