Not sure I understand. Why does this need to be as complex as this? When you say encoding of data, surely all this data is, is a byte[]? On 18 Feb 2010, at 15:26, Galder Zamarreno wrote: > Hi, > > I've been looking again at the encoding of data in Hot Rod > (http://community.jboss.org/wiki/HotRodProtocol) and there's a few > things > I'm not too happy about or they're not totally clear: > > 1. The type appears to be pretty wasteful because the majority of types > cannot combined with others, for example: You cannot have a type that is > Boolean and Long at the same time, or Double and Character. However, you > can potentially have an Arrays of Serialized. So, I propose instead > separating between a meta type and type. > > Meta types would be: Array, Map, Primitive and Compressed. The meta type > would be encoded using bit ops, so you could combine them in diff ways, > i.e. Array of primitives. I don't think we should support combinations > off > diff collections, i.e Map of Array, or Array of Maps. It would > complexity > and don't forsee an immediate req for this. > > Type would be: Byte, Boolean, Character, String, Date, Double, Float, > Integer, Long, Short, Serialized, StringBuilder, and Any. These would > literals from 1 to N. Note that I've added Any to separate between two > different collections. For example, if you send an Array of String, each > individual element just follows together with its size. However, if you > send an Array of Any, each individual entry must define its type. > > For Maps, we've got two options: First, no type assumptions made and let > each key/value define its own type. Or allow maps meta-type definitions > to > be followed by not one but two type fields. Even if the map was of mixed > types, you could have Any, Any. My preference is for the latter. > > Both type and metatype would be variable length integers. > > 2. Serialized will be stored as byte[] internally, no attempt to > unmarshalling will be done in Hot Rod. Clients decide how they wanna > marshall this Serialized types. They just need to gives us a byte[] and > its length. > > 3. To clarify something that Alex mentioned in the previous encoding > data > email, Arrays and Maps are followed by the number of items in Hot Rod > and > not the number of bytes. In case of Arrays of Any, each individual field > gives us its size, type and the data. In case of Array of Booleans, each > individual field comes with size and data. Size might have been optional > in each field of Array of Boolean, but it simplifies deadling with Array > of Serialized, where each individual field is a byte[] of arbitrary > length. > > For Maps, a similar thing happens. If we have a Map of String, Boolean, > we > get key/value pair like: k=[size+data]v=[size+data]. If it's a Map of > Any, > Any, we get k=[type+size+data],v=[size+data] -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org

On 18 Feb 2010, at 15:51, Galder Zamarreno wrote: > That's indeed an option but then debugging of logs could be quite a > nightmare if all we store in the cache a byte[] and there was no > readable > presentation for that. This would also be a problem if in the future we > decided to allow some tool to inspect the contents of the cache. > > The idea behind the encoding is to at least have a way to encode the > simplest objects that can be passed: primitives and Strings and > collections of these. Obviously, if users want to simply store a blog, > i.e. an image, then that's a byte[] and nothing can be done about that. > > I agree that maybe some stuff can be simplified. We could simply support > primitives and Strings and treat the rest as a byte[]. That makes more sense, although you still then have the issue of collections and arrays.

Personally, I'd be in favour of just using byte[]s, unless we have a really good argument against this. No point in reimplementing a marshalling layer and object encoding for HotRod.

Cheers Manik > > Thoughts? > > On Thu, 18 Feb 2010 16:40:32 +0100, Manik Surtani <manik(a)jboss.org&gt; > wrote: > >> Not sure I understand. Why does this need to be as complex as this? >> When you say encoding of data, surely all this data is, is a byte[]? >> >> On 18 Feb 2010, at 15:26, Galder Zamarreno wrote: >> >>> Hi, >>> >>> I've been looking again at the encoding of data in Hot Rod >>> (http://community.jboss.org/wiki/HotRodProtocol) and there's a few >>> things >>> I'm not too happy about or they're not totally clear: >>> >>> 1. The type appears to be pretty wasteful because the majority of >>> types >>> cannot combined with others, for example: You cannot have a type that >>> is >>> Boolean and Long at the same time, or Double and Character. However, >>> you >>> can potentially have an Arrays of Serialized. So, I propose instead >>> separating between a meta type and type. >>> >>> Meta types would be: Array, Map, Primitive and Compressed. The meta >>> type >>> would be encoded using bit ops, so you could combine them in diff >>> ways, >>> i.e. Array of primitives. I don't think we should support combinations >>> off >>> diff collections, i.e Map of Array, or Array of Maps. It would >>> complexity >>> and don't forsee an immediate req for this. >>> >>> Type would be: Byte, Boolean, Character, String, Date, Double, Float, >>> Integer, Long, Short, Serialized, StringBuilder, and Any. These would >>> literals from 1 to N. Note that I've added Any to separate between two >>> different collections. For example, if you send an Array of String, >>> each >>> individual element just follows together with its size. However, if >>> you >>> send an Array of Any, each individual entry must define its type. >>> >>> For Maps, we've got two options: First, no type assumptions made and >>> let >>> each key/value define its own type. Or allow maps meta-type >>> definitions >>> to >>> be followed by not one but two type fields. Even if the map was of >>> mixed >>> types, you could have Any, Any. My preference is for the latter. >>> >>> Both type and metatype would be variable length integers. >>> >>> 2. Serialized will be stored as byte[] internally, no attempt to >>> unmarshalling will be done in Hot Rod. Clients decide how they wanna >>> marshall this Serialized types. They just need to gives us a byte[] >>> and >>> its length. >>> >>> 3. To clarify something that Alex mentioned in the previous encoding >>> data >>> email, Arrays and Maps are followed by the number of items in Hot Rod >>> and >>> not the number of bytes. In case of Arrays of Any, each individual >>> field >>> gives us its size, type and the data. In case of Array of Booleans, >>> each >>> individual field comes with size and data. Size might have been >>> optional >>> in each field of Array of Boolean, but it simplifies deadling with >>> Array >>> of Serialized, where each individual field is a byte[] of arbitrary >>> length. >>> >>> For Maps, a similar thing happens. If we have a Map of String, >>> Boolean, >>> we >>> get key/value pair like: k=[size+data]v=[size+data]. If it's a Map of >>> Any, >>> Any, we get k=[type+size+data],v=[size+data] >> >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> > > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 18 Feb 2010, at 16:39, Galder Zamarreno wrote: > On Thu, 18 Feb 2010 17:11:49 +0100, Manik Surtani <manik(a)jboss.org&gt; > wrote: > >> >> On 18 Feb 2010, at 15:51, Galder Zamarreno wrote: >> >>> That's indeed an option but then debugging of logs could be quite a >>> nightmare if all we store in the cache a byte[] and there was no >>> readable >>> presentation for that. This would also be a problem if in the future >>> we >>> decided to allow some tool to inspect the contents of the cache. >>> >>> The idea behind the encoding is to at least have a way to encode the >>> simplest objects that can be passed: primitives and Strings and >>> collections of these. Obviously, if users want to simply store a blog, >>> i.e. an image, then that's a byte[] and nothing can be done about >>> that. >>> >>> I agree that maybe some stuff can be simplified. We could simply >>> support >>> primitives and Strings and treat the rest as a byte[]. >> >> That makes more sense, although you still then have the issue of >> collections and arrays. > > What issue exactly? Anyone sending collections or arrays needs to > marshall > them into a byte[] and pass that. I don't see a problem. Mircea, maybe > you > have thought of how you expect to present this to the clients? Well, if you want to store strings and primitives as they are for for debugging purposes, then you lose that benefit the moment someone passes in an array of strings or primitives. :) > >> Personally, I'd be in favour of just using byte[]s, unless we have a >> really good argument against this. No point in reimplementing a >> marshalling layer and object encoding for HotRod. > > Is debugging not enough of a reason to at least support the basic types? > Imagine having to debug through a Infinispan Hot Rod server storing > String > k,v pairs but having to figure out each String's byte[] format to find > the > one where things fail. Anything other than primitive types we have no > other option. We'd just have to make sure that if client logs are > available, things are logged properly. At the very least, we should make > it easy to pass Strings and maybe forget about the rest of types. You will also need to deal with collections of {Strings, primitives, byte[]s}. See above. > >> >> Cheers >> Manik >> >>> >>> Thoughts? >>> >>> On Thu, 18 Feb 2010 16:40:32 +0100, Manik Surtani <manik(a)jboss.org&gt; >>> wrote: >>> >>>> Not sure I understand. Why does this need to be as complex as this? >>>> When you say encoding of data, surely all this data is, is a byte[]? >>>> >>>> On 18 Feb 2010, at 15:26, Galder Zamarreno wrote: >>>> >>>>> Hi, >>>>> >>>>> I've been looking again at the encoding of data in Hot Rod >>>>> (http://community.jboss.org/wiki/HotRodProtocol) and there's a few >>>>> things >>>>> I'm not too happy about or they're not totally clear: >>>>> >>>>> 1. The type appears to be pretty wasteful because the majority of >>>>> types >>>>> cannot combined with others, for example: You cannot have a type >>>>> that >>>>> is >>>>> Boolean and Long at the same time, or Double and Character. However, >>>>> you >>>>> can potentially have an Arrays of Serialized. So, I propose instead >>>>> separating between a meta type and type. >>>>> >>>>> Meta types would be: Array, Map, Primitive and Compressed. The meta >>>>> type >>>>> would be encoded using bit ops, so you could combine them in diff >>>>> ways, >>>>> i.e. Array of primitives. I don't think we should support >>>>> combinations >>>>> off >>>>> diff collections, i.e Map of Array, or Array of Maps. It would >>>>> complexity >>>>> and don't forsee an immediate req for this. >>>>> >>>>> Type would be: Byte, Boolean, Character, String, Date, Double, >>>>> Float, >>>>> Integer, Long, Short, Serialized, StringBuilder, and Any. These >>>>> would >>>>> literals from 1 to N. Note that I've added Any to separate between >>>>> two >>>>> different collections. For example, if you send an Array of String, >>>>> each >>>>> individual element just follows together with its size. However, if >>>>> you >>>>> send an Array of Any, each individual entry must define its type. >>>>> >>>>> For Maps, we've got two options: First, no type assumptions made and >>>>> let >>>>> each key/value define its own type. Or allow maps meta-type >>>>> definitions >>>>> to >>>>> be followed by not one but two type fields. Even if the map was of >>>>> mixed >>>>> types, you could have Any, Any. My preference is for the latter. >>>>> >>>>> Both type and metatype would be variable length integers. >>>>> >>>>> 2. Serialized will be stored as byte[] internally, no attempt to >>>>> unmarshalling will be done in Hot Rod. Clients decide how they wanna >>>>> marshall this Serialized types. They just need to gives us a byte[] >>>>> and >>>>> its length. >>>>> >>>>> 3. To clarify something that Alex mentioned in the previous encoding >>>>> data >>>>> email, Arrays and Maps are followed by the number of items in Hot >>>>> Rod >>>>> and >>>>> not the number of bytes. In case of Arrays of Any, each individual >>>>> field >>>>> gives us its size, type and the data. In case of Array of Booleans, >>>>> each >>>>> individual field comes with size and data. Size might have been >>>>> optional >>>>> in each field of Array of Boolean, but it simplifies deadling with >>>>> Array >>>>> of Serialized, where each individual field is a byte[] of arbitrary >>>>> length. >>>>> >>>>> For Maps, a similar thing happens. If we have a Map of String, >>>>> Boolean, >>>>> we >>>>> get key/value pair like: k=[size+data]v=[size+data]. If it's a Map >>>>> of >>>>> Any, >>>>> Any, we get k=[type+size+data],v=[size+data] >>>> >>>> >>>> -- >>>> Manik Surtani >>>> manik(a)jboss.org >>>> Lead, Infinispan >>>> Lead, JBoss Cache >>>> http://www.infinispan.org >>>> http://www.jbosscache.org >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> Galder Zamarreño >>> Sr. Software Engineer >>> Infinispan, JBoss Cache >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> manik(a)jboss.org >> Lead, Infinispan >> Lead, JBoss Cache >> http://www.infinispan.org >> http://www.jbosscache.org >> >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

This means that they byte representation has to contain the object type, and if it is a more complex type, it's size. We can use a custom encoding, or we can use serialization, there just isn't much else. Serialization struck me as a little heavy weight, and our application has a lot of primitive values, and arrays of primitive values, and a few maps that get cached, so this encoding scheme is working well for us.