I think we should differentiate two use cases: - serialize / marshall basic types or a composition of basic types (JDK's basic wrapper classes + a few of the collection types) - serialize / marshall custom object structures I don't see #2 as fully solvable as some language don't support some concepts other languages offer. In this area a non-portable solution seems ok. However #1 is an important use case that should be addressed in a portable way. In particular OGM does deconstruct an object into it's composing structures and pass an AtomicMap made of these basic components.

On 21 juil. 2010, at 13:06, Galder Zamarreño wrote: > Hi all, > > Re: https://jira.jboss.org/browse/ISPN-508. This is a bit of a long one, so take your time :) > > Over past few days I've been trying to come up with a Infinispan Marshaller implementation based on one of those portable serialization libraries (i.e. protobuf, thrift...etc) and I've reached the conclusion that it is possible to do so but requires a fair bit of work on our side and performance/usability would decrease. Before explaining my findings, let me explain the model that I'm trying to achieve: > > I want to build a generic marshalling/unmarshalling implementation that's based on language neutral serialization libraries and which does not require the unmarshalling code to have knowledge of the target type. Think about Java Serialization or JBoss Marshalling where the code written can be generic enough w/o needing to know what we're trying to deserialize. The payload in these cases has enough information for the underlying library to figure out the type, instantiate it and populate it accordingly. > > The problem is that there's no such library in the space of portable serialization libraries that provides this out of the box and let me explain why: > > Pretty much any code using those libraries must use class type or schema name to deserialize the payload. However, Infinispan based Marshaller does not have such information when it's trying to deserialize the payload. Infinispan Marshaller based on JDK serialization or JBoss Marshalling uses information in the payload to instantiate the object. However, none of the libraries out there use this mechanism and instead some (i.e. Protobufs) force the client code to do things like: Pojo.parseFrom(byte[]) to generate instances of Pojo. In these cases for example, nothing stops you from writing a UTF-8 string with the class name and put it at the beginning of the payload so the deserialization part can be class agnostic, but this payload would not be portable. What would a python client do with a String containing the java class name? > > Based on the FileDescriptorSet information in http://code.google.com/apis/protocolbuffers/docs/techniques.html I was able to hack something that might work in a portable way. Given a FileDescriptorSet generated a class compile time, I was able to match the protobuf name of a class with its java counter part. So, before writing the protobuf generated byte[], i prepend it with protobuf class name so that when reading, I can take the name, get the java class name and using reflection call parseFrom method to convert the byte[] into a pojo. Note that DynamicMessage class hinted in the the techniques page won't work cos it cannot create instances of pojos. It can only create generic objects with fields that are accessed in a reflection style. > > I also looked at what Avro offers but it does not fully fit either. They have a reflection based serialization mechanism that doesn't require any precompilation, but it requires some kind of type knowledge on the client code to deserialize, plus Avro themselves recommend against it and I'm not sure how performant it'd be. Avro also includes other marshalling mechanisms called specific (like protobufs one with precompiled classes) and generic used to build dynamic objects on the fly. None of these two fit the bill. The specific one is like protobufs with the disadvantage of having ugly code like http://is.gd/dzwj3 where a static object has a strong reference to a <String,Class> CHM, which would leak in an AS env. > > Thrift has the same problems as stated for Protobufs but coudn't see an equivalent way to get find the file descriptor set. Docu is way below what Protobuf offers and latest version which is 0.2 has issues generating classes as stated in JIRA. > > MessagePack has the capability to deserialize an object given a String representation of the schema, so a solution like the protobuf one might be hackable. However, the generated classes do not have an equals implementation (??) which is rather odd. Maybe it's due to lack of maturity? Latest version is 0.3, so that might explain it. API wise, MessagePack provides the API that suits best to what we'd want to do and avoids having to use reflection to resolve the payload. However, looking around I couldn't see similar API for the python language for example and similar to Protobuf, we'd have to prepend the schema name key to the payload to then have the reading part lookup the entire schema based on it. > > So, I can see two solutions here bearing in mind that pretty much any solution would require precompiling some classes: > > - Try to build a marshaller using Protobuf or MessagePack where we enhance it to pass a string key that permits the reading part to deserialize the payload in a generic way. > > - Or try to build some wikis on how to integrate Protobuf/Thrift/MessagePack with Hot Rod client so that they can generate the byte[] with these libraries and pass it to the corresponding Hot Rod client. For the moment, we'd do this for the Hot Rod java client. We would add more info once other language clients are available. The reason I said about potentially showing various libraries is cos whereas Protobuf only supports Java/Python/C++, Thrift supports loads more languages ( C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml) > > In terms of usability , I think the second option is a bit better because building the marshaller requires steps for clients to generate the FileDescriptorSet and somehow pass this to Infinispan so that the marshaller can use it. The second option has none of this and the only added code on the client side is retrieving the byte[] and calling Pojo.parseFrom(byte[]). Performance wise, the 2nd option would be better cos there's no extra hacking needed to pass the String key around and so less bytes are sent around, and no need to use reflection to resolve the payload. > > It's a pity that I couldn't find a tool that fully fits our use case and I wonder whether the need to deal with different languages makes coming up with such solution difficult. I believe reflection based instantiation is present in C++ or Python but not sure about other languages. It's also true that our use case is very specific where we're building a tool where we don't have control over what people will put in the cache. Generics could give us some hints for example but it's not mandatory and would not solve the issue entirely. > > Thoughts? > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 21 juil. 2010, at 15:23, Galder Zamarreño wrote: > I'm not so sure I understand the type of solution that would work for #2 though. A java client can generate a payload that can be read by the java server. So, a non-portable solution could just use normal serialization to take an example. However, what would you do with a python client that sends a custom object? How is the java server gonna transform that into a java class? You'd be screwed but that's my point it's not useful that often. But when it's useful, I kinda like the approach taken by Coherence: ie a special (de)serializer implementation for a given conceptual type: the user develop the native type for each of its client language + the serializer / deserializer for this type.

In your example and with this method, you would get a python code reading the stack of serialized primitive types and populate the equivalent python type.

We're back to the case #1 in essence where the stored data is really an intermediary format and the conversion to the final object is handled by manual code. Note that they don't support circularity, which indeed is a bitch.

_______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Ok, dealing with simple types/collections is clear. We delegate to the corresponding library and that's it. As far as custom types that are controlled by user, so I've gone and compared 3 different methods to deal with them: Option 1. User/client code generates the byte[] from a precompiled class. Code example: http://pastebin.com/iqX9rVqV Option 2. User/client gives us the Pojo and our marshaller converts it into a byte[] with the help of the precompiled class methods. Code example: http://pastebin.com/FTTjcfR4 Option 3. User/client gives us the Pojo and our marshaller converts it into a byte using client provided serializer/deserializer impl that relies on portable simple type marshaller. I don't have a code example but it'd be very similar to option 2 with the following changes: At startup, ids and class names would be mapped from a given configuration. At writing time, the id would be written and the rest would be delegated to user provided impl. At reading time, once the id has been determined, the class would be instantiated and it would delegate to it the rest of the reading part. With this in mind, I've created a spreadsheet comparing them based on different factors: http://spreadsheets.google.com/pub?key=0Ag5RGdzR_GsldEJkUEExX2JPTHdsSF93M... I'm leaning towards 1 for its simplicity (less code and less configuration in spite of required precompilation) and performance (no extra bytes, no reflection or class loading). Thoughts? On Jul 22, 2010, at 3:11 AM, Mircea Markus wrote: > > On 21 Jul 2010, at 14:40, Emmanuel Bernard wrote: > >> >> On 21 juil. 2010, at 15:23, Galder Zamarreño wrote: >>> I'm not so sure I understand the type of solution that would work for #2 though. A java client can generate a payload that can be read by the java server. So, a non-portable solution could just use normal serialization to take an example. However, what would you do with a python client that sends a custom object? How is the java server gonna transform that into a java class? >> >> You'd be screwed but that's my point it's not useful that often. >> >> But when it's useful, I kinda like the approach taken by Coherence: ie a special (de)serializer implementation for a given conceptual type: the user develop the native type for each of its client language + the serializer / deserializer for this type. > they also need the xml describing the object (their version of .proto). Compared with Coherenece's POF, I think using ProtoBuf is better as you only need to define metadata: then types + serializer are generated for you using protoc. Less effort. > Where they might be better is if their core is aware of POF definition and it can index properties of portable objects - which would be cool. >> In your example and with this method, you would get a python code reading the stack of serialized primitive types and populate the equivalent python type. > I think an equivalent solution can be based on ProtoBuf. >> We're back to the case #1 in essence where the stored data is really an intermediary format and the conversion to the final object is handled by manual code. >> >> Note that they don't support circularity, which indeed is a bitch. > +1. Not sure protobuf supports it either >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Yeah definitely a good idea to break the problem down into specific cases. On 22 Jul 2010, at 10:47, Galder Zamarreño wrote: > Ok, dealing with simple types/collections is clear. We delegate to the corresponding library and that's it. > > As far as custom types that are controlled by user, so I've gone and compared 3 different methods to deal with them: > > Option 1. User/client code generates the byte[] from a precompiled class. Code example: http://pastebin.com/iqX9rVqV > Option 2. User/client gives us the Pojo and our marshaller converts it into a byte[] with the help of the precompiled class methods. Code example: http://pastebin.com/FTTjcfR4 > Option 3. User/client gives us the Pojo and our marshaller converts it into a byte using client provided serializer/deserializer impl that relies on portable simple type marshaller. I don't have a code example but it'd be very similar to option 2 with the following changes: At startup, ids and class names would be mapped from a given configuration. At writing time, the id would be written and the rest would be delegated to user provided impl. At reading time, once the id has been determined, the class would be instantiated and it would delegate to it the rest of the reading part. > > With this in mind, I've created a spreadsheet comparing them based on different factors: http://spreadsheets.google.com/pub?key=0Ag5RGdzR_GsldEJkUEExX2JPTHdsSF93M... > > I'm leaning towards 1 for its simplicity (less code and less configuration in spite of required precompilation) and performance (no extra bytes, no reflection or class loading). I would lean towards 1 as well. Just to re-cap on the cases here: 1) JVM <--> JVM interaction: just use our default marshallers. Indexing supported.

2) JVM <--> non-JVM interaction, just storing primitives + collections. We provide a marshaller for this? Indexing supported.

3) JVM <--> non-JVM interaction, storing custom objects. User needs to serialize/de-serialize, and store byte[]'s. Indexing NOT supported.

Does that sum it up?

Cheers Manik -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 23 Jul 2010, at 15:27, Galder Zamarreño wrote: > > On Jul 23, 2010, at 1:47 PM, Manik Surtani wrote: > >> Yeah definitely a good idea to break the problem down into specific cases. >> >> On 22 Jul 2010, at 10:47, Galder Zamarreño wrote: >> >>> Ok, dealing with simple types/collections is clear. We delegate to the corresponding library and that's it. >>> >>> As far as custom types that are controlled by user, so I've gone and compared 3 different methods to deal with them: >>> >>> Option 1. User/client code generates the byte[] from a precompiled class. Code example: http://pastebin.com/iqX9rVqV >>> Option 2. User/client gives us the Pojo and our marshaller converts it into a byte[] with the help of the precompiled class methods. Code example: http://pastebin.com/FTTjcfR4 >>> Option 3. User/client gives us the Pojo and our marshaller converts it into a byte using client provided serializer/deserializer impl that relies on portable simple type marshaller. I don't have a code example but it'd be very similar to option 2 with the following changes: At startup, ids and class names would be mapped from a given configuration. At writing time, the id would be written and the rest would be delegated to user provided impl. At reading time, once the id has been determined, the class would be instantiated and it would delegate to it the rest of the reading part. >>> >>> With this in mind, I've created a spreadsheet comparing them based on different factors: http://spreadsheets.google.com/pub?key=0Ag5RGdzR_GsldEJkUEExX2JPTHdsSF93M... >>> >>> I'm leaning towards 1 for its simplicity (less code and less configuration in spite of required precompilation) and performance (no extra bytes, no reflection or class loading). >> >> I would lean towards 1 as well. >> >> Just to re-cap on the cases here: >> >> 1) JVM <--> JVM interaction: just use our default marshallers. Indexing supported. > > Yup. At the moment, the default marshaller for HotRod client/server is based on std Java serialization which deals with class indexing. Nope, the default (as of CR2) is based on JBoss Marshalling. http://fisheye.jboss.org/browse/Infinispan/trunk/client/hotrod-client/src... > We could also plug JBoss Marshalling based marshaller but we'd need the ability for Hot Rod clients to be configurable with user provided externalizers. Yup, this is possible. See above. > As a side note, https://jira.jboss.org/browse/ISPN-244 would not be needed here cos Hot Rod server treats stuff as merely a byte[], so no need for deserialization on the Hot Rod server side. But, any client externalizer would be encouraged to follow the pattern in that JIRA. > > However, one interesting thing would be how to configure Hot Rod clients that are part of RemoteCacheStore. In this case, you could use https://jira.jboss.org/browse/ISPN-244 as a way to configure both inter Infinispan JVM externalizer marshalling and Hot Rod client. In fact, if you used lazy deserialization here, Hot Rod client would not need any configuration, cos it could take the byte[] in memory and directly ship it to the remote Hot Rod server. > >> 2) JVM <--> non-JVM interaction, just storing primitives + collections. We provide a marshaller for this? Indexing supported. > > Yeah, we'd need a marshaller to take basic types and common collections that supports indexing. Find attached a very rough version of what it might look like if we used Protobufs. > > <BasicTypeMarshallingProtobuf.java> > > I'd need to verify whether whatever that marshaller writes can be read successfully read say from an python marshaller equivalent. > >> 3) JVM <--> non-JVM interaction, storing custom objects. User needs to serialize/de-serialize, and store byte[]'s. Indexing NOT supported. > > Yup, no indexing here. Client code knows what it retrieves from the cache and applies the corresponding method, i.e. static method Pojo.parseFrom(). > >> >> Does that sum it up? > > Yeah, pretty much. > > Something else to decide is which of the libraries out there to go for. I've done a little comparison in: http://spreadsheets.google.com/pub?key=0Ag5RGdzR_GsldDdOVnpwbDUyT20xZlZPZ... > > I'm still undecided on which one I prefer, particularly cos I just discovered a hidden feature in MessagePack that might make the basic type/collection marshaller very simple to code. > > I'll report as soon as I know more. Ok, cool! Cheers Manik -- Manik Surtani manik(a)jboss.org Lead, Infinispan Lead, JBoss Cache http://www.infinispan.org http://www.jbosscache.org _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I think we should differentiate two use cases: - serialize / marshall basic types or a composition of basic types (JDK's basic wrapper classes + a few of the collection types) - serialize / marshall custom object structures I don't see #2 as fully solvable as some language don't support some concepts other languages offer. In this area a non-portable solution seems ok. However #1 is an important use case that should be addressed in a portable way. In particular OGM does deconstruct an object into it's composing structures and pass an AtomicMap made of these basic components.

On 21 juil. 2010, at 13:06, Galder Zamarreño wrote: > Hi all, > > Re: https://jira.jboss.org/browse/ISPN-508. This is a bit of a long one, so take your time :) > > Over past few days I've been trying to come up with a Infinispan Marshaller implementation based on one of those portable serialization libraries (i.e. protobuf, thrift...etc) and I've reached the conclusion that it is possible to do so but requires a fair bit of work on our side and performance/usability would decrease. Before explaining my findings, let me explain the model that I'm trying to achieve: > > I want to build a generic marshalling/unmarshalling implementation that's based on language neutral serialization libraries and which does not require the unmarshalling code to have knowledge of the target type. Think about Java Serialization or JBoss Marshalling where the code written can be generic enough w/o needing to know what we're trying to deserialize. The payload in these cases has enough information for the underlying library to figure out the type, instantiate it and populate it accordingly. > > The problem is that there's no such library in the space of portable serialization libraries that provides this out of the box and let me explain why: > > Pretty much any code using those libraries must use class type or schema name to deserialize the payload. However, Infinispan based Marshaller does not have such information when it's trying to deserialize the payload. Infinispan Marshaller based on JDK serialization or JBoss Marshalling uses information in the payload to instantiate the object. However, none of the libraries out there use this mechanism and instead some (i.e. Protobufs) force the client code to do things like: Pojo.parseFrom(byte[]) to generate instances of Pojo. In these cases for example, nothing stops you from writing a UTF-8 string with the class name and put it at the beginning of the payload so the deserialization part can be class agnostic, but this payload would not be portable. What would a python client do with a String containing the java class name? > > Based on the FileDescriptorSet information in http://code.google.com/apis/protocolbuffers/docs/techniques.html I was able to hack something that might work in a portable way. Given a FileDescriptorSet generated a class compile time, I was able to match the protobuf name of a class with its java counter part. So, before writing the protobuf generated byte[], i prepend it with protobuf class name so that when reading, I can take the name, get the java class name and using reflection call parseFrom method to convert the byte[] into a pojo. Note that DynamicMessage class hinted in the the techniques page won't work cos it cannot create instances of pojos. It can only create generic objects with fields that are accessed in a reflection style. > > I also looked at what Avro offers but it does not fully fit either. They have a reflection based serialization mechanism that doesn't require any precompilation, but it requires some kind of type knowledge on the client code to deserialize, plus Avro themselves recommend against it and I'm not sure how performant it'd be. Avro also includes other marshalling mechanisms called specific (like protobufs one with precompiled classes) and generic used to build dynamic objects on the fly. None of these two fit the bill. The specific one is like protobufs with the disadvantage of having ugly code like http://is.gd/dzwj3 where a static object has a strong reference to a <String,Class> CHM, which would leak in an AS env. > > Thrift has the same problems as stated for Protobufs but coudn't see an equivalent way to get find the file descriptor set. Docu is way below what Protobuf offers and latest version which is 0.2 has issues generating classes as stated in JIRA. > > MessagePack has the capability to deserialize an object given a String representation of the schema, so a solution like the protobuf one might be hackable. However, the generated classes do not have an equals implementation (??) which is rather odd. Maybe it's due to lack of maturity? Latest version is 0.3, so that might explain it. API wise, MessagePack provides the API that suits best to what we'd want to do and avoids having to use reflection to resolve the payload. However, looking around I couldn't see similar API for the python language for example and similar to Protobuf, we'd have to prepend the schema name key to the payload to then have the reading part lookup the entire schema based on it. > > So, I can see two solutions here bearing in mind that pretty much any solution would require precompiling some classes: > > - Try to build a marshaller using Protobuf or MessagePack where we enhance it to pass a string key that permits the reading part to deserialize the payload in a generic way. > > - Or try to build some wikis on how to integrate Protobuf/Thrift/MessagePack with Hot Rod client so that they can generate the byte[] with these libraries and pass it to the corresponding Hot Rod client. For the moment, we'd do this for the Hot Rod java client. We would add more info once other language clients are available. The reason I said about potentially showing various libraries is cos whereas Protobuf only supports Java/Python/C++, Thrift supports loads more languages ( C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml) > > In terms of usability , I think the second option is a bit better because building the marshaller requires steps for clients to generate the FileDescriptorSet and somehow pass this to Infinispan so that the marshaller can use it. The second option has none of this and the only added code on the client side is retrieving the byte[] and calling Pojo.parseFrom(byte[]). Performance wise, the 2nd option would be better cos there's no extra hacking needed to pass the String key around and so less bytes are sent around, and no need to use reflection to resolve the payload. > > It's a pity that I couldn't find a tool that fully fits our use case and I wonder whether the need to deal with different languages makes coming up with such solution difficult. I believe reflection based instantiation is present in C++ or Python but not sure about other languages. It's also true that our use case is very specific where we're building a tool where we don't have control over what people will put in the cache. Generics could give us some hints for example but it's not mandatory and would not solve the issue entirely. > > Thoughts? > -- > Galder Zamarreño > Sr. Software Engineer > Infinispan, JBoss Cache > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 22 juil. 2010, at 02:56, Mircea Markus <mircea.markus(a)jboss.com&gt; wrote: > The OGM code that deconstructs the object works on a java object (e.g. Person). To be able to use this Person from C++ you'd need to have C++ code to assemble the AtomicMap (i.e. C++ OGM), or? Yes

_______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

>>> The OGM code that deconstructs the object works on a java object (e.g. Person). To be able to use this Person from C++ you'd need to have C++ code to assemble the AtomicMap (i.e. C++ OGM), or? >> >> Yes > > Yeah but does not need to be anything complicated. On the client side, it'll be just a custom type that looks exactly like a Map. The difference comes when in the server this is converted into an AtomicMap vs a HashMap. The serialized conceptual structure of a HashMap is different from the serialized structure of an AtomicMap? Why?

_______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi all, Re: https://jira.jboss.org/browse/ISPN-508. This is a bit of a long one, so take your time :) Over past few days I've been trying to come up with a Infinispan Marshaller implementation based on one of those portable serialization libraries (i.e. protobuf, thrift...etc) and I've reached the conclusion that it is possible to do so but requires a fair bit of work on our side and performance/usability would decrease. Before explaining my findings, let me explain the model that I'm trying to achieve: I want to build a generic marshalling/unmarshalling implementation that's based on language neutral serialization libraries and which does not require the unmarshalling code to have knowledge of the target type. Think about Java Serialization or JBoss Marshalling where the code written can be generic enough w/o needing to know what we're trying to deserialize. The payload in these cases has enough information for the underlying library to figure out the type, instantiate it and populate it accordingly. The problem is that there's no such library in the space of portable serialization libraries that provides this out of the box and let me explain why: Pretty much any code using those libraries must use class type or schema name to deserialize the payload. However, Infinispan based Marshaller does not have such information when it's trying to deserialize the payload. Infinispan Marshaller based on JDK serialization or JBoss Marshalling uses information in the payload to instantiate the object. However, none of the libraries out there use this mechanism and instead some (i.e. Protobufs) force the client code to do things like: Pojo.parseFrom(byte[]) to generate instances of Pojo. In these cases for example, nothing stops you from writing a UTF-8 string with the class name and put it at the beginning of the payload so the deserialization part can be class agnostic, but this payload would not be portable. What would a python client do with a String containing the java class name?

Based on the FileDescriptorSet information in http://code.google.com/apis/protocolbuffers/docs/techniques.html I was able to hack something that might work in a portable way. Given a FileDescriptorSet generated a class compile time, I was able to match the protobuf name of a class with its java counter part. So, before writing the protobuf generated byte[], i prepend it with protobuf class name so that when reading, I can take the name, get the java class name and using reflection call parseFrom method to convert the byte[] into a pojo. Note that DynamicMessage class hinted in the the techniques page won't work cos it cannot create instances of pojos. It can only create generic objects with fields that are accessed in a reflection style. I also looked at what Avro offers but it does not fully fit either. They have a reflection based serialization mechanism that doesn't require any precompilation, but it requires some kind of type knowledge on the client code to deserialize, plus Avro themselves recommend against it and I'm not sure how performant it'd be. Avro also includes other marshalling mechanisms called specific (like protobufs one with precompiled classes) and generic used to build dynamic objects on the fly. None of these two fit the bill. The specific one is like protobufs with the disadvantage of having ugly code like http://is.gd/dzwj3 where a static object has a strong reference to a <String,Class> CHM, which would leak in an AS env. Thrift has the same problems as stated for Protobufs but coudn't see an equivalent way to get find the file descriptor set. Docu is way below what Protobuf offers and latest version which is 0.2 has issues generating classes as stated in JIRA. MessagePack has the capability to deserialize an object given a String representation of the schema, so a solution like the protobuf one might be hackable. However, the generated classes do not have an equals implementation (??) which is rather odd. Maybe it's due to lack of maturity? Latest version is 0.3, so that might explain it. API wise, MessagePack provides the API that suits best to what we'd want to do and avoids having to use reflection to resolve the payload. However, looking around I couldn't see similar API for the python language for example and similar to Protobuf, we'd have to prepend the schema name key to the payload to then have the reading part lookup the entire schema based on it. So, I can see two solutions here bearing in mind that pretty much any solution would require precompiling some classes: - Try to build a marshaller using Protobuf or MessagePack where we enhance it to pass a string key that permits the reading part to deserialize the payload in a generic way. - Or try to build some wikis on how to integrate Protobuf/Thrift/MessagePack with Hot Rod client so that they can generate the byte[] with these libraries and pass it to the corresponding Hot Rod client. For the moment, we'd do this for the Hot Rod java client. We would add more info once other language clients are available. The reason I said about potentially showing various libraries is cos whereas Protobuf only supports Java/Python/C++, Thrift supports loads more languages ( C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml) In terms of usability , I think the second option is a bit better because building the marshaller requires steps for clients to generate the FileDescriptorSet and somehow pass this to Infinispan so that the marshaller can use it. The second option has none of this and the only added code on the client side is retrieving the byte[] and calling Pojo.parseFrom(byte[]). Performance wise, the 2nd option would be better cos there's no extra hacking needed to pass the String key around and so less bytes are sent around, and no need to use reflection to resolve the payload. It's a pity that I couldn't find a tool that fully fits our use case and I wonder whether the need to deal with different languages makes coming up with such solution difficult. I believe reflection based instantiation is present in C++ or Python but not sure about other languages. It's also true that our use case is very specific where we're building a tool where we don't have control over what people will put in the cache. Generics could give us some hints for example but it's not mandatory and would not solve the issue entirely. Thoughts?

-- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev