[infinispan-dev] Hot Rod encoding

[Emmanuel Bernard]

Note that in a future version of hibernate search - soon I hope, the encoding will no longer be a problem. You will be able to provide a navigation/traversal API that will know how to read things from your blob. It's called free form entity by its friends. 

> On 17 févr. 2016, at 12:35, Galder Zamarreño <galder at redhat.com> wrote:
> 
> I like the idea of pluggable serialization/marshalling, but as you rightly explained, what flexibility gives you you lose by lack of functionality. E.g. querying is only available for protostream based encoding.
> 
> So, I think we need to remove the hard bind between functionality and encoding to be able to have a trully pluggable encoding mechanism.
> 
> Cheers,
> --
> Galder Zamarreño
> Infinispan, Red Hat
> 
>> On 3 Feb 2016, at 11:38, Gustavo Fernandes <gustavo at infinispan.org> wrote:
>> 
>> 
>> 
>> On Mon, Jan 25, 2016 at 2:03 PM, Galder Zamarreño <galder at redhat.com> wrote:
>> Hi all,
>> 
>> As I write the Javascript client for Hot Rod, and Vittorio writes the C++ client, the question the encoding of the byte arrays has popped up.
>> 
>> The reason why encoding matters is mainly because of compatibility mode. How does a Hot Rod client know how it should transform something a REST client set?
>> 
>> To be able to answer this question correctly, the Hot Rod client needs to know the type of the data.
>> 
>> Although we could consider adding encoding information to the Hot Rod protocol long term, in the short term this question might already been answered by Protostream.
>> 
>> 
>> Compatibility between disparate clients should certainly be supported, but what about a pluggable marshaller mechanism? Let's consider all usage scenarios:
>> 
>> * Only Java clients: in most cases, JBoss marshalling is adequate, no need to involve protobuf or json and since JBoss Marshalling is default, no need to configure anything.
>> 
>> * Mix of Java and C++/C#/Python clients: the Protobuf encoding works wonderfully for that. The client could be configured to use the protostream marshaller, the same for the server.
>> 
>> * Mix of Java, C++/C#/Python and REST clients: protobuf with json [1] encoding is an interesting option. Same as above, a protostream marshaller with json support could 
>> be configured both on client and server.
>> 
>> * Custom marshallers: consider FlatBuffers [2], for example, an interesting new cross-platform serializer that allows to access serialized data without de-serializing the whole payload, 
>> and without generating any transient memory, and it optionally supports JSON. It could be interesting to write a marshaller based on it and plug it on Infinispan.
>> 
>> Although we have a way of configuring the marshaller on client (via RemoteCacheManager) and server (deployable jar), there'd be more work to do in order make them really pluggable:
>> 
>> - Some serialization formats like Protobuf and [2] require interface description to be available on both client and server. This is not supported for all marshallers, just for Protostream
>> - Remote query requires some metadata regarding which fields to index and how, and this would need to be possible on every language regardless of the marshalling.
>> Currently, this info can be encoded in the .proto file only, a custom marshaller would not work.
>> - Currently remote query is settled on protobuf not only for the byte array but for the the HotRod query request and response [3]
>> 
>> Is having a pluggable serializer mechanism too far fetched, given the effort already invested towards protobuf? 
>> 
>> Gustavo
>> 
>> [1] https://developers.google.com/protocol-buffers/docs/proto3#json
>> [2] https://google.github.io/flatbuffers/
>> [2] https://code.facebook.com/posts/872547912839369/improving-facebook-s-performance-on-android-with-flatbuffers/
>> [3] https://github.com/infinispan/infinispan/blob/master/remote-query/remote-query-client/src/main/resources/org/infinispan/query/remote/client/query.proto 
>> 
>> 
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev