[infinispan-dev] Hot Rod encoding

[Sanne Grinovero]

It sounds like a good idea, almost like a natural evolution, but to
play devil's advocate I'll try to find some drawbacks for such a
decision.

One negative argument is overall complexity: there are many points in
code in which one needs to consider that the encoding might be
"something else". This isn't a new problem as we already support two
modes, but we've soon how much more difficult it makes things and this
is making things even a bit more complex.

Another point which I don't like much is that people will have to
reconfigure the server depending on specific needs of the clients; if
we go this way there should be a way in the Hot Rod protocol to
"negotiate encoding", so to avoid such configuration tasks for end
users (at least when there's no need to actually deploy dependencies
on the server to handle some custom encoder..).

The other problem I see is that this severely complicates
interoperability between different clients. Suppose two applications
are being developed to use the Infinispan Server and decide to use two
different encoders, I suspect that at some point they'll regret it
when needing one to access some data from the other... not suggesting
that this should be a good practice, but still it would be very
inconvenient.

Finally, tooling. We'll eventually need to work on better tooling and
supporting multiple encoders would spread efforts thinly.

That said, I'm not against the idea of toying with such options and
make other encoders as an *experimental* feature: protobuf is the most
suited choice but forever is a long time and we should not hinder
research on better alternatives.

Thanks,
Sanne




On 3 February 2016 at 10:38, Gustavo Fernandes <gustavo at infinispan.org> wrote:
>
>
> On Mon, Jan 25, 2016 at 2:03 PM, Galder Zamarreño <galder at redhat.com> wrote:
>>
>> Hi all,
>>
>> As I write the Javascript client for Hot Rod, and Vittorio writes the C++
>> client, the question the encoding of the byte arrays has popped up.
>>
>> The reason why encoding matters is mainly because of compatibility mode.
>> How does a Hot Rod client know how it should transform something a REST
>> client set?
>>
>> To be able to answer this question correctly, the Hot Rod client needs to
>> know the type of the data.
>>
>> Although we could consider adding encoding information to the Hot Rod
>> protocol long term, in the short term this question might already been
>> answered by Protostream.
>
>
>
> Compatibility between disparate clients should certainly be supported, but
> what about a pluggable marshaller mechanism? Let's consider all usage
> scenarios:
>
> * Only Java clients: in most cases, JBoss marshalling is adequate, no need
> to involve protobuf or json and since JBoss Marshalling is default, no need
> to configure anything.
>
> * Mix of Java and C++/C#/Python clients: the Protobuf encoding works
> wonderfully for that. The client could be configured to use the protostream
> marshaller, the same for the server.
>
> * Mix of Java, C++/C#/Python and REST clients: protobuf with json [1]
> encoding is an interesting option. Same as above, a protostream marshaller
> with json support could
> be configured both on client and server.
>
> * Custom marshallers: consider FlatBuffers [2], for example, an interesting
> new cross-platform serializer that allows to access serialized data without
> de-serializing the whole payload,
> and without generating any transient memory, and it optionally supports
> JSON. It could be interesting to write a marshaller based on it and plug it
> on Infinispan.
>
> Although we have a way of configuring the marshaller on client (via
> RemoteCacheManager) and server (deployable jar), there'd be more work to do
> in order make them really pluggable:
>
> - Some serialization formats like Protobuf and [2] require interface
> description to be available on both client and server. This is not supported
> for all marshallers, just for Protostream
> - Remote query requires some metadata regarding which fields to index and
> how, and this would need to be possible on every language regardless of the
> marshalling.
> Currently, this info can be encoded in the .proto file only, a custom
> marshaller would not work.
> - Currently remote query is settled on protobuf not only for the byte array
> but for the the HotRod query request and response [3]
>
> Is having a pluggable serializer mechanism too far fetched, given the effort
> already invested towards protobuf?
>
> Gustavo
>
> [1] https://developers.google.com/protocol-buffers/docs/proto3#json
> [2] https://google.github.io/flatbuffers/
> [2]
> https://code.facebook.com/posts/872547912839369/improving-facebook-s-performance-on-android-with-flatbuffers/
> [3]
> https://github.com/infinispan/infinispan/blob/master/remote-query/remote-query-client/src/main/resources/org/infinispan/query/remote/client/query.proto
>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev