[infinispan-dev] Hot Rod encoding

Wed Feb 17 06:48:23 EST 2016

> On 3 Feb 2016, at 14:07, Adrian Nistor <anistor at redhat.com> wrote:
> 
> Hi all,
> 
> </snip>
> 
> Re Protostream, it is just plain Protobuf folks! It's in no way a 
> customly-modified-encoding-loosely-based-on-protobuf. The only apparent 
> twist is our usage of Protobuf, which btw also follows Google's 
> recommendations (see [1]). All keys and values are wrapped into a struct 
> like this [2] (actually it is a union when looking closer). And for good 
> reason. Adding that metadata as a separate header at hot rod protocol 
> level would be another possibility, but really, having it stored in 
> cache with the key/value is much more appropriate. This type metadata 
> (the type name or type id) is needed for indexing too. And could also be 
> used by some smart cache stores to make some sense out of that blob and 
> map it to database columns or suchlike. And would also be needed by non 
> HR clients, like REST. I can go on .. :)  The only downside I see is 
> space. Our caches are not homogeneous, unfortunately [let's not debate 
> now why], so the type info needs to be stored for each single key, value 
> instead of storing it once per cache.

^ I'm not sure Protostream is just plain Protobuf... If you look at [3], you see very specific encoding/decoding steps about the wrapped message.

Could I just read Protostream stuff from a JS client with only a .proto file? Looking at [3], it seems like that's not the case and hence I think Protostream imposes certain type of encoding. 

It looks to me I'd have to encode/decode each basic primitive types manually in the JS client encoder to support Protostream, and only for custom types provided via a .proto file I can delegate to Protobuf itself.

@Adrian, am I reading this properly?

Cheers,

[3] https://github.com/infinispan/protostream/blob/master/core/src/main/java/org/infinispan/protostream/WrappedMessage.java#L54

> 
> Adrian
> 
> [1] 
> https://developers.google.com/protocol-buffers/docs/techniques#self-description
> [2] 
> https://raw.githubusercontent.com/anistor/protostream/master/core/src/main/resources/org/infinispan/protostream/message-wrapping.proto
> 
> On 02/03/2016 01:34 PM, Sanne Grinovero wrote:
>> It sounds like a good idea, almost like a natural evolution, but to
>> play devil's advocate I'll try to find some drawbacks for such a
>> decision.
>> 
>> One negative argument is overall complexity: there are many points in
>> code in which one needs to consider that the encoding might be
>> "something else". This isn't a new problem as we already support two
>> modes, but we've soon how much more difficult it makes things and this
>> is making things even a bit more complex.
>> 
>> Another point which I don't like much is that people will have to
>> reconfigure the server depending on specific needs of the clients; if
>> we go this way there should be a way in the Hot Rod protocol to
>> "negotiate encoding", so to avoid such configuration tasks for end
>> users (at least when there's no need to actually deploy dependencies
>> on the server to handle some custom encoder..).
>> 
>> The other problem I see is that this severely complicates
>> interoperability between different clients. Suppose two applications
>> are being developed to use the Infinispan Server and decide to use two
>> different encoders, I suspect that at some point they'll regret it
>> when needing one to access some data from the other... not suggesting
>> that this should be a good practice, but still it would be very
>> inconvenient.
>> 
>> Finally, tooling. We'll eventually need to work on better tooling and
>> supporting multiple encoders would spread efforts thinly.
>> 
>> That said, I'm not against the idea of toying with such options and
>> make other encoders as an *experimental* feature: protobuf is the most
>> suited choice but forever is a long time and we should not hinder
>> research on better alternatives.
>> 
>> Thanks,
>> Sanne
>> 
>> 
>> 
>> 
>> On 3 February 2016 at 10:38, Gustavo Fernandes <gustavo at infinispan.org> wrote:
>>> 
>>> On Mon, Jan 25, 2016 at 2:03 PM, Galder Zamarreño <galder at redhat.com> wrote:
>>>> Hi all,
>>>> 
>>>> As I write the Javascript client for Hot Rod, and Vittorio writes the C++
>>>> client, the question the encoding of the byte arrays has popped up.
>>>> 
>>>> The reason why encoding matters is mainly because of compatibility mode.
>>>> How does a Hot Rod client know how it should transform something a REST
>>>> client set?
>>>> 
>>>> To be able to answer this question correctly, the Hot Rod client needs to
>>>> know the type of the data.
>>>> 
>>>> Although we could consider adding encoding information to the Hot Rod
>>>> protocol long term, in the short term this question might already been
>>>> answered by Protostream.
>>> 
>>> 
>>> Compatibility between disparate clients should certainly be supported, but
>>> what about a pluggable marshaller mechanism? Let's consider all usage
>>> scenarios:
>>> 
>>> * Only Java clients: in most cases, JBoss marshalling is adequate, no need
>>> to involve protobuf or json and since JBoss Marshalling is default, no need
>>> to configure anything.
>>> 
>>> * Mix of Java and C++/C#/Python clients: the Protobuf encoding works
>>> wonderfully for that. The client could be configured to use the protostream
>>> marshaller, the same for the server.
>>> 
>>> * Mix of Java, C++/C#/Python and REST clients: protobuf with json [1]
>>> encoding is an interesting option. Same as above, a protostream marshaller
>>> with json support could
>>> be configured both on client and server.
>>> 
>>> * Custom marshallers: consider FlatBuffers [2], for example, an interesting
>>> new cross-platform serializer that allows to access serialized data without
>>> de-serializing the whole payload,
>>> and without generating any transient memory, and it optionally supports
>>> JSON. It could be interesting to write a marshaller based on it and plug it
>>> on Infinispan.
>>> 
>>> Although we have a way of configuring the marshaller on client (via
>>> RemoteCacheManager) and server (deployable jar), there'd be more work to do
>>> in order make them really pluggable:
>>> 
>>> - Some serialization formats like Protobuf and [2] require interface
>>> description to be available on both client and server. This is not supported
>>> for all marshallers, just for Protostream
>>> - Remote query requires some metadata regarding which fields to index and
>>> how, and this would need to be possible on every language regardless of the
>>> marshalling.
>>> Currently, this info can be encoded in the .proto file only, a custom
>>> marshaller would not work.
>>> - Currently remote query is settled on protobuf not only for the byte array
>>> but for the the HotRod query request and response [3]
>>> 
>>> Is having a pluggable serializer mechanism too far fetched, given the effort
>>> already invested towards protobuf?
>>> 
>>> Gustavo
>>> 
>>> [1] https://developers.google.com/protocol-buffers/docs/proto3#json
>>> [2] https://google.github.io/flatbuffers/
>>> [2]
>>> https://code.facebook.com/posts/872547912839369/improving-facebook-s-performance-on-android-with-flatbuffers/
>>> [3]
>>> https://github.com/infinispan/infinispan/blob/master/remote-query/remote-query-client/src/main/resources/org/infinispan/query/remote/client/query.proto
>>> 
>>>> 
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> infinispan-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> 
>>> 
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev