[infinispan-dev] Infinispan client/server architecture based on gRPC

Wed May 30 06:22:39 EDT 2018

On Wed, May 30, 2018 at 10:56 AM, Adrian Nistor <anistor at redhat.com> wrote:

> The oneof and WrappedMessage solve the same problem but in a different way.
> Oneof has the nasty effect that in ties the service model to the user data
> model.
>

The user data model is only "static" at storage level (guided by
configuration), and the user data can travel on the wire in any format the
user wants [1]

[1]
https://github.com/infinispan/infinispan/blob/master/client/hotrod-client/src/test/java/org/infinispan/client/hotrod/transcoding/DataFormatTest.java#L109

So better not to assume it will be marshalled and unmarshalled in a
specific way.

> Even if it seems like just one more line of code to add when a new user
> type is introduced, it is one line of code in the wrong place because
> you'll have to re-generate the service. IE user run protoc again on OUR
> IDLs. Should a user do that? This coupling between the infinispan's service
> model and the user's data model bothers me.
>
> WrappedMessage is just a wrapper around an array of bytes + information
> regarding what message type or what scalar type is in there. Something very
> similar to a VARIANT [1]. The reason it is needed is explained here [2].
>
> You are correct, this is not a gRPC limitation, it is a by-design protobuf
> protocol limitation, that was very thoughtfully introduced to reduce wire
> level bandwitdth for the common case where types are static. Unfortunately
> it leaves generic/dynamic types in mid-air. But it is fairly easy to solve,
> as you can see with WrappedMessage. At the time I introduced WrappedMessage
> we were using protobuf 2.
>
> protobuf 3 introduces type Any, which solves the issue in a similar way
> with WrappedMessage. The difference is Any seems to have been created to
> wrap either a plain byte[] or a message type that has been marshalled to a
> byte[]. No support for scalars in sight. Can we solve that? Sure, put a
> WrappedMessage inside that byte[] :)))) That is the reason I did not jump
> immediately at using Any and stayed with WrappedMessage.
>
> Can a 150 lines PoC be a proposal for the ISPN object model? No, but we
> need to explore the pain points of gRPC and protobuf that are relevant to
> our usage, and this thing with genericly typed services is one of them.
> I think we already have a good solution in sight, before giving up and
> going with byte[] for key and value as it was suggested earlier here. I can
> make a PR to the grpc PoC to show it by the end of the week.
>
> Adrian
>
> [1] https://en.wikipedia.org/wiki/Variant_type
> [2] https://developers.google.com/protocol-buffers/docs/
> techniques#streaming
>
>
> On 05/30/2018 11:34 AM, Vittorio Rigamonti wrote:
>
>
>
> On Tue, May 29, 2018 at 8:59 PM, Adrian Nistor <anistor at redhat.com> wrote:
>
>> So you assume the two are separate, Emmanuel. So do I.
>>
>> But in the current PoC the user data model is directly referenced by the
>> service model interface (KeyMsg and ValueMsg are oneofs listing all
>> possible user application types???). I was assuming this hard dependency
>> was there just to make things simple for the scope of the PoC. But let's
>> not make this too simple because it will stop being useful. My expectation
>> is to see a generic yet fully typed 'cache service' interface that does not
>> depend on the key and value types that come from userland, using maybe
>> 'google.protobuf.Any' or our own 'WrappedMessage' type instead. I'm not
>> sure what to believe now because discussing my hopes and assumptions on the
>> gRPC topic on zulip I think I understood the opposite is desired.
>> Vittorio, please comment on this.
>>
>
> Yep that was my design choice. Well my first goal was to keep the
> framework language independent: to reach that I tried to define in
> grpc/protobuf as much as possible (that's why I didn't use the Any clause).
> Then I realized that with very little effort I could design a framework
> that works only with user data from the user side to the cache storage and
> I'd  liked to investigate this, manly for two reasons:
>
> - from the user point of view I like the idea that I can found my objects
> types in the cache
> - the embeddedCache<object,object> is transparently exposed
>
> but this is my 150 lines of code grpc server prototype, not a proposal for
> the ISPN object model. However it's ok to use it as starting point for a
> wider discussion
>
>
>>
>> I'm still hoping we want to keep the service interface generic and
>> separated from the user model. And if we do it, would you expect to be able
>> to marshall the service call using gRPC lib and at the same time be able to
>> marshall the user model using whatever other library? Would be nice but
>> that seems to be a no-no with gRPC, or I did not search deep enough. I only
>> looked at the java implementation anyway. It seems to be forcing you to go
>> with protoc generated code and protobuf-java.jar all the way, for
>> marshalling both the service and its arguments. And this goes infinitely
>> deeper. If a service argument of type A has a nested field of type B and
>> the marshaller for A is generated with protobuf-java then so is B. Using
>> oneofs or type 'Any' still do not save you from this.  The only escape is
>> to pretend the user payload is of type 'bytes'. At that point you are left
>> to do your marshaling to and from bytes yourself. And you are also left
>> with the question, what the heck is the contents of that byte array next
>> time you unmarshall it, which is currently answered by WrappedMessage.
>>
>
> And indeed the "oneof" clause in my message definition solves the same
> problem solved by the WrappedMessage message: what I have to do with these
> bytes? Actually I'm not sure this is a gRPC limitation: if I receive a
> stream of bytes I also need some info on what I have to reconstruct.... I'm
> just guessing
>
>>
>> So the more I look at gRPC it seems elegant for most purposes but lacking
>> for ours. And again, as with protocol buffers, the wire protocol and the
>> IDL are really nice. It is the implementation that is lacking, IMHO.
>>
>> I think to be really on the same page we should first make a clear
>> statement of what we intend to achieve here in a bit more detail. Also,
>> since this is not a clean slate effort, we should think right from the
>> start what are the expected interactions with existing code base, like what
>> are we willing to sacrifice. Somebody mention hot rod please!
>>
>> Adrian
>>
>>
>>
>> On 05/29/2018 07:20 PM, Emmanuel Bernard wrote:
>>
>> Right. Here we are talking about a gRPC representation of the client
>> server interactions. Not the data schema stored in ISPN. In that model, the
>> API is compiled by us and handed over as a package.
>>
>> On 29 May 2018, at 15:51, Sanne Grinovero <sanne at infinispan.org> wrote:
>>
>>
>>
>> On 29 May 2018 at 13:45, Vittorio Rigamonti <vrigamon at redhat.com> wrote:
>>
>>> Thanks Adrian,
>>>
>>> of course there's a marshalling work under the cover and that is
>>> reflected into the generated code (specially the accessor methods generated
>>> from the oneof clause).
>>>
>>> My opinion is that on the client side this could be accepted, as long as
>>> the API are well defined and documented: application developer can build an
>>> adhoc decorator on the top if needed. The alternative to this is to develop
>>> a protostream equivalent for each supported language and it doesn't seem
>>> really feasible to me.
>>>
>>
>> This might indeed be reasonable for some developers, some languages.
>>
>> Just please make sure it's not the only option, as many other developers
>> will not expect to need a compiler at hand in various stages of the
>> application lifecycle.
>>
>> For example when deploying a JPA model into an appserver, or just booting
>> Hibernate in JavaSE as well, there is a strong expectation that we'll be
>> able - at runtime - to inspect the listed Java POJOs via reflection and
>> automatically generate whatever Infinispan will need.
>>
>> Perhaps a key differentiator is between invoking Infinispan APIs (RPC) vs
>> defining the object models and related CODECs for keys, values, streams and
>> query results? It might get a bit more fuzzy to differentiate them for
>> custom functions but I guess we can draw a line somewhere.
>>
>> Thanks,
>> Sanne
>>
>>
>>
>>>
>>> On the server side (java only) the situation is different: protobuf is
>>> optimized for streaming not for storing so probably a Protostream layer is
>>> needed.
>>>
>>> On Mon, May 28, 2018 at 4:47 PM, Adrian Nistor <anistor at redhat.com>
>>> wrote:
>>>
>>>> Hi Vittorio,
>>>> thanks for exploring gRPC. It seems like a very elegant solution for
>>>> exposing services. I'll have a look at your PoC soon.
>>>>
>>>> I feel there are some remarks that need to be made regarding gRPC. gRPC
>>>> is just some nice cheesy topping on top of protobuf. Google's
>>>> implementation of protobuf, to be more precise.
>>>> It does not need handwritten marshallers, but the 'No need for
>>>> marshaller' does not accurately describe it. Marshallers are needed and are
>>>> generated under the cover by the library and so are the data objects and
>>>> you are unfortunately forced to use them. That's both the good news and the
>>>> bad news:) The whole thing looks very promising and friendly for many uses
>>>> cases, especially for demos and PoCs :))). Nobody wants to write those
>>>> marshallers. But it starts to become a nuisance if you want to use your own
>>>> data objects.
>>>> There is also the ugliness and excessive memory footprint of the
>>>> generated code, which is the reason Infinispan did not adopt the
>>>> protobuf-java library although it did adopt protobuf as an encoding format.
>>>> The Protostream library was created as an alternative implementation to
>>>> solve the aforementioned problems with the generated code. It solves this
>>>> by letting the user provide their own data objects. And for the marshallers
>>>> it gives you two options: a) write the marshaller yourself (hated), b)
>>>> annotated your data objects and the marshaller gets generated (loved).
>>>> Protostream does not currently support service definitions right now but
>>>> this is something I started to investigate recently after Galder asked me
>>>> if I think it's doable. I think I'll only find out after I do it:)
>>>>
>>>> Adrian
>>>>
>>>>
>>>> On 05/28/2018 04:15 PM, Vittorio Rigamonti wrote:
>>>>
>>>> Hi Infinispan developers,
>>>>
>>>> I'm working on a solution for developers who need to access Infinispan
>>>> services  through different programming languages.
>>>>
>>>> The focus is not on developing a full featured client, but rather
>>>> discover the value and the limits of this approach.
>>>>
>>>> - is it possible to automatically generate useful clients in different
>>>> languages?
>>>> - can that clients interoperate on the same cache with the same data
>>>> types?
>>>>
>>>> I came out with a small prototype that I would like to submit to you
>>>> and on which I would like to gather your impressions.
>>>>
>>>>  You can found the project here [1]: is a gRPC-based client/server
>>>> architecture for Infinispan based on and EmbeddedCache, with very few
>>>> features exposed atm.
>>>>
>>>> Currently the project is nothing more than a poc with the following
>>>> interesting features:
>>>>
>>>> - client can be generated in all the grpc supported language: java, go,
>>>> c++ examples are provided;
>>>> - the interface is full typed. No need for marshaller and clients build
>>>> in different language can cooperate on the same cache;
>>>>
>>>> The second item is my preferred one beacuse it frees the developer from
>>>> data marshalling.
>>>>
>>>> What do you think about?
>>>> Sounds interesting?
>>>> Can you see any flaw?
>>>>
>>>> There's also a list of issues for the future [2], basically I would
>>>> like to investigate these questions:
>>>> How far this architecture can go?
>>>> Topology, events, queries... how many of the Infinispan features can be
>>>> fit in a grpc architecture?
>>>>
>>>> Thank you
>>>> Vittorio
>>>>
>>>> [1] https://github.com/rigazilla/ispn-grpc
>>>> [2] https://github.com/rigazilla/ispn-grpc/issues
>>>>
>>>> --
>>>>
>>>> Vittorio Rigamonti
>>>>
>>>> Senior Software Engineer
>>>>
>>>> Red Hat
>>>>
>>>> <https://www.redhat.com>
>>>>
>>>> Milan, Italy
>>>>
>>>> vrigamon at redhat.com
>>>>
>>>> irc: rigazilla
>>>> <https://red.ht/sig>
>>>>
>>>>
>>>> _______________________________________________
>>>> infinispan-dev mailing listinfinispan-dev at lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Vittorio Rigamonti
>>>
>>> Senior Software Engineer
>>>
>>> Red Hat
>>>
>>> <https://www.redhat.com>
>>>
>>> Milan, Italy
>>>
>>> vrigamon at redhat.com
>>>
>>> irc: rigazilla
>>> <https://red.ht/sig>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing listinfinispan-dev at lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>>
>
>
> --
>
> Vittorio Rigamonti
>
> Senior Software Engineer
>
> Red Hat
>
> <https://www.redhat.com>
>
> Milan, Italy
>
> vrigamon at redhat.com
>
> irc: rigazilla
> <https://red.ht/sig>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20180530/0585b3da/attachment-0001.html 