[infinispan-dev] Infinispan client/server architecture based on gRPC
Adrian Nistor
anistor at redhat.com
Tue May 29 14:59:13 EDT 2018
So you assume the two are separate, Emmanuel. So do I.
But in the current PoC the user data model is directly referenced by the
service model interface (KeyMsg and ValueMsg are oneofs listing all
possible user application types???). I was assuming this hard dependency
was there just to make things simple for the scope of the PoC. But let's
not make this too simple because it will stop being useful. My
expectation is to see a generic yet fully typed 'cache service'
interface that does not depend on the key and value types that come from
userland, using maybe 'google.protobuf.Any' or our own 'WrappedMessage'
type instead. I'm not sure what to believe now because discussing my
hopes and assumptions on the gRPC topic on zulip I think I understood
the opposite is desired. Vittorio, please comment on this.
I'm still hoping we want to keep the service interface generic and
separated from the user model. And if we do it, would you expect to be
able to marshall the service call using gRPC lib and at the same time be
able to marshall the user model using whatever other library? Would be
nice but that seems to be a no-no with gRPC, or I did not search deep
enough. I only looked at the java implementation anyway. It seems to be
forcing you to go with protoc generated code and protobuf-java.jar all
the way, for marshalling both the service and its arguments. And this
goes infinitely deeper. If a service argument of type A has a nested
field of type B and the marshaller for A is generated with protobuf-java
then so is B. Using oneofs or type 'Any' still do not save you from
this. The only escape is to pretend the user payload is of type
'bytes'. At that point you are left to do your marshaling to and from
bytes yourself. And you are also left with the question, what the heck
is the contents of that byte array next time you unmarshall it, which is
currently answered by WrappedMessage.
So the more I look at gRPC it seems elegant for most purposes but
lacking for ours. And again, as with protocol buffers, the wire protocol
and the IDL are really nice. It is the implementation that is lacking, IMHO.
I think to be really on the same page we should first make a clear
statement of what we intend to achieve here in a bit more detail. Also,
since this is not a clean slate effort, we should think right from the
start what are the expected interactions with existing code base, like
what are we willing to sacrifice. Somebody mention hot rod please!
Adrian
On 05/29/2018 07:20 PM, Emmanuel Bernard wrote:
> Right. Here we are talking about a gRPC representation of the client
> server interactions. Not the data schema stored in ISPN. In that
> model, the API is compiled by us and handed over as a package.
>
> On 29 May 2018, at 15:51, Sanne Grinovero <sanne at infinispan.org
> <mailto:sanne at infinispan.org>> wrote:
>
>>
>>
>> On 29 May 2018 at 13:45, Vittorio Rigamonti <vrigamon at redhat.com
>> <mailto:vrigamon at redhat.com>> wrote:
>>
>> Thanks Adrian,
>>
>> of course there's a marshalling work under the cover and that is
>> reflected into the generated code (specially the accessor methods
>> generated from the oneof clause).
>>
>> My opinion is that on the client side this could be accepted, as
>> long as the API are well defined and documented: application
>> developer can build an adhoc decorator on the top if needed. The
>> alternative to this is to develop a protostream equivalent for
>> each supported language and it doesn't seem really feasible to me.
>>
>>
>> This might indeed be reasonable for some developers, some languages.
>>
>> Just please make sure it's not the only option, as many other
>> developers will not expect to need a compiler at hand in various
>> stages of the application lifecycle.
>>
>> For example when deploying a JPA model into an appserver, or just
>> booting Hibernate in JavaSE as well, there is a strong expectation
>> that we'll be able - at runtime - to inspect the listed Java POJOs
>> via reflection and automatically generate whatever Infinispan will need.
>>
>> Perhaps a key differentiator is between invoking Infinispan APIs
>> (RPC) vs defining the object models and related CODECs for keys,
>> values, streams and query results? It might get a bit more fuzzy to
>> differentiate them for custom functions but I guess we can draw a
>> line somewhere.
>>
>> Thanks,
>> Sanne
>>
>>
>> On the server side (java only) the situation is different:
>> protobuf is optimized for streaming not for storing so probably a
>> Protostream layer is needed.
>>
>> On Mon, May 28, 2018 at 4:47 PM, Adrian Nistor
>> <anistor at redhat.com <mailto:anistor at redhat.com>> wrote:
>>
>> Hi Vittorio,
>> thanks for exploring gRPC. It seems like a very elegant
>> solution for exposing services. I'll have a look at your PoC
>> soon.
>>
>> I feel there are some remarks that need to be made regarding
>> gRPC. gRPC is just some nice cheesy topping on top of
>> protobuf. Google's implementation of protobuf, to be more
>> precise.
>> It does not need handwritten marshallers, but the 'No need
>> for marshaller' does not accurately describe it. Marshallers
>> are needed and are generated under the cover by the library
>> and so are the data objects and you are unfortunately forced
>> to use them. That's both the good news and the bad news:) The
>> whole thing looks very promising and friendly for many uses
>> cases, especially for demos and PoCs :))). Nobody wants to
>> write those marshallers. But it starts to become a nuisance
>> if you want to use your own data objects.
>> There is also the ugliness and excessive memory footprint of
>> the generated code, which is the reason Infinispan did not
>> adopt the protobuf-java library although it did adopt
>> protobuf as an encoding format.
>> The Protostream library was created as an alternative
>> implementation to solve the aforementioned problems with the
>> generated code. It solves this by letting the user provide
>> their own data objects. And for the marshallers it gives you
>> two options: a) write the marshaller yourself (hated), b)
>> annotated your data objects and the marshaller gets generated
>> (loved). Protostream does not currently support service
>> definitions right now but this is something I started to
>> investigate recently after Galder asked me if I think it's
>> doable. I think I'll only find out after I do it:)
>>
>> Adrian
>>
>>
>> On 05/28/2018 04:15 PM, Vittorio Rigamonti wrote:
>>> Hi Infinispan developers,
>>>
>>> I'm working on a solution for developers who need to access
>>> Infinispan services through different programming languages.
>>>
>>> The focus is not on developing a full featured client, but
>>> rather discover the value and the limits of this approach.
>>>
>>> - is it possible to automatically generate useful clients in
>>> different languages?
>>> - can that clients interoperate on the same cache with the
>>> same data types?
>>>
>>> I came out with a small prototype that I would like to
>>> submit to you and on which I would like to gather your
>>> impressions.
>>>
>>> You can found the project here [1]: is a gRPC-based
>>> client/server architecture for Infinispan based on and
>>> EmbeddedCache, with very few features exposed atm.
>>>
>>> Currently the project is nothing more than a poc with the
>>> following interesting features:
>>>
>>> - client can be generated in all the grpc supported
>>> language: java, go, c++ examples are provided;
>>> - the interface is full typed. No need for marshaller and
>>> clients build in different language can cooperate on the
>>> same cache;
>>>
>>> The second item is my preferred one beacuse it frees the
>>> developer from data marshalling.
>>>
>>> What do you think about?
>>> Sounds interesting?
>>> Can you see any flaw?
>>>
>>> There's also a list of issues for the future [2], basically
>>> I would like to investigate these questions:
>>> How far this architecture can go?
>>> Topology, events, queries... how many of the Infinispan
>>> features can be fit in a grpc architecture?
>>>
>>> Thank you
>>> Vittorio
>>>
>>> [1] https://github.com/rigazilla/ispn-grpc
>>> <https://github.com/rigazilla/ispn-grpc>
>>> [2] https://github.com/rigazilla/ispn-grpc/issues
>>> <https://github.com/rigazilla/ispn-grpc/issues>
>>>
>>> --
>>>
>>> Vittorio Rigamonti
>>>
>>> Senior Software Engineer
>>>
>>> Red Hat
>>>
>>> <https://www.redhat.com>
>>>
>>> Milan, Italy
>>>
>>> vrigamon at redhat.com <mailto:vrigamon at redhat.com>
>>>
>>> irc: rigazilla
>>>
>>> <https://red.ht/sig>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> <mailto:infinispan-dev at lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>> <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
>>
>>
>>
>>
>>
>> --
>>
>> Vittorio Rigamonti
>>
>> Senior Software Engineer
>>
>> Red Hat
>>
>> <https://www.redhat.com>
>>
>> Milan, Italy
>>
>> vrigamon at redhat.com <mailto:vrigamon at redhat.com>
>>
>> irc: rigazilla
>>
>> <https://red.ht/sig>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> <mailto:infinispan-dev at lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> <https://lists.jboss.org/mailman/listinfo/infinispan-dev>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org <mailto:infinispan-dev at lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20180529/8ffc1f58/attachment-0001.html
More information about the infinispan-dev
mailing list