[infinispan-dev] protobuf as a marshalling format for infinispan remote-query

[Galder Zamarreño]

On Jun 28, 2013, at 8:33 PM, Adrian Nistor <anistor at redhat.com> wrote:

> Hi all,
> 
> as some probably know protobuf was chosen as the serialization format for the remote query [1]. Them main reason for choosing it was that is simple, time tested, has good support for schema evolution, is nearly ubiquitous (some people say google uses it :)), has multiple language support and most importantly it mandates the existence of a schema for our objects - the proto file. We need that schema on the server side to be able to extract indexable fields from those binary cache values and index them without the need to unmarshall them into plain java domain objects.
> 
> And I need to stress that the format was chosen, rather than the actual API/library provided by Google [2]. While the protobuf wire format is superb, Google's approach to create a library for marshaling objects to/from a protobuf stream is heavily based on code generation via the protoc tool. Both the marshalling code and the entities to be marshaled (your beloved domain model) are generated. This does not work well if you want to bring your own domain classes to the party.

Side note: One thing I've noticed recently is Dagger [1], it's a fast CDI implementation for Java, and here they've used annotation processing to generate code (as opposed to having to run a `protoc` command). The way they test it too is quite interesting (I've used a similar pattern for the JCache TCK). There's a really good presentation on Dagger and the design decisions behind this in [2]. If you haven't seen it, you might wanna watch it to see if the annotation processor method could work for us too… it's not fully ideal (e.g. annotation processing has issues when you need to generate code for dependency jars) but it's good food for thought :)

[1] https://github.com/square/dagger
[2] http://www.infoq.com/presentations/Dagger

> So what we did is create a small set of support classes on top of google's low level wire format classes to assist users in marshaling their own domain model to the protobuf wire format without using google's code generator. These attempts are hosted on a small github project experiment [3] that will be moved to infinispan once we have a conclusion. This project contains several modules demonstrating the attempts. [4] explains the purpose of each module. So far, the approach in module stream-like offers the best user experience. I would name this plan A, which is going to be implemented for infinispan 6.0. 
> 
> Plan A can also use the classes generated by google's protoc code generator tool. So if somebody prefers to go old school it can still work well. Also, the amount of new support code we added on top of google's library is small, so I don't foresee any nightmare in porting this to another language.
> 
> I would like to get some feedback from anyone who has some time to have a look at [3], specifically at the stream-like module.
> Stream-like is about 89.9% implemented. There are still some unimplemented methods but do not mind them for the review. 

I'll have a look and comment about it more specifically in a separate email.

> 
> And finally, if anyone could suggest a better name for the stream-like package? I can't think of any other option except streamlike :) (which might be trade-marked) So any other options? If no options then we'll just call it marshaling. I'll move this to the ispn branch once I know the package name :)
> 
> Have a nice weekend guys!
> 
> --------------
> [1] https://community.jboss.org/wiki/RemoteQueryDesignInInfinispan
> [2] https://developers.google.com/protocol-buffers/
> [3] https://github.com/anistor/protobuf-playground
> [4] https://github.com/anistor/protobuf-playground/blob/master/README.md
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Galder Zamarreño
galder at redhat.com
twitter.com/galderz

Project Lead, Escalante
http://escalante.io

Engineer, Infinispan
http://infinispan.org