[infinispan-dev] protobuf as a marshalling format for infinispan remote-query

[Adrian Nistor]

Hi all,

as some probably know protobuf was chosen as the serialization format 
for the remote query [1]. Them main reason for choosing it was that is 
/simple/, time tested, has good support for schema evolution, is nearly 
ubiquitous (some people say google uses it :)), has multiple language 
support and most importantly it mandates the existence of a schema for 
our objects - the proto file. We need that schema on the server side to 
be able to extract indexable fields from those binary cache values and 
index them without the need to unmarshall them into plain java domain 
objects.

And I need to stress that the format was chosen, rather than the actual 
API/library provided by Google [2]. While the protobuf wire format is 
superb, Google's approach to create a library for marshaling objects 
to/from a protobuf stream is heavily based on code generation via the 
protoc tool. Both the marshalling code /and the entities/ to be 
marshaled (your beloved domain model) are generated. This does not work 
well if you want to bring your own domain classes to the party.

So what we did is create a small set of support classes on top of 
google's low level wire format classes to assist users in marshaling 
their own domain model to the protobuf wire format without using 
google's code generator. These attempts are hosted on a small github 
project experiment [3] that will be moved to infinispan once we have a 
conclusion. This project contains several modules demonstrating the 
attempts. [4] explains the purpose of each module. So far, the approach 
in module /stream-like/ offers the best user experience. I would name 
this plan A, which is going to be implemented for infinispan 6.0.

Plan A can also use the classes generated by google's protoc code 
generator tool. So if somebody prefers to go old school it can still 
work well. Also, the amount of new support code we added on top of 
google's library is small, so I don't foresee any nightmare in porting 
this to another language.

I would like to get some feedback from anyone who has some time to have 
a look at [3], specifically at the /stream-like/ module.
Stream-like is about 89.9% implemented. There are still some 
unimplemented methods but do not mind them for the review.

And finally, if anyone could suggest a better name for the stream-like 
package? I can't think of any other option except /streamlike/ :) (which 
might be trade-marked) So any other options? If no options then we'll 
just call it /marshaling/. I'll move this to the ispn branch once I know 
the package name :)

Have a nice weekend guys!

--------------
[1] https://community.jboss.org/wiki/RemoteQueryDesignInInfinispan
[2] https://developers.google.com/protocol-buffers/
[3] https://github.com/anistor/protobuf-playground
[4] https://github.com/anistor/protobuf-playground/blob/master/README.md
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/infinispan-dev/attachments/20130628/4779a056/attachment.html