6:06 a.m.
Hi all,
Re: https://jira.jboss.org/browse/ISPN-508. This is a bit of a long one, so take your time
:)
Over past few days I've been trying to come up with a Infinispan Marshaller
implementation based on one of those portable serialization libraries (i.e. protobuf,
thrift...etc) and I've reached the conclusion that it is possible to do so but
requires a fair bit of work on our side and performance/usability would decrease. Before
explaining my findings, let me explain the model that I'm trying to achieve:
I want to build a generic marshalling/unmarshalling implementation that's based on
language neutral serialization libraries and which does not require the unmarshalling code
to have knowledge of the target type. Think about Java Serialization or JBoss Marshalling
where the code written can be generic enough w/o needing to know what we're trying to
deserialize. The payload in these cases has enough information for the underlying library
to figure out the type, instantiate it and populate it accordingly.
The problem is that there's no such library in the space of portable serialization
libraries that provides this out of the box and let me explain why:
Pretty much any code using those libraries must use class type or schema name to
deserialize the payload. However, Infinispan based Marshaller does not have such
information when it's trying to deserialize the payload. Infinispan Marshaller based
on JDK serialization or JBoss Marshalling uses information in the payload to instantiate
the object. However, none of the libraries out there use this mechanism and instead some
(i.e. Protobufs) force the client code to do things like: Pojo.parseFrom(byte[]) to
generate instances of Pojo. In these cases for example, nothing stops you from writing a
UTF-8 string with the class name and put it at the beginning of the payload so the
deserialization part can be class agnostic, but this payload would not be portable. What
would a python client do with a String containing the java class name?
Based on the FileDescriptorSet information in
http://code.google.com/apis/protocolbuffers/docs/techniques.html I was able to hack
something that might work in a portable way. Given a FileDescriptorSet generated a class
compile time, I was able to match the protobuf name of a class with its java counter part.
So, before writing the protobuf generated byte[], i prepend it with protobuf class name so
that when reading, I can take the name, get the java class name and using reflection call
parseFrom method to convert the byte[] into a pojo. Note that DynamicMessage class hinted
in the the techniques page won't work cos it cannot create instances of pojos. It can
only create generic objects with fields that are accessed in a reflection style.
I also looked at what Avro offers but it does not fully fit either. They have a reflection
based serialization mechanism that doesn't require any precompilation, but it requires
some kind of type knowledge on the client code to deserialize, plus Avro themselves
recommend against it and I'm not sure how performant it'd be. Avro also includes
other marshalling mechanisms called specific (like protobufs one with precompiled classes)
and generic used to build dynamic objects on the fly. None of these two fit the bill. The
specific one is like protobufs with the disadvantage of having ugly code like
http://is.gd/dzwj3 where a static object has a strong reference to a <String,Class>
CHM, which would leak in an AS env.
Thrift has the same problems as stated for Protobufs but coudn't see an equivalent way
to get find the file descriptor set. Docu is way below what Protobuf offers and latest
version which is 0.2 has issues generating classes as stated in JIRA.
MessagePack has the capability to deserialize an object given a String representation of
the schema, so a solution like the protobuf one might be hackable. However, the generated
classes do not have an equals implementation (??) which is rather odd. Maybe it's due
to lack of maturity? Latest version is 0.3, so that might explain it. API wise,
MessagePack provides the API that suits best to what we'd want to do and avoids having
to use reflection to resolve the payload. However, looking around I couldn't see
similar API for the python language for example and similar to Protobuf, we'd have to
prepend the schema name key to the payload to then have the reading part lookup the entire
schema based on it.
So, I can see two solutions here bearing in mind that pretty much any solution would
require precompiling some classes:
- Try to build a marshaller using Protobuf or MessagePack where we enhance it to pass a
string key that permits the reading part to deserialize the payload in a generic way.
- Or try to build some wikis on how to integrate Protobuf/Thrift/MessagePack with Hot Rod
client so that they can generate the byte[] with these libraries and pass it to the
corresponding Hot Rod client. For the moment, we'd do this for the Hot Rod java
client. We would add more info once other language clients are available. The reason I
said about potentially showing various libraries is cos whereas Protobuf only supports
Java/Python/C++, Thrift supports loads more languages ( C++, Java, Python, PHP, Ruby,
Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml)
In terms of usability , I think the second option is a bit better because building the
marshaller requires steps for clients to generate the FileDescriptorSet and somehow pass
this to Infinispan so that the marshaller can use it. The second option has none of this
and the only added code on the client side is retrieving the byte[] and calling
Pojo.parseFrom(byte[]). Performance wise, the 2nd option would be better cos there's
no extra hacking needed to pass the String key around and so less bytes are sent around,
and no need to use reflection to resolve the payload.
It's a pity that I couldn't find a tool that fully fits our use case and I wonder
whether the need to deal with different languages makes coming up with such solution
difficult. I believe reflection based instantiation is present in C++ or Python but not
sure about other languages. It's also true that our use case is very specific where
we're building a tool where we don't have control over what people will put in the
cache. Generics could give us some hints for example but it's not mandatory and would
not solve the issue entirely.
Thoughts?
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache