8:03 a.m.
Hi all,
As I write the Javascript client for Hot Rod, and Vittorio writes the C++ client, the
question the encoding of the byte arrays has popped up.
The reason why encoding matters is mainly because of compatibility mode. How does a Hot
Rod client know how it should transform something a REST client set?
To be able to answer this question correctly, the Hot Rod client needs to know the type of
the data.
Although we could consider adding encoding information to the Hot Rod protocol long term,
in the short term this question might already been answered by Protostream.
Protostream based marshaller adds encoding to the byte arrays by adding information of
what the type of that is written, whether raw type or complex. Protostream is based around
Protobuf which works well as medium for compatibility mode.
So, in essence, if we want get different clients working together, the simplest thing to
do is to implement the Protostream marshaller in each client lang, and they should work
together without probs.
This option has the downside that by limiting our compatibility marshalling to
Protostream, how would a standard REST client would understand this format? They'd
need to have a way to understand Protostream...
Another option would be to setlle on UTF-8 String as compatibility medium and get the
languages to convert types into JSON UTF-8 String. I think this would work better for
REST-based clients. Also, AFAIK protobuf objects can be transformed into JSON, so if
there's already a protostream marshaller available, that can be used to transform to
JSON.
Down the line we might want to add encoding to Hot Rod or switch to a technology where
encoding is sorted out for us, e.g. HTTP-based protocols...
Thoughts?
Cheers,
--
Galder Zamarreño
Infinispan, Red Hat
4:38 a.m.
On Mon, Jan 25, 2016 at 2:03 PM, Galder Zamarreño <galder(a)redhat.com> wrote:
Hi all,
As I write the Javascript client for Hot Rod, and Vittorio writes the C++
client, the question the encoding of the byte arrays has popped up.
The reason why encoding matters is mainly because of compatibility mode.
How does a Hot Rod client know how it should transform something a REST
client set?
To be able to answer this question correctly, the Hot Rod client needs to
know the type of the data.
Although we could consider adding encoding information to the Hot Rod
protocol long term, in the short term this question might already been
answered by Protostream.
Compatibility between disparate clients should certainly be supported, but
what about a pluggable marshaller mechanism? Let's consider all usage
scenarios:
* Only Java clients: in most cases, JBoss marshalling is adequate, no need
to involve protobuf or json and since JBoss Marshalling is default, no need
to configure anything.
* Mix of Java and C++/C#/Python clients: the Protobuf encoding works
wonderfully for that. The client could be configured to use the protostream
marshaller, the same for the server.
* Mix of Java, C++/C#/Python and REST clients: protobuf with json [1]
encoding is an interesting option. Same as above, a protostream marshaller
with json support could
be configured both on client and server.
* Custom marshallers: consider FlatBuffers [2], for example, an interesting
new cross-platform serializer that allows to access serialized data without
de-serializing the whole payload,
and without generating any transient memory, and it optionally supports
JSON. It could be interesting to write a marshaller based on it and plug it
on Infinispan.
Although we have a way of configuring the marshaller on client (via
RemoteCacheManager) and server (deployable jar), there'd be more work to do
in order make them really pluggable:
- Some serialization formats like Protobuf and [2] require interface
description to be available on both client and server. This is not
supported for all marshallers, just for Protostream
- Remote query requires some metadata regarding which fields to index and
how, and this would need to be possible on every language regardless of the
marshalling.
Currently, this info can be encoded in the .proto file only, a custom
marshaller would not work.
- Currently remote query is settled on protobuf not only for the byte array
but for the the HotRod query request and response [3]
Is having a pluggable serializer mechanism too far fetched, given the
effort already invested towards protobuf?
Gustavo
[1] https://developers.google.com/protocol-buffers/docs/proto3#json
[2] https://google.github.io/flatbuffers/
[2]
https://code.facebook.com/posts/872547912839369/improving-facebook-s-perf...
[3]
https://github.com/infinispan/infinispan/blob/master/remote-query/remote-...
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
5:27 a.m.
It sounds like a good idea, almost like a natural evolution, but to
play devil's advocate I'll try to find some drawbacks for such a
decision.
One negative argument is overall complexity: there are many points in
code in which one needs to consider that the encoding might be
"something else". This isn't a new problem as we already support two
modes, but we've soon how much more difficult it makes things and this
is making things even a bit more complex.
Another point which I don't like much is that people will have to
reconfigure the server depending on specific needs of the clients; if
we go this way there should be a way in the Hot Rod protocol to
"negotiate encoding", so to avoid such configuration tasks for end
users (at least when there's no need to actually deploy dependencies
on the server to handle some custom encoder..).
The other problem I see is that this severely complicates
interoperability between different clients. Suppose two applications
are being developed to use the Infinispan Server and decide to use two
different encoders, I suspect that at some point they'll regret it
when needing one to access some data from the other... not suggesting
that this should be a good practice, but still it would be very
inconvenient.
Finally, tooling. We'll eventually need to work on better tooling and
supporting multiple encoders would spread efforts thinly.
That said, I'm not against the idea of toying with such options and
make other encoders as an *experimental* feature: protobuf is the most
suited choice but forever is a long time and we should not hinder
research on better alternatives.
Thanks,
Sanne
On 3 February 2016 at 10:38, Gustavo Fernandes <gustavo(a)infinispan.org> wrote:
On Mon, Jan 25, 2016 at 2:03 PM, Galder Zamarreño <galder(a)redhat.com> wrote:
>
> Hi all,
>
> As I write the Javascript client for Hot Rod, and Vittorio writes the C++
> client, the question the encoding of the byte arrays has popped up.
>
> The reason why encoding matters is mainly because of compatibility mode.
> How does a Hot Rod client know how it should transform something a REST
> client set?
>
> To be able to answer this question correctly, the Hot Rod client needs to
> know the type of the data.
>
> Although we could consider adding encoding information to the Hot Rod
> protocol long term, in the short term this question might already been
> answered by Protostream.
Compatibility between disparate clients should certainly be supported, but
what about a pluggable marshaller mechanism? Let's consider all usage
scenarios:
* Only Java clients: in most cases, JBoss marshalling is adequate, no need
to involve protobuf or json and since JBoss Marshalling is default, no need
to configure anything.
* Mix of Java and C++/C#/Python clients: the Protobuf encoding works
wonderfully for that. The client could be configured to use the protostream
marshaller, the same for the server.
* Mix of Java, C++/C#/Python and REST clients: protobuf with json [1]
encoding is an interesting option. Same as above, a protostream marshaller
with json support could
be configured both on client and server.
* Custom marshallers: consider FlatBuffers [2], for example, an interesting
new cross-platform serializer that allows to access serialized data without
de-serializing the whole payload,
and without generating any transient memory, and it optionally supports
JSON. It could be interesting to write a marshaller based on it and plug it
on Infinispan.
Although we have a way of configuring the marshaller on client (via
RemoteCacheManager) and server (deployable jar), there'd be more work to do
in order make them really pluggable:
- Some serialization formats like Protobuf and [2] require interface
description to be available on both client and server. This is not supported
for all marshallers, just for Protostream
- Remote query requires some metadata regarding which fields to index and
how, and this would need to be possible on every language regardless of the
marshalling.
Currently, this info can be encoded in the .proto file only, a custom
marshaller would not work.
- Currently remote query is settled on protobuf not only for the byte array
but for the the HotRod query request and response [3]
Is having a pluggable serializer mechanism too far fetched, given the effort
already invested towards protobuf?
Gustavo
[1] https://developers.google.com/protocol-buffers/docs/proto3#json
[2] https://google.github.io/flatbuffers/
[2]
https://code.facebook.com/posts/872547912839369/improving-facebook-s-perf...
[3]
https://github.com/infinispan/infinispan/blob/master/remote-query/remote-...
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
7:08 a.m.
On Wed, Feb 3, 2016 at 11:27 AM, Sanne Grinovero <sanne(a)infinispan.org>
wrote:
It sounds like a good idea, almost like a natural evolution, but to
play devil's advocate I'll try to find some drawbacks for such a
decision.
One negative argument is overall complexity: there are many points in
code in which one needs to consider that the encoding might be
"something else". This isn't a new problem as we already support two
modes, but we've soon how much more difficult it makes things and this
is making things even a bit more complex.
Ideally those several points would need to be formalized into an SPI,
allowing
customizations like it was done for Avro [1] a bit easier maybe.
I am not familiar with all those points, but I imagine it'd be a complex
refactor
nonetheless :)
The other problem I see is that this severely complicates
interoperability between different clients. Suppose two applications
are being developed to use the Infinispan Server and decide to use two
different encoders, I suspect that at some point they'll regret it
when needing one to access some data from the other... not suggesting
that this should be a good practice, but still it would be very
inconvenient.
I think the issue is the same that happens today: a remote java client
using
JBoss marshaller wants to start doing queries, it will not work because the
server
expects Protobuf.
Finally, tooling. We'll eventually need to work on better tooling
and
supporting multiple encoders would spread efforts thinly.
I assume you mean tooling to build interface definitions in the case of
protobuf or similar? They might already be out there [2][3][4]
Gustavo
[1] https://github.com/leads-project/infinispan-avro
[2] https://code.google.com/p/protobuf-dt/
[3] https://plugins.jetbrains.com/plugin/4942?pr=idea
[4] https://plugins.jetbrains.com/plugin/7971?pr=idea
5:34 a.m.
It sounds like a good idea, almost like a natural evolution, but to
play devil's advocate I'll try to find some drawbacks for such a
decision.
One negative argument is overall complexity: there are many points in
code in which one needs to consider that the encoding might be
"something else". This isn't a new problem as we already support two
modes, but we've soon how much more difficult it makes things and this
is making things even a bit more complex.
Another point which I don't like much is that people will have to
reconfigure the server depending on specific needs of the clients; if
we go this way there should be a way in the Hot Rod protocol to
"negotiate encoding", so to avoid such configuration tasks for end
users (at least when there's no need to actually deploy dependencies
on the server to handle some custom encoder..).
The other problem I see is that this severely complicates
interoperability between different clients. Suppose two applications
are being developed to use the Infinispan Server and decide to use two
different encoders, I suspect that at some point they'll regret it
when needing one to access some data from the other... not suggesting
that this should be a good practice, but still it would be very
inconvenient.
Finally, tooling. We'll eventually need to work on better tooling and
supporting multiple encoders would spread efforts thinly.
That said, I'm not against the idea of toying with such options and
make other encoders as an *experimental* feature: protobuf is the most
suited choice but forever is a long time and we should not hinder
research on better alternatives.
Thanks,
Sanne
On 3 February 2016 at 10:38, Gustavo Fernandes <gustavo(a)infinispan.org> wrote:
On Mon, Jan 25, 2016 at 2:03 PM, Galder Zamarreño <galder(a)redhat.com> wrote:
>
> Hi all,
>
> As I write the Javascript client for Hot Rod, and Vittorio writes the C++
> client, the question the encoding of the byte arrays has popped up.
>
> The reason why encoding matters is mainly because of compatibility mode.
> How does a Hot Rod client know how it should transform something a REST
> client set?
>
> To be able to answer this question correctly, the Hot Rod client needs to
> know the type of the data.
>
> Although we could consider adding encoding information to the Hot Rod
> protocol long term, in the short term this question might already been
> answered by Protostream.
Compatibility between disparate clients should certainly be supported, but
what about a pluggable marshaller mechanism? Let's consider all usage
scenarios:
* Only Java clients: in most cases, JBoss marshalling is adequate, no need
to involve protobuf or json and since JBoss Marshalling is default, no need
to configure anything.
* Mix of Java and C++/C#/Python clients: the Protobuf encoding works
wonderfully for that. The client could be configured to use the protostream
marshaller, the same for the server.
* Mix of Java, C++/C#/Python and REST clients: protobuf with json [1]
encoding is an interesting option. Same as above, a protostream marshaller
with json support could
be configured both on client and server.
* Custom marshallers: consider FlatBuffers [2], for example, an interesting
new cross-platform serializer that allows to access serialized data without
de-serializing the whole payload,
and without generating any transient memory, and it optionally supports
JSON. It could be interesting to write a marshaller based on it and plug it
on Infinispan.
Although we have a way of configuring the marshaller on client (via
RemoteCacheManager) and server (deployable jar), there'd be more work to do
in order make them really pluggable:
- Some serialization formats like Protobuf and [2] require interface
description to be available on both client and server. This is not supported
for all marshallers, just for Protostream
- Remote query requires some metadata regarding which fields to index and
how, and this would need to be possible on every language regardless of the
marshalling.
Currently, this info can be encoded in the .proto file only, a custom
marshaller would not work.
- Currently remote query is settled on protobuf not only for the byte array
but for the the HotRod query request and response [3]
Is having a pluggable serializer mechanism too far fetched, given the effort
already invested towards protobuf?
Gustavo
[1] https://developers.google.com/protocol-buffers/docs/proto3#json
[2] https://google.github.io/flatbuffers/
[2]
https://code.facebook.com/posts/872547912839369/improving-facebook-s-perf...
[3]
https://github.com/infinispan/infinispan/blob/master/remote-query/remote-...
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
7:07 a.m.
Hi all,
Experimenting with custom marshallers is possible right now. People just
loose query support doing so. Freedom has its cost ... (kidding)
There are a few simple things we can do right now to better support at
least two client categories (and not loose query support) without much
changes to current state of the code:
1. provide json support on top of protobuf, via a canonical json <->
protobuf translation. We've been talking about this since ispn 7.x.
Meanwhile protobuf3 is out and provides it out of the box, as Gustavo
pointed out, so there is even less effort. This would be useful for both
REST and hot rod.
2. better support for remote query for java clients using
jbossmarshalling (in compat mode only). Would require hibernate-search
annotated entities be available on server too, but would spare the user
from touching protobuf if he's not interested in interop. The only
reason this does not work already is because it was not among the
initial set of requirements when designing remote query. 2.5 years later
it is clearly a must.
Infinispan 9 is not too far, so I propose to implement 1 and 2 as a
first step, then see what is the right time frame for the more involved
step of decoupling query module's source of metadata form the protobuf
schema/marshaller. I think this refactoring deserves to be done even for
the sake of elegance and maintainability, if not for plugability.
Re Protostream, it is just plain Protobuf folks! It's in no way a
customly-modified-encoding-loosely-based-on-protobuf. The only apparent
twist is our usage of Protobuf, which btw also follows Google's
recommendations (see [1]). All keys and values are wrapped into a struct
like this [2] (actually it is a union when looking closer). And for good
reason. Adding that metadata as a separate header at hot rod protocol
level would be another possibility, but really, having it stored in
cache with the key/value is much more appropriate. This type metadata
(the type name or type id) is needed for indexing too. And could also be
used by some smart cache stores to make some sense out of that blob and
map it to database columns or suchlike. And would also be needed by non
HR clients, like REST. I can go on .. :) The only downside I see is
space. Our caches are not homogeneous, unfortunately [let's not debate
now why], so the type info needs to be stored for each single key, value
instead of storing it once per cache.
Adrian
[1]
https://developers.google.com/protocol-buffers/docs/techniques#self-descr...
[2]
https://raw.githubusercontent.com/anistor/protostream/master/core/src/mai...
On 02/03/2016 01:34 PM, Sanne Grinovero wrote:
It sounds like a good idea, almost like a natural evolution, but to
play devil's advocate I'll try to find some drawbacks for such a
decision.
One negative argument is overall complexity: there are many points in
code in which one needs to consider that the encoding might be
"something else". This isn't a new problem as we already support two
modes, but we've soon how much more difficult it makes things and this
is making things even a bit more complex.
Another point which I don't like much is that people will have to
reconfigure the server depending on specific needs of the clients; if
we go this way there should be a way in the Hot Rod protocol to
"negotiate encoding", so to avoid such configuration tasks for end
users (at least when there's no need to actually deploy dependencies
on the server to handle some custom encoder..).
The other problem I see is that this severely complicates
interoperability between different clients. Suppose two applications
are being developed to use the Infinispan Server and decide to use two
different encoders, I suspect that at some point they'll regret it
when needing one to access some data from the other... not suggesting
that this should be a good practice, but still it would be very
inconvenient.
Finally, tooling. We'll eventually need to work on better tooling and
supporting multiple encoders would spread efforts thinly.
That said, I'm not against the idea of toying with such options and
make other encoders as an *experimental* feature: protobuf is the most
suited choice but forever is a long time and we should not hinder
research on better alternatives.
Thanks,
Sanne
On 3 February 2016 at 10:38, Gustavo Fernandes <gustavo(a)infinispan.org> wrote:
>
> On Mon, Jan 25, 2016 at 2:03 PM, Galder Zamarreño <galder(a)redhat.com> wrote:
>> Hi all,
>>
>> As I write the Javascript client for Hot Rod, and Vittorio writes the C++
>> client, the question the encoding of the byte arrays has popped up.
>>
>> The reason why encoding matters is mainly because of compatibility mode.
>> How does a Hot Rod client know how it should transform something a REST
>> client set?
>>
>> To be able to answer this question correctly, the Hot Rod client needs to
>> know the type of the data.
>>
>> Although we could consider adding encoding information to the Hot Rod
>> protocol long term, in the short term this question might already been
>> answered by Protostream.
>
>
> Compatibility between disparate clients should certainly be supported, but
> what about a pluggable marshaller mechanism? Let's consider all usage
> scenarios:
>
> * Only Java clients: in most cases, JBoss marshalling is adequate, no need
> to involve protobuf or json and since JBoss Marshalling is default, no need
> to configure anything.
>
> * Mix of Java and C++/C#/Python clients: the Protobuf encoding works
> wonderfully for that. The client could be configured to use the protostream
> marshaller, the same for the server.
>
> * Mix of Java, C++/C#/Python and REST clients: protobuf with json [1]
> encoding is an interesting option. Same as above, a protostream marshaller
> with json support could
> be configured both on client and server.
>
> * Custom marshallers: consider FlatBuffers [2], for example, an interesting
> new cross-platform serializer that allows to access serialized data without
> de-serializing the whole payload,
> and without generating any transient memory, and it optionally supports
> JSON. It could be interesting to write a marshaller based on it and plug it
> on Infinispan.
>
> Although we have a way of configuring the marshaller on client (via
> RemoteCacheManager) and server (deployable jar), there'd be more work to do
> in order make them really pluggable:
>
> - Some serialization formats like Protobuf and [2] require interface
> description to be available on both client and server. This is not supported
> for all marshallers, just for Protostream
> - Remote query requires some metadata regarding which fields to index and
> how, and this would need to be possible on every language regardless of the
> marshalling.
> Currently, this info can be encoded in the .proto file only, a custom
> marshaller would not work.
> - Currently remote query is settled on protobuf not only for the byte array
> but for the the HotRod query request and response [3]
>
> Is having a pluggable serializer mechanism too far fetched, given the effort
> already invested towards protobuf?
>
> Gustavo
>
> [1] https://developers.google.com/protocol-buffers/docs/proto3#json
> [2] https://google.github.io/flatbuffers/
> [2]
>
https://code.facebook.com/posts/872547912839369/improving-facebook-s-perf...
> [3]
>
https://github.com/infinispan/infinispan/blob/master/remote-query/remote-...
>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
5:48 a.m.
On 3 Feb 2016, at 14:07, Adrian Nistor <anistor(a)redhat.com>
wrote:
Hi all,
</snip>
Re Protostream, it is just plain Protobuf folks! It's in no way a
customly-modified-encoding-loosely-based-on-protobuf. The only apparent
twist is our usage of Protobuf, which btw also follows Google's
recommendations (see [1]). All keys and values are wrapped into a struct
like this [2] (actually it is a union when looking closer). And for good
reason. Adding that metadata as a separate header at hot rod protocol
level would be another possibility, but really, having it stored in
cache with the key/value is much more appropriate. This type metadata
(the type name or type id) is needed for indexing too. And could also be
used by some smart cache stores to make some sense out of that blob and
map it to database columns or suchlike. And would also be needed by non
HR clients, like REST. I can go on .. :) The only downside I see is
space. Our caches are not homogeneous, unfortunately [let's not debate
now why], so the type info needs to be stored for each single key, value
instead of storing it once per cache.
^ I'm not sure Protostream is just plain Protobuf... If you look at [3], you see very
specific encoding/decoding steps about the wrapped message.
Could I just read Protostream stuff from a JS client with only a .proto file? Looking at
[3], it seems like that's not the case and hence I think Protostream imposes certain
type of encoding.
It looks to me I'd have to encode/decode each basic primitive types manually in the JS
client encoder to support Protostream, and only for custom types provided via a .proto
file I can delegate to Protobuf itself.
@Adrian, am I reading this properly?
Cheers,
[3]
https://github.com/infinispan/protostream/blob/master/core/src/main/java/...
Adrian
[1]
https://developers.google.com/protocol-buffers/docs/techniques#self-descr...
[2]
https://raw.githubusercontent.com/anistor/protostream/master/core/src/mai...
On 02/03/2016 01:34 PM, Sanne Grinovero wrote:
> It sounds like a good idea, almost like a natural evolution, but to
> play devil's advocate I'll try to find some drawbacks for such a
> decision.
>
> One negative argument is overall complexity: there are many points in
> code in which one needs to consider that the encoding might be
> "something else". This isn't a new problem as we already support two
> modes, but we've soon how much more difficult it makes things and this
> is making things even a bit more complex.
>
> Another point which I don't like much is that people will have to
> reconfigure the server depending on specific needs of the clients; if
> we go this way there should be a way in the Hot Rod protocol to
> "negotiate encoding", so to avoid such configuration tasks for end
> users (at least when there's no need to actually deploy dependencies
> on the server to handle some custom encoder..).
>
> The other problem I see is that this severely complicates
> interoperability between different clients. Suppose two applications
> are being developed to use the Infinispan Server and decide to use two
> different encoders, I suspect that at some point they'll regret it
> when needing one to access some data from the other... not suggesting
> that this should be a good practice, but still it would be very
> inconvenient.
>
> Finally, tooling. We'll eventually need to work on better tooling and
> supporting multiple encoders would spread efforts thinly.
>
> That said, I'm not against the idea of toying with such options and
> make other encoders as an *experimental* feature: protobuf is the most
> suited choice but forever is a long time and we should not hinder
> research on better alternatives.
>
> Thanks,
> Sanne
>
>
>
>
> On 3 February 2016 at 10:38, Gustavo Fernandes <gustavo(a)infinispan.org> wrote:
>>
>> On Mon, Jan 25, 2016 at 2:03 PM, Galder Zamarreño <galder(a)redhat.com>
wrote:
>>> Hi all,
>>>
>>> As I write the Javascript client for Hot Rod, and Vittorio writes the C++
>>> client, the question the encoding of the byte arrays has popped up.
>>>
>>> The reason why encoding matters is mainly because of compatibility mode.
>>> How does a Hot Rod client know how it should transform something a REST
>>> client set?
>>>
>>> To be able to answer this question correctly, the Hot Rod client needs to
>>> know the type of the data.
>>>
>>> Although we could consider adding encoding information to the Hot Rod
>>> protocol long term, in the short term this question might already been
>>> answered by Protostream.
>>
>>
>> Compatibility between disparate clients should certainly be supported, but
>> what about a pluggable marshaller mechanism? Let's consider all usage
>> scenarios:
>>
>> * Only Java clients: in most cases, JBoss marshalling is adequate, no need
>> to involve protobuf or json and since JBoss Marshalling is default, no need
>> to configure anything.
>>
>> * Mix of Java and C++/C#/Python clients: the Protobuf encoding works
>> wonderfully for that. The client could be configured to use the protostream
>> marshaller, the same for the server.
>>
>> * Mix of Java, C++/C#/Python and REST clients: protobuf with json [1]
>> encoding is an interesting option. Same as above, a protostream marshaller
>> with json support could
>> be configured both on client and server.
>>
>> * Custom marshallers: consider FlatBuffers [2], for example, an interesting
>> new cross-platform serializer that allows to access serialized data without
>> de-serializing the whole payload,
>> and without generating any transient memory, and it optionally supports
>> JSON. It could be interesting to write a marshaller based on it and plug it
>> on Infinispan.
>>
>> Although we have a way of configuring the marshaller on client (via
>> RemoteCacheManager) and server (deployable jar), there'd be more work to do
>> in order make them really pluggable:
>>
>> - Some serialization formats like Protobuf and [2] require interface
>> description to be available on both client and server. This is not supported
>> for all marshallers, just for Protostream
>> - Remote query requires some metadata regarding which fields to index and
>> how, and this would need to be possible on every language regardless of the
>> marshalling.
>> Currently, this info can be encoded in the .proto file only, a custom
>> marshaller would not work.
>> - Currently remote query is settled on protobuf not only for the byte array
>> but for the the HotRod query request and response [3]
>>
>> Is having a pluggable serializer mechanism too far fetched, given the effort
>> already invested towards protobuf?
>>
>> Gustavo
>>
>> [1] https://developers.google.com/protocol-buffers/docs/proto3#json
>> [2] https://google.github.io/flatbuffers/
>> [2]
>>
https://code.facebook.com/posts/872547912839369/improving-facebook-s-perf...
>> [3]
>>
https://github.com/infinispan/infinispan/blob/master/remote-query/remote-...
>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
4:07 a.m.
On Wed, Feb 3, 2016 at 1:07 PM, Adrian Nistor <anistor(a)redhat.com> wrote:
2. better support for remote query for java clients using
jbossmarshalling (in compat mode only).
Would require hibernate-search
annotated entities be available on server too, but would spare the
user
from touching protobuf if he's not interested in interop. The only
reason this does not work already is because it was not among the
initial set of requirements when designing remote query. 2.5 years later
it is clearly a must.
Further down the line, we'd probably benefit from having a language neutral
way of describing index
configuration (analyzers,fields, classes etc). This config would not be
tied to hibernate search or any particular
marshaller and it could be automatically inferred for java users if they
already have hibernate search annotated
classes. Having this would allow fulltext capabilities from any language
and across marhsallers. WDYT?
Infinispan 9 is not too far, so I propose to implement 1 and 2 as a
first step, then see what is the right time frame for the more involved
step of decoupling query module's source of metadata form the protobuf
schema/marshaller. I think this refactoring deserves to be done even for
the sake of elegance and maintainability, if not for plugability.
+1
Re Protostream, it is just plain Protobuf folks! It's in no way a
customly-modified-encoding-loosely-based-on-protobuf.
That's clear :) , but looking at the Hot Rod protocol documentation, the
Query request itself
is encoded in protobuf [1], not only the struct with key/values. Wouldn't
it be better to isolate
what is marshaller byproduct and what is Hot Rod protocol related? This will
make the job of the Hot Rod client implementors easier, and it'd be another
step towards a pluggable
marshaller.
[1]
https://github.com/infinispan/infinispan/blob/master/remote-query/remote-...
Gustavo
5:35 a.m.
I like the idea of pluggable serialization/marshalling, but as you rightly explained, what
flexibility gives you you lose by lack of functionality. E.g. querying is only available
for protostream based encoding.
So, I think we need to remove the hard bind between functionality and encoding to be able
to have a trully pluggable encoding mechanism.
Cheers,
--
Galder Zamarreño
Infinispan, Red Hat
On 3 Feb 2016, at 11:38, Gustavo Fernandes
<gustavo(a)infinispan.org> wrote:
On Mon, Jan 25, 2016 at 2:03 PM, Galder Zamarreño <galder(a)redhat.com> wrote:
Hi all,
As I write the Javascript client for Hot Rod, and Vittorio writes the C++ client, the
question the encoding of the byte arrays has popped up.
The reason why encoding matters is mainly because of compatibility mode. How does a Hot
Rod client know how it should transform something a REST client set?
To be able to answer this question correctly, the Hot Rod client needs to know the type
of the data.
Although we could consider adding encoding information to the Hot Rod protocol long term,
in the short term this question might already been answered by Protostream.
Compatibility between disparate clients should certainly be supported, but what about a
pluggable marshaller mechanism? Let's consider all usage scenarios:
* Only Java clients: in most cases, JBoss marshalling is adequate, no need to involve
protobuf or json and since JBoss Marshalling is default, no need to configure anything.
* Mix of Java and C++/C#/Python clients: the Protobuf encoding works wonderfully for
that. The client could be configured to use the protostream marshaller, the same for the
server.
* Mix of Java, C++/C#/Python and REST clients: protobuf with json [1] encoding is an
interesting option. Same as above, a protostream marshaller with json support could
be configured both on client and server.
* Custom marshallers: consider FlatBuffers [2], for example, an interesting new
cross-platform serializer that allows to access serialized data without de-serializing the
whole payload,
and without generating any transient memory, and it optionally supports JSON. It could be
interesting to write a marshaller based on it and plug it on Infinispan.
Although we have a way of configuring the marshaller on client (via RemoteCacheManager)
and server (deployable jar), there'd be more work to do in order make them really
pluggable:
- Some serialization formats like Protobuf and [2] require interface description to be
available on both client and server. This is not supported for all marshallers, just for
Protostream
- Remote query requires some metadata regarding which fields to index and how, and this
would need to be possible on every language regardless of the marshalling.
Currently, this info can be encoded in the .proto file only, a custom marshaller would
not work.
- Currently remote query is settled on protobuf not only for the byte array but for the
the HotRod query request and response [3]
Is having a pluggable serializer mechanism too far fetched, given the effort already
invested towards protobuf?
Gustavo
[1] https://developers.google.com/protocol-buffers/docs/proto3#json
[2] https://google.github.io/flatbuffers/
[2]
https://code.facebook.com/posts/872547912839369/improving-facebook-s-perf...
[3]
https://github.com/infinispan/infinispan/blob/master/remote-query/remote-...
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
7:53 a.m.
Note that in a future version of hibernate search - soon I hope, the encoding will no
longer be a problem. You will be able to provide a navigation/traversal API that will know
how to read things from your blob. It's called free form entity by its friends.
On 17 févr. 2016, at 12:35, Galder Zamarreño
<galder(a)redhat.com> wrote:
I like the idea of pluggable serialization/marshalling, but as you rightly explained,
what flexibility gives you you lose by lack of functionality. E.g. querying is only
available for protostream based encoding.
So, I think we need to remove the hard bind between functionality and encoding to be able
to have a trully pluggable encoding mechanism.
Cheers,
--
Galder Zamarreño
Infinispan, Red Hat
> On 3 Feb 2016, at 11:38, Gustavo Fernandes <gustavo(a)infinispan.org> wrote:
>
>
>
> On Mon, Jan 25, 2016 at 2:03 PM, Galder Zamarreño <galder(a)redhat.com> wrote:
> Hi all,
>
> As I write the Javascript client for Hot Rod, and Vittorio writes the C++ client, the
question the encoding of the byte arrays has popped up.
>
> The reason why encoding matters is mainly because of compatibility mode. How does a
Hot Rod client know how it should transform something a REST client set?
>
> To be able to answer this question correctly, the Hot Rod client needs to know the
type of the data.
>
> Although we could consider adding encoding information to the Hot Rod protocol long
term, in the short term this question might already been answered by Protostream.
>
>
> Compatibility between disparate clients should certainly be supported, but what about
a pluggable marshaller mechanism? Let's consider all usage scenarios:
>
> * Only Java clients: in most cases, JBoss marshalling is adequate, no need to involve
protobuf or json and since JBoss Marshalling is default, no need to configure anything.
>
> * Mix of Java and C++/C#/Python clients: the Protobuf encoding works wonderfully for
that. The client could be configured to use the protostream marshaller, the same for the
server.
>
> * Mix of Java, C++/C#/Python and REST clients: protobuf with json [1] encoding is an
interesting option. Same as above, a protostream marshaller with json support could
> be configured both on client and server.
>
> * Custom marshallers: consider FlatBuffers [2], for example, an interesting new
cross-platform serializer that allows to access serialized data without de-serializing the
whole payload,
> and without generating any transient memory, and it optionally supports JSON. It
could be interesting to write a marshaller based on it and plug it on Infinispan.
>
> Although we have a way of configuring the marshaller on client (via
RemoteCacheManager) and server (deployable jar), there'd be more work to do in order
make them really pluggable:
>
> - Some serialization formats like Protobuf and [2] require interface description to
be available on both client and server. This is not supported for all marshallers, just
for Protostream
> - Remote query requires some metadata regarding which fields to index and how, and
this would need to be possible on every language regardless of the marshalling.
> Currently, this info can be encoded in the .proto file only, a custom marshaller
would not work.
> - Currently remote query is settled on protobuf not only for the byte array but for
the the HotRod query request and response [3]
>
> Is having a pluggable serializer mechanism too far fetched, given the effort already
invested towards protobuf?
>
> Gustavo
>
> [1] https://developers.google.com/protocol-buffers/docs/proto3#json
> [2] https://google.github.io/flatbuffers/
> [2]
https://code.facebook.com/posts/872547912839369/improving-facebook-s-perf...
> [3]
https://github.com/infinispan/infinispan/blob/master/remote-query/remote-...
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
3510
days inactive
3540
days old
infinispan-dev@lists.jboss.org
9 comments
5 participants
participants (5)
-
Adrian Nistor -
Emmanuel Bernard -
Galder Zamarreño -
Gustavo Fernandes -
Sanne Grinovero