--
Galder Zamarreño
Infinispan, Red Hat
On 30 Mar 2017, at 17:15, Gustavo Fernandes
<gustavo(a)infinispan.org> wrote:
On Thu, Mar 30, 2017 at 1:51 PM, Galder Zamarreño <galder(a)redhat.com> wrote:
Hi all,
For a demo I'm giving next week, I'd like to show how to use distributed streams
via a remote server task. All server tasks that we have in testsuite rely on primitives
but in my case I wanted to use POJOs.
To do that, I needed to get compatibility mode working in such way that those POJOs could
be unmarshalled for the server task. Since in another demo I'm showing Protostream
based POJOs, I thought I'd try to use that as mechanism to unmarshall POJOs server
side.
We have a test for such scenario [1], but the reality (running on a proper server) is
anything that simple. Here's a list of things I've found out while creating a
WordCount example that relies on a POJO:
1. Out of the box, it's impossible to set compatibility marshaller to
org.infinispan.query.remote.CompatibilityProtoStreamMarshaller [1] because
"org.infinispan.main" classloader can't access that class. I worked around
that by tweaking the module.xml to have an optional dependency to
"org.infinispan.remote-query.server" module.
2. After doing that, I had to register the protofile and associated classes remotely in
the server. Again, there's no out of the box mechanism for that, so I created a remote
server task that would do that [3].
AFAICT, you should be able to do that doing a PUT in the Protobuf_Metadata cache, which
entails having auth enabled. This cache should be REPL_SYNC, so no need to run a server
task.
Good point but not so sure it completely removes the need for the task. The task does two
things:
1. Call ProtobufMetadataManager.registerProtofile, which as you say could be swapped with
a cache.put on the metadata cache.
2. Call ProtobufMetadataManager.registerMarshaller. This goes deep into updating
SerializationContextImpl, which seems independent of any replicated cache.
In fact, I had originally set up the task to execute in only in one node, but when I did
that I found that marshallers were not registered in all nodes, so I had to execute the
task in all nodes.
I guess the task could be just limited to only executing 2.) in all nodes (and store the
protofile contents by accessing the cache remotely), but I can't see how I can avoid
the task altogether right now.
3. Finally, with all that in place, I was able to complete the WordCount test [4] with a
final caveat: the return of the word count, and words protofile registration, tasks return
objects that are not marshalled by the compatibility marshaller, so I had to make sure
that the remote cache manager used for those tasks uses the default marshaller.
Clearly we need to improve on this, and we have plans to address these issues (with new
upcoming transcoding capabilities), but I thought it'd be worth mentioning the
problems found in case anyone else encounters them before transcoding is in place.
Cheers,
[1]
https://github.com/galderz/datagrid-patterns/blob/master/server-config/do...
[2]
https://github.com/galderz/datagrid-patterns/blob/master/server-config/or...
[3]
https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream...
[4]
https://github.com/galderz/datagrid-patterns/blob/master/analytics-stream...
--
Galder Zamarreño
Infinispan, Red Hat
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev