[infinispan-dev] input JSON -> convert -> put into ISPN cache & ready for Queries

Wed Oct 2 03:35:28 EDT 2013

OData, RDF and to some extend GData try to offer a schema, API and query
capability atop HTTP and other "web" techs.

It would be an interesting project to build OData support on top of
Infinispan and see where that goes in the community. That support would
live alongside memcache and HotRod.

But when looking at these protocols, it feels like they are designed for
portability more than raw performance which is what Infinispan Core is after. Oh and
who cares about transactions, it's the web and we are in 2013 ;) More
seriously AFAIK, OData only supports batched operation which is a poor
man's version of transaction but nowhere near what's necessary.

On another note, the JSON wrapper you describe is how we have done the
query support. In Hibernate Search 5, we are planning to internalize
this kind of support and allow any data besides plain objects (JSON, Protobuf etc)
to be indexed.

Emmanuel

On Fri 2013-09-27  3:25, Tomas Sykora wrote:
> Hi Adrian! (and devs)
> Thank you very much for your input and for your blog post around remote queries as well.. it helped me a lot.
> 
> Please, let me shortly introduce you my idea of OData Infinispan server.
> 
> At first, What's ODATA? http://www.odata.org/introduction/
> If I should point out one paragraph from -Why use the Open Data Protocol?- section: "...OData solves this problem by providing a uniform way to expose, structure, query and manipulate data using REST practices and JSON or ATOM syntax to describe the payload. OData also provides a uniform way to represent metadata about the data, allowing computers to know more about the type system, relationships and structure of the data."
> 
> I think this is the way how to expose our cached data in infinispan to wide ecosystem of clients. These are capable to interact with OData service and they understand it.
> 
> Clients will be storing "JSON objects" into infinispan cache and will be able to get it back based on the String key.
> 
> OData provides its query language which helps you to filter entities. And when you are storing byte[] into the cache or some heterogenous objects it's hard to query anything. Also client and server needs to be aware of what's exactly stored into the cache.
> When used JSON this is (maybe?) solved. 
> 
> The question is how to qeury through JSON objects in the cache.
> Some clients wants to store cars there, some others books, users etc. 
> 
> That's why I was firstly thinking about dynamic generation of Java classes and annotate them by @Indexed etc. So deserialize JSON stuff into Indexed Java object instance and put it into the cache. But this is very complex and if I can manage it, it would be VERY slow in the end.
> 
> I had a brainstorming with Anna and we had idea of JSONWrapper.
> 
> @Indexed
> class JSONWrapper {
>   @ Field String JSONString;
>   @ Field String entityKey or entityClass/Type; ?? maybe ?? 
> }
> 
> So once we got JSON from client we put in to the cache wrapped inside this. => put(stringKey, jsonWrappedObject);
> 
> Now clients will query OData service using OData query conventions. I will map these OData queries to meaningful infinispan queries and query our cache for the results.
> 
> Sorry for pseudo-language only:
> 
> onField(JSONString).match(something).wildCard("*"carColor": "red"*"); and this should select all objects which contains "carColor": "red". 
> 
> If needed entity type can be added into every JSON object to have "entityType": "Car.class" and add it as AND condition to query to be sure about returning only car objects, because we can have basically anything in our cache.
> 
> Then OData service is providing $metadata about itself so clients know where they can find their data, how to query it etc. etc. and this approach is uniform.
> 
> Thank you very much Adrian for your help.
> 
> Tomas 
> 
> 
> ----- Original Message -----
> > From: "Adrian Nistor" <anistor at redhat.com>
> > To: "infinispan -Dev List" <infinispan-dev at lists.jboss.org>, tsykora at redhat.com
> > Sent: Thursday, September 26, 2013 9:01:29 PM
> > Subject: Re: [infinispan-dev] input JSON -> convert -> put into ISPN cache & ready for Queries
> > 
> > Tomas, it seems you're looking at remote query over REST (via JSON).
> > This has not been implemented yet but we might have it in 6.1.
> > 
> > In 6.0 we already have remote query over Hot Rod implemented (using
> > protobuf encoding - much more compact than JSON). Some rough details
> > available here:
> > https://community.jboss.org/wiki/RemoteQueryDesignInInfinispan
> > We did not go the bytecode generation of annotated classes route as
> > you're probably trying to do. Instead we use a pretty nifty generic
> > entity (ProtobufValueWrapper) and a HSearch class bridge
> > (ProtobufValueWrapperFieldBridge) - all configured programatically, no
> > annotations.
> > 
> > Let's chat tomorrow on #infinispan and see how we may help you.
> > 
> > Cheers
> > 
> > On 09/25/2013 11:13 PM, Tomas Sykora wrote:
> > > Hi team!
> > >
> > > I need to ask for your help.
> > > It's connected to the OData endpoint.
> > > (https://issues.jboss.org/browse/ISPN-2109) I was thinking about the
> > > design etc. and it would be nice to map OData queries to Infinispan
> > > queries so clients can get their results based on particular query.
> > >
> > > You know, there is basically not much to do with only schema-less key-value
> > > store. Like exposing only values to clients based on their key requests
> > > does not fully use OData capabilities.
> > >
> > > So I was thinking about something like that...
> > >
> > > >From any client you are sending JSON object (for example a Book, with
> > > >variables: title, author, description) to OData service and would like to
> > > >store query-able Book Object value into the cache "under" some key.
> > >
> > > So you go: JSON --> to query-able Book.class Object --> cache.put("key",
> > > bookFromJson);
> > > Then in pseudo query: get-me-books-filter-description-contains-"great IT
> > > book"-top-N-results --> issue query on cache, get results --> transform
> > > returned Books.class into JSON, return to client
> > >
> > > My question is:
> > >
> > > How to transform JSON input, which is in most cases simple String build
> > > according to JSON rules, into object, which is query-able and can be put
> > > into the cache.
> > >
> > > The thing is that you usually have java class:
> > >
> > > @Indexed
> > > Book {
> > >
> > > @Filed String title;
> > > @Filed String author;
> > >
> > > etc. etc.
> > >
> > > I simply don't know how to create an object ready for queries, or even
> > > annotated class and instantiate it for further put into the cache.
> > > I'm discovering this, recently: http://www.jboss.org/javassist
> > >
> > > Or can you see there any other, maybe totally different, approach how to do
> > > it?
> > >
> > > THANK YOU very much for any input!
> > > I'm stuck on this right now... that's why I'm asking for a help.
> > >
> > > Have a nice day all!
> > > Tomas
> > > _______________________________________________
> > > infinispan-dev mailing list
> > > infinispan-dev at lists.jboss.org
> > > https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > 
> > 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev