On Wed 2014-02-05 15:53, Mircea Markus wrote: > On Feb 3, 2014, at 9:32 AM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > >> Sure searching for any cache is useful. What I was advocating is that if you search for more than one cache transparently, then you probably need to CRUD for more than one cache transparently as well. And this is not being discussed. > Not sure what you mean by CRUD over multiple caches? ATM one can run a TX over multiple caches, but I think there's something else you have in mind :-) //some unified query giving me entries pointing by fk copy to bar and //buz objects. So I need to manually load these references. //happy emmanuel Cache unifiedCache = cacheManager.getMotherOfAllCaches(); Bar bar = unifiedCache.get(foo); Buz buz = unifiedCache.get(baz); //not so happy emmanuel Cache fooCache = cacheManager.getCache("foo"); Bar bar = fooCache.get(foo); Cache bazCache = cacheManager.getCache("baz"); Buz buz = bazCache.put(baz);

>> I have to admit that having to add a cache name to the stored elements of the index documents makes me a bit sad. > sad because of the increased index size? It makes the index non natural and less reusable using direct Lucene APIs. But that might be less of a concern for Infinispan. >> I was already unhappy when I had to do it for class names. Renaming a cache will be a heavy operation too. >> Sanne, if we know that we don't share the semi index for different caches, can we avoid the need to store the cache name in each document? _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

What I am saying is that the idea of search across caches as appealing as it is is is not the whole story. People search, read, navigate and M/R their data in interleaved ways. You need to project and think about a 100-200 lines of code that would use that feature in combination with other related features to see if that will be useful in the end (or gimmicky) and if the user experience (API mostly in our case) will be good or make people kill themselves.

On Feb 5, 2014, at 7:34 PM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > On Wed 2014-02-05 17:44, Radim Vansa wrote: >> On 02/05/2014 05:30 PM, Emmanuel Bernard wrote: >>> On Wed 2014-02-05 15:53, Mircea Markus wrote: >>>> On Feb 3, 2014, at 9:32 AM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: >>>> >>>>> Sure searching for any cache is useful. What I was advocating is that if you search for more than one cache transparently, then you probably need to CRUD for more than one cache transparently as well. And this is not being discussed. >>>> Not sure what you mean by CRUD over multiple caches? ATM one can run a TX over multiple caches, but I think there's something else you have in mind :-) >>> >>> //some unified query giving me entries pointing by fk copy to bar and >>> //buz objects. So I need to manually load these references. >>> >>> //happy emmanuel >>> Cache unifiedCache = cacheManager.getMotherOfAllCaches(); >>> Bar bar = unifiedCache.get(foo); >>> Buz buz = unifiedCache.get(baz); >>> >>> //not so happy emmanuel >>> Cache fooCache = cacheManager.getCache("foo"); >>> Bar bar = fooCache.get(foo); >>> Cache bazCache = cacheManager.getCache("baz"); >>> Buz buz = bazCache.put(baz); >> >> cacheManager.getCache("foo").put("xxx", "yyy"); >> cacheManager.getCache("foo").put("xxx", "zzz"); >> >> String xxx = cacheManager.getMotherOfAllCaches().get("xxx"); >> System.out.println(xxx); >> >> What should it print? Should an exception be thrown? Or should get on >> mother of all caches return Map<Cache<String, String>, String>? >> > > Yes I'm aware of that. > What I am saying is that the idea of search across caches as > appealing as it is is is not the whole story. > > People search, read, navigate and M/R their data in interleaved ways. In all the non-trivial deployments I saw people used multiple caches for different data, instead of one. That's why for me this came as the straight forward way of structuring data and naturally I thought that querying multiple caches makes sense in this context: to allow querying to run over a model that is already in use and not to change the model to accommodate querying.

On Wed 2014-02-05 15:53, Mircea Markus wrote: > > On Feb 3, 2014, at 9:32 AM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > >> Sure searching for any cache is useful. What I was advocating is that if you search for more than one cache transparently, then you probably need to CRUD for more than one cache transparently as well. And this is not being discussed. > > Not sure what you mean by CRUD over multiple caches? ATM one can run a TX over multiple caches, but I think there's something else you have in mind :-) //some unified query giving me entries pointing by fk copy to bar and //buz objects. So I need to manually load these references. //happy emmanuel Cache unifiedCache = cacheManager.getMotherOfAllCaches(); Bar bar = unifiedCache.get(foo); Buz buz = unifiedCache.get(baz); //not so happy emmanuel Cache fooCache = cacheManager.getCache("foo"); Bar bar = fooCache.get(foo); Cache bazCache = cacheManager.getCache("baz"); Buz buz = bazCache.put(baz);

> >> >> I have to admit that having to add a cache name to the stored elements of the index documents makes me a bit sad. > > sad because of the increased index size? It makes the index non natural and less reusable using direct Lucene APIs. But that might be less of a concern for Infinispan. > >> I was already unhappy when I had to do it for class names. Renaming a cache will be a heavy operation too. >> Sanne, if we know that we don't share the semi index for different caches, can we avoid the need to store the cache name in each document? _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 17 févr. 2014, at 18:51, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > On Mon 2014-02-17 18:43, Galder Zamarreño wrote: > >> On 05 Feb 2014, at 17:30, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: >> >>> On Wed 2014-02-05 15:53, Mircea Markus wrote: >>> >>>> On Feb 3, 2014, at 9:32 AM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: >>>> >>>> Sure searching for any cache is useful. What I was advocating is that if you search for more than one cache transparently, then you probably need to CRUD for more than one cache transparently as well. And this is not being discussed. >>> >>> Not sure what you mean by CRUD over multiple caches? ATM one can run a TX over multiple caches, but I think there's something else you have in mind :-) >> >> >> //some unified query giving me entries pointing by fk copy to bar and >> //buz objects. So I need to manually load these references. >> >> //happy emmanuel >> Cache unifiedCache = cacheManager.getMotherOfAllCaches(); >> Bar bar = unifiedCache.get(foo); >> Buz buz = unifiedCache.get(baz); >> >> //not so happy emmanuel >> Cache fooCache = cacheManager.getCache("foo"); >> Bar bar = fooCache.get(foo); >> Cache bazCache = cacheManager.getCache("baz"); >> Buz buz = bazCache.put(baz); > > Would something like what Paul suggests in https://issues.jboss.org/browse/ISPN-3640 help you better? IOW, have a single cache, and then have a filtered view for Bar or Buz types? Not sure I understand the differences in your code changes in terms of what makes you happy vs not. Not really. What makes me unhappy is to have to keep in my app all the references to these specific cache store instances. The filtering approach only moves the problem. _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Well, OGM and Infinispan are different species :) So, Infinispan being what it is today - a non-homogenous, schema-less KV store, without support for entity associations (except embedding) - which simplifies the whole thing a lot, should we or should we not provide transparent cross-cacheManager search capabilities, in this exact context? Vote?

There were some points raised previously like /"if you search for more than one cache transparently, then you probably need to CRUD for more than one cache transparently as well"/. In the SQL world you would also probably CRUD against a table or set of tables and then query against a view - a bit like what we're doing here. I don't see any problem with this in principle. There is however something currently missing in the query result set API - it currently does not provide you the keys of the matching entities. People

work around this by storing the key in the entity. Now with the addition of the cross-cacheManager search we'll probably need to fix the result api and also provide a reference to the cache (or just the name?) where the entity is stored.

On 18 February 2014 13:01, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > On Tue 2014-02-18 14:02, Adrian Nistor wrote: >> There were some points raised previously like /"if you search for more than >> one cache transparently, then you probably need to CRUD for more than one >> cache transparently as well"/. In the SQL world you would also probably CRUD >> against a table or set of tables and then query against a view - a bit like >> what we're doing here. I don't see any problem with this in principle. There >> is however something currently missing in the query result set API - it >> currently does not provide you the keys of the matching entities. People > > Really? I think we have the info in the index at least when the > "ProvidedId" and the keys are the same. We have this info in the engine, but the results to the user don't usually include the keys. For some this is a bit unnatural: a different perspective would be to return _only_ the keys and avoid doing the lookup. We provide a "LazyIterator" on the results which fetches only each matching entry on demand, which I think covers a good deal of use cases but there might be other usages for these keys. I would be great if we had Lambda support to allow users to say what they want us to do with the resultset, rather than fetching it.

By the way, Mircea, Sanne and I had quite a long discussion about this one and the idea of one cache per entity. It turns out that the right (as in easy) solution does involve a higher level programming model like OGM provides. You can simulate it yourself using the Infinispan APIs but it is just cumbersome.

> On 17 févr. 2014, at 18:51, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > >> On Mon 2014-02-17 18:43, Galder Zamarreño wrote: >> >>> On 05 Feb 2014, at 17:30, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: >>> >>>> On Wed 2014-02-05 15:53, Mircea Markus wrote: >>>> >>>>> On Feb 3, 2014, at 9:32 AM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: >>>>> >>>>> Sure searching for any cache is useful. What I was advocating is that if you search for more than one cache transparently, then you probably need to CRUD for more than one cache transparently as well. And this is not being discussed. >>>> >>>> Not sure what you mean by CRUD over multiple caches? ATM one can run a TX over multiple caches, but I think there's something else you have in mind :-) >>> >>> >>> //some unified query giving me entries pointing by fk copy to bar and >>> //buz objects. So I need to manually load these references. >>> >>> //happy emmanuel >>> Cache unifiedCache = cacheManager.getMotherOfAllCaches(); >>> Bar bar = unifiedCache.get(foo); >>> Buz buz = unifiedCache.get(baz); >>> >>> //not so happy emmanuel >>> Cache fooCache = cacheManager.getCache("foo"); >>> Bar bar = fooCache.get(foo); >>> Cache bazCache = cacheManager.getCache("baz"); >>> Buz buz = bazCache.put(baz); >> >> Would something like what Paul suggests in https://issues.jboss.org/browse/ISPN-3640 help you better? IOW, have a single cache, and then have a filtered view for Bar or Buz types? Not sure I understand the differences in your code changes in terms of what makes you happy vs not. > > Not really. > What makes me unhappy is to have to keep in my app all the > references to these specific cache store instances. The filtering > approach only moves the problem. > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> On 24 févr. 2014, at 17:39, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > > >> On Feb 17, 2014, at 10:13 PM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: >> >> By the way, Mircea, Sanne and I had quite a long discussion about this one and the idea of one cache per entity. It turns out that the right (as in easy) solution does involve a higher level programming model like OGM provides. You can simulate it yourself using the Infinispan APIs but it is just cumbersome. > > Curious to hear the whole story :-) > We cannot mandate all the suers to use OGM though, one of the reasons being OGM is not platform independent (hotrod). Then solve all the issues I have raised with a magic wand and come back to me when you have done it, I'm interested.

On 25 Feb 2014, at 16:08, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > > On Feb 25, 2014, at 9:28 AM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > >>> On 24 févr. 2014, at 17:39, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: >>> >>> >>>> On Feb 17, 2014, at 10:13 PM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: >>>> >>>> By the way, Mircea, Sanne and I had quite a long discussion about this one and the idea of one cache per entity. It turns out that the right (as in easy) solution does involve a higher level programming model like OGM provides. You can simulate it yourself using the Infinispan APIs but it is just cumbersome. >>> >>> Curious to hear the whole story :-) >>> We cannot mandate all the suers to use OGM though, one of the reasons being OGM is not platform independent (hotrod). >> >> Then solve all the issues I have raised with a magic wand and come back to me when you have done it, I'm interested. > > People are going to use infinispan with one cache per entity, because it makes sense: > - different config (repl/dist | persistent/non-persistent) for different data types > - have map/reduce tasks running only the Person entires not on Dog as well, when you want to select (Person) where age > 18 > I don't see a reason to forbid this, on the contrary. The way I see it the relation between (OGM, ISPN) <=> (Hibernate, JDBC). Indeed OGM would be a better abstraction and should be recommended as such for the Java clients, but ultimately we're a general purpose storage engine that is available to different platforms as well. > I do disagree on your assessment. I did write a whole essay on why I think your view is problematic - I was getting tired of repeating myself ;P https://github.com/infinispan/infinispan/wiki/A-continuum-of-data-structu...

To anecdotally answer your specific example, yes different configs for different entities is an interesting benefit but it has to outweigh the drawbacks.

If you have to do a map reduce for tasks so simple as age > 18, I think you system better have to be prepared to run gazillions of M/R jobs.

I think that Dogs and any domestic animal is fundamentally related to humans - Person in your case. So queries involving both will be required - a cross cache M/R is not doable today AFAIK and even if it was, it’s still M/R and all its drawbacks. To me, the Cache API and Hot Rod are well suited for what I call self contained object graph (i.e. where Dog would be an embedded object of Person and not a separate Entity). In that situation, there is a single cache.

One cache per entity does make sense for API that do support what I call connected entities. Hibernate OGM specifically.

But please read the wiki page first before commenting. I did spend a lot of time on it https://github.com/infinispan/infinispan/wiki/A-continuum-of-data-structu...

Emmanuel _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On Feb 26, 2014, at 5:14 PM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > > On 25 Feb 2014, at 16:08, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > >> >> On Feb 25, 2014, at 9:28 AM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: >> >>>> On 24 févr. 2014, at 17:39, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: >>>> >>>> >>>>> On Feb 17, 2014, at 10:13 PM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: >>>>> >>>>> By the way, Mircea, Sanne and I had quite a long discussion about this one and the idea of one cache per entity. It turns out that the right (as in easy) solution does involve a higher level programming model like OGM provides. You can simulate it yourself using the Infinispan APIs but it is just cumbersome. >>>> >>>> Curious to hear the whole story :-) >>>> We cannot mandate all the suers to use OGM though, one of the reasons being OGM is not platform independent (hotrod). >>> >>> Then solve all the issues I have raised with a magic wand and come back to me when you have done it, I'm interested. >> >> People are going to use infinispan with one cache per entity, because it makes sense: >> - different config (repl/dist | persistent/non-persistent) for different data types >> - have map/reduce tasks running only the Person entires not on Dog as well, when you want to select (Person) where age > 18 >> I don't see a reason to forbid this, on the contrary. The way I see it the relation between (OGM, ISPN) <=> (Hibernate, JDBC). Indeed OGM would be a better abstraction and should be recommended as such for the Java clients, but ultimately we're a general purpose storage engine that is available to different platforms as well. >> > > I do disagree on your assessment. > I did write a whole essay on why I think your view is problematic - I was getting tired of repeating myself ;P > https://github.com/infinispan/infinispan/wiki/A-continuum-of-data-structu... Thanks for writing this up, it is a good taxonomy of data storage schemes and querying. > > To anecdotally answer your specific example, yes different configs for different entities is an interesting benefit but it has to outweigh the drawbacks. Using a single cache for all the types is practical at all :-) Just to expand my idea, people prefer using different caches for many reasons: - security: Account cache has a different security requirements than the News cache - data consistency: News is a non-transactional cache, Account require pessimistic XA transactions - expiry: expire last year's news from the system. Not the same for Accounts - availability: I want the Accounts cache to be backed up to another site. I don't want that for the News cache - logical data grouping: mixing Accounts with News doesn't make sense. I might want to know which account appeared in the news, though.

> If you have to do a map reduce for tasks so simple as age > 18, I think you system better have to be prepared to run gazillions of M/R jobs. I want to run a simple M/R job in the evening to determine who turns 18 tomorrow, to congratulate them. Once a day, not gazzilions of times, and I don't need to index the age filed just for that. Also when it comes to Map/Reduce, the drawback of holding all the data in a single cache is two-folded: - performance: you iterate over the data that is not related to your query.

- programming model: the Map/Reduce implementation has a dependency on both Dog and Person. If I add Cats to the cache, I'll need to update the M/R code to be aware of that as well. Same if I rename/remove Dog. Not nice.

> I think that Dogs and any domestic animal is fundamentally related to humans - Person in your case. So queries involving both will be required - a cross cache M/R is not doable today AFAIK and even if it was, it’s still M/R and all its drawbacks. > To me, the Cache API and Hot Rod are well suited for what I call self contained object graph (i.e. where Dog would be an embedded object of Person and not a separate Entity). In that situation, there is a single cache. I see where you come from but I don't think requiring people to use a single cache for all the entities is an option. Besides a natural logical separation, different data has different storage requirements: security, access patterns, consistency, durability, availability etc. For most of the non-trivial use cases, using a single cache just wont do.

> One cache per entity does make sense for API that do support what I call connected entities. Hibernate OGM specifically. OGM does a great job covering this, but it is very specific: java only and OOP - our C/S mode, hotrod specifically, is language independent and not OOP.

Also I would like to comment on the following statements: "I believe a cache API and Hot Rod are well suited to address up to the self contained object graph use case with a couple of relations maintained manually by the application but that cannot be queried. For the connected entities use case, only a high level paradigm is suited like JPA." I don't think storing object graphs should be under scrutiny here: Infinispan C/S mode (and there's where most of the client focus is BTW) has a schema (prtobuf) that does not support object graphs. I also think expecting people to use multiple caches for multiple data types is a solid assumption to start from. And here's me speculating: these data types have logical relations between them so people will ask for querying. In order to queries on multiple data types, you can either merge them together (your suggestion) or support some sort of new cross-cache indexing/querying/api. x-cache querying is more flexible and less restraining than merging data, but from what I understand from you has certain implementation challenges. There's no pressure to take a decision now around supporting queries spreading multiple caches - just something to keep an eye on when dealing with use cases/users. ATM merging data is the only solution available, let's wait and see if people ask for more.

My impression is that this thread is boiling down to an identity problem of the project. It is a key/value store with some properties which make it a great fit for a Cache; I don't think there are doubts that relational queries are useful to have to resolve a certain category of problems, and no doubt someone will like to do that on Infinispan, but it should be stated clearly that this is beyond the scope of the project: we're not in the business of relational queries, because the fundamental building blocks are not fit for it. So this should be a clear no-go, as it should be clear for many other features which will be requested over time: just because we could implement some part doesn't make the project any better. That said, it is possible for us to provide some additional features which are a good fit to build on top of these fundamental building blocks, and also happen to make the fundamental API much more useful. I'm referring to the simple queries we have today, the various remoting protocols, management extensions, better eviction policies,.. So I'm not questioning many of the nice features we have today, but it's a dangerous game if the limitations of what it can or can not do start to blur. To make an example, it was four years ago that I suggested that Infinispan should not expose the full Map interface without actually implementing it in its full glory, and people regularly bang their head against it. The limitations are clear in the javadoc, but still the type is compatible with Map and so sometimes the client wasn't written with Infinispan in mind: just an example, but it's dangerous to expose a contract which we can't maintain in all its intricate complex forms.

To quote Emmanuel: "So back to the root of that discussion now, what was the use case(s) that lead to believe we need cross-cache query support?" The intent I had in my original email was having several bullet points, and none are related to use cases but rather to internal efficiency concerns regarding sharing Search services, bootstrap time, configuration simplifications. The configuration one used to be quite painful but that improved recently, and for most of these points we can find alternative or even better solutions.

I even have additional reasons to not do it: as previously discussed, Map/Reduce tasks should be able to output results to a different cache. That's reasonable as you need to "park" a potentially large output somewhere, but also doesn't imply cross-cache M/R search,

which would be a requirement for this feature to be consistent. If any updates on different Caches - and therefore its content - should be considered highly unrelated: might be using a different distribution, different cachestore, a different Transaction. If any kind of soft-join was "implied" across caches, we would never be able to maintain consistency, a causality order (in future), a consistent backup/snapshot, etc.. a nightmare for users and for us to firefight "bugs". So my conclusion is that it's dangerous, confusing, and also complex to achieve: we won't implement it.

Sanne On 4 March 2014 18:02, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > > On 28 Feb 2014, at 22:14, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > > > On Feb 26, 2014, at 5:14 PM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; > wrote: > > > On 25 Feb 2014, at 16:08, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > > > On Feb 25, 2014, at 9:28 AM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; > wrote: > > On 24 févr. 2014, at 17:39, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > > > On Feb 17, 2014, at 10:13 PM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; > wrote: > > By the way, Mircea, Sanne and I had quite a long discussion about this one > and the idea of one cache per entity. It turns out that the right (as in > easy) solution does involve a higher level programming model like OGM > provides. You can simulate it yourself using the Infinispan APIs but it is > just cumbersome. > > > Curious to hear the whole story :-) > We cannot mandate all the suers to use OGM though, one of the reasons being > OGM is not platform independent (hotrod). > > > Then solve all the issues I have raised with a magic wand and come back to > me when you have done it, I'm interested. > > > People are going to use infinispan with one cache per entity, because it > makes sense: > - different config (repl/dist | persistent/non-persistent) for different > data types > - have map/reduce tasks running only the Person entires not on Dog as well, > when you want to select (Person) where age > 18 > I don't see a reason to forbid this, on the contrary. The way I see it the > relation between (OGM, ISPN) <=> (Hibernate, JDBC). Indeed OGM would be a > better abstraction and should be recommended as such for the Java clients, > but ultimately we're a general purpose storage engine that is available to > different platforms as well. > > > I do disagree on your assessment. > I did write a whole essay on why I think your view is problematic - I was > getting tired of repeating myself ;P > https://github.com/infinispan/infinispan/wiki/A-continuum-of-data-structu... > > > Thanks for writing this up, it is a good taxonomy of data storage schemes > and querying. > > > To anecdotally answer your specific example, yes different configs for > different entities is an interesting benefit but it has to outweigh the > drawbacks. > > > Using a single cache for all the types is practical at all :-) Just to > expand my idea, people prefer using different caches for many reasons: > - security: Account cache has a different security requirements than the > News cache > - data consistency: News is a non-transactional cache, Account require > pessimistic XA transactions > - expiry: expire last year's news from the system. Not the same for Accounts > - availability: I want the Accounts cache to be backed up to another site. I > don't want that for the News cache > - logical data grouping: mixing Accounts with News doesn't make sense. I > might want to know which account appeared in the news, though. > > > This kind of reasons reminds me in the RDBMS world of why people use > different databases. > In fact, I have had experience where literally News was a different database > than Accounts. > > But again in this model, in one database, you have many tables. > > > If you have to do a map reduce for tasks so simple as age > 18, I think you > system better have to be prepared to run gazillions of M/R jobs. > > > I want to run a simple M/R job in the evening to determine who turns 18 > tomorrow, to congratulate them. Once a day, not gazzilions of times, and I > don't need to index the age filed just for that. Also when it comes to > Map/Reduce, the drawback of holding all the data in a single cache is > two-folded: > - performance: you iterate over the data that is not related to your query. > > > If the data are never related (query wise), then we are in the database > split category. Which is fine. But if some of your queries are related, what > do you do? Deny the user the ability to do them? > > - programming model: the Map/Reduce implementation has a dependency on both > Dog and Person. If I add Cats to the cache, I'll need to update the M/R code > to be aware of that as well. Same if I rename/remove Dog. Not nice. > > > Well it’s called type safety, some people find it good ;) > By the way, OGM does abstract a class from it’s representation in the > datastore (including its name). But that’s another story ;) > > > I think that Dogs and any domestic animal is fundamentally related to humans > - Person in your case. So queries involving both will be required - a cross > cache M/R is not doable today AFAIK and even if it was, it’s still M/R and > all its drawbacks. > To me, the Cache API and Hot Rod are well suited for what I call self > contained object graph (i.e. where Dog would be an embedded object of Person > and not a separate Entity). In that situation, there is a single cache. > > > I see where you come from but I don't think requiring people to use a single > cache for all the entities is an option. Besides a natural logical > separation, different data has different storage requirements: security, > access patterns, consistency, durability, availability etc. For most of the > non-trivial use cases, using a single cache just wont do. > > > Let me rephrase and sum up my position. > If you are storing unrelated data, use different caches if you want, that’s > fine. > If you are storing related data, store it as one root entity and embeddable > objects (ie one cache entry for the whole graph) > you can have one root entity per cache, that’s fine. > If you are storing related entities and want to do queries on it: you are > more or less screwed today with Infinispan and need a higher level > abstraction. > > So _recommending_ one entity = one cache to me is wrong. It’s more one > entity graph = one cache which is vastly different and has deep consequences > (see my wiki page). > > > One cache per entity does make sense for API that do support what I call > connected entities. Hibernate OGM specifically. > > > OGM does a great job covering this, but it is very specific: java only and > OOP - our C/S mode, hotrod specifically, is language independent and not > OOP. > > > Sure. Outside of Java consider that connected objects are not a supported > feature of Infinispan. When I mean not supported, I mean: > - no inter object query > - no automatically handled relation between objects > > Also I would like to comment on the following statements: > "I believe a cache API and Hot Rod are well suited to address up to the self > contained object graph use case with a couple of relations maintained > manually by the application but that cannot be queried. For the connected > entities use case, only a high level paradigm is suited like JPA." > > I don't think storing object graphs should be under scrutiny here: > Infinispan C/S mode (and there's where most of the client focus is BTW) has > a schema (prtobuf) that does not support object graphs. I also think > expecting people to use multiple caches for multiple data types is a solid > assumption to start from. And here's me speculating: these data types have > logical relations between them so people will ask for querying. In order to > queries on multiple data types, you can either merge them together (your > suggestion) or support some sort of new cross-cache indexing/querying/api. > x-cache querying is more flexible and less restraining than merging data, > but from what I understand from you has certain implementation challenges. > There's no pressure to take a decision now around supporting queries > spreading multiple caches - just something to keep an eye on when dealing > with use cases/users. ATM merging data is the only solution available, let's > wait and see if people ask for more. > > > That’s fine I guess. My problem is that nowhere in your documentations do > you guys list what can and cannot be done in that regard. And when you call > a data structure an entity, it comes with an implicit bagage: relation > between entities, polymorphism etc. > So this needs to be clarified. > > Now circling back to the main topic. Doing cross-cache query on non related > data (as they are in different caches) might be useful in some situations > but is generally not interesting as we specifically don’t support joins > (they are not related). > So back to the root of that discussion now, what was the use case(s) that > lead to believe we need cross-cache query support? > > Emmanuel > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Sanne came with a good follow up to this email, just some small clarifications: On Mar 4, 2014, at 6:02 PM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: >>> If you have to do a map reduce for tasks so simple as age > 18, I think you system better have to be prepared to run gazillions of M/R jobs. >> >> I want to run a simple M/R job in the evening to determine who turns 18 tomorrow, to congratulate them. Once a day, not gazzilions of times, and I don't need to index the age filed just for that. Also when it comes to Map/Reduce, the drawback of holding all the data in a single cache is two-folded: >> - performance: you iterate over the data that is not related to your query. > > If the data are never related (query wise), then we are in the database split category. Which is fine. But if some of your queries are related, what do you do? Deny the user the ability to do them? Here's where cross-site query would have been used. As Sanne suggested (next post) these limitations overcome the advantages.

> >> - programming model: the Map/Reduce implementation has a dependency on both Dog and Person. If I add Cats to the cache, I'll need to update the M/R code to be aware of that as well. Same if I rename/remove Dog. Not nice. > > Well it’s called type safety, some people find it good ;) If anything, this model reduces type safety and reusability. E.g. say you want a M/R task to see how many Persons speak French. With the single cache model(both Dog and Person int he Cache<String, Mammal>) it would look something like: a) //pseudocode map (String k, Mammal value) { if (value instanceof Person)) { //this is the ugly part if (((Person)value).speaks("French")) ... } else { //ignore it, it's an Dog } } Same thing written for a Cache<String, Person>: b) map (String k, Person value) { if (value.speaks("French")) ... } I don't think people would prefer writing a) instead of b) ;)

On Wed 2014-03-05 17:16, Mircea Markus wrote: > Sanne came with a good follow up to this email, just some small clarifications: > > On Mar 4, 2014, at 6:02 PM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > >>>> If you have to do a map reduce for tasks so simple as age > 18, I think you system better have to be prepared to run gazillions of M/R jobs. >>> >>> I want to run a simple M/R job in the evening to determine who turns 18 tomorrow, to congratulate them. Once a day, not gazzilions of times, and I don't need to index the age filed just for that. Also when it comes to Map/Reduce, the drawback of holding all the data in a single cache is two-folded: >>> - performance: you iterate over the data that is not related to your query. >> >> If the data are never related (query wise), then we are in the database split category. Which is fine. But if some of your queries are related, what do you do? Deny the user the ability to do them? > > Here's where cross-site query would have been used. As Sanne suggested (next post) these limitations overcome the advantages. No. Cross-cache query if implemented will not support (efficiently enough) that kind of query. Cf my wiki page.

On 7 March 2014 14:54, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > > On Mar 6, 2014, at 9:21 AM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > >> On Wed 2014-03-05 17:16, Mircea Markus wrote: >>> Sanne came with a good follow up to this email, just some small clarifications: >>> >>> On Mar 4, 2014, at 6:02 PM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: >>> >>>>>> If you have to do a map reduce for tasks so simple as age > 18, I think you system better have to be prepared to run gazillions of M/R jobs. >>>>> >>>>> I want to run a simple M/R job in the evening to determine who turns 18 tomorrow, to congratulate them. Once a day, not gazzilions of times, and I don't need to index the age filed just for that. Also when it comes to Map/Reduce, the drawback of holding all the data in a single cache is two-folded: >>>>> - performance: you iterate over the data that is not related to your query. >>>> >>>> If the data are never related (query wise), then we are in the database split category. Which is fine. But if some of your queries are related, what do you do? Deny the user the ability to do them? >>> >>> Here's where cross-site query would have been used. As Sanne suggested (next post) these limitations overcome the advantages. >> >> No. Cross-cache query if implemented will not support (efficiently >> enough) that kind of query. Cf my wiki page. > > yes, non-indexed joins would be exponential on the number of caches involved. Technically non-indexed joins would be exponential on the number of caches (joins) involves *and* on the amount of entries you have stored: I know you wheren't suggesting doing it, but to confirm it's even worse than an horrible idea ;-) And that's not even considering the subtle design catch of "load it all from all cachestores".. combined with "multiple times per join"..

> Is it possible to use an index for x-cache joins with linear index update time and query? Index update cost is not linear but LogN: approximates to a constant cost.

And we could cut this constant by 4 orders of magnitude if only I could safely differentiate between a put of a new entry vs. an update -> something which we'll need to brainstorm about. Query time is also significantly sub-linear in practice, but specifics will vary on the query type. Yes you could use indexes to improve x-cache joins, but you'll need an additional engine to coordinate that correctly, not least to manage data size buffers; essentially I think you'd need Teiid. Sanne > > Cheers, > -- > Mircea Markus > Infinispan lead (www.infinispan.org) > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> Is it possible to use an index for x-cache joins with linear index update time and query? Index update cost is not linear but LogN: approximates to a constant cost. And we could cut this constant by 4 orders of magnitude if only I could safely differentiate between a put of a new entry vs. an update -> something which we'll need to brainstorm about. Query time is also significantly sub-linear in practice, but specifics will vary on the query type. Yes you could use indexes to improve x-cache joins, but you'll need an additional engine to coordinate that correctly, not least to manage data size buffers; essentially I think you'd need Teiid.

Sanne came with a good follow up to this email, just some small clarifications: On Mar 4, 2014, at 6:02 PM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: >>> If you have to do a map reduce for tasks so simple as age > 18, I think you system better have to be prepared to run gazillions of M/R jobs. >> >> I want to run a simple M/R job in the evening to determine who turns 18 tomorrow, to congratulate them. Once a day, not gazzilions of times, and I don't need to index the age filed just for that. Also when it comes to Map/Reduce, the drawback of holding all the data in a single cache is two-folded: >> - performance: you iterate over the data that is not related to your query. > > If the data are never related (query wise), then we are in the database split category. Which is fine. But if some of your queries are related, what do you do? Deny the user the ability to do them? Here's where cross-site query would have been used. As Sanne suggested (next post) these limitations overcome the advantages. > >> - programming model: the Map/Reduce implementation has a dependency on both Dog and Person. If I add Cats to the cache, I'll need to update the M/R code to be aware of that as well. Same if I rename/remove Dog. Not nice. > > Well it’s called type safety, some people find it good ;) If anything, this model reduces type safety and reusability. E.g. say you want a M/R task to see how many Persons speak French. With the single cache model(both Dog and Person int he Cache<String, Mammal>) it would look something like: a) //pseudocode map (String k, Mammal value) { if (value instanceof Person)) { //this is the ugly part if (((Person)value).speaks("French")) ... } else { //ignore it, it's an Dog } } Same thing written for a Cache<String, Person>: b) map (String k, Person value) { if (value.speaks("French")) ... } I don't think people would prefer writing a) instead of b) ;)

Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org) _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 05 Mar 2014, at 18:16, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > Sanne came with a good follow up to this email, just some small clarifications: > > On Mar 4, 2014, at 6:02 PM, Emmanuel Bernard <emmanuel(a)hibernate.org&gt; wrote: > >>>> If you have to do a map reduce for tasks so simple as age > 18, I think you system better have to be prepared to run gazillions of M/R jobs. >>> >>> I want to run a simple M/R job in the evening to determine who turns 18 tomorrow, to congratulate them. Once a day, not gazzilions of times, and I don't need to index the age filed just for that. Also when it comes to Map/Reduce, the drawback of holding all the data in a single cache is two-folded: >>> - performance: you iterate over the data that is not related to your query. >> >> If the data are never related (query wise), then we are in the database split category. Which is fine. But if some of your queries are related, what do you do? Deny the user the ability to do them? > > Here's where cross-site query would have been used. As Sanne suggested (next post) these limitations overcome the advantages. > >> >>> - programming model: the Map/Reduce implementation has a dependency on both Dog and Person. If I add Cats to the cache, I'll need to update the M/R code to be aware of that as well. Same if I rename/remove Dog. Not nice. >> >> Well it’s called type safety, some people find it good ;) > > If anything, this model reduces type safety and reusability. E.g. say you want a M/R task to see how many Persons speak French. With the single cache model(both Dog and Person int he Cache<String, Mammal>) it would look something like: > > a) > //pseudocode > map (String k, Mammal value) { > if (value instanceof Person)) { //this is the ugly part > if (((Person)value).speaks("French")) ... > } else { > //ignore it, it's an Dog > } > } > > Same thing written for a Cache<String, Person>: > > b) > map (String k, Person value) { > if (value.speaks("French")) ... > } > > I don't think people would prefer writing a) instead of b) ;) It’s a pity that having discussed in Mallorca and having referenced a fair few times already, you could not think of an option based on what Paul suggsted in https://issues.jboss.org/browse/ISPN-3640 Here’s my attempt: Cache<?, ?> cache = …; ValueFilter filter = … // filter would check those values that are persons... Cache<String, Person> view = cache.filter(filter); view.map(String k, Person value) { ... } Paul referred to per key type filters, but maybe per value type filters, such as the one I did here, might be useful.

Cheers, > > > Cheers, > -- > Mircea Markus > Infinispan lead (www.infinispan.org) > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño galder(a)redhat.com twitter.com/galderz Project Lead, Escalante http://escalante.io Engineer, Infinispan http://infinispan.org

On 05 Mar 2014, at 18:16, Mircea Markus <mmarkus(a)redhat.com&gt; wrote: > If anything, this model reduces type safety and reusability. E.g. say you want a M/R task to see how many Persons speak French. With the single cache model(both Dog and Person int he Cache<String, Mammal>) it would look something like: > > a) > //pseudocode > map (String k, Mammal value) { > if (value instanceof Person)) { //this is the ugly part > if (((Person)value).speaks("French")) ... > } else { > //ignore it, it's an Dog > } > } > > Same thing written for a Cache<String, Person>: > > b) > map (String k, Person value) { > if (value.speaks("French")) ... > } > > I don't think people would prefer writing a) instead of b) ;) It’s a pity that having discussed in Mallorca and having referenced a fair few times already, you could not think of an option based on what Paul suggsted in https://issues.jboss.org/browse/ISPN-3640

> </snip> >> >> To anecdotally answer your specific example, yes different configs for different entities is an interesting benefit but it has to outweigh the drawbacks. > > Using a single cache for all the types is practical at all :-) Just to expand my idea, people prefer using different caches for many reasons: > - security: Account cache has a different security requirements than the News cache > - data consistency: News is a non-transactional cache, Account require pessimistic XA transactions > - expiry: expire last year's news from the system. Not the same for Accounts > - availability: I want the Accounts cache to be backed up to another site. I don't want that for the News cache > - logical data grouping: mixing Accounts with News doesn't make sense. I might want to know which account appeared in the news, though. This kind of reasons reminds me in the RDBMS world of why people use different databases. In fact, I have had experience where literally News was a different database than Accounts. But again in this model, in one database, you have many tables. > >> If you have to do a map reduce for tasks so simple as age > 18, I think you system better have to be prepared to run gazillions of M/R jobs. > > I want to run a simple M/R job in the evening to determine who turns 18 tomorrow, to congratulate them. Once a day, not gazzilions of times, and I don't need to index the age filed just for that. Also when it comes to Map/Reduce, the drawback of holding all the data in a single cache is two-folded: > - performance: you iterate over the data that is not related to your query.

If the data are never related (query wise), then we are in the database split category. Which is fine. But if some of your queries are related, what do you do? Deny the user the ability to do them? > - programming model: the Map/Reduce implementation has a dependency on both Dog and Person. If I add Cats to the cache, I'll need to update the M/R code to be aware of that as well. Same if I rename/remove Dog. Not nice. Well it’s called type safety, some people find it good ;) By the way, OGM does abstract a class from it’s representation in the datastore (including its name). But that’s another story ;) > >> I think that Dogs and any domestic animal is fundamentally related to humans - Person in your case. So queries involving both will be required - a cross cache M/R is not doable today AFAIK and even if it was, it’s still M/R and all its drawbacks. >> To me, the Cache API and Hot Rod are well suited for what I call self contained object graph (i.e. where Dog would be an embedded object of Person and not a separate Entity). In that situation, there is a single cache. > > I see where you come from but I don't think requiring people to use a single cache for all the entities is an option. Besides a natural logical separation, different data has different storage requirements: security, access patterns, consistency, durability, availability etc. For most of the non-trivial use cases, using a single cache just wont do. Let me rephrase and sum up my position. If you are storing unrelated data, use different caches if you want, that’s fine. If you are storing related data, store it as one root entity and embeddable objects (ie one cache entry for the whole graph) you can have one root entity per cache, that’s fine. If you are storing related entities and want to do queries on it: you are more or less screwed today with Infinispan and need a higher level abstraction. So _recommending_ one entity = one cache to me is wrong.

It’s more one entity graph = one cache which is vastly different and has deep consequences (see my wiki page).