[infinispan-dev] Integration between HotRod and OGM

Emmanuel Bernard emmanuel at hibernate.org
Wed Jan 22 08:41:11 EST 2014 

On the other hand if the group feature adds a way to:
- get all the keys,
- get a subset of the keys based on a filter

Infinispan will be able to start supporting use cases restricted to Cassandra in the past (esp around time series).

I am assuming groups offer offer a way to add / change and remove a key from / to a group without having to load all the group or even the group keys.

On 22 Jan 2014, at 14:33, Emmanuel Bernard <emmanuel at hibernate.org> wrote:

> BTW query support on groups (by entry of each group) is an interesting non covered use case today.
> 
> On 22 Jan 2014, at 14:26, Emmanuel Bernard <emmanuel at hibernate.org> wrote:
> 
>> Conceptually I like the grouping API better than AtomicMap as I don’t have to rely on a specific Infinispan type.
>> 
>> We do use FineGrainedAtomicMap both for the entity and the association persistence (not AtomicMap). It is particularly critical for how we store the association navigation information. I don’t want one update to literally prevent the whole association from being updated. This is the same semantic a RDBMS has and that’s why Manik and I designed the FGAM requirements.
>> 
>> So my question is what are the differences between the grouping API and the FGAM in particular for:
>> 
>> - the amount of data sent back and forth (seems like grouping is sending the data naturally per key as “delta compared to the group"
>> - the locking level when a new entry is added to the FGAM / Grouping API
>> - the locking level when a new entry is removed to the FGAM / Grouping API
>> - the locking level when a new entry is updated to the FGAM / Grouping API
>> - the overall network verbosity
>> - does grouping offer the same repeatable read protection that AtomicMap offers within a transaction?
>> 
>> I think retrying as a transaction workaround is quite fragile. We can offer it as a solution but supporting or encouraging it is another story. Unless each OGM nodes do behave like a transaction but that would be wrong. I am also concerned about reading data form a group that are inconsistent.
>> 
>> Emmanuel
>> 
>> On 21 Jan 2014, at 16:07, Mircea Markus <mmarkus at redhat.com> wrote:
>> 
>>> Hi Emmanuel,
>>> 
>>> Just had a good chat with Davide on this and one solution to overcome the shortcoming you mentioned in the above email would be to enhance the hotrod client to support grouping:
>>> 
>>> RemoteClient.put(G g, K k, V v); //first param is the group
>>> RemoteClinet.getGroup(G g) : Map<K,V>;
>>> 
>>> It requires an enhancement on our local grouping API: EmbeddedCache.getGroup(G). This is something useful for us in a broader context, as it is the step needed to be able to deprecated AtomicMaps and get suggest them being replaced with Grouping. 
>>> 
>>> This approach still has some limitations compared to the current embedded integration: 
>>> - performance caused by the lack of transactions: this means increased TCP chattiness between the Hot Rod client and the server. 
>>> - you'd have to handle atomicity, potentially by retrying an operation
>>> 
>>> What do you think?
>>> 
>>> 
>>> On Dec 3, 2013, at 3:10 AM, Mircea Markus <mmarkus at redhat.com> wrote:
>>> 
>>>> 
>>>> On Nov 19, 2013, at 10:22 AM, Emmanuel Bernard <emmanuel at hibernate.org> wrote:
>>>> 
>>>>> It's an interesting approach that would work fine-ish for entities
>>>>> assuming the Hot Rod client is multi threaded and assuming the client
>>>>> uses Future to parallelize the calls.
>>>> 
>>>> The Java Hotrod client is both multithreaded and exposes an Async API.
>>>> 
>>>>> 
>>>>> But it won't work for associations as we have them designed today.
>>>>> Each association - or more precisely the query results to go from an
>>>>> entity A1 to the list of entities B associated to it - is represented by
>>>>> an AtomicMap.
>>>>> Each entry in this map does correspond to an entry in the association.
>>>>> 
>>>>> While we can "guess" the column names and build from the metadata the
>>>>> list of composed keys for entities, we cannot do the same for
>>>>> associations as the key is literally the (composite) id of the
>>>>> association and we cannot guess that most of the time (we can in very
>>>>> pathological cases).
>>>>> We could imagine that we list the association row keys in a special
>>>>> entry to work around that but this approach is just as problematic and
>>>>> is conceptually the same.
>>>>> The only solution would be to lock the whole association for each
>>>>> operation and I guess impose some versioning / optimistic lock.
>>>>> 
>>>>> That is not a pattern that scales sufficiently from my experience.
>>>> 
>>>> I think so too :-)
>>>> 
>>>>> That's the problem with interconnected data :)
>>>>> 
>>>>> Emmanuel
>>>>> 
>>>>> On Mon 2013-11-18 23:05, Mircea Markus wrote:
>>>>>> Neither the grouping API nor the AtomicMap work over hotrod.
>>>>>> Between the grouping API and AtomicMap, I think the one that would make more sense migrating is the grouping API.
>>>>>> One way or the other, I think the hotrod protocol would require an enhancement - mind raising a JIRA for that?
>>>>>> For now I guess you can sacrifice performance and always sending the entire object across on every update instead of only the deltas?
>>>>>> 
>>>>>> On Nov 18, 2013, at 9:56 AM, Emmanuel Bernard <emmanuel at hibernate.org> wrote:
>>>>>> 
>>>>>>> Someone mentioned the grouping API as some sort of alternative to
>>>>>>> AtomicMap. Maybe we should use that?
>>>>>>> Note that if we don't have a fine-grained approach we will need to
>>>>>>> make sure we *copy* the complex data structure upon reads to mimic
>>>>>>> proper transaction isolation.
>>>>>>> 
>>>>>>> On Tue 2013-11-12 15:14, Sanne Grinovero wrote:
>>>>>>>> On 12 November 2013 14:54, Emmanuel Bernard <emmanuel at hibernate.org> wrote:
>>>>>>>>> On the transaction side, we can start without them.
>>>>>>>> 
>>>>>>>> +1 on omitting transactions for now.
>>>>>>>> 
>>>>>>>> And on the missing AtomicMaps, I hope the Infinispan will want to implement it?
>>>>>>>> Would be good to eventually converge on similar featuresets on remote
>>>>>>>> vs embedded APIs.
>>>>>>>> 
>>>>>>>> I know the embedded version relies on batching/transactions, but I
>>>>>>>> guess we could obtain a similar effect with some ad-hoc commands in
>>>>>>>> Hot Rod?
>>>>>>>> 
>>>>>>>> Sanne
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Tue 2013-11-12 14:34, Davide D'Alto wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> I'm working on the integration between HotRod and OGM.
>>>>>>>>>> 
>>>>>>>>>> We already have a dialect for Inifinispan and I'm trying to follow the same
>>>>>>>>>> logic.
>>>>>>>>>> At the moment I'm having two problems:
>>>>>>>>>> 
>>>>>>>>>> 1) In the Infinispan dialect we are using the AtomicMap and the
>>>>>>>>>> AtomicMapLookup but this classes don't work with the RemoteCache. Is there
>>>>>>>>>> an equivalent for HotRod?
>>>>>>>>>> 
>>>>>>>>>> 2) As far as I know HotRod does not support transactions. I've found a link
>>>>>>>>>> to a branch on Mircea repository:
>>>>>>>>>> https://github.com/mmarkus/ops_over_hotrod/wiki/Usage-guide
>>>>>>>>>> Is this something I could/should use?
>>>>>>>>>> 
>>>>>>>>>> Any help is appreciated.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> Davide
>>>>>>>>> 
>>>>>>>>>> _______________________________________________
>>>>>>>>>> infinispan-dev mailing list
>>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> infinispan-dev mailing list
>>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>>> _______________________________________________
>>>>>>>> infinispan-dev mailing list
>>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>>> _______________________________________________
>>>>>>> infinispan-dev mailing list
>>>>>>> infinispan-dev at lists.jboss.org
>>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>>> 
>>>>>> Cheers,
>>>>>> -- 
>>>>>> Mircea Markus
>>>>>> Infinispan lead (www.infinispan.org)
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> infinispan-dev mailing list
>>>>>> infinispan-dev at lists.jboss.org
>>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>>> _______________________________________________
>>>>> infinispan-dev mailing list
>>>>> infinispan-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>> 
>>>> Cheers,
>>>> -- 
>>>> Mircea Markus
>>>> Infinispan lead (www.infinispan.org)
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> Cheers,
>>> -- 
>>> Mircea Markus
>>> Infinispan lead (www.infinispan.org)
>>> 
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>> 
>> 
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> 
> 
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

More information about the infinispan-dev mailing list