[hibernate-dev] Proposal for a Redis module in Hibernate OGM

[Mark Paluch]

Hi Gunnar,

thanks for your helpful feedback. I will come up with a PR as soon as we
agree on the strategies issue - well, perhaps in a couple days. I can and
will support you and I can help you with support/maintenance for the Redis
module in the future. I will have to ask you in return on some specific
issues, such as why enum-ordinals cannot be mapped back.

> * I saw you create a hash per entity type, using the ids as keys within
this hash and a JSON document modelling the properties of a given entity
instance.
>
> Have you considered to store JSON documents on the "top-level" instead?
This would require to encode the table name into the key as well (similar
to as we do it for CouchDB): "Order:123" -> ... . The reason I am asking is
that I am wondering whether it may be a source of contention and/or whether
it may pose scaling issues if all entities of one type are stored within a
single hash.

Yes, I was thinking about that issue. I ended up with the hash per entity
type in the first place. This is not a final thing, but rather a draft.
Pulling the JSON strategy to the global level is a good idea and key
patterns like "Order:123" are a recommended way. The top-level approach
tackles also another issue: Redis can be operated as a cluster in terms of
sharding. The sharding is key-based. A top-level key encourages a nice
distribution amongst the cluster. Cluster support is not yet included in my
code. I wanted to check with you guys first, whether a Redis module could
be included in OGM.

> Using JSON for the actual entity contents seems reasonable and a common
approach in the Redis world from what I've read, so +1 for doing this from
me. How would the "more native" approach look like you describe below?
Would we have to serialise everything ourselves into String/byte[]? If so,
I am not sure whether that's of much usefulness actually and I'd be
inclined to go the JSON route exclusively. As you say, it also works nicely
for hierarchical structures.

My first idea on the more native approach was storing all basic data types
as key-value pairs. I noticed, that embedded properties are transported
with dot-notation, which comes handy. Converting the properties to String
is also a common Redis style and interoperability between different
systems/languages is encouraged hereby. I'm just not sure, how to deal with
external and embedded associations. Simulating associations by distributing
the lists to Redis lists with referencing other hashes looks like an
overkill to Redis. A lot of meta-data would have to be held. Feasible Yes,
sensible No. The ROM framework (https://github.com/josiahcarlson/rom/) goes
for this approach.

So if omitting associations in the more native approach is not a
deal-breaker, I'd vote to go this way as well. I'm also fine if we start
with just the JSON approach.

> * The id is used as key and also part of the JSON value. Could it be
removed from the latter?
Yes, this is due to the draft state

> * Do you have any ideas how Redis-specific structures could be exposed
which do not map exactly to entities, e.g. Lists stored directly via some
key?

Yes. Redis-backed Collections, Maps, Locks, Queues, Caching, and many more
are possible. I'm not sure, whether you want to have that features because
they do not look like they fit the Hibernate OGM concept. There are
implementations (Redisson: https://github.com/mrniko/redisson, Spring Data
Redis:
https://github.com/spring-projects/spring-data-redis/tree/master/src/main/java/org/springframework/data/redis/support/collections)
that cover the Redis-backed Collections etc.


> * Our option system could be used to expose specific settings such as TTL
for given entities
Great! Redis supports TTL's, I was not aware of that OGM feature

> * Is there any form of transaction support in Redis we could leverage?
Yes. Redis supports transactions with the MULTI/EXEC/DISCARD commands.
Transactions require a connection affinity and imply the use of multiple
connections. That's nothing bad per se, just a thing to be aware of.

Thanks and best regards, Mark


Gunnar Morling <gunnar at hibernate.org> schrieb am Di., 21. Juli 2015 um
13:03 Uhr:

> Hi Mark,
>
> Many thanks for giving this a try; Your work looks impressive, it's very
> advanced for an initial dialect contribution, covering many advanced cases
> already. So kudos for that!
>
> Please find some remarks and questions below.
>
> Cheers,
>
> --Gunnar
>
>
> 2015-07-17 17:16 GMT+02:00 Mark Paluch <mpaluch at paluch.biz>:
>
>> Hi there,
>>
>> I created a Redis module for Hibernate OGM, based on OGM 4.2. Redis is a
>> Key-Value store supporting Lists, Hashes (Maps), Sorted Sets, ...
>>
>> I took the Infinispan/CouchDB approach storing entities and associations
>> as
>> JSON documents within Redis Hashes. A Map<Object, Map<Object, Object>>
>> corresponds with a Redis Hash.
>>
>> The entity table maps to the Hash Id, the entity Id to the key of the hash
>> and the set of columns is enclosed within a JSON (currently). Supported
>> features are:
>>
>> * Composite Id's
>> * Embedded associations
>> * External associations
>>
>
> * I saw you create a hash per entity type, using the ids as keys within
> this hash and a JSON document modelling the properties of a given entity
> instance.
>
> Have you considered to store JSON documents on the "top-level" instead?
> This would require to encode the table name into the key as well (similar
> to as we do it for CouchDB): "Order:123" -> ... . The reason I am asking is
> that I am wondering whether it may be a source of contention and/or whether
> it may pose scaling issues if all entities of one type are stored within a
> single hash.
>
> Using JSON for the actual entity contents seems reasonable and a common
> approach in the Redis world from what I've read, so +1 for doing this from
> me. How would the "more native" approach look like you describe below?
> Would we have to serialise everything ourselves into String/byte[]? If so,
> I am not sure whether that's of much usefulness actually and I'd be
> inclined to go the JSON route exclusively. As you say, it also works nicely
> for hierarchical structures.
>
> * The id is used as key and also part of the JSON value. Could it be
> removed from the latter?
>
> * Do you have any ideas how Redis-specific structures could be exposed
> which do not map exactly to entities, e.g. Lists stored directly via some
> key?
>
> * Our option system could be used to expose specific settings such as TTL
> for given entities
>
> * Is there any form of transaction support in Redis we could leverage?
>
> and I guess many more, that I'm currently not aware of :-)
>>
>> There are various ways (JSON, Java-serialized simulating Documents) how to
>> store data within Redis.
>> Another approach could be storing Tuples within Redis more native, since
>> they are in the end Maps, somehow. However, mapping tuples to Redis fit a
>> native storage but raises questions about Id's, limited data type support
>> (there is no equivalent for nested lists/maps). So the more native
>> approach
>> would require distributing data amongst various data types, depending on
>> the supported features, and would require more IO's.
>>
>> My impl can be used to interoperate with other clients from the C, Ruby,
>> Python, Node.js, ... worlds. The JSON is clean and does not look awkward
>> on
>> the first look.
>>
>> The current code can be found below
>> https://github.com/mp911de/hibernate-ogm/tree/redis. Most TCK tests pass,
>> only some 15 tests fail. Mostly due to enum and JTA support. Couldn't
>> figure out, how to fix it, so need your help to figure it out.
>>
>
> I'll take a closer look at your change asap. Could you create a pull
> request from your branch against the upstream repo? Then we can have a
> discussion right there.
>
> Hope, I made you hungry for the Redis OGM module. Let's start a discussion
>> based on my proposal for inclusion and let's see, where we end up.
>>
>
> Yep, it looks very promising and I think it'd be great to have support for
> Redis in OGM :) We'll need your continued help though to make it happen and
> evolve it in the future as Redis itself and OGM core advance.
>
> Best regards, Mark
>> _______________________________________________
>> hibernate-dev mailing list
>> hibernate-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>
>
>