[hibernate-dev] Proposal for a Redis module in Hibernate OGM

Wed Jul 22 04:41:12 EDT 2015

2015-07-21 14:14 GMT+02:00 Mark Paluch <mpaluch at paluch.biz>:

> Hi Gunnar,
>
> thanks for your helpful feedback. I will come up with a PR as soon as we
> agree on the strategies issue - well, perhaps in a couple days. I can and
> will support you and I can help you with support/maintenance for the Redis
> module in the future.
>

That's great to hear. We are no Redis experts, so the help of one is needed
to make this fly :)

> I will have to ask you in return on some specific issues, such as why
> enum-ordinals cannot be mapped back.
>

Sure, absolutely. I will take a look at the enum thingy.

> > * I saw you create a hash per entity type, using the ids as keys within
> this hash and a JSON document modelling the properties of a given entity
> instance.
> >
> > Have you considered to store JSON documents on the "top-level" instead?
> This would require to encode the table name into the key as well (similar
> to as we do it for CouchDB): "Order:123" -> ... . The reason I am asking is
> that I am wondering whether it may be a source of contention and/or whether
> it may pose scaling issues if all entities of one type are stored within a
> single hash.
>
> Yes, I was thinking about that issue. I ended up with the hash per entity
> type in the first place. This is not a final thing, but rather a draft.
> Pulling the JSON strategy to the global level is a good idea and key
> patterns like "Order:123" are a recommended way. The top-level approach
> tackles also another issue: Redis can be operated as a cluster in terms of
> sharding. The sharding is key-based. A top-level key encourages a nice
> distribution amongst the cluster. Cluster support is not yet included in my
> code. I wanted to check with you guys first, whether a Redis module could
> be included in OGM.
>

Yes, clustering was part of my concern with the original approach. The
top-level approach would help with that.

>
> > Using JSON for the actual entity contents seems reasonable and a common
> approach in the Redis world from what I've read, so +1 for doing this from
> me. How would the "more native" approach look like you describe below?
> Would we have to serialise everything ourselves into String/byte[]? If so,
> I am not sure whether that's of much usefulness actually and I'd be
> inclined to go the JSON route exclusively. As you say, it also works nicely
> for hierarchical structures.
>
> My first idea on the more native approach was storing all basic data types
> as key-value pairs. I noticed, that embedded properties are transported
> with dot-notation, which comes handy. Converting the properties to String
> is also a common Redis style and interoperability between different
> systems/languages is encouraged hereby.
>

Are there conventions for mapping more complex types such as Date for
instance? You'd need custom GridTypes to map everything into String (we
have String-mapping implementations for some types such as Date and Long,
but not for all of them).

> I'm just not sure, how to deal with external and embedded associations.
> Simulating associations by distributing the lists to Redis lists with
> referencing other hashes looks like an overkill to Redis. A lot of
> meta-data would have to be held. Feasible Yes, sensible No. The ROM
> framework (https://github.com/josiahcarlson/rom/) goes for this approach.
>
> So if omitting associations in the more native approach is not a
> deal-breaker, I'd vote to go this way as well.
>

Not supporting associations at all in that mode seems like a heavy
limitation to me. I am not sure what you mean by "lot of meta-data would
have to be held"; Where, in the driver, or in the dialect? OGM engine
should give you all you need, I suggest to check out the Infinispan/Ehcache
backends and follow the same approach.

> I'm also fine if we start with just the JSON approach.
>

+1 Let's start from there and let things evolve.

>
> > * The id is used as key and also part of the JSON value. Could it be
> removed from the latter?
> Yes, this is due to the draft state
>
> > * Do you have any ideas how Redis-specific structures could be exposed
> which do not map exactly to entities, e.g. Lists stored directly via some
> key?
>
> Yes. Redis-backed Collections, Maps, Locks, Queues, Caching, and many more
> are possible. I'm not sure, whether you want to have that features because
> they do not look like they fit the Hibernate OGM concept. There are
> implementations (Redisson: https://github.com/mrniko/redisson, Spring
> Data Redis:
> https://github.com/spring-projects/spring-data-redis/tree/master/src/main/java/org/springframework/data/redis/support/collections)
> that cover the Redis-backed Collections etc.
>

People will use Redis for the powers of its specific features, so I think
they should be accessible in one way or the other. Not everything can be
mapped to/exposed through JPA in a sensible way. I guess one needs to look
at it on a case-by-case basis, maybe it makes sense to somehow expose the
native API for certain advanced cases.

>
> > * Our option system could be used to expose specific settings such as
> TTL for given entities
> Great! Redis supports TTL's, I was not aware of that OGM feature
>

OGM does not directly support this, but it provides an extensible option
system which dialects can leverage to let the user configure specific
settings and apply these when talking to the datastore. E.g. one could
imagine something like this:

@TTL(value=7, unit=TimeUnit.DAY)
@Entity
class LogRecord { ... }

@TTL would be a Redis-backend specific setting based on said option system,
and the Redis backend would apply this setting when writing tuples of the
LogRecord entity. You can find examples with @ReadPreference/@WriteConcern
in the MongoDB backend. Of course that's not required for a first version,
but would make for a nice addition in later iterations.

>
> > * Is there any form of transaction support in Redis we could leverage?
> Yes. Redis supports transactions with the MULTI/EXEC/DISCARD commands.
> Transactions require a connection affinity and imply the use of multiple
> connections. That's nothing bad per se, just a thing to be aware of.
>

Ah, that's interesting. So it should be possible to link this with the
JPA/JTA transaction APIs?

>
> Thanks and best regards, Mark
>
>
> Gunnar Morling <gunnar at hibernate.org> schrieb am Di., 21. Juli 2015 um
> 13:03 Uhr:
>
>> Hi Mark,
>>
>> Many thanks for giving this a try; Your work looks impressive, it's very
>> advanced for an initial dialect contribution, covering many advanced cases
>> already. So kudos for that!
>>
>> Please find some remarks and questions below.
>>
>> Cheers,
>>
>> --Gunnar
>>
>>
>> 2015-07-17 17:16 GMT+02:00 Mark Paluch <mpaluch at paluch.biz>:
>>
>>> Hi there,
>>>
>>> I created a Redis module for Hibernate OGM, based on OGM 4.2. Redis is a
>>> Key-Value store supporting Lists, Hashes (Maps), Sorted Sets, ...
>>>
>>> I took the Infinispan/CouchDB approach storing entities and associations
>>> as
>>> JSON documents within Redis Hashes. A Map<Object, Map<Object, Object>>
>>> corresponds with a Redis Hash.
>>>
>>> The entity table maps to the Hash Id, the entity Id to the key of the
>>> hash
>>> and the set of columns is enclosed within a JSON (currently). Supported
>>> features are:
>>>
>>> * Composite Id's
>>> * Embedded associations
>>> * External associations
>>>
>>
>> * I saw you create a hash per entity type, using the ids as keys within
>> this hash and a JSON document modelling the properties of a given entity
>> instance.
>>
>> Have you considered to store JSON documents on the "top-level" instead?
>> This would require to encode the table name into the key as well (similar
>> to as we do it for CouchDB): "Order:123" -> ... . The reason I am asking is
>> that I am wondering whether it may be a source of contention and/or whether
>> it may pose scaling issues if all entities of one type are stored within a
>> single hash.
>>
>> Using JSON for the actual entity contents seems reasonable and a common
>> approach in the Redis world from what I've read, so +1 for doing this from
>> me. How would the "more native" approach look like you describe below?
>> Would we have to serialise everything ourselves into String/byte[]? If so,
>> I am not sure whether that's of much usefulness actually and I'd be
>> inclined to go the JSON route exclusively. As you say, it also works nicely
>> for hierarchical structures.
>>
>> * The id is used as key and also part of the JSON value. Could it be
>> removed from the latter?
>>
>> * Do you have any ideas how Redis-specific structures could be exposed
>> which do not map exactly to entities, e.g. Lists stored directly via some
>> key?
>>
>> * Our option system could be used to expose specific settings such as TTL
>> for given entities
>>
>> * Is there any form of transaction support in Redis we could leverage?
>>
>> and I guess many more, that I'm currently not aware of :-)
>>>
>>> There are various ways (JSON, Java-serialized simulating Documents) how
>>> to
>>> store data within Redis.
>>> Another approach could be storing Tuples within Redis more native, since
>>> they are in the end Maps, somehow. However, mapping tuples to Redis fit a
>>> native storage but raises questions about Id's, limited data type support
>>> (there is no equivalent for nested lists/maps). So the more native
>>> approach
>>> would require distributing data amongst various data types, depending on
>>> the supported features, and would require more IO's.
>>>
>>> My impl can be used to interoperate with other clients from the C, Ruby,
>>> Python, Node.js, ... worlds. The JSON is clean and does not look awkward
>>> on
>>> the first look.
>>>
>>> The current code can be found below
>>> https://github.com/mp911de/hibernate-ogm/tree/redis. Most TCK tests
>>> pass,
>>> only some 15 tests fail. Mostly due to enum and JTA support. Couldn't
>>> figure out, how to fix it, so need your help to figure it out.
>>>
>>
>> I'll take a closer look at your change asap. Could you create a pull
>> request from your branch against the upstream repo? Then we can have a
>> discussion right there.
>>
>> Hope, I made you hungry for the Redis OGM module. Let's start a discussion
>>> based on my proposal for inclusion and let's see, where we end up.
>>>
>>
>> Yep, it looks very promising and I think it'd be great to have support
>> for Redis in OGM :) We'll need your continued help though to make it happen
>> and evolve it in the future as Redis itself and OGM core advance.
>>
>> Best regards, Mark
>>> _______________________________________________
>>> hibernate-dev mailing list
>>> hibernate-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>>
>>
>>