[hibernate-dev] Proposal for a Redis module in Hibernate OGM

[Mark Paluch]

Hi,
I fixed the enum thing. It was a mismatch between int and long. D'oh!

> Are there conventions for mapping more complex types such as Date for
instance?
No (at least I don't know any), so we should go as we need it.

> TTL
Implemented.

> Transactions: Ah, that's interesting. So it should be possible to link
this with the JPA/JTA transaction APIs?

Redis transactions work a bit different (was not aware of that, as I told
you about transactions). Redis transactions are more batch than
transactions. Redis transactions work very nice when writing, but there's a
reading issue. Reads are queued until the transaction is submitted. So the
read will come back with a result only if the transaction is confirmed with
`EXEC`. I'm pretty sure, this can be a deal-breaker for using transactions
in the JTA way.

I submitted a PR https://github.com/hibernate/hibernate-ogm/pull/560 and
started to work on the docs. Let's start a discussion based on the PR :-)

Best regards, viele Grüße, Mark


Gunnar Morling <gunnar at hibernate.org> schrieb am Mi., 22. Juli 2015 um
10:41 Uhr:

> 2015-07-21 14:14 GMT+02:00 Mark Paluch <mpaluch at paluch.biz>:
>
>> Hi Gunnar,
>>
>> thanks for your helpful feedback. I will come up with a PR as soon as we
>> agree on the strategies issue - well, perhaps in a couple days. I can and
>> will support you and I can help you with support/maintenance for the Redis
>> module in the future.
>>
>
> That's great to hear. We are no Redis experts, so the help of one is
> needed to make this fly :)
>
>
>> I will have to ask you in return on some specific issues, such as why
>> enum-ordinals cannot be mapped back.
>>
>
> Sure, absolutely. I will take a look at the enum thingy.
>
>
>> > * I saw you create a hash per entity type, using the ids as keys within
>> this hash and a JSON document modelling the properties of a given entity
>> instance.
>> >
>> > Have you considered to store JSON documents on the "top-level" instead?
>> This would require to encode the table name into the key as well (similar
>> to as we do it for CouchDB): "Order:123" -> ... . The reason I am asking is
>> that I am wondering whether it may be a source of contention and/or whether
>> it may pose scaling issues if all entities of one type are stored within a
>> single hash.
>>
>> Yes, I was thinking about that issue. I ended up with the hash per entity
>> type in the first place. This is not a final thing, but rather a draft.
>> Pulling the JSON strategy to the global level is a good idea and key
>> patterns like "Order:123" are a recommended way. The top-level approach
>> tackles also another issue: Redis can be operated as a cluster in terms of
>> sharding. The sharding is key-based. A top-level key encourages a nice
>> distribution amongst the cluster. Cluster support is not yet included in my
>> code. I wanted to check with you guys first, whether a Redis module could
>> be included in OGM.
>>
>
> Yes, clustering was part of my concern with the original approach. The
> top-level approach would help with that.
>
>>
>> > Using JSON for the actual entity contents seems reasonable and a common
>> approach in the Redis world from what I've read, so +1 for doing this from
>> me. How would the "more native" approach look like you describe below?
>> Would we have to serialise everything ourselves into String/byte[]? If so,
>> I am not sure whether that's of much usefulness actually and I'd be
>> inclined to go the JSON route exclusively. As you say, it also works nicely
>> for hierarchical structures.
>>
>> My first idea on the more native approach was storing all basic data
>> types as key-value pairs. I noticed, that embedded properties are
>> transported with dot-notation, which comes handy. Converting the properties
>> to String is also a common Redis style and interoperability between
>> different systems/languages is encouraged hereby.
>>
>
> Are there conventions for mapping more complex types such as Date for
> instance? You'd need custom GridTypes to map everything into String (we
> have String-mapping implementations for some types such as Date and Long,
> but not for all of them).
>
>
>> I'm just not sure, how to deal with external and embedded associations.
>> Simulating associations by distributing the lists to Redis lists with
>> referencing other hashes looks like an overkill to Redis. A lot of
>> meta-data would have to be held. Feasible Yes, sensible No. The ROM
>> framework (https://github.com/josiahcarlson/rom/) goes for this
>> approach.
>>
>> So if omitting associations in the more native approach is not a
>> deal-breaker, I'd vote to go this way as well.
>>
>
> Not supporting associations at all in that mode seems like a heavy
> limitation to me. I am not sure what you mean by "lot of meta-data would
> have to be held"; Where, in the driver, or in the dialect? OGM engine
> should give you all you need, I suggest to check out the Infinispan/Ehcache
> backends and follow the same approach.
>
>
>> I'm also fine if we start with just the JSON approach.
>>
>
> +1 Let's start from there and let things evolve.
>
>>
>> > * The id is used as key and also part of the JSON value. Could it be
>> removed from the latter?
>> Yes, this is due to the draft state
>>
>> > * Do you have any ideas how Redis-specific structures could be exposed
>> which do not map exactly to entities, e.g. Lists stored directly via some
>> key?
>>
>> Yes. Redis-backed Collections, Maps, Locks, Queues, Caching, and many
>> more are possible. I'm not sure, whether you want to have that
>> features because they do not look like they fit the Hibernate OGM concept.
>> There are implementations (Redisson: https://github.com/mrniko/redisson,
>> Spring Data Redis:
>> https://github.com/spring-projects/spring-data-redis/tree/master/src/main/java/org/springframework/data/redis/support/collections)
>> that cover the Redis-backed Collections etc.
>>
>
> People will use Redis for the powers of its specific features, so I think
> they should be accessible in one way or the other. Not everything can be
> mapped to/exposed through JPA in a sensible way. I guess one needs to look
> at it on a case-by-case basis, maybe it makes sense to somehow expose the
> native API for certain advanced cases.
>
>>
>> > * Our option system could be used to expose specific settings such as
>> TTL for given entities
>> Great! Redis supports TTL's, I was not aware of that OGM feature
>>
>
> OGM does not directly support this, but it provides an extensible option
> system which dialects can leverage to let the user configure specific
> settings and apply these when talking to the datastore. E.g. one could
> imagine something like this:
>
> @TTL(value=7, unit=TimeUnit.DAY)
> @Entity
> class LogRecord { ... }
>
> @TTL would be a Redis-backend specific setting based on said option
> system, and the Redis backend would apply this setting when writing tuples
> of the LogRecord entity. You can find examples with
> @ReadPreference/@WriteConcern in the MongoDB backend. Of course that's not
> required for a first version, but would make for a nice addition in later
> iterations.
>
>
>>
>> > * Is there any form of transaction support in Redis we could leverage?
>> Yes. Redis supports transactions with the MULTI/EXEC/DISCARD commands.
>> Transactions require a connection affinity and imply the use of multiple
>> connections. That's nothing bad per se, just a thing to be aware of.
>>
>
> Ah, that's interesting. So it should be possible to link this with the
> JPA/JTA transaction APIs?
>
>
>>
>> Thanks and best regards, Mark
>>
>>
>> Gunnar Morling <gunnar at hibernate.org> schrieb am Di., 21. Juli 2015 um
>> 13:03 Uhr:
>>
>>> Hi Mark,
>>>
>>> Many thanks for giving this a try; Your work looks impressive, it's very
>>> advanced for an initial dialect contribution, covering many advanced cases
>>> already. So kudos for that!
>>>
>>> Please find some remarks and questions below.
>>>
>>> Cheers,
>>>
>>> --Gunnar
>>>
>>>
>>> 2015-07-17 17:16 GMT+02:00 Mark Paluch <mpaluch at paluch.biz>:
>>>
>>>> Hi there,
>>>>
>>>> I created a Redis module for Hibernate OGM, based on OGM 4.2. Redis is a
>>>> Key-Value store supporting Lists, Hashes (Maps), Sorted Sets, ...
>>>>
>>>> I took the Infinispan/CouchDB approach storing entities and
>>>> associations as
>>>> JSON documents within Redis Hashes. A Map<Object, Map<Object, Object>>
>>>> corresponds with a Redis Hash.
>>>>
>>>> The entity table maps to the Hash Id, the entity Id to the key of the
>>>> hash
>>>> and the set of columns is enclosed within a JSON (currently). Supported
>>>> features are:
>>>>
>>>> * Composite Id's
>>>> * Embedded associations
>>>> * External associations
>>>>
>>>
>>> * I saw you create a hash per entity type, using the ids as keys within
>>> this hash and a JSON document modelling the properties of a given entity
>>> instance.
>>>
>>> Have you considered to store JSON documents on the "top-level" instead?
>>> This would require to encode the table name into the key as well (similar
>>> to as we do it for CouchDB): "Order:123" -> ... . The reason I am asking is
>>> that I am wondering whether it may be a source of contention and/or whether
>>> it may pose scaling issues if all entities of one type are stored within a
>>> single hash.
>>>
>>> Using JSON for the actual entity contents seems reasonable and a common
>>> approach in the Redis world from what I've read, so +1 for doing this from
>>> me. How would the "more native" approach look like you describe below?
>>> Would we have to serialise everything ourselves into String/byte[]? If so,
>>> I am not sure whether that's of much usefulness actually and I'd be
>>> inclined to go the JSON route exclusively. As you say, it also works nicely
>>> for hierarchical structures.
>>>
>>> * The id is used as key and also part of the JSON value. Could it be
>>> removed from the latter?
>>>
>>> * Do you have any ideas how Redis-specific structures could be exposed
>>> which do not map exactly to entities, e.g. Lists stored directly via some
>>> key?
>>>
>>> * Our option system could be used to expose specific settings such as
>>> TTL for given entities
>>>
>>> * Is there any form of transaction support in Redis we could leverage?
>>>
>>> and I guess many more, that I'm currently not aware of :-)
>>>>
>>>> There are various ways (JSON, Java-serialized simulating Documents) how
>>>> to
>>>> store data within Redis.
>>>> Another approach could be storing Tuples within Redis more native, since
>>>> they are in the end Maps, somehow. However, mapping tuples to Redis fit
>>>> a
>>>> native storage but raises questions about Id's, limited data type
>>>> support
>>>> (there is no equivalent for nested lists/maps). So the more native
>>>> approach
>>>> would require distributing data amongst various data types, depending on
>>>> the supported features, and would require more IO's.
>>>>
>>>> My impl can be used to interoperate with other clients from the C, Ruby,
>>>> Python, Node.js, ... worlds. The JSON is clean and does not look
>>>> awkward on
>>>> the first look.
>>>>
>>>> The current code can be found below
>>>> https://github.com/mp911de/hibernate-ogm/tree/redis. Most TCK tests
>>>> pass,
>>>> only some 15 tests fail. Mostly due to enum and JTA support. Couldn't
>>>> figure out, how to fix it, so need your help to figure it out.
>>>>
>>>
>>> I'll take a closer look at your change asap. Could you create a pull
>>> request from your branch against the upstream repo? Then we can have a
>>> discussion right there.
>>>
>>> Hope, I made you hungry for the Redis OGM module. Let's start a
>>>> discussion
>>>> based on my proposal for inclusion and let's see, where we end up.
>>>>
>>>
>>> Yep, it looks very promising and I think it'd be great to have support
>>> for Redis in OGM :) We'll need your continued help though to make it happen
>>> and evolve it in the future as Redis itself and OGM core advance.
>>>
>>> Best regards, Mark
>>>> _______________________________________________
>>>> hibernate-dev mailing list
>>>> hibernate-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>>>
>>>
>>>