2015-07-21 14:14 GMT+02:00 Mark Paluch <mpaluch(a)paluch.biz>:
Hi Gunnar,
thanks for your helpful feedback. I will come up with a PR as soon as we
agree on the strategies issue - well, perhaps in a couple days. I can and
will support you and I can help you with support/maintenance for the Redis
module in the future.
That's great to hear. We are no Redis experts, so the help of one is needed
to make this fly :)
I will have to ask you in return on some specific issues, such as
why
enum-ordinals cannot be mapped back.
Sure, absolutely. I will take a look at the enum thingy.
> * I saw you create a hash per entity type, using the ids as keys
within
this hash and a JSON document modelling the properties of a given entity
instance.
>
> Have you considered to store JSON documents on the "top-level" instead?
This would require to encode the table name into the key as well (similar
to as we do it for CouchDB): "Order:123" -> ... . The reason I am asking is
that I am wondering whether it may be a source of contention and/or whether
it may pose scaling issues if all entities of one type are stored within a
single hash.
Yes, I was thinking about that issue. I ended up with the hash per entity
type in the first place. This is not a final thing, but rather a draft.
Pulling the JSON strategy to the global level is a good idea and key
patterns like "Order:123" are a recommended way. The top-level approach
tackles also another issue: Redis can be operated as a cluster in terms of
sharding. The sharding is key-based. A top-level key encourages a nice
distribution amongst the cluster. Cluster support is not yet included in my
code. I wanted to check with you guys first, whether a Redis module could
be included in OGM.
Yes, clustering was part of my concern with the original approach. The
top-level approach would help with that.
> Using JSON for the actual entity contents seems reasonable and a common
approach in the Redis world from what I've read, so +1 for doing this from
me. How would the "more native" approach look like you describe below?
Would we have to serialise everything ourselves into String/byte[]? If so,
I am not sure whether that's of much usefulness actually and I'd be
inclined to go the JSON route exclusively. As you say, it also works nicely
for hierarchical structures.
My first idea on the more native approach was storing all basic data types
as key-value pairs. I noticed, that embedded properties are transported
with dot-notation, which comes handy. Converting the properties to String
is also a common Redis style and interoperability between different
systems/languages is encouraged hereby.
Are there conventions for mapping more complex types such as Date for
instance? You'd need custom GridTypes to map everything into String (we
have String-mapping implementations for some types such as Date and Long,
but not for all of them).
I'm just not sure, how to deal with external and embedded
associations.
Simulating associations by distributing the lists to Redis lists with
referencing other hashes looks like an overkill to Redis. A lot of
meta-data would have to be held. Feasible Yes, sensible No. The ROM
framework (
https://github.com/josiahcarlson/rom/) goes for this approach.
So if omitting associations in the more native approach is not a
deal-breaker, I'd vote to go this way as well.
Not supporting associations at all in that mode seems like a heavy
limitation to me. I am not sure what you mean by "lot of meta-data would
have to be held"; Where, in the driver, or in the dialect? OGM engine
should give you all you need, I suggest to check out the Infinispan/Ehcache
backends and follow the same approach.
I'm also fine if we start with just the JSON approach.
+1 Let's start from there and let things evolve.
> * The id is used as key and also part of the JSON value. Could it be
removed from the latter?
Yes, this is due to the draft state
> * Do you have any ideas how Redis-specific structures could be exposed
which do not map exactly to entities, e.g. Lists stored directly via some
key?
Yes. Redis-backed Collections, Maps, Locks, Queues, Caching, and many more
are possible. I'm not sure, whether you want to have that features because
they do not look like they fit the Hibernate OGM concept. There are
implementations (Redisson:
https://github.com/mrniko/redisson, Spring
Data Redis:
https://github.com/spring-projects/spring-data-redis/tree/master/src/main...)
that cover the Redis-backed Collections etc.
People will use Redis for the powers of its specific features, so I think
they should be accessible in one way or the other. Not everything can be
mapped to/exposed through JPA in a sensible way. I guess one needs to look
at it on a case-by-case basis, maybe it makes sense to somehow expose the
native API for certain advanced cases.
> * Our option system could be used to expose specific settings such as
TTL for given entities
Great! Redis supports TTL's, I was not aware of that OGM feature
OGM does not directly support this, but it provides an extensible option
system which dialects can leverage to let the user configure specific
settings and apply these when talking to the datastore. E.g. one could
imagine something like this:
@TTL(value=7, unit=TimeUnit.DAY)
@Entity
class LogRecord { ... }
@TTL would be a Redis-backend specific setting based on said option system,
and the Redis backend would apply this setting when writing tuples of the
LogRecord entity. You can find examples with @ReadPreference/@WriteConcern
in the MongoDB backend. Of course that's not required for a first version,
but would make for a nice addition in later iterations.
> * Is there any form of transaction support in Redis we could leverage?
Yes. Redis supports transactions with the MULTI/EXEC/DISCARD commands.
Transactions require a connection affinity and imply the use of multiple
connections. That's nothing bad per se, just a thing to be aware of.
Ah, that's interesting. So it should be possible to link this with the
JPA/JTA transaction APIs?
Thanks and best regards, Mark
Gunnar Morling <gunnar(a)hibernate.org> schrieb am Di., 21. Juli 2015 um
13:03 Uhr:
> Hi Mark,
>
> Many thanks for giving this a try; Your work looks impressive, it's very
> advanced for an initial dialect contribution, covering many advanced cases
> already. So kudos for that!
>
> Please find some remarks and questions below.
>
> Cheers,
>
> --Gunnar
>
>
> 2015-07-17 17:16 GMT+02:00 Mark Paluch <mpaluch(a)paluch.biz>:
>
>> Hi there,
>>
>> I created a Redis module for Hibernate OGM, based on OGM 4.2. Redis is a
>> Key-Value store supporting Lists, Hashes (Maps), Sorted Sets, ...
>>
>> I took the Infinispan/CouchDB approach storing entities and associations
>> as
>> JSON documents within Redis Hashes. A Map<Object, Map<Object,
Object>>
>> corresponds with a Redis Hash.
>>
>> The entity table maps to the Hash Id, the entity Id to the key of the
>> hash
>> and the set of columns is enclosed within a JSON (currently). Supported
>> features are:
>>
>> * Composite Id's
>> * Embedded associations
>> * External associations
>>
>
> * I saw you create a hash per entity type, using the ids as keys within
> this hash and a JSON document modelling the properties of a given entity
> instance.
>
> Have you considered to store JSON documents on the "top-level" instead?
> This would require to encode the table name into the key as well (similar
> to as we do it for CouchDB): "Order:123" -> ... . The reason I am asking
is
> that I am wondering whether it may be a source of contention and/or whether
> it may pose scaling issues if all entities of one type are stored within a
> single hash.
>
> Using JSON for the actual entity contents seems reasonable and a common
> approach in the Redis world from what I've read, so +1 for doing this from
> me. How would the "more native" approach look like you describe below?
> Would we have to serialise everything ourselves into String/byte[]? If so,
> I am not sure whether that's of much usefulness actually and I'd be
> inclined to go the JSON route exclusively. As you say, it also works nicely
> for hierarchical structures.
>
> * The id is used as key and also part of the JSON value. Could it be
> removed from the latter?
>
> * Do you have any ideas how Redis-specific structures could be exposed
> which do not map exactly to entities, e.g. Lists stored directly via some
> key?
>
> * Our option system could be used to expose specific settings such as TTL
> for given entities
>
> * Is there any form of transaction support in Redis we could leverage?
>
> and I guess many more, that I'm currently not aware of :-)
>>
>> There are various ways (JSON, Java-serialized simulating Documents) how
>> to
>> store data within Redis.
>> Another approach could be storing Tuples within Redis more native, since
>> they are in the end Maps, somehow. However, mapping tuples to Redis fit a
>> native storage but raises questions about Id's, limited data type support
>> (there is no equivalent for nested lists/maps). So the more native
>> approach
>> would require distributing data amongst various data types, depending on
>> the supported features, and would require more IO's.
>>
>> My impl can be used to interoperate with other clients from the C, Ruby,
>> Python, Node.js, ... worlds. The JSON is clean and does not look awkward
>> on
>> the first look.
>>
>> The current code can be found below
>>
https://github.com/mp911de/hibernate-ogm/tree/redis. Most TCK tests
>> pass,
>> only some 15 tests fail. Mostly due to enum and JTA support. Couldn't
>> figure out, how to fix it, so need your help to figure it out.
>>
>
> I'll take a closer look at your change asap. Could you create a pull
> request from your branch against the upstream repo? Then we can have a
> discussion right there.
>
> Hope, I made you hungry for the Redis OGM module. Let's start a discussion
>> based on my proposal for inclusion and let's see, where we end up.
>>
>
> Yep, it looks very promising and I think it'd be great to have support
> for Redis in OGM :) We'll need your continued help though to make it happen
> and evolve it in the future as Redis itself and OGM core advance.
>
> Best regards, Mark
>> _______________________________________________
>> hibernate-dev mailing list
>> hibernate-dev(a)lists.jboss.org
>>
https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>
>
>