8:51 p.m.
After more feedback about sharding from IRC and the forum [1], I
believe we should bind sharding implementations to indexed types, as
opposing to index names.
Currently a sharding strategy is strongly related to the index, but
configured on the index name and so it will affect all entities using
that same index name: there's no need to enforce this.
This also means we could change the sharding strategy interface to:
a - deal with Entity instances instead of org.apache.lucene.Document
and string-encoded ids
b - be a typesafe interface using generics
This would affect configuration: instead of configuring them on the
index name, an annotation should be placed on the type (or an optional
parameter for existing @Indexed).
So we won't be able to figure out in how many shards an index is
divided at boot time (unless we ask the sharding strategy via a new
method for this purpose), but this is good provided we finally support
dynamic sharding: start IndexManagers on demand as needed.
Configuration should have been reworked anyway, as it currently
supports only numbers as sub-indexes identifiers.
On top of a greater flexibility in sharding, but will also avoid
exposing the o.a.l.Document yet in another API. Not last, I've seen
cases in which people where forced to encode some token in the
Document only for the sake of their sharding strategy decision logic.
Sanne
1 - https://forum.hibernate.org/viewtopic.php?f=9&t=1011836
9:09 p.m.
On Thu, 11 Aug 2011 13:51:33 +0200, Sanne Grinovero <sanne(a)hibernate.org>
wrote:
This also means we could change the sharding strategy interface to:
a - deal with Entity instances instead of org.apache.lucene.Document
and string-encoded ids
Are there not valid use-cases where I want to make the decision using the
Document. What if I have a custom class bridge which I apply on all
entities
in order to add a field to the document ((partly)independent of the entity
itself)
which is then used for sharding. I guess I could always write the same
logic
in the ShardingStrategy!? Not sure.
This would affect configuration: instead of configuring them on the
index name, an annotation should be placed on the type (or an optional
parameter for existing @Indexed).
Are changing the sharding configuration and the ShardingStrategy really
coupled? We could change the way shards are configured and still keep
passing the Document to the sharding strategy, maybe in combination with
the actual entity. Wouldn't that give most flexibility?
On top of a greater flexibility in sharding, but will also avoid
exposing the o.a.l.Document yet in another API.
Is that such a bad thing?
--Hardy
9:27 p.m.
2011/8/11 Hardy Ferentschik <hardy(a)hibernate.org>:
On Thu, 11 Aug 2011 13:51:33 +0200, Sanne Grinovero
<sanne(a)hibernate.org>
wrote:
> This also means we could change the sharding strategy interface to:
> a - deal with Entity instances instead of org.apache.lucene.Document
> and string-encoded ids
Are there not valid use-cases where I want to make the decision using the
Document. What if I have a custom class bridge which I apply on all
entities
in order to add a field to the document ((partly)independent of the entity
itself)
which is then used for sharding. I guess I could always write the same
logic
in the ShardingStrategy!? Not sure.
I've thought about that same, and concluded than since the Document is
the output of some stateless transformation from the entity, I'm sure
that all what you could do in a custom bridge & sharding strategy
could be implemented in the sharding strategy alone, provided it gets
access to the entity. By definition the entity has more information,
not less, so it's definitely an added value.
Also if you had to change both code places to achieve your desired
sharding strategy, now you only need to do it once, in something
properly named "ShardingStrategy", instead of hacking around and
possibly instead of inserting extra unneeded tokens in the Document
for consumption of the ShardingStrategy.
> This would affect configuration: instead of configuring them on
the
> index name, an annotation should be placed on the type (or an optional
> parameter for existing @Indexed).
Are changing the sharding configuration and the ShardingStrategy really
coupled? We could change the way shards are configured and still keep
passing the Document to the sharding strategy, maybe in combination with
the actual entity. Wouldn't that give most flexibility?
As above, I don't think that will give you more functional options,
but maybe you're right that some operations might be easier: I'm
thinking of those cases in which the decisions is related to the
string-encoded form of some complex type, but also I don't see a
practical use of it, and nothing prevents me to re-encode it as needed
(other than repeating the encoding).
Still, not passing the Document provides you more advantages:
1 - we often felt the need of using a different representation, but
can't experiment with it as this is part of public API
2 - we would not be tight to build the Document before applying the
sharding logic: building the full Document often entails a lot of
work, not least loading extra entities for the sake of relations
indexing, and the strategy could decide to skip it. (I know I'm
trespassing in the area of another popular feature request which I
wouldn't implement this way, but as an example of kind of things we
can do with this added flexibility!)
> On top of a greater flexibility in sharding, but will also avoid
> exposing the o.a.l.Document yet in another API.
Is that such a bad thing?
Not extremely evil, but I hope above answers this.
Sanne
10:06 p.m.
Answers inline
On Thu, 11 Aug 2011 14:27:58 +0200, Sanne Grinovero <sanne(a)hibernate.org>
wrote:
>> This also means we could change the sharding strategy
interface to:
>> a - deal with Entity instances instead of org.apache.lucene.Document
>> and string-encoded ids
>
> Are there not valid use-cases where I want to make the decision using
> the
> Document. What if I have a custom class bridge which I apply on all
> entities
> in order to add a field to the document ((partly)independent of the
> entity
> itself)
> which is then used for sharding. I guess I could always write the same
> logic
> in the ShardingStrategy!? Not sure.
I've thought about that same, and concluded than since the Document is
the output of some stateless transformation from the entity, I'm sure
that all what you could do in a custom bridge & sharding strategy
could be implemented in the sharding strategy alone, provided it gets
access to the entity. By definition the entity has more information,
not less, so it's definitely an added value.
Also if you had to change both code places to achieve your desired
sharding strategy, now you only need to do it once, in something
properly named "ShardingStrategy", instead of hacking around and
possibly instead of inserting extra unneeded tokens in the Document
for consumption of the ShardingStrategy.
I agree. If you pass along the entity you are always do the same
transformations/decisions, but do you need to do it again if you already
applied
the logic in the bridge (provided you want the information indexed as well)
>> This would affect configuration: instead of configuring them
on the
>> index name, an annotation should be placed on the type (or an optional
>> parameter for existing @Indexed).
>
> Are changing the sharding configuration and the ShardingStrategy really
> coupled? We could change the way shards are configured and still keep
> passing the Document to the sharding strategy, maybe in combination with
> the actual entity. Wouldn't that give most flexibility?
As above, I don't think that will give you more functional options,
but maybe you're right that some operations might be easier: I'm
thinking of those cases in which the decisions is related to the
string-encoded form of some complex type,
That's what I am thinking about as well.
>> On top of a greater flexibility in sharding, but will also
avoid
>> exposing the o.a.l.Document yet in another API.
>
> Is that such a bad thing?
Not extremely evil, but I hope above answers this.
I am not too concerned about exposing the Document, but overall don't have
a strong
argument against your suggestion. Just wanted to share my initial thoughts
and
concerns.
--Hardy
4:54 p.m.
I'm a bit lost. Maybe we could do an IRC session on the subject to get everyone (ahem
me) up to speed.
On 11 août 2011, at 13:51, Sanne Grinovero wrote:
After more feedback about sharding from IRC and the forum [1], I
believe we should bind sharding implementations to indexed types, as
opposing to index names.
Currently a sharding strategy is strongly related to the index, but
configured on the index name and so it will affect all entities using
that same index name: there's no need to enforce this.
This also means we could change the sharding strategy interface to:
a - deal with Entity instances instead of org.apache.lucene.Document
and string-encoded ids
b - be a typesafe interface using generics
This would affect configuration: instead of configuring them on the
index name, an annotation should be placed on the type (or an optional
parameter for existing @Indexed).
So we won't be able to figure out in how many shards an index is
divided at boot time (unless we ask the sharding strategy via a new
method for this purpose), but this is good provided we finally support
dynamic sharding: start IndexManagers on demand as needed.
Configuration should have been reworked anyway, as it currently
supports only numbers as sub-indexes identifiers.
On top of a greater flexibility in sharding, but will also avoid
exposing the o.a.l.Document yet in another API. Not last, I've seen
cases in which people where forced to encode some token in the
Document only for the sake of their sharding strategy decision logic.
Sanne
1 - https://forum.hibernate.org/viewtopic.php?f=9&t=1011836
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
5:48 p.m.
2011/8/17 Emmanuel Bernard <emmanuel(a)hibernate.org>:
I'm a bit lost. Maybe we could do an IRC session on the subject
to get everyone (ahem me) up to speed.
whenever you want, I'm online and available.
Are you specifically lost about this discussion related to sharding,
or general-purpose lost about the state of Search?
Cheers,
Sanne
7:04 p.m.
On 17 août 2011, at 10:48, Sanne Grinovero wrote:
2011/8/17 Emmanuel Bernard <emmanuel(a)hibernate.org>:
> I'm a bit lost. Maybe we could do an IRC session on the subject to get everyone
(ahem me) up to speed.
whenever you want, I'm online and available.
Are you specifically lost about this discussion related to sharding,
or general-purpose lost about the state of Search?
Only this discussion on sharding, I'm not that old ;)
I'll be quite busy today but we can chat more freely tomorrow.
4649
days inactive
4655
days old
6 comments
3 participants
participants (3)
-
Emmanuel Bernard -
Hardy Ferentschik -
Sanne Grinovero