[Search] Dynamic Sharding the second
by Hardy Ferentschik
Hi,
here comes a follow up on my previous email regarding configuration of dynamic sharding.
Now I would like to get some feedback on ShardIdentifierProvider. This interface was added for dynamic sharding on
top of DynamicShardingStrategy. Gunnar and I had a discussion around it today [1] and we came to the conclusion that
this interface is actually not needed and just adds confusion in the API.
Really ShardIdentifierProvider is a IndexShardingStrategy in disguise. Add 'forAddtion' and 'forDeletion' to the two 'getShardIdentifier' methods
and you have (almost) an IndexShardingStrategy. The problem seems to be, that in order to implement dynamic sharding the
IndexManagerHolder and EntityIndexBinding are needed. IndexShardingStrategy#initialize does not provide access to these objects which maybe
led to the current design.
In DynamicShardingStrategy this is taken care of by passing IndexManagerHolder and EntityIndexBinding as part of the constructor arguments.
Otherwise DynamicShardingStrategy is just a very thin wrapper delegating to the ShardIdentifierProvider.
If the current IndexShardingStrategy#initialize method would get passed the required information for dynamic sharding, there would be no need
for an additional interface like ShardIdentifierProvider. Dynamic sharding could be implemented with the existing extension points and implementations
would fit better into the current pattern of providing custom implementations.
What we could do is to make DynamicShardingStrategy an interface extending IndexShardingStrategy and adding a second initialise contract
of sorts. This would keep backwards compatibility, but also allow for dynamic sharding by users implementing DynamicShardingStrategy.
At the downside the user would have to write a bit more code, but I think that's acceptable given the more consistent approach towards sharding.
Thoughts?
--Hardy
P.S. In case you guys think that we really should hold on to ShardIdentifierProvider, I would at least suggest to either rename
the two 'getShardIdentifier' methods adding 'forAddition' and 'forDeletion' or even collapse the two into a single method (not sure
whether this is easily possible)
[1] http://transcripts.jboss.org/channel/irc.freenode.org/%23hibernate-dev/20...
12 years, 2 months
Should we deprecate @Similarity
by Sanne Grinovero
Discussing about some hibernate-search-engine complexities with Hardy
on IRC, we came to the agreement that the way @Similarity behaves
today was a mistake, so we're proposing to deprecate it in 4.4.
Reasoning follows.
There is a strong requirement in Lucene that all operations on the
same index need to use the same Similarity implementation. We infer
the org.apache.lucene.search.Similarity to be used on a specific index
from either:
- the similarity property from the configuration file, which is
positioned on an index name
- the @Similarity annotation positioned on an indexed entity
The trouble is about all these options needing to be consistent; first
problem is there isn't a precedence rule: if one of them is not
specified, we assume the user is using the other way to configure
things. But also different entities might be sharing the same index,
which leads to needing to verify that at least the user isn't
specifying some contradictory option.
This all gets more confusing when you introduce the notion of
dynamically added new entities (a feature used by Infinispan) and/or
Dynamic Sharding, in which the properties of some indexes might
conflict with the specified @Similarity, but we might notice this only
at runtime rather than at bootstrap. Failing to load a class at this
point is far more annoying to the users than to fail the health-check
at bootstrap time.
So the proposal:
drop the @Similarity annotation and use properties exclusively.
Properties are more suited for this as they are set on a per-index
base, which is what matters in this case.
Downside:
I guess lots of people where using a single index per type, and for
those there was no danger to simply specify a @Similarity. We lose
this straight-forward way of things, but I'd argue that if you're in
the business of specifying a custom Similarity, you're a very advanced
user and wouldn't mind setting a one-liner in your configuration file.
Do we agree?
Sanne
12 years, 2 months
HSEARCH-1270 : indexing (still) happening after Session#clear()
by Sanne Grinovero
I'm having a half-baked solution for HSEARCH-1270, and it looks like I
can't do better than that without waiting for HHH-8577 to be included
in ORM [JIRA links at the bottom].
# The problem described by HSEARCH-1270
After the user has enqueued some indexing operations in Hibernate
Search's Worker, if he then invokes Session.clear() the queue is not
cleared.
I realize that clear() might not be as popular as most other methods
on Session, still I consider this an annoying limitation: it
introduces an inconsistency in the index and the user might not even
realize the mistake, and neither do we have an opportunity to warn
him.
# Partial solution
This is working well, and the test passes:
https://github.com/Sanne/hibernate-search/compare/HSEARCH-1270?expand=1
#Why it's not complete
The test requires to invoke the clear() operation *after* having
wrapped the Session in a FullTextSession.
In other words, if I'm just using the Session the bug would still manifest.
#What now
Considering that we can't do better than that - at least without
patching ORM with HHH-8577 - would you still merge such a patch?
I would propose to merge it, and add a warning in the docs, at least
the user has an option to workaround the issue.
The downside is that so far, at least Session and FullTextSession
behave consistently.. was that a good thing?
Another approach to the problem could be to provide an explicit
FullTextSession#clearIndexingQueue()
I think I'd like that, especially as it can be useful in other
situations, for example to avoid triggering any indexing operation for
a specific transaction.
Also, it would be a good companion for
FullTextSession#flushToIndexes()
-- Sanne
https://hibernate.atlassian.net/browse/HSEARCH-1270
https://hibernate.atlassian.net/browse/HHH-8577
12 years, 2 months
LoadPlans, EntityGraphs, 4.3 and moving forward
by Steve Ebersole
At this point I think we are far enough along with LoadPlans and
EntityGraph work to move that over to upstream master. Given that the
rest of the 2.1 TCK is in good shape I'd like to do that as soon as
possible. So I'd like to plan out how/when to get this moved over from
my fork to upstream. I have no problem doing the actual move if y'all
would prefer, we all just need to agree on a time to do it.
At that point we also need to start breaking down the remaining work
into granular chunks so all of us can pitch in and get 4.3 rolling.
Gail, Strong, since y'all have been working on this lately can y'all
give some thought to this and start breaking it down into tasks/chunks?
12 years, 2 months
