Detached Query causes stale query cache entries
by Lei Gu
Hibernate version:
3.2.6
We ran into an issue where detached query causes stale query cache
entries. Our example is to retrieve a list of our favorite dogs. We use
a separate table called favorite_animal to store our favorite animals,
including dogs.
For example, our main query is the following,
from Dog
And we use a detached query to select our favorite Dog,
select fa.animal_id from favoriate_animal fa where fa.user_id = :user_id
The combined SQL looks like the following,
from Dog dog where dog.id in (select fa.animal_id from favoriate_animal
fa where fa.user_id = :user_id)
There is our problem. We execute the above query once and get a list of
dogs back. We then remove one dog from our favorite list and then run
the above query again. We are expected to see one less dog in our result
but we got the same list as before!
It turns out that the query cache entry was not invalidated when we
removed one dog from the favorite list. I dug a little and it seems like
the query spaces used to look up invalidated entity names are only built
from associated criteria, but not from detached query or detached query
based criterion.
In our example, the query spaces will only contain [DOG] entity.
I had made some changes to CriteriaQueryTranslator to include additional
query spaces from detached queries in getQuerySpaces method in
CriteriaQueryTranslator but want to make sure that I am on the right
track.
After my fix, the query spaces will contain [DOG,FAVORITE_ANIMAL].
I am attaching a patch as well.
Thanks for your feedback and comments.
-- Lei
16 years, 4 months
HSearch: Using sharding and avoiding query on multiple shards
by Emmanuel Bernard
Today, in Hibernate Search, a query is applied on all shards. We use a
MultiReader to wrap them together.
In some sharding scenario, it makes sense to apply the query on a
single shard or a subset of the shards.
We could add the following API to IndexShardingStrategy
public DirectoryProvider<?>[]
getDirectoryProvidersForQuery(o.a.l.search.Query query);
The query could be analyzed by the sharding strategy to detect boolean
queries on their sharding criteria
//query building
BooleanQuery bQuery = new BooleanQuery();
bQuery.add(regularQuery, Occur.MUST);
bQuery.add( new TermQuery( new Term("distributor.id", "2"),
Occur.MUST ); //only occurs in shard 1
public DirectoryProvider<?>[]
getDirectoryProvidersForQuery(o.a.l.search.Query query) {
if (query instanceof BooleanQuery) {
List<BooleanClause> clauses =
BooleanQuery.class.cast(query).clauses
}
int restrictedShard;
boolean isAllMust = true;
for (BooleanClause clause : clauses) {
if (clause.getOccur() != Occur.MUST) { isAllMust = false; break; }
if ( clause.getQuery() instanceof TermQuery ) {
Term term = TermQuery.class.cast( clause.getQuery() ).getTerm();
if (term.field().equals("distributor.id")) { restrictedShard =
Integer.parse(term.text(); }
}
}
if (isAllMust && restrictedShard != null) return new Provider[]
{ providers[restrictedShard-1] };
else return providers;
}
That's very flexibile but quite hard to implement correctly especially
since the query tree structure might not be trivial
The alternative strategy is to have the following API on
IndexShardingStrategy
public DirectoryProvider<?>[] getDirectoryProvidersForQuery(Object
hint);
and a corresponding fullTextQuery.setShardHint(Object);
A query could "know it targets shard 2 and pass the information to the
strategy through a standard language:
fullTextQuery.setShardHint("Sony");
public DirectoryProvider<?>[] getDirectoryProvidersForQuery(Object
hint) {
if (String.class.isInstance(hint) &&
String.class.cast(hint).equals("Sony")) {
return new Provider[] { providers[2] }
}
else {
return providers;
}
}
WDYT? How useful would that be?
--
Emmanuel Bernard
http://in.relation.to/Bloggers/Emmanuel | http://blog.emmanuelbernard.com
| http://twitter.com/emmanuelbernard
Hibernate Search in Action (http://is.gd/Dl1)
16 years, 4 months
RE: HSearch: Using sharding and avoiding query on multiple shards
by John Griffin
Performance is ALWAYS an issue.
But am I correct in assuming that the performance increase would be
approximately inversely proportional to the number of shards queried?
My point is that the amount of work required to implement this should be
worth what we think the expected gain will be.
John G.
http://thediningphilosopher.blogspot.com
-----Original Message-----
From: hibernate-dev-bounces(a)lists.jboss.org
[mailto:hibernate-dev-bounces@lists.jboss.org] On Behalf Of
hibernate-dev-request(a)lists.jboss.org
Sent: Thursday, July 31, 2008 10:00 AM
To: hibernate-dev(a)lists.jboss.org
Subject: hibernate-dev Digest, Vol 25, Issue 28
Send hibernate-dev mailing list submissions to
hibernate-dev(a)lists.jboss.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.jboss.org/mailman/listinfo/hibernate-dev
or, via email, send a message with subject or body 'help' to
hibernate-dev-request(a)lists.jboss.org
You can reach the person managing the list at
hibernate-dev-owner(a)lists.jboss.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of hibernate-dev digest..."
Today's Topics:
1. HSearch: Using sharding and avoiding query on multiple shards
(Emmanuel Bernard)
2. Re: HSearch: Using sharding and avoiding query on multiple
shards (Sanne Grinovero)
----------------------------------------------------------------------
Message: 1
Date: Wed, 30 Jul 2008 20:36:13 -0400
From: Emmanuel Bernard <emmanuel(a)hibernate.org>
Subject: [hibernate-dev] HSearch: Using sharding and avoiding query on
multiple shards
To: hibernate-dev(a)lists.jboss.org, Aaron Walker
<a.walker(a)base2services.com>
Message-ID: <165DCC1F-BE42-49BA-8537-C8C41F834677(a)hibernate.org>
Content-Type: text/plain; charset="us-ascii"
Today, in Hibernate Search, a query is applied on all shards. We use a
MultiReader to wrap them together.
In some sharding scenario, it makes sense to apply the query on a
single shard or a subset of the shards.
We could add the following API to IndexShardingStrategy
public DirectoryProvider<?>[]
getDirectoryProvidersForQuery(o.a.l.search.Query query);
The query could be analyzed by the sharding strategy to detect boolean
queries on their sharding criteria
//query building
BooleanQuery bQuery = new BooleanQuery();
bQuery.add(regularQuery, Occur.MUST);
bQuery.add( new TermQuery( new Term("distributor.id", "2"),
Occur.MUST ); //only occurs in shard 1
public DirectoryProvider<?>[]
getDirectoryProvidersForQuery(o.a.l.search.Query query) {
if (query instanceof BooleanQuery) {
List<BooleanClause> clauses =
BooleanQuery.class.cast(query).clauses
}
int restrictedShard;
boolean isAllMust = true;
for (BooleanClause clause : clauses) {
if (clause.getOccur() != Occur.MUST) { isAllMust = false; break; }
if ( clause.getQuery() instanceof TermQuery ) {
Term term = TermQuery.class.cast( clause.getQuery() ).getTerm();
if (term.field().equals("distributor.id")) { restrictedShard =
Integer.parse(term.text(); }
}
}
if (isAllMust && restrictedShard != null) return new Provider[]
{ providers[restrictedShard-1] };
else return providers;
}
That's very flexibile but quite hard to implement correctly especially
since the query tree structure might not be trivial
The alternative strategy is to have the following API on
IndexShardingStrategy
public DirectoryProvider<?>[] getDirectoryProvidersForQuery(Object
hint);
and a corresponding fullTextQuery.setShardHint(Object);
A query could "know it targets shard 2 and pass the information to the
strategy through a standard language:
fullTextQuery.setShardHint("Sony");
public DirectoryProvider<?>[] getDirectoryProvidersForQuery(Object
hint) {
if (String.class.isInstance(hint) &&
String.class.cast(hint).equals("Sony")) {
return new Provider[] { providers[2] }
}
else {
return providers;
}
}
WDYT? How useful would that be?
--
Emmanuel Bernard
http://in.relation.to/Bloggers/Emmanuel | http://blog.emmanuelbernard.com
| http://twitter.com/emmanuelbernard
Hibernate Search in Action (http://is.gd/Dl1)
16 years, 5 months
