9:10 a.m.
Maybe it's time to revive this one since we have started to go down the rabbit hole
with indexing only when dirty searched properties are present.
Begin forwarded message:
From: "lantz moore (JIRA)" <noreply(a)atlassian.com>
Date: 14 janvier 2011 01:52:05 HNEC
To: emmanuel(a)hibernate.org
Subject: [Hibernate-JIRA] Commented: (HSEARCH-471) Ability to selectively index an entity
based on its state
Issue (View Online)
Key: HSEARCH-471
Issue Type: New Feature
Status: Open
Priority: Minor
Assignee: Unassigned
Reporter: Dobes Vandermeer
Operations
View all
View comments
View history
Ability to selectively index an entity based on its state
Updated: 13/Jan/11 6:51 PM Created: 16/Mar/10 12:10 PM
The following comment has been added to this issue: [ Permalink ]
Author: lantz moore
Date: 13/Jan/11 6:51 PM
Comment:
up until some recent restructuring of FullTextIndexEventListener and
EventSourceTransactionContext i was able to implement something like this very easily.
first, my use case: we have a few objects that are expensive to index but are modified
"a lot". we only want them indexed at a few well defined transitions: initial
creation and when certain members are modified. otherwise, we don't need them to be
reindexed.
to solve this issue with earlier versions of hib-search, we extended
FullTextIndexEventListener and overrode processWork. we map classes to MVEL expressions
and use the MVEL expressions to evaluate whether or not to index the entity.
we define the FullTextIndexEventListener in spring and configure it like:
<bean id="fullTextIndexer"
class="com.example.dao.hibernate.FullTextIndexEventListener">
<property name="entityChecks">
<map>
<entry key="com.example.model.RetailTransaction">
<value>
if (event is org.hibernate.event.PostUpdateEvent) { return (oldState.suspended !=
newState.suspended || entity.getTransactionState().name() != "PENDING") }
return true;
</value>
</entry>
</map>
</property>
</bean>
then instead of autoregistering the default listeners, we register this new listener with
the appropriate events. for our particular use case this helped us out quite a bit.
however, it looks as though we aren't going to be able to do this in the near future
because of the redesign of EventSourceTransactionContext and FullTextIndexEventListener.
EventSourceTransactionContext.getIndexWorkFlushEventListener looks thru the listeners for
the FullTextIndexEventListener like so:
for (FlushEventListener listener : flushEventListeners) {
if ( listener.getClass().equals( FullTextIndexEventListener.class ) ) { return
(FullTextIndexEventListener) listener; }
}
which means our subclass of FTIEL isn't be found. plus, it looks as though FTIEL will
soon be final.
so, i don't know if we've been "abusing" the system here, or what, but
this seemed like a fairly straight forward way of implementing conditional indexing and
has worked really well for us for quite some time.
i'm not suggesting that hib-search make a dependency on MVEL, etc; however, it would
be nice if we were able to continue to subclass FullTextIndexEventListener in the way that
we have been.
Project: Hibernate Search
Components: mapping
Affects Versions: 3.1.1.GA
Fix Versions: 3.4.0
Description
In our system we have entities that are searched but not all of them are available for
search - some of them are flagged as "removed". It would improve the efficiency
of our search subsystem if we could implement a kind of "filter" that blocked
these entities from being added to the search index, since we wouldn't have to make
that a search term and our indexes would be somewhat smaller.
This message was automatically generated by Atlassian JIRA Enterprise Edition, Version:
4.1-519 - Bug/feature request.
If you think it was sent incorrectly, contact one of this server's administrators.
11:33 a.m.
Agreed, also there where additional similar requests recently via the
forums. As usual, alternatives where found but we should make this
easier.
So these are all alternatives I've been considering, not wanting to
implement necessarily all of them:
* SearchFactory.enableIndexing(boolean) // enables/disables globally on the fly
could operate on existing boolean
org.hibernate.search.event.FullTextIndexEventListener.used
* FullTextSession.enableIndexing(boolean)
toggles indexing for the currently open session
* entity controlled, via an entity-level annotation pointing to some
custom implementation respecting interface:
boolean indexingEnabled(T entity)
This one is more tricky, should we evaluate it twice (before and after
changes) to see if an index delete operation should trigger?
As an alternative, could it have sense to provide a more complex
interface which controls add/update/delete triggers prevention.
* entity controlled, via an annotated boolean property
I'd mandate a boolean property, or something which can be easily
evaluated via a SQL fragment / Criteria / filterable, so that the same
information could be reused
by the MassIndexer when picking all values to be indexed.
others, improvements?
Cheers,
Sanne
2011/1/14 Emmanuel Bernard <emmanuel(a)hibernate.org>:
Maybe it's time to revive this one since we have started to go
down the rabbit hole with indexing only when dirty searched properties are present.
Begin forwarded message:
> From: "lantz moore (JIRA)" <noreply(a)atlassian.com>
> Date: 14 janvier 2011 01:52:05 HNEC
> To: emmanuel(a)hibernate.org
> Subject: [Hibernate-JIRA] Commented: (HSEARCH-471) Ability to selectively index an
entity based on its state
>
>
> Issue (View Online)
> Key: HSEARCH-471
> Issue Type: New Feature
> Status: Open
> Priority: Minor
> Assignee: Unassigned
> Reporter: Dobes Vandermeer
>
> Operations
> View all
> View comments
> View history
> Ability to selectively index an entity based on its state
> Updated: 13/Jan/11 6:51 PM Created: 16/Mar/10 12:10 PM
>
> The following comment has been added to this issue: [ Permalink ]
> Author: lantz moore
> Date: 13/Jan/11 6:51 PM
> Comment:
> up until some recent restructuring of FullTextIndexEventListener and
EventSourceTransactionContext i was able to implement something like this very easily.
>
> first, my use case: we have a few objects that are expensive to index but are
modified "a lot". we only want them indexed at a few well defined transitions:
initial creation and when certain members are modified. otherwise, we don't need them
to be reindexed.
>
> to solve this issue with earlier versions of hib-search, we extended
FullTextIndexEventListener and overrode processWork. we map classes to MVEL expressions
and use the MVEL expressions to evaluate whether or not to index the entity.
>
> we define the FullTextIndexEventListener in spring and configure it like:
>
> <bean id="fullTextIndexer"
class="com.example.dao.hibernate.FullTextIndexEventListener">
> <property name="entityChecks">
> <map>
> <entry key="com.example.model.RetailTransaction">
> <value>
> if (event is org.hibernate.event.PostUpdateEvent) { return (oldState.suspended !=
newState.suspended || entity.getTransactionState().name() != "PENDING") }
> return true;
> </value>
> </entry>
> </map>
> </property>
> </bean>
>
> then instead of autoregistering the default listeners, we register this new listener
with the appropriate events. for our particular use case this helped us out quite a bit.
>
> however, it looks as though we aren't going to be able to do this in the near
future because of the redesign of EventSourceTransactionContext and
FullTextIndexEventListener.
>
> EventSourceTransactionContext.getIndexWorkFlushEventListener looks thru the listeners
for the FullTextIndexEventListener like so:
>
> for (FlushEventListener listener : flushEventListeners) {
> if ( listener.getClass().equals( FullTextIndexEventListener.class ) ) { return
(FullTextIndexEventListener) listener; }
> }
>
> which means our subclass of FTIEL isn't be found. plus, it looks as though FTIEL
will soon be final.
>
> so, i don't know if we've been "abusing" the system here, or what,
but this seemed like a fairly straight forward way of implementing conditional indexing
and has worked really well for us for quite some time.
>
> i'm not suggesting that hib-search make a dependency on MVEL, etc; however, it
would be nice if we were able to continue to subclass FullTextIndexEventListener in the
way that we have been.
>
>
> Project: Hibernate Search
> Components: mapping
> Affects Versions: 3.1.1.GA
> Fix Versions: 3.4.0
>
> Description
> In our system we have entities that are searched but not all of them are available
for search - some of them are flagged as "removed". It would improve the
efficiency of our search subsystem if we could implement a kind of "filter" that
blocked these entities from being added to the search index, since we wouldn't have to
make that a search term and our indexes would be somewhat smaller.
>
>
> This message was automatically generated by Atlassian JIRA Enterprise Edition,
Version: 4.1-519 - Bug/feature request.
> If you think it was sent incorrectly, contact one of this server's
administrators.
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
1:47 p.m.
On Fri, 14 Jan 2011 11:33:25 +0100, Sanne Grinovero
<sanne.grinovero(a)gmail.com> wrote:
Agreed, also there where additional similar requests recently via
the
forums. As usual, alternatives where found but we should make this
easier.
* SearchFactory.enableIndexing(boolean) // enables/disables globally
on
the fly
could operate on existing boolean
org.hibernate.search.event.FullTextIndexEventListener.used
+1 This is something I was wishing for some time back. It should be easy
to implement as well. This option should then also be exposed via JMX.
* FullTextSession.enableIndexing(boolean)
toggles indexing for the currently open session
I don't find this very useful. Could be skipped imo
* entity controlled, via an entity-level annotation pointing to some
custom implementation respecting interface:
boolean indexingEnabled(T entity)
This one is more tricky, should we evaluate it twice (before and after
changes) to see if an index delete operation should trigger?
As an alternative, could it have sense to provide a more complex
interface which controls add/update/delete triggers prevention.
That would really be the solution for the HSEARCH-471 usecase, hence +1
Of course we would have to settle the details.
* entity controlled, via an annotated boolean property
I'd mandate a boolean property, or something which can be easily
evaluated via a SQL fragment / Criteria / filterable, so that the same
information could be reused
by the MassIndexer when picking all values to be indexed.
Not sure I understand exactly what you mean.
--Hardy
2:24 p.m.
On 14 janv. 2011, at 13:47, Hardy Ferentschik wrote:
On Fri, 14 Jan 2011 11:33:25 +0100, Sanne Grinovero
<sanne.grinovero(a)gmail.com> wrote:
> Agreed, also there where additional similar requests recently via the
> forums. As usual, alternatives where found but we should make this
> easier.
> * SearchFactory.enableIndexing(boolean) // enables/disables globally on
> the fly
> could operate on existing boolean
> org.hibernate.search.event.FullTextIndexEventListener.used
+1 This is something I was wishing for some time back. It should be easy
to implement as well. This option should then also be exposed via JMX.
What's the use case behind it?
> * FullTextSession.enableIndexing(boolean)
> toggles indexing for the currently open session
I don't find this very useful. Could be skipped imo
What's the use case behind it?
> * entity controlled, via an entity-level annotation pointing to some
> custom implementation respecting interface:
> boolean indexingEnabled(T entity)
> This one is more tricky, should we evaluate it twice (before and after
> changes) to see if an index delete operation should trigger?
> As an alternative, could it have sense to provide a more complex
> interface which controls add/update/delete triggers prevention.
That would really be the solution for the HSEARCH-471 usecase, hence +1
Of course we would have to settle the details.
It seems to me in the use cases we have seen so far was to minimize the numbers of
indexing as an optimization. In this case, delete operations should always be applied.
It's just Update that is impacted (maybe insert).
> * entity controlled, via an annotated boolean property
> I'd mandate a boolean property, or something which can be easily
> evaluated via a SQL fragment / Criteria / filterable, so that the same
> information could be reused
> by the MassIndexer when picking all values to be indexed.
Not sure I understand exactly what you mean.
@Entity @Indexed
public class MyEntity {
...
@SkipIndexing
boolean isIndexed() {
return status == TEMP;
}
}
--Hardy
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
3:06 p.m.
2011/1/14 Emmanuel Bernard <emmanuel(a)hibernate.org>:
On 14 janv. 2011, at 13:47, Hardy Ferentschik wrote:
> On Fri, 14 Jan 2011 11:33:25 +0100, Sanne Grinovero
> <sanne.grinovero(a)gmail.com> wrote:
>
>> Agreed, also there where additional similar requests recently via the
>> forums. As usual, alternatives where found but we should make this
>> easier.
>
>> * SearchFactory.enableIndexing(boolean) // enables/disables globally on
>> the fly
>> could operate on existing boolean
>> org.hibernate.search.event.FullTextIndexEventListener.used
>
> +1 This is something I was wishing for some time back. It should be easy
> to implement as well. This option should then also be exposed via JMX.
What's the use case behind it?
Could be useful for emergency situations, like filesystem needs to be
set in readonly mode and people don't care for index sync temporarily.
I'm fine to skip/postpone this, people have been asking for it but
AFAIR always to workaround some other crazy scenario.
>> * FullTextSession.enableIndexing(boolean)
>> toggles indexing for the currently open session
>
> I don't find this very useful. Could be skipped imo
What's the use case behind it?
This approach is the most close to what users have been asking for,
just disabling it with a single method call.
Is not as polished and consistent as the next two, but I still see
this ease of use compelling.
>> * entity controlled, via an entity-level annotation pointing
to some
>> custom implementation respecting interface:
>> boolean indexingEnabled(T entity)
>> This one is more tricky, should we evaluate it twice (before and after
>> changes) to see if an index delete operation should trigger?
>> As an alternative, could it have sense to provide a more complex
>> interface which controls add/update/delete triggers prevention.
>
> That would really be the solution for the HSEARCH-471 usecase, hence +1
> Of course we would have to settle the details.
It seems to me in the use cases we have seen so far was to minimize the numbers of
indexing as an optimization. In this case, delete operations should always be applied.
It's just Update that is impacted (maybe insert).
First, it depends if you want to minimize the documents in the index,
or if you want to minimize changes to the index (including deletions).
Second, people have been asking to "hide" some information for
security reasons; they have been suggested to use a filter, but nobody
is ever going to look at all those documents: a waste of space; for
this security requirement we can't avoid or postpone delete
operations.
>> * entity controlled, via an annotated boolean property
>> I'd mandate a boolean property, or something which can be easily
>> evaluated via a SQL fragment / Criteria / filterable, so that the same
>> information could be reused
>> by the MassIndexer when picking all values to be indexed.
>
> Not sure I understand exactly what you mean.
@Entity @Indexed
public class MyEntity {
...
@SkipIndexing
boolean isIndexed() {
return status == TEMP;
}
}
exactly! simple and reusable by the mass indexer.
Am I missing other approaches? we've been collecting quite some
feedback on these.
Sanne
>
> --Hardy
10:43 p.m.
On Fri, 14 Jan 2011 14:24:07 +0100, Emmanuel Bernard
<emmanuel(a)hibernate.org> wrote:
>> * SearchFactory.enableIndexing(boolean) // enables/disables
globally on
>> the fly
>> could operate on existing boolean
>> org.hibernate.search.event.FullTextIndexEventListener.used
>
> +1 This is something I was wishing for some time back. It should be easy
> to implement as well. This option should then also be exposed via JMX.
What's the use case behind it?
I think it would allow for application which are completely manageable via
some user interface. At the moment you set the auto indexing option or not.
If I could programmatically change it I could do things like stop
auto indexing, do db modifications, purge the index, re-index, enable
event indexing.
>> * entity controlled, via an annotated boolean property
>> I'd mandate a boolean property, or something which can be easily
>> evaluated via a SQL fragment / Criteria / filterable, so that the same
>> information could be reused
>> by the MassIndexer when picking all values to be indexed.
>
> Not sure I understand exactly what you mean.
@Entity @Indexed
public class MyEntity {
...
@SkipIndexing
boolean isIndexed() {
return status == TEMP;
}
}
Wouldn't the other solution be more generic?
--Hardy
11:59 p.m.
2011/1/14 Hardy Ferentschik <hibernate(a)ferentschik.de>:
On Fri, 14 Jan 2011 14:24:07 +0100, Emmanuel Bernard
<emmanuel(a)hibernate.org> wrote:
>>> * SearchFactory.enableIndexing(boolean) // enables/disables globally on
>>> the fly
>>> could operate on existing boolean
>>> org.hibernate.search.event.FullTextIndexEventListener.used
>>
>> +1 This is something I was wishing for some time back. It should be easy
>> to implement as well. This option should then also be exposed via JMX.
>
> What's the use case behind it?
I think it would allow for application which are completely manageable via
some user interface. At the moment you set the auto indexing option or not.
If I could programmatically change it I could do things like stop
auto indexing, do db modifications, purge the index, re-index, enable
event indexing.
>>> * entity controlled, via an annotated boolean property
>>> I'd mandate a boolean property, or something which can be easily
>>> evaluated via a SQL fragment / Criteria / filterable, so that the same
>>> information could be reused
>>> by the MassIndexer when picking all values to be indexed.
>>
>> Not sure I understand exactly what you mean.
>
> @Entity @Indexed
> public class MyEntity {
> ...
>
> @SkipIndexing
> boolean isIndexed() {
> return status == TEMP;
> }
> }
Wouldn't the other solution be more generic?
yes, but the more flexibility we give, the less clever things we can
do automatically.
Consider the dirty checks we implemented for HSEARCH-361, I had to
disable every optimization
both in case there's a ClassBridge or a BoostStrategy as in those
cases we can't predict
if the index would change.
Similarly, in this case if we allow for a flexible custom made
implementation we should
disable dirty checks on such entities and would have no clue to
provide to the MassIndexer,
at least not without more help from the user;
we'd need to consider at least those cases, something along:
void addRestrictions(Criteria loadingCriteria);
Set<String> getConsideredPropertyNames();
Who knows what else we'll need later.
I'd say @SkipIndexing is more straight-forward, but we can also
support the more flexible approach
as an "advanced" alternative which requires some more coding and care.
Sanne
6:16 p.m.
Reviving again;
I think we should split the solution among two main use cases:
1)entities which should stay out of the index for security/filtering reasons
In this case we need to accomodate on-commit deletion of entities which where
indexed, and are no more. We need to invoke the check for "needs
indexing" on the
entity as it was during loading and again at commit to see if changes
are needed.
2)entities people would like to skip index operations because they
don't care about consistency
of this specific index, but want to improve indexing performance.
In this case "@SkipIndexing" is a more suitable name, and if the check
returns "we have to delete this",
we might actually skip the delete operation.
I don't like option 2), it's not clear what to do for example when the
entity is updated and the
@SkipIndexing-annotated-method returns true, while for example it was
false before the update -> a mess.
Also it seems likely that even while performing some additional
deletes, it still is a performance improvement.
So focusing on 1) only ATM, are we able at flush/commit time to invoke
the annotated method twice, using once the original entity, in the
state it was at transaction begin?
Or shall we mandate an additional persistent attribute, so we can
basically compare it as we do with dirty checking?
Regards,
Sanne
2011/1/14 Sanne Grinovero <sanne.grinovero(a)gmail.com>:
2011/1/14 Hardy Ferentschik <hibernate(a)ferentschik.de>:
> On Fri, 14 Jan 2011 14:24:07 +0100, Emmanuel Bernard
> <emmanuel(a)hibernate.org> wrote:
>
>>>> * SearchFactory.enableIndexing(boolean) // enables/disables globally on
>>>> the fly
>>>> could operate on existing boolean
>>>> org.hibernate.search.event.FullTextIndexEventListener.used
>>>
>>> +1 This is something I was wishing for some time back. It should be easy
>>> to implement as well. This option should then also be exposed via JMX.
>>
>> What's the use case behind it?
>
> I think it would allow for application which are completely manageable via
> some user interface. At the moment you set the auto indexing option or not.
> If I could programmatically change it I could do things like stop
> auto indexing, do db modifications, purge the index, re-index, enable
> event indexing.
>
>>>> * entity controlled, via an annotated boolean property
>>>> I'd mandate a boolean property, or something which can be easily
>>>> evaluated via a SQL fragment / Criteria / filterable, so that the same
>>>> information could be reused
>>>> by the MassIndexer when picking all values to be indexed.
>>>
>>> Not sure I understand exactly what you mean.
>>
>> @Entity @Indexed
>> public class MyEntity {
>> ...
>>
>> @SkipIndexing
>> boolean isIndexed() {
>> return status == TEMP;
>> }
>> }
>
> Wouldn't the other solution be more generic?
yes, but the more flexibility we give, the less clever things we can
do automatically.
Consider the dirty checks we implemented for HSEARCH-361, I had to
disable every optimization
both in case there's a ClassBridge or a BoostStrategy as in those
cases we can't predict
if the index would change.
Similarly, in this case if we allow for a flexible custom made
implementation we should
disable dirty checks on such entities and would have no clue to
provide to the MassIndexer,
at least not without more help from the user;
we'd need to consider at least those cases, something along:
void addRestrictions(Criteria loadingCriteria);
Set<String> getConsideredPropertyNames();
Who knows what else we'll need later.
I'd say @SkipIndexing is more straight-forward, but we can also
support the more flexible approach
as an "advanced" alternative which requires some more coding and care.
Sanne
4826
days inactive
4844
days old
7 comments
4 participants
participants (4)
-
Emmanuel Bernard -
Hardy Ferentschik -
Sanne Grinovero -
Sanne Grinovero