8:06 a.m.
Hi all,
to solve HSEARCH-284 (Lucene locks left around in application restart/crash)
I am investigating the differences in all Lucene's LockFactory implementations;
some of them are interesting and I would like to add a configuration option to
replace the default one in Hibernate Search.
1) property name:
I'm naming the property key "locking_strategy", scoped for each
DirectoryProvider.
2) values:
At first I was planning to use the usual design to be able to specify
any implementation
by specifying the qualified class name, but actually only 4
implementations make sense
and if someone really needed the flexibility to implement their own
they can define
their own DirectoryProvider and use whatever they want.
Also 2 of the base Lucene implementations are missing a public no-args
constructor,
so I would use only simple names:
simple -> org.apache.lucene.store.SimpleFSLockFactory
native -> org.apache.lucene.store.NativeFSLockFactory
single -> org.apache.lucene.store.SingleInstanceLockFactory
none -> org.apache.lucene.store.NoLockFactory
3) support for "none":
I don't think we should let the user select something which is not compatible
with H.Searche's design, so I am still trying to figure out if the NoLockFactory
could be an option in case Search is the only process accessing the index.
Currently inside search the IndexReaders/Searchers are read-only, and at most
one IndexWriter is using the DP, so the locking strategy could be set
to "none" IMHO.
opinions?
4) default:
The current default is "simple", which has the problem as reported in
HSEARCH-284,
so I was puzzled by the idea to define the "native" as the default one to have
Search beginners avoid the locking problem, but I am hitting some
problems under load testing,
so I think the default should stay to "simple" at least for now.
Sanne
3:01 a.m.
On Mon, 16 Feb 2009 15:06:27 +0100, Sanne Grinovero
<sanne.grinovero(a)gmail.com> wrote:
1) property name:
I'm naming the property key "locking_strategy", scoped for each
DirectoryProvider.
Is it really useful to be able to configure it per DirectoryProvider? How
useful is it
to mix the locking strategies between providers?
I understand that from a consistency point of view it is the way to go. On
the other hand
by providing all these options you bet that people will start using
"stupid" combinations.
2) values:
At first I was planning to use the usual design to be able to specify
any implementation
by specifying the qualified class name, but actually only 4
implementations make sense
and if someone really needed the flexibility to implement their own
they can define
their own DirectoryProvider and use whatever they want.
Also 2 of the base Lucene implementations are missing a public no-args
constructor,
so I would use only simple names:
simple -> org.apache.lucene.store.SimpleFSLockFactory
native -> org.apache.lucene.store.NativeFSLockFactory
single -> org.apache.lucene.store.SingleInstanceLockFactory
none -> org.apache.lucene.store.NoLockFactory
What's about a approach similar to "Reader strategy configuration", where
shared and not-shared are
basically alias names for the underlying class names. This gives you names
for the
most common lock factories and you can plug your own strategy.
And of course there is the consistency argument ;-)
3) support for "none":
I don't think we should let the user select something which is not
compatible
with H.Searche's design, so I am still trying to figure out if the
NoLockFactory
could be an option in case Search is the only process accessing the
index.
Currently inside search the IndexReaders/Searchers are read-only, and at
most
one IndexWriter is using the DP, so the locking strategy could be set
to "none" IMHO.
opinions?
If there is only one single Hibernate Search process running on the
machine AND
no other process using Lucene natively it should be ok.
4) default:
Search beginners avoid the locking problem, but I am hitting some
problems under load testing,
so I think the default should stay to "simple" at least for now.
What are the problems?
--Hardy
3:23 a.m.
2009/2/17 Hardy Ferentschik <hibernate(a)ferentschik.de>:
On Mon, 16 Feb 2009 15:06:27 +0100, Sanne Grinovero
<sanne.grinovero(a)gmail.com> wrote:
> 1) property name:
> I'm naming the property key "locking_strategy", scoped for each
> DirectoryProvider.
Is it really useful to be able to configure it per DirectoryProvider? How
useful is it
to mix the locking strategies between providers?
I understand that from a consistency point of view it is the way to go. On
the other hand
by providing all these options you bet that people will start using "stupid"
combinations.
I'm not really worried about "stupid" combinations as different DPs
are indipendent;
somebody could use a RAM-DP for something and a FS-DP for something else,
and maybe have one favourite on a NFS share.. for each one there are
more suitable locking strategies.
If you wanted to use only one strategy, then no code change is required
as Lucene is reading a system property to define the implementation
as you don't define one (and Search is not defining, so you could experiment
with it without code changes); so this feature is really to provide
the flexibility
to have different ones per DP (and make is easy).
> 2) values:
> At first I was planning to use the usual design to be able to specify
> any implementation
> by specifying the qualified class name, but actually only 4
> implementations make sense
> and if someone really needed the flexibility to implement their own
> they can define
> their own DirectoryProvider and use whatever they want.
> Also 2 of the base Lucene implementations are missing a public no-args
> constructor,
> so I would use only simple names:
>
> simple -> org.apache.lucene.store.SimpleFSLockFactory
> native -> org.apache.lucene.store.NativeFSLockFactory
> single -> org.apache.lucene.store.SingleInstanceLockFactory
> none -> org.apache.lucene.store.NoLockFactory
What's about a approach similar to "Reader strategy configuration", where
shared and not-shared are
basically alias names for the underlying class names. This gives you names
for the
most common lock factories and you can plug your own strategy.
And of course there is the consistency argument ;-)
Yes that's what I am proposing, the four names above is my proposal of
" simple name mapping to factories", to the implementations listed.
> 3) support for "none":
> I don't think we should let the user select something which is not
> compatible
> with H.Searche's design, so I am still trying to figure out if the
> NoLockFactory
> could be an option in case Search is the only process accessing the index.
> Currently inside search the IndexReaders/Searchers are read-only, and at
> most
> one IndexWriter is using the DP, so the locking strategy could be set
> to "none" IMHO.
> opinions?
If there is only one single Hibernate Search process running on the machine
AND
no other process using Lucene natively it should be ok.
> 4) default:
> Search beginners avoid the locking problem, but I am hitting some
> problems under load testing,
> so I think the default should stay to "simple" at least for now.
What are the problems?
basically it looks like (IMHO) a bug in Lucene's implementation: it is
trying to acquire
the lock, if it's locked it tries again - only once - after a second,
and then it raises
a timeout exception if it fails again; even if the lock has been
available most of the time between the two checks.
I'll ask on lucene's forums if we could change that; anyway the
current 2.4 implementation is
not my favourite candidate for our "default" implementation;
I'm still investigating if my stress test was actually a situation
that's not occurring normally, we should have only
one thread (the writer one) accessing the index, but only in the "one
search instance per index" scenario.
--Hardy
Sanne
3:50 a.m.
On Tue, 17 Feb 2009 10:23:28 +0100, Sanne Grinovero
<sanne.grinovero(a)gmail.com> wrote:
I'm not really worried about "stupid" combinations as
different DPs
are indipendent;
somebody could use a RAM-DP for something and a FS-DP for something else,
and maybe have one favourite on a NFS share.. for each one there are
more suitable locking strategies.
If you wanted to use only one strategy, then no code change is required
as Lucene is reading a system property to define the implementation
as you don't define one (and Search is not defining, so you could
experiment
with it without code changes); so this feature is really to provide
the flexibility
to have different ones per DP (and make is easy).
Fair enough.
basically it looks like (IMHO) a bug in Lucene's implementation:
it is
trying to acquire
the lock, if it's locked it tries again - only once - after a second,
and then it raises
a timeout exception if it fails again; even if the lock has been
available most of the time between the two checks.
I'll ask on lucene's forums if we could change that; anyway the
current 2.4 implementation is
not my favourite candidate for our "default" implementation;
I'm still investigating if my stress test was actually a situation
that's not occurring normally, we should have only
one thread (the writer one) accessing the index, but only in the "one
In this case we should definitely stay with 'simple' for now.
--Hardy
4:21 a.m.
On Feb 16, 2009, at 15:06, Sanne Grinovero wrote:
Hi all,
to solve HSEARCH-284 (Lucene locks left around in application
restart/crash)
I am investigating the differences in all Lucene's LockFactory
implementations;
some of them are interesting and I would like to add a configuration
option to
replace the default one in Hibernate Search.
1) property name:
I'm naming the property key "locking_strategy", scoped for each
DirectoryProvider.
sounds good
2) values:
At first I was planning to use the usual design to be able to specify
any implementation
by specifying the qualified class name, but actually only 4
implementations make sense
and if someone really needed the flexibility to implement their own
they can define
their own DirectoryProvider and use whatever they want.
hum, does that makes our codebase more complex to support custom
LockFactories? I would like this flexibility/
Also 2 of the base Lucene implementations are missing a public no-args
constructor,
so I would use only simple names:
simple -> org.apache.lucene.store.SimpleFSLockFactory
native -> org.apache.lucene.store.NativeFSLockFactory
single -> org.apache.lucene.store.SingleInstanceLockFactory
none -> org.apache.lucene.store.NoLockFactory
yes, simple names for those that make sense and fqcn for the others.
We might also support a "factory" so that impl wo no-arg constructor
can be used, but that's an enhancement we can do later if needed.
3) support for "none":
I don't think we should let the user select something which is not
compatible
with H.Searche's design, so I am still trying to figure out if the
NoLockFactory
could be an option in case Search is the only process accessing the
index.
Currently inside search the IndexReaders/Searchers are read-only,
and at most
one IndexWriter is using the DP, so the locking strategy could be set
to "none" IMHO.
opinions?
that sounds reasonable but how does that differ from
SingleInstanceLockFactory?
4) default:
The current default is "simple", which has the problem as reported in
HSEARCH-284,
so I was puzzled by the idea to define the "native" as the default
one to have
Search beginners avoid the locking problem, but I am hitting some
problems under load testing,
so I think the default should stay to "simple" at least for now.
OK
4:49 a.m.
The issue is closed already:
http://fisheye.labs.jboss.com/browse/Hibernate/search/trunk
I'll add support to create LockFactories by fqcn later, I didn't
consider that important
at first.
answers inline:
2009/2/18 Emmanuel Bernard <emmanuel(a)hibernate.org>:
On Feb 16, 2009, at 15:06, Sanne Grinovero wrote:
> Hi all,
> to solve HSEARCH-284 (Lucene locks left around in application
> restart/crash)
> I am investigating the differences in all Lucene's LockFactory
> implementations;
> some of them are interesting and I would like to add a configuration
> option to
> replace the default one in Hibernate Search.
>
> 1) property name:
> I'm naming the property key "locking_strategy", scoped for each
> DirectoryProvider.
sounds good
>
>
> 2) values:
> At first I was planning to use the usual design to be able to specify
> any implementation
> by specifying the qualified class name, but actually only 4
> implementations make sense
> and if someone really needed the flexibility to implement their own
> they can define
> their own DirectoryProvider and use whatever they want.
hum, does that makes our codebase more complex to support custom
LockFactories? I would like this flexibility/
>
> Also 2 of the base Lucene implementations are missing a public no-args
> constructor,
> so I would use only simple names:
>
> simple -> org.apache.lucene.store.SimpleFSLockFactory
> native -> org.apache.lucene.store.NativeFSLockFactory
> single -> org.apache.lucene.store.SingleInstanceLockFactory
> none -> org.apache.lucene.store.NoLockFactory
yes, simple names for those that make sense and fqcn for the others. We
might also support a "factory" so that impl wo no-arg constructor can be
used, but that's an enhancement we can do later if needed.
I avoided support for "fqcn" and factories to keep it simple;
I'll add support for both for consistency and because you seem to like them.
>
>
> 3) support for "none":
> I don't think we should let the user select something which is not
> compatible
> with H.Searche's design, so I am still trying to figure out if the
> NoLockFactory
> could be an option in case Search is the only process accessing the index.
> Currently inside search the IndexReaders/Searchers are read-only, and at
> most
> one IndexWriter is using the DP, so the locking strategy could be set
> to "none" IMHO.
> opinions?
that sounds reasonable but how does that differ from
SingleInstanceLockFactory?
the single instance locking has a java in memory lock, so to protect
yourself in case of using several indexwriter/readers on the same index
making changes when all running in the same JVM.
The "none" doesn't protect at all, so you have to make sure to never open
a second indexwriter or use any indexwriter when you are using an indexreader
to make changes.
Sounds a bit dangerous IMHO but I don't think we should stop people shooting
at their own feet.
As we are enforcing a single indexwriter and all our indexreaders are
readonly since
last release, you may want to use it: I think the current design of H.Search
doesn't stop us from offering this option.
I've added a note in the docs to use it only if you really understand it.
>
>
> 4) default:
> The current default is "simple", which has the problem as reported in
> HSEARCH-284,
> so I was puzzled by the idea to define the "native" as the default one to
> have
> Search beginners avoid the locking problem, but I am hitting some
> problems under load testing,
> so I think the default should stay to "simple" at least for now.
OK
8:19 a.m.
On Feb 18, 2009, at 11:49, Sanne Grinovero wrote:
>>
>> Also 2 of the base Lucene implementations are missing a public no-
>> args
>> constructor,
>> so I would use only simple names:
>>
>> simple -> org.apache.lucene.store.SimpleFSLockFactory
>> native -> org.apache.lucene.store.NativeFSLockFactory
>> single -> org.apache.lucene.store.SingleInstanceLockFactory
>> none -> org.apache.lucene.store.NoLockFactory
>
> yes, simple names for those that make sense and fqcn for the
> others. We
> might also support a "factory" so that impl wo no-arg constructor
> can be
> used, but that's an enhancement we can do later if needed.
I avoided support for "fqcn" and factories to keep it simple;
I'll add support for both for consistency and because you seem to
like them.
The Hibernate Search mantra is Keep It Simple, Keep it Extensible. So
there is an answer for everyone.
8:24 a.m.
On Feb 18, 2009, at 11:49, Sanne Grinovero wrote:
The "none" doesn't protect at all, so you have to make
sure to never
open
a second indexwriter or use any indexwriter when you are using an
indexreader
to make changes.
Sounds a bit dangerous IMHO but I don't think we should stop people
shooting
at their own feet.
As we are enforcing a single indexwriter and all our indexreaders are
readonly since
last release, you may want to use it: I think the current design of
H.Search
doesn't stop us from offering this option.
I've added a note in the docs to use it only if you really
understand it.
Well, if you are only using Hibernate Search to access indexes (90% of
our user base) and you deploy on one server (80% of our user base),
none is a compelling solution right? (that's 70% of deployments :))
5758
days inactive
5760
days old
7 comments
3 participants
participants (3)
-
Emmanuel Bernard -
Hardy Ferentschik -
Sanne Grinovero