7:44 p.m.
Hi guys,
I've been trying to get jbc-searchable working with JbossCache 3, but then I
noticed this thread for Infinispan. I am trying to build a proof-of-concept
for a distributed searchable cache, and obviously infinispan with hibernate
search would be an ideal solution. At what stage of development is this?
Is it possible to use the existing jbc-searchable code with infinispan?
Should I perhaps stick with jbc for now?
On a related note, I like the idea that each node builds and stores its own
indices for data it currently hosts. It would also be nice if these indices
could be further partitioned (in our case, we would like to do this per-day,
other situations maybe based on a modulous of the hash code) to reduce the
size of the indices. Replicated indices are ok if they dont change that
often, but ours will be updated almost constantly, and so you either need to
nominate one node to do the work, or have some distributed locking in place,
which would probably kill performance. Building an index locally also
provides protection against index corruption by having redundant copies
around the grid. We are intending to do deduplication and score
normalization ourselves.
I also tried using the jbosscache-lucene Directory implementation that Manik
wrote, but things started to die when lucene tried to merge segments. Has
there been any work to create a infinispan-backed lucene Directory?
Regards,
Ray Hilton
4:56 a.m.
New subject: Infinispan and search APIs
On 19 May 2009, at 01:44, Ray Hilton wrote:
Hi guys,
I've been trying to get jbc-searchable working with JbossCache 3,
but then I noticed this thread for Infinispan. I am trying to build
a proof-of-concept for a distributed searchable cache, and obviously
infinispan with hibernate search would be an ideal solution. At
what stage of development is this? Is it possible to use the
existing jbc-searchable code with infinispan? Should I perhaps
stick with jbc for now?
This isn't in Infinispan as yet. The plan is to add querying support,
but it won't make our next major release. Unless you feel like
contributing it. :-)
On a related note, I like the idea that each node builds and stores
its own indices for data it currently hosts. It would also be nice
if these indices could be further partitioned (in our case, we would
like to do this per-day, other situations maybe based on a modulous
of the hash code) to reduce the size of the indices. Replicated
indices are ok if they dont change that often, but ours will be
updated almost constantly, and so you either need to nominate one
node to do the work, or have some distributed locking in place,
which would probably kill performance. Building an index locally
also provides protection against index corruption by having
redundant copies around the grid. We are intending to do
deduplication and score normalization ourselves.
I also tried using the jbosscache-lucene Directory implementation
that Manik wrote, but things started to die when lucene tried to
merge segments. Has there been any work to create a infinispan-
backed lucene Directory?
There hasn't been any effort as yet, and with Vladimir's recent work
on the lock() API, I expect Infinispan to be a far better fit for a
Lucene directory than JBoss Cache is. So again, if you are interested
in contributing this, perhaps based on my JBoss Cache implementation,
it would be much appreciated.
Cheers
Manik
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
6:15 p.m.
New subject: Infinispan and search APIs
On 19/05/2009, at 7:56 PM, Manik Surtani wrote:
I've been trying to get jbc-searchable working with JbossCache 3, but then I
noticed this thread for Infinispan. I am trying to build a proof-of-concept
for a distributed searchable cache, and obviously infinispan with hibernate
search would be an ideal solution. At what stage of development is this?
Is it possible to use the existing jbc-searchable code with infinispan?
Should I perhaps stick with jbc for now?
This isn't in Infinispan as yet. The plan is to add querying support, but
it won't make our next major release. Unless you feel like contributing it.
:-)
I'd be happy to take a look. What would be the preferred to monitor what
data is being stored on the local node? I took a look at the interceptors,
but Im unsure as to where to start. In theory, I guess you just need to
know what data is being stored or evicted from the local node and update an
index accordingly.
I also tried using the jbosscache-lucene Directory implementation that Manik
wrote, but things started to die when lucene tried to merge segments. Has
there been any work to create a infinispan-backed lucene Directory?
There hasn't been any effort as yet, and with Vladimir's recent work on the
lock() API, I expect Infinispan to be a far better fit for a Lucene
directory than JBoss Cache is. So again, if you are interested in
contributing this, perhaps based on my JBoss Cache implementation, it would
be much appreciated.
I'm leaning away from storing the Directory in-grid at the moment, feels
like a more complex and error-prone solution than keeping indices on each
machine (no distributed locks, etc). Redundant indices also protects
against index corruption, something we seem to get a lot of in our current
implementation (using ruby's ferret, although I am sure Lucene is more
reliable).
Ray
5:12 a.m.
New subject: Infinispan and search APIs
On 21 May 2009, at 00:15, Ray Hilton wrote:
On 19/05/2009, at 7:56 PM, Manik Surtani wrote:
>> I've been trying to get jbc-searchable working with JbossCache 3,
>> but then I noticed this thread for Infinispan. I am trying to
>> build a proof-of-concept for a distributed searchable cache, and
>> obviously infinispan with hibernate search would be an ideal
>> solution. At what stage of development is this? Is it possible
>> to use the existing jbc-searchable code with infinispan? Should I
>> perhaps stick with jbc for now?
>
> This isn't in Infinispan as yet. The plan is to add querying
> support, but it won't make our next major release. Unless you feel
> like contributing it. :-)
I'd be happy to take a look. What would be the preferred to monitor
what data is being stored on the local node? I took a look at the
interceptors, but Im unsure as to where to start. In theory, I
guess you just need to know what data is being stored or evicted
from the local node and update an index accordingly.
I would expect we do this in an interceptor. The interceptor adds or
updates indexes in Hibernate Search accordingly.
We'd need a new API to deal with queries, you should take a look at
JBoss Cache - Searchable [1] and use it as a starting point. JBCS
does *not* use interceptors though, instead it attaches a listener to
JBC. I jotted some notes in improving this to use interceptors [2].
I believe the approach for Infinispan should use interceptors.
Cheers
Manik
[1] http://anonsvn.jboss.org/repos/jbosscache/searchable/trunk/
[2] http://www.jboss.org/community/wiki/JBossCache-SearchableEdition-Designs
>> I also tried using the jbosscache-lucene Directory implementation
>> that Manik wrote, but things started to die when lucene tried to
>> merge segments. Has there been any work to create a infinispan-
>> backed lucene Directory?
>
> There hasn't been any effort as yet, and with Vladimir's recent
> work on the lock() API, I expect Infinispan to be a far better fit
> for a Lucene directory than JBoss Cache is. So again, if you are
> interested in contributing this, perhaps based on my JBoss Cache
> implementation, it would be much appreciated.
I'm leaning away from storing the Directory in-grid at the moment,
feels like a more complex and error-prone solution than keeping
indices on each machine (no distributed locks, etc). Redundant
indices also protects against index corruption, something we seem to
get a lot of in our current implementation (using ruby's ferret,
although I am sure Lucene is more reliable).
Ray
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
5:27 a.m.
New subject: Infinispan and search APIs
On 21 May 2009, at 00:15, Ray Hilton wrote:
On 19/05/2009, at 7:56 PM, Manik Surtani wrote:
>> I've been trying to get jbc-searchable working with JbossCache 3,
>> but then I noticed this thread for Infinispan. I am trying to
>> build a proof-of-concept for a distributed searchable cache, and
>> obviously infinispan with hibernate search would be an ideal
>> solution. At what stage of development is this? Is it possible
>> to use the existing jbc-searchable code with infinispan? Should I
>> perhaps stick with jbc for now?
>
> This isn't in Infinispan as yet. The plan is to add querying
> support, but it won't make our next major release. Unless you feel
> like contributing it. :-)
I'd be happy to take a look. What would be the preferred to monitor
what data is being stored on the local node? I took a look at the
interceptors, but Im unsure as to where to start. In theory, I
guess you just need to know what data is being stored or evicted
from the local node and update an index accordingly.
I would expect we do this in an interceptor. The interceptor adds or
updates indexes in Hibernate Search accordingly.
We'd need a new API to deal with queries, you should take a look at
JBoss Cache - Searchable [1] and use it as a starting point. JBCS
does *not* use interceptors though, instead it attaches a listener to
JBC. I jotted some notes in improving this to use interceptors [2].
I believe the approach for Infinispan should use interceptors.
Cheers
Manik
[1] http://anonsvn.jboss.org/repos/jbosscache/searchable/trunk/
[2] http://www.jboss.org/community/wiki/JBossCache-SearchableEdition-Designs
>> I also tried using the jbosscache-lucene Directory implementation
>> that Manik wrote, but things started to die when lucene tried to
>> merge segments. Has there been any work to create a infinispan-
>> backed lucene Directory?
>
> There hasn't been any effort as yet, and with Vladimir's recent
> work on the lock() API, I expect Infinispan to be a far better fit
> for a Lucene directory than JBoss Cache is. So again, if you are
> interested in contributing this, perhaps based on my JBoss Cache
> implementation, it would be much appreciated.
I'm leaning away from storing the Directory in-grid at the moment,
feels like a more complex and error-prone solution than keeping
indices on each machine (no distributed locks, etc). Redundant
indices also protects against index corruption, something we seem to
get a lot of in our current implementation (using ruby's ferret,
although I am sure Lucene is more reliable).
Ray
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
1:43 a.m.
New subject: Infinispan and search APIs
On May 19, 2009, at 02:44, Ray Hilton wrote:
On a related note, I like the idea that each node builds and stores
its own indices for data it currently hosts. It would also be nice
if these indices could be further partitioned (in our case, we would
like to do this per-day, other situations maybe based on a modulous
of the hash code) to reduce the size of the indices. Replicated
indices are ok if they dont change that often, but ours will be
updated almost constantly, and so you either need to nominate one
node to do the work, or have some distributed locking in place,
which would probably kill performance. Building an index locally
also provides protection against index corruption by having
redundant copies around the grid. We are intending to do
deduplication and score normalization ourselves.
You can partition in Hibernate Search by using what is called sharding
(that would be for your day type partition). Otherwise the drawback of
using distributed indices is that you need to read them all to answer
your query. Not sure it's a good fit, at least it's unlikely a 1-1 fit
with where the data is stored.
6176
days inactive
6182
days old
infinispan-dev@lists.jboss.org
5 comments
3 participants
participants (3)
-
Emmanuel Bernard -
Manik Surtani -
Ray Hilton