3:55 p.m.
Hello,
as I had anticipated in some chats, my current feeling with the Lucene
Directory is that I got a marvelous new device but I can't completely
use it,
like currently in my home town they installed a new pair of train
trails for high speed trains to connect to the city but they can't
afford the high-speed trains.
Or, more fitting, like getting a 64-way server for a service which
can't use more than one thread, because of some old dependency. You
would definitely want to solve this limitation :D
The Lucene Directory is doing well in read-most situations, and a nice
to have for huge sized indexes to easily sync them in dynamic-topology
clusters.
But when the many participant nodes all potentially apply changes, a
global index lock needs to be acquired; this is safely achievable by
using the LockFactory implementations provided in the lucene-directory
package but of course doesn't scale.
In practice, it's worse than a performance problem: Lucene expects
full ownership of all resources, and so the IndexWriter implementation
doesn't expect the Lock to timeout, and has no notion of fairness in
the wait process, so if your architecture does something like "each
node can ask for the lock and apply changes" without some external
coordination, this only works in case the contention on this lock is
low enough; if it piles up, it will blow up the application with
exceptions on indexing.
Best solution I've seen so far is what Emmanuel implemented years ago
for Hibernate Search by using JMS to send changes to a single master
node. Currently I think the state-of-the-art installation should
combine such a queue-like solution to delegate all changes to a single
node, and this single node should apply the changes to an Infinispan
Directory - so making changes to all other nodes visible through
efficient Infinispan distribution/replication.
Replication was done in the past by using an rsync-like file copy, so
the new benefit would be to ease the setup of Directory replication,
but you still need a dedicated master node and work on setting this
up.
Now I would like to use Infinispan to replace the JMS approach,
especially as in cloud environments it's useful that the different
participants which make up the service are all equally configured:
having a Master Writer node is fine as long as it's auto-elected.
(not mandatory, just very nice to have)
The problems to solve:
A) Locking
The current locking solution is implemented by atomically adding a
marker value in the cache. I can't use transactions as the lock could
span several transactions and must be visible.
It's Lucene's responsibility to clear this lock properly, and I can
trust it for that as far as the code can do. But what happens if the
node dies? Other nodes taking over the writer role should be able to
detect the situation and remove the lock.
Proposals:
- A1) We don't lock, but find a way to elect one and only one writer
node. This would be very cool, but I have no idea about how to
implement it.
- A2) We could store the node-address as value in the marker object,
if the address isn't part of the members the lock is cleared. (can I
trust the members view? the index will corrupt if changed by two
nodes)
B) Sending changes to writer
As for Lucene's design the IndexWriter is threadsafe and is an heavy
to build object, it should be reused as much as possible to insert
many Documents at once. So when a node managed to acquire the Lock it
should keep the IndexWriter open a relatively long time, and possibly
receive changes from other nodes to be applied on the index.
Proposal:
- B1) Use JMS or JGroups directly (as Hibernate Search is currently
capable to use both), again I face the problem of IndexWriter node
election, and have the messages sent to the correct node.
In both cases I would like to receive enough information from
Infinispan to now where to send messages from the queue.
- B2) Exploit the ConcurrentMap nature of Infinispan: I don't need
strict ordering of change-requests if we can make an assumption on the
Lucene Documents: to have each document identifiable.
This is usually the case, and always is in Hibernate Search where
each Document entry is identified by (typeOfEntity, PrimaryKey).
Assuming we find out how to start the IndexWriting process on a
single node, we could have a Cache to store change requests on the
index: if for each changed entity I was inserting a key made of
(documentID,typeOfOperation) and value containing eventually the new
Document to be written and some timestamp/counter. typeOfOperation
could be "delete" or "add", which are the only supported operations
by
Lucene. I believe I could read the timestamp from the entry?
The timestamp would be needed to recognize what to do in case of
having both a delete and a add operation on same entity (add,delete ->
noop; delete,add --> update).
So the node which is running the Lucene IndexWriting task could
periodically iterate on all entries of this cache and apply latest
needed operations, using atomic removeIfUnchanged . It would be fine,
even better, in case of overwritten entries as I would write only
latest version of a Document in case it's being changed quicker than
what we can write.
The drawback of this approach is that it assumes that, while it will
buffer temporary spikes of load, it needs to be on average faster to
write out all changes, and that changes on different entities are
applied in unpredictable order, still I'm liking this solution the
most as it looks like it can best use Infinispan for maximum
performance.
Thoughts? Problems?
I'd also appreciate some example of a job that needs to be running on
a single node only if you have one, and would love to avoid depending
on more than Infinispan.
Cheers,
Sanne
7:33 a.m.
I'm commenting on this without being a Lucene Hibernate Search expert,
so bear with me...
Sanne Grinovero wrote:
The Lucene Directory is doing well in read-most situations, and a
nice
to have for huge sized indexes to easily sync them in dynamic-topology
clusters.
But when the many participant nodes all potentially apply changes, a
global index lock needs to be acquired; this is safely achievable by
using the LockFactory implementations provided in the lucene-directory
package but of course doesn't scale.
In practice, it's worse than a performance problem: Lucene expects
full ownership of all resources, and so the IndexWriter implementation
doesn't expect the Lock to timeout, and has no notion of fairness in
the wait process, so if your architecture does something like "each
node can ask for the lock and apply changes" without some external
coordination, this only works in case the contention on this lock is
low enough; if it piles up, it will blow up the application with
exceptions on indexing.
Best solution I've seen so far is what Emmanuel implemented years ago
for Hibernate Search by using JMS to send changes to a single master
node. Currently I think the state-of-the-art installation should
combine such a queue-like solution to delegate all changes to a single
node, and this single node should apply the changes to an Infinispan
Directory - so making changes to all other nodes visible through
efficient Infinispan distribution/replication.
By sending the changes to a single master, who then applies them, you
essentially pick a total order concept: all changes are applies in the
same order across the cluster. Total order eliminates the need to
acquire a lock, because the changes from different nodes will always be
seen in the same order, therefore you don't need to lock.
This could also be done by using the SEQUENCER protocol in JGroups: say
A, B and C need to update an index. If B wanted to make an update, it
would have to acquire the cluster wide lock, update the index, replicate
across the cluster, and the release the lock. The lock is needed to make
sure that B's changes don't overlap with C's or A's changes.
However, if we guaranteed that everybody would receive B's modification,
followed by A's modification, followed by C's modification, we would not
need a lock at all !
In other words, if we have to apply a change across a cluster, with 1
message, total order can replace a two-phase lock-unlock protocol.
Now I would like to use Infinispan to replace the JMS approach,
especially as in cloud environments it's useful that the different
participants which make up the service are all equally configured:
having a Master Writer node is fine as long as it's auto-elected.
(not mandatory, just very nice to have)
The problems to solve:
A) Locking
The current locking solution is implemented by atomically adding a
marker value in the cache. I can't use transactions as the lock could
span several transactions and must be visible.
The issue with this is that you (ab)use the cache as a messaging /
signalling system ! This is not recommended. A better approach would be
to use a separate channel for communication, or to even reuse the
channel Infinispan uses. The latter is something I've been discussing
with Brian.
It's Lucene's responsibility to clear this lock properly, and
I can
trust it for that as far as the code can do. But what happens if the
node dies? Other nodes taking over the writer role should be able to
detect the situation and remove the lock.
You get a new view and release all locks held by nodes which are not
part of the new view. This is how it's implemented in JBossCache and
Infinispan today.
Proposals:
- A1) We don't lock, but find a way to elect one and only one writer
node. This would be very cool, but I have no idea about how to
implement it.
You could use the cluster view, which is the same across all nodes, and
pick the first element in the list. Or you could run an agreement
protocol, which deterministically elects a master.
- A2) We could store the node-address as value in the marker object,
if the address isn't part of the members the lock is cleared. (can I
trust the members view? the index will corrupt if changed by two
nodes)
Not so good, don't abuse the cache as a communication bus
Proposal:
- B1) Use JMS or JGroups directly (as Hibernate Search is currently
capable to use both), again I face the problem of IndexWriter node
election, and have the messages sent to the correct node.
In both cases I would like to receive enough information from
Infinispan to know where to send messages from the queue.
I'd use JGroups (of course I'm biased :-)), but JMS really has no means
of cluster views to elect a master. Plus, you could use SEQUENCER as a
means to establish a global order between messages, so maybe the need to
acquire a lock could be dropped altogether.
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
2:20 p.m.
See inline
Sanne Grinovero wrote:
> The Lucene Directory is doing well in read-most situations, and a nice
> to have for huge sized indexes to easily sync them in dynamic-topology
> clusters.
> But when the many participant nodes all potentially apply changes, a
> global index lock needs to be acquired; this is safely achievable by
> using the LockFactory implementations provided in the lucene-directory
> package but of course doesn't scale.
>
> In practice, it's worse than a performance problem: Lucene expects
> full ownership of all resources, and so the IndexWriter implementation
> doesn't expect the Lock to timeout, and has no notion of fairness in
> the wait process, so if your architecture does something like "each
> node can ask for the lock and apply changes" without some external
> coordination, this only works in case the contention on this lock is
> low enough; if it piles up, it will blow up the application with
> exceptions on indexing.
>
> Best solution I've seen so far is what Emmanuel implemented years ago
> for Hibernate Search by using JMS to send changes to a single master
> node. Currently I think the state-of-the-art installation should
> combine such a queue-like solution to delegate all changes to a single
> node, and this single node should apply the changes to an Infinispan
> Directory - so making changes to all other nodes visible through
> efficient Infinispan distribution/replication.
By sending the changes to a single master, who then applies them, you
essentially pick a total order concept: all changes are applies in the
same order across the cluster. Total order eliminates the need to
acquire a lock, because the changes from different nodes will always be
seen in the same order, therefore you don't need to lock.
This could also be done by using the SEQUENCER protocol in JGroups: say
A, B and C need to update an index. If B wanted to make an update, it
would have to acquire the cluster wide lock, update the index, replicate
across the cluster, and the release the lock. The lock is needed to make
sure that B's changes don't overlap with C's or A's changes.
However, if we guaranteed that everybody would receive B's modification,
followed by A's modification, followed by C's modification, we would not
need a lock at all !
In other words, if we have to apply a change across a cluster, with 1
message, total order can replace a two-phase lock-unlock protocol.
In Lucene's term that would mean that an index copy is present on all nodes (not store
in Infinispan necessarily) and all changes are passed and executed on all nodes (being
passed in total ordering via JGroups).
Intuitively, I don't think it's the right approach:
- that would mean CPU and I/O consumption on all nodes: indexing / analyzing is a heavy
work potentially
- I am not sure Lucene is 100% deterministic in its way of indexing
> Now I would like to use Infinispan to replace the JMS approach,
> especially as in cloud environments it's useful that the different
> participants which make up the service are all equally configured:
> having a Master Writer node is fine as long as it's auto-elected.
> (not mandatory, just very nice to have)
>
> The problems to solve:
>
> A) Locking
> The current locking solution is implemented by atomically adding a
> marker value in the cache. I can't use transactions as the lock could
> span several transactions and must be visible.
The issue with this is that you (ab)use the cache as a messaging /
signalling system ! This is not recommended. A better approach would be
to use a separate channel for communication, or to even reuse the
channel Infinispan uses. The latter is something I've been discussing
with Brian.
> It's Lucene's responsibility to clear this lock properly, and I can
> trust it for that as far as the code can do. But what happens if the
> node dies? Other nodes taking over the writer role should be able to
> detect the situation and remove the lock.
You get a new view and release all locks held by nodes which are not
part of the new view. This is how it's implemented in JBossCache and
Infinispan today.
> Proposals:
>
> - A1) We don't lock, but find a way to elect one and only one writer
> node. This would be very cool, but I have no idea about how to
> implement it.
You could use the cluster view, which is the same across all nodes, and
pick the first element in the list. Or you could run an agreement
protocol, which deterministically elects a master.
Looks simple, deterministic and elegant.
> - A2) We could store the node-address as value in the marker object,
> if the address isn't part of the members the lock is cleared. (can I
> trust the members view? the index will corrupt if changed by two
> nodes)
Not so good, don't abuse the cache as a communication bus
> Proposal:
>
> - B1) Use JMS or JGroups directly (as Hibernate Search is currently
> capable to use both), again I face the problem of IndexWriter node
> election, and have the messages sent to the correct node.
> In both cases I would like to receive enough information from
> Infinispan to know where to send messages from the queue.
I'd use JGroups (of course I'm biased :-)), but JMS really has no means
of cluster views to elect a master. Plus, you could use SEQUENCER as a
means to establish a global order between messages, so maybe the need to
acquire a lock could be dropped altogether.
So basically, following Bela's advice, you would:
- use the cluster view to elect the master
- send changes to the master using JGroups
We still need one guaranty: make sure queued changes not applied on the master are going
to be processed by the new master (that's why Sanne was thining of using Infinispan as
a messaging system I guess).
5:16 p.m.
New subject: asynchronous change manager for high load Lucene
Emmanuel Bernard wrote:
> In other words, if we have to apply a change across a cluster,
with 1
> message, total order can replace a two-phase lock-unlock protocol.
In Lucene's term that would mean that an index copy is present on all
nodes (not store in Infinispan necessarily) and all changes are passed
and executed on all nodes (being passed in total ordering via JGroups).
Ah, that changes the game ! I thought that the index is some kind of
metadata, present on every node. Of course, if that'snot the case, then
SEQUENCER makes no sense...
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
1:14 p.m.
I've cut large parts of text and quoting from different answers, hope it helps:
Bela Ban wrote:
>I'm commenting on this without being a Lucene Hibernate Search
expert,
so bear with me...
thanks for all the insight, it's very appreciated.
Manik Surtani wrote:
> This is because each time an update is made, all parts of the
index need to be updated? Can this not be done in a finer-grained manner where nodes only
lock and update the chunks they need to update? Caveat: I have no idea of Lucene index
formats, so this may make no sense whatsoever. :)
The index is structured in a series of large files; one of them is
small and is always updated by all changes, the others are either
deleted or added, never updated - depending on the amount of segments:
if some thresholds are crossed, more updates might be needed.
The most frequent operation is to add a new segment and edit the small
one to add a pointer to the new segment.
In some cases a group of smaller segments is deleted and a new one is
added containing the merged data - the new segment will have a new
name/key.
It's quite possible the case that they are all deleted and new
segments are rewritten containing a new optimized copy of the same
information - will use a different filename in this case (filename is
our key).
Manik Surtani wrote:
> Not nice. Are there plans to support retries in Lucene in
future?
I don't think retries are planned, but if we wanted to do that we
could ignore the typical semantics of the Lucene lock as we implement
that and blok longer than what the application asks for.
Still that means that all nodes are going to have high contention on
the global index lock - in theory it's still more efficient to send
over all changes to a single node and have it handle it as it can
buffer and aggregate changes.
The IndexWritter was designed to be highly efficient and
multithreaded, we just need to make sure there's one instance - a
longer term idea I'd like to explore is to totally override the
IndexWriter but that's going to be complex and every change in Lucene
would require an update.
Would be nice to override the IndexReader too and use an ad-hoc index
structure - I'll play with it but doubt I could deliver something
stable in the near future as I'm working on this in my spare time.
> Sanne Grinovero wrote:
> The current locking solution is implemented by atomically adding a
> marker value in the cache. I can't use transactions as the lock could
> span several transactions and must be visible.
Bela Ban wrote:
> The issue with this is that you (ab)use the cache as a messaging
/
> signalling system ! This is not recommended. A better approach would be
> to use a separate channel for communication, or to even reuse the
> channel Infinispan uses. The latter is something I've been discussing
> with Brian.
I know, I clearly remember your recommendations about this, please
forgive me :) This was done so that the lock state should be coupled
to the index and this Lucene Directory could be used in other
scenarios, like to keep it simple enough to work as a drop-in
replacement in existing applications which don't want to implement the
enhancements discussed on this thread.
In most cases Lucene is almost read-only, not updated very often as
updates are very costly, and so the cost of such a lock is quite
negligible compared to all that what's happening in the middle:
potentially fetching, processing and distributing many gigabytes of
index.
It's possible to plug it's own locking implementation, the marker
value stored in the cache is the default as a safety net.
In fact I'd like to plug a no-op Lock implementation and just make
sure the election stuff works - but need strong guarantees.
To make another analogy to "traditional lucene" locks, a well designed
application could disable them too, but by default a file-level lock
is written to protect from the unexpected.
Manik Surtani wrote:
> This is because each time an update is made, all parts of the
index need to be updated? Can this not be done in a finer-grained manner where nodes only
lock and update the chunks they need to update? Caveat: I have no idea of Lucene index
formats, so this may make no sense whatsoever. :)
Lucene chooses which parts it needs to update, as mentioned above I'd
like to investigate this path and take control of the index structure,
but that's far-future planning :)
Emmanuel Bernard wrote:
Bela Ban wrote:
> You could use the cluster view, which is the same across all nodes, and
> pick the first element in the list. Or you could run an agreement
> protocol, which deterministically elects a master.
Looks simple, deterministic and elegant.
Agreed, that looks like a clean solution, but would scale better if we
could decide a master for each different index, to spread the load
instead of choosing always the same node for all the work, ideas on
that?
I could hash the indexname (the identifier) and % cluster view list size, WDYT ?
Emmanuel Bernard wrote:
So basically, following Bela's advice, you would:
- use the cluster view to elect the master
- send changes to the master using JGroups
We still need one guaranty: make sure queued changes not applied on the master are going
to be processed by the new master (that's why Sanne was thining of using Infinispan as
a messaging system I guess).
Yes that's what I thought, if I store changes-to-be-done in Infinispan
I expect that to be reliable enough - as much as people wants to
configure. You're right about networking inefficiency, ideally I
should need something that was able to select the main storing node,
and have secondary (third..) "buddies" elected by hash.
Total ordering is not needed, but we need to make sure that different
works applying to the same entity instance are applied in the same
order - and it's also fine if they happen to cancel out each other, so
writing changes in a map I could atomically remove work which is done,
or atomically replace outdated work which can be replaced to a new
version when the writer is lagging behind the work production - this
lag could accumulate the whole day, and still have updates to the
index reflected as fast as possible (in current design terms) and
possibly have guarantess that at night all work will be processed -
and duplicates removed as a side effect of key collisions during the
storing.
When using a global queue I'll have to guarantee that all received
work is processed in the same order as they where produced - order has
to match the database transactions to which Hibernate Search was
listening to - when using a map approach I'm confident that MVCC will
help in "seeing" only the most recently put work, which always is the
correct ordering.
The short to identify JGroups ordering contexts is way too limited for
this problem, the context should be defined as unique
type+id+operation to solve this same problem, which is a nice key for
my Map.
I don't want to push for the "Cache bus" approach, but in other cases
I would definitely need some help on the issues above, while the Map
sounds easy to implement now that I also see an easy way to know if
I'm the master or not.
I got puzzled by this SEQUENCER and just read the docs, I'll ask some
more details in an appropriate thread.
Manik, you have an example of how I should send a message to a
selected member by integrating in Infinispan? I think I can figure out
how to send it with some code reading, but I'm mostly lost about how
to receive it? Or maybe you suggest I should share the
JGroupsTransport with totally infinispan-independent code?
thanks all for the excellent feedback,
Sanne
2 p.m.
On 30 mars 2010, at 13:14, Sanne Grinovero wrote:
Emmanuel Bernard wrote:
> Bela Ban wrote:
>> You could use the cluster view, which is the same across all nodes, and
>> pick the first element in the list. Or you could run an agreement
>> protocol, which deterministically elects a master.
>
> Looks simple, deterministic and elegant.
Agreed, that looks like a clean solution, but would scale better if we
could decide a master for each different index, to spread the load
instead of choosing always the same node for all the work, ideas on
that?
I could hash the indexname (the identifier) and % cluster view list size, WDYT ?
That would work assuming you take the actual directory provider name (ie the one that can
potentially contain the sharded information like Account.1, Account.2 etc).
6:15 p.m.
On 30 Mar 2010, at 13:00, Emmanuel Bernard wrote:
On 30 mars 2010, at 13:14, Sanne Grinovero wrote:
>
> Emmanuel Bernard wrote:
>> Bela Ban wrote:
>>> You could use the cluster view, which is the same across all nodes, and
>>> pick the first element in the list. Or you could run an agreement
>>> protocol, which deterministically elects a master.
>>
>> Looks simple, deterministic and elegant.
>
> Agreed, that looks like a clean solution, but would scale better if we
> could decide a master for each different index, to spread the load
> instead of choosing always the same node for all the work, ideas on
> that?
> I could hash the indexname (the identifier) and % cluster view list size, WDYT ?
That would work assuming you take the actual directory provider name (ie the one that can
potentially contain the sharded information like Account.1, Account.2 etc).
This sounds very workable, yes.
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
3:09 p.m.
New subject: asynchronous change manager for high load Lucene
Sanne Grinovero wrote:
>> You could use the cluster view, which is the same across all
nodes,
>> and pick the first element in the list. Or you could run an
>> agreement protocol, which deterministically elects a master.
> Looks simple, deterministic and elegant.
Bela Ban wrote:
Agreed, that looks like a clean solution, but would scale better if we
could decide a master for each different index, to spread the load
instead of choosing always the same node for all the work, ideas on that?
I could hash the indexname (the identifier) and % cluster view list
size, WDYT ?
Absolutely, this makes a lot of sense. However, this will require moving
indices around when the cluster view changes.
With a simple modulo based algorithm you'll have a lot of rebalancing
going on, so maybe something like a consistent hash could determine the
server or servers on which a given index is stored. You could even store
an index on more than 1 nodes, similar to what Infinispan does with DIST
and numOwners=2 (for example).
Even better: *if* an index can be recreated from the real data, you
could use a consistent hash and only store an index *once*, and you
could use a Cache instance for it !
Emmanuel Bernard wrote:
> So basically, following Bela's advice, you would:
> - use the cluster view to elect the master
> - send changes to the master using JGroups
>
> We still need one guaranty: make sure queued changes not applied on
> the master are going to be processed by the new master (that's why
> Sanne was thining of using Infinispan as a messaging system I guess)
This is something SEQUENCER guarantees: a sender sends the message to
the sequencer (coordinator), but queues the message locally as well.
Only when it receives its own message wil it be removed from the queue.
When a coordinator crashes, all queues messages will get resent to the
new coordinator.
Because this is done above NAKACK (multicast messages) or UNICAST
(unicast messages), we're also guaranteed that no duplicates will be
delivered and no message will get dropped.
Yes that's what I thought, if I store changes-to-be-done in
Infinispan
I expect that to be reliable enough - as much as people wants to
configure. You're right about networking inefficiency, ideally I
should need something that was able to select the main storing node,
and have secondary (third..) "buddies" elected by hash.
Total ordering is not needed
Yes, in this case you actually implement global ordering by always
sending everything for a given index to the same master node. This is fine.
but we need to make sure that different
works applying to the same entity instance are applied in the same
order - and it's also fine if they happen to cancel out each other, so
writing changes in a map I could atomically remove work which is done,
or atomically replace outdated work which can be replaced to a new
version when the writer is lagging behind the work production - this
lag could accumulate the whole day, and still have updates to the
index reflected as fast as possible (in current design terms) and
possibly have guarantess that at night all work will be processed -
and duplicates removed as a side effect of key collisions during the
storing.
When using a global queue I'll have to guarantee that all received
work is processed in the same order as they where produced - order has
to match the database transactions to which Hibernate Search was
listening to - when using a map approach I'm confident that MVCC will
help in "seeing" only the most recently put work, which always is the
correct ordering.
The short to identify JGroups ordering contexts is way too limited for
this problem, the context should be defined as unique
type+id+operation to solve this same problem, which is a nice key for
my Map.
I don't want to push for the "Cache bus" approach, but in other cases
I would definitely need some help on the issues above, while the Map
sounds easy to implement now that I also see an easy way to know if
I'm the master or not.
I got puzzled by this SEQUENCER and just read the docs, I'll ask some
more details in an appropriate thread
Sure, or ping me on skype or IRC (#jgroups @ irc.freenode.net)
Manik, you have an example of how I should send a message to a
selected member by integrating in Infinispan? I think I can figure out
how to send it with some code reading, but I'm mostly lost about how
to receive it? Or maybe you suggest I should share the
JGroupsTransport with totally infinispan-independent code?
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
6:17 p.m.
New subject: asynchronous change manager for high load Lucene
On 30 Mar 2010, at 14:09, Bela Ban wrote:
Sanne Grinovero wrote:
>>> You could use the cluster view, which is the same across all nodes,
>>> and pick the first element in the list. Or you could run an
>>> agreement protocol, which deterministically elects a master.
>> Looks simple, deterministic and elegant.
> Bela Ban wrote:
>
> Agreed, that looks like a clean solution, but would scale better if we
> could decide a master for each different index, to spread the load
> instead of choosing always the same node for all the work, ideas on that?
> I could hash the indexname (the identifier) and % cluster view list
> size, WDYT ?
Absolutely, this makes a lot of sense. However, this will require moving
indices around when the cluster view changes.
With a simple modulo based algorithm you'll have a lot of rebalancing
going on, so maybe something like a consistent hash could determine the
server or servers on which a given index is stored. You could even store
an index on more than 1 nodes, similar to what Infinispan does with DIST
and numOwners=2 (for example).
Even better: *if* an index can be recreated from the real data, you
could use a consistent hash and only store an index *once*, and you
could use a Cache instance for it !
Yes, this would be the plan. The indexes themselves are *read* by every node in the
cluster and hence is stored in the cache. It is just the updates that will be processed
by a selected processor singleton.
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
6:20 p.m.
On 30 Mar 2010, at 12:14, Sanne Grinovero wrote:
Manik, you have an example of how I should send a message to a
selected member by integrating in Infinispan? I think I can figure out
how to send it with some code reading, but I'm mostly lost about how
to receive it? Or maybe you suggest I should share the
JGroupsTransport with totally infinispan-independent code?
Hmm, I'd really like to not have to expose the RpcManager as a client API. The reason
why is that we do a lot of Infinispan-specific optimisations on this layer and this needs
to be carefully considered if we want to expose it as a mechanism for people to do
arbitrary RPC.
The RpcManager does sit on top of a JGroups channel though, and what Bela mentioned is the
ability for a JGroups channel to have different RPC systems built on top of it and to
dispatch inbound RPC to the appropriate handler. JGroups has an RpcDispatcher
building-block that you could use.
Cheers
Manik
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
7:50 a.m.
New subject: asynchronous change manager for high load Lucene
Manik Surtani wrote:
On 30 Mar 2010, at 12:14, Sanne Grinovero wrote:
> Manik, you have an example of how I should send a message to a
> selected member by integrating in Infinispan? I think I can figure out
> how to send it with some code reading, but I'm mostly lost about how
> to receive it? Or maybe you suggest I should share the
> JGroupsTransport with totally infinispan-independent code?
Hmm, I'd really like to not have to expose the RpcManager as a client
API. The reason why is that we do a lot of Infinispan-specific
optimisations on this layer and this needs to be carefully considered
if we want to expose it as a mechanism for people to do arbitrary RPC.
The RpcManager does sit on top of a JGroups channel though, and what
Bela mentioned is the ability for a JGroups channel to have different
RPC systems built on top of it and to dispatch inbound RPC to the
appropriate handler. JGroups has an RpcDispatcher building-block that
you could use.
Brian and I are discussing this, and I prototyped an example in [1].
We're currently debating whether a shared transport or [1] is better.
We're probably going to end up with a mixture: unrelated components will
each use a separate channel on top of the same shared transport, related
components might end up sharing a channel (on top of the same shared
transport).
I'm cautious with [1] as I don't want to get back into the same mess we
had with the Multiplexer !
[1] https://jira.jboss.org/jira/browse/JGRP-1177
--
Bela Ban
Lead JGroups / Clustering Team
JBoss
2:04 p.m.
Hi Sanne.
Wow, talk about a detailed analysis email! My comments inline.
On 26 Mar 2010, at 14:55, Sanne Grinovero wrote:
Hello,
as I had anticipated in some chats, my current feeling with the Lucene
Directory is that I got a marvelous new device but I can't completely
use it,
like currently in my home town they installed a new pair of train
trails for high speed trains to connect to the city but they can't
afford the high-speed trains.
Or, more fitting, like getting a 64-way server for a service which
can't use more than one thread, because of some old dependency. You
would definitely want to solve this limitation :D
An interesting analogy... :)
The Lucene Directory is doing well in read-most situations, and a
nice
to have for huge sized indexes to easily sync them in dynamic-topology
clusters.
But when the many participant nodes all potentially apply changes, a
global index lock needs to be acquired; this is safely achievable by
using the LockFactory implementations provided in the lucene-directory
package but of course doesn't scale.
This is because each time an update is made, all parts of the index need to be updated?
Can this not be done in a finer-grained manner where nodes only lock and update the chunks
they need to update? Caveat: I have no idea of Lucene index formats, so this may make no
sense whatsoever. :)
In practice, it's worse than a performance problem: Lucene
expects
full ownership of all resources, and so the IndexWriter implementation
doesn't expect the Lock to timeout, and has no notion of fairness in
the wait process, so if your architecture does something like "each
node can ask for the lock and apply changes" without some external
coordination, this only works in case the contention on this lock is
low enough; if it piles up, it will blow up the application with
exceptions on indexing.
Not nice. Are there plans to support retries in Lucene in future?
Best solution I've seen so far is what Emmanuel implemented years
ago
for Hibernate Search by using JMS to send changes to a single master
node. Currently I think the state-of-the-art installation should
combine such a queue-like solution to delegate all changes to a single
node, and this single node should apply the changes to an Infinispan
Directory - so making changes to all other nodes visible through
efficient Infinispan distribution/replication.
Replication was done in the past by using an rsync-like file copy, so
the new benefit would be to ease the setup of Directory replication,
but you still need a dedicated master node and work on setting this
up.
That would still not scale though - still a bottleneck on the single master updater node?
Probably better than your current impl, though, so I see the motivations here. :)
Now I would like to use Infinispan to replace the JMS approach,
especially as in cloud environments it's useful that the different
participants which make up the service are all equally configured:
having a Master Writer node is fine as long as it's auto-elected.
(not mandatory, just very nice to have)
The problems to solve:
A) Locking
The current locking solution is implemented by atomically adding a
marker value in the cache. I can't use transactions as the lock could
span several transactions and must be visible.
It's Lucene's responsibility to clear this lock properly, and I can
trust it for that as far as the code can do. But what happens if the
node dies? Other nodes taking over the writer role should be able to
detect the situation and remove the lock.
Proposals:
- A1) We don't lock, but find a way to elect one and only one writer
node. This would be very cool, but I have no idea about how to
implement it.
Infinispan exposes JGroups coordinator information. You can peg this task to the
coordinator, and you can register for view change events to be notified of potential
changes in the coordinator.
- A2) We could store the node-address as value in the marker object,
if the address isn't part of the members the lock is cleared. (can I
trust the members view? the index will corrupt if changed by two
nodes)
Not necessary, see above.
B) Sending changes to writer
As for Lucene's design the IndexWriter is threadsafe and is an heavy
to build object, it should be reused as much as possible to insert
many Documents at once. So when a node managed to acquire the Lock it
should keep the IndexWriter open a relatively long time, and possibly
receive changes from other nodes to be applied on the index.
Proposal:
- B1) Use JMS or JGroups directly (as Hibernate Search is currently
capable to use both), again I face the problem of IndexWriter node
election, and have the messages sent to the correct node.
In both cases I would like to receive enough information from
Infinispan to now where to send messages from the queue.
I agree that using JGroups may be the best way to do this. No more dependencies since you
already have JGroups to run Infinispan. One comment re: Bela's SEQUENCER proposal:
make sure you set up a separate channel for this communication. SEQUENCER in
Infinispan's JGroups channel will add unnecessary overhead for your indexes.
- B2) Exploit the ConcurrentMap nature of Infinispan: I don't
need
strict ordering of change-requests if we can make an assumption on the
Lucene Documents: to have each document identifiable.
This is usually the case, and always is in Hibernate Search where
each Document entry is identified by (typeOfEntity, PrimaryKey).
Assuming we find out how to start the IndexWriting process on a
single node, we could have a Cache to store change requests on the
index: if for each changed entity I was inserting a key made of
(documentID,typeOfOperation) and value containing eventually the new
Document to be written and some timestamp/counter. typeOfOperation
could be "delete" or "add", which are the only supported operations
by
Lucene. I believe I could read the timestamp from the entry?
The timestamp would be needed to recognize what to do in case of
having both a delete and a add operation on same entity (add,delete ->
noop; delete,add --> update).
So the node which is running the Lucene IndexWriting task could
periodically iterate on all entries of this cache and apply latest
needed operations, using atomic removeIfUnchanged . It would be fine,
even better, in case of overwritten entries as I would write only
latest version of a Document in case it's being changed quicker than
what we can write.
The drawback of this approach is that it assumes that, while it will
buffer temporary spikes of load, it needs to be on average faster to
write out all changes, and that changes on different entities are
applied in unpredictable order, still I'm liking this solution the
most as it looks like it can best use Infinispan for maximum
performance.
B2 is effectively using Infinispan for message passing. I think that will hamper
scalability even further - whart are your options? REPL? Then everyone will get these
"tasks", even though (all - 1) nodes will ignore them, DIST? How can you
guarantee that the coordinator node is always in the owner group? Otherwise the
coordinator will still need to fetch these objects from a remote source.
Thoughts? Problems?
I'd also appreciate some example of a job that needs to be running on
a single node only if you have one, and would love to avoid depending
on more than Infinispan.
Cheers,
Sanne
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
3:31 p.m.
On 29 mars 2010, at 14:04, Manik Surtani wrote:
>
> The Lucene Directory is doing well in read-most situations, and a nice
> to have for huge sized indexes to easily sync them in dynamic-topology
> clusters.
> But when the many participant nodes all potentially apply changes, a
> global index lock needs to be acquired; this is safely achievable by
> using the LockFactory implementations provided in the lucene-directory
> package but of course doesn't scale.
This is because each time an update is made, all parts of the index need to be updated?
Can this not be done in a finer-grained manner where nodes only lock and update the chunks
they need to update? Caveat: I have no idea of Lucene index formats, so this may make no
sense whatsoever. :)
Nope. The finest granularity is the Lucene index. We do split the indexed data in multiple
indexes to reduce the granularity but at the index level, you need one exclusive lock.
5148
days inactive
5153
days old
infinispan-dev@lists.jboss.org
12 comments
4 participants
participants (4)
-
Bela Ban -
Emmanuel Bernard -
Manik Surtani -
Sanne Grinovero