Design change in Infinispan Query
by Sanne Grinovero
Hello all,
currently Infinispan Query is an interceptor registering on the
specific Cache instance which has indexing enabled; one such
interceptor is doing all what it needs to do in the sole scope of the
cache it was registered in.
If you enable indexing - for example - on 3 different caches, there
will be 3 different Hibernate Search engines started in background,
and they are all unaware of each other.
After some design discussions with Ales for CapeDwarf, but also
calling attention on something that bothered me since some time, I'd
evaluate the option to have a single Hibernate Search Engine
registered in the CacheManager, and have it shared across indexed
caches.
Current design limitations:
A- If they are all configured to use the same base directory to
store indexes, and happen to have same-named indexes, they'll share
the index without being aware of each other. This is going to break
unless the user configures some tricky parameters, and even so
performance won't be great: instances will lock each other out, or at
best write in alternate turns.
B- The search engine isn't particularly "heavy", still it would be
nice to share some components and internal services.
C- Configuration details which need some care - like injecting a
JGroups channel for clustering - needs to be done right isolating each
instance (so large parts of configuration would be quite similar but
not totally equal)
D- Incoming messages into a JGroups Receiver need to be routed not
only among indexes, but also among Engine instances. This prevents
Query to reuse code from Hibernate Search.
Problems with a unified Hibernate Search Engine:
1#- Isolation of types / indexes. If the same indexed class is
stored in different (indexed) caches, they'll share the same index. Is
it a problem? I'm tempted to consider this a good thing, but wonder if
it would surprise some users. Would you expect that?
2#- configuration format overhaul: indexing options won't be set on
the cache section but in the global section. I'm looking forward to
use the schema extensions anyway to provide a better configuration
experience than the current <properties />.
3#- Assuming 1# is fine, when a search hit is found I'd need to be
able to figure out from which cache the value should be loaded.
3#A we could have the cache name encoded in the index, as part
of the identifier: {PK,cacheName}
3#B we actually shard the index, keeping a physically separate
index per cache. This would mean searching on the joint index view but
extracting hits from specific indexes to keep track of "which index"..
I think we can do that but it's definitely tricky.
It's likely easier to keep indexed values from different caches in
different indexes. that would mean to reject #1 and mess with the user
defined index name, to add for example the cache name to the user
defined string.
Any comment?
Cheers,
Sanne
10 years, 10 months
Re: [infinispan-dev] Removing Infinispan dependency on the Hibernate-Infinispan module in 4.x
by Galder Zamarreño
Scott, what do you suggest doing instead then? Without the commands, evictAll invalidation won't work.
Are you suggesting that I revert back to using the cache as a notification bus so that regions are invalidated?
On Feb 8, 2012, at 4:13 PM, Scott Marlow wrote:
> http://lists.jboss.org/pipermail/infinispan-dev/2012-February/010125.html has more context.
>
> Since there are no easy/quick fixes that can be applied at this time, to remove the AS7 Infinispan dependency on the Hibernate-Infinispan module, I think we should avoid depending on the service loader way to supply the custom commands (in the Hibernate-Infinispan module), at least until this can be addressed elsewhere.
>
> I propose that the Hibernate-Infinispan second level cache should not use the Service Loader to pass custom commands into Infinispan. If we agree, I'll create a jira for this.
>
> Scott
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
11 years, 6 months
Map Reduce 2.0
by Vladimir Blagojevic
Hey guys,
Before moving forward with next iteration of map reduce I wanted to hear
your thoughts about the following proposal. After we agree on the
general direction I will transcribe the agreed design on a wiki page and
start implementation.
Shortcoming of current map reduce implementation
While our current map reduce implementation is more than a proof of a
concept there are several drawbacks preventing it from being an
industrial grade map reduce solution. The main drawback is the inability
of the current solution to deal with a large data (in GB/TB) map reduce
problems. This shortcoming is mainly around our reduce phase execution.
Reduce phase, as you might know, is currently done on a single
Infinispan master task node; reduce phase of map reduce problems we can
support (data size wise) is therefore shrunk to a working memory of a
single node.
Proposed solution
The proposed solution involves distributing execution of reduce phase
tasks across the cluster thus effectively achieving higher reduce task
parallelization and at the same time removing the above mentioned reduce
phase restriction. Effectively leveraging our consistent hashing
solution even further we can parallelize reduce phase and elevate our
map reduce solution to an industrial level. Here is how we can achieve that.
Map phase
MapReduceTask, as it currently does, will hash task input keys and group
them by execution node N they are hashed to. For each node N and its
grouped input KIn keys MapReduceTask creates a MapCombineCommand which
is migrated to an execution target node N. MapCombineCommand is similar
to current MapReduceCommand. MapCombineCommand takes an instance of a
Mapper and an instance of a Reducer, which is a combiner [1].
Once loaded into target execution node MapCombineCommand takes each
local KIn key and executes Mapper method void map(KIn key, VIn value,
Collector<KOut, VOut> collector). Results are collected to a common
Collector<KOut, VOut> collector and combine phase is initiated. A
Combiner, if specified, takes KOut keys and imediatelly invokes reduce
phase on keys. The result of mapping phase executed on each node is
<KOut, VOut> map M. There will be one resulting M map per execution node N.
At the end of combine phase instead of returning map M to the master
task node (as we currently do), we now hash each KOut in map M and group
KOut keys by the execution node N they are hashed to. Each group of KOut
keys and its VOut values, hashed to the same node, is wrapped with a new
command Migrate. Command Migrate, which is very similar to
PutKeyValueCommand,executed on Infinispan target node N esentially
maintains KOut K -> List<VOut> mapping, i.e all KOut/VOut pairs from all
executed MapCombineCommands will be collocated on a node N where KOut is
hashed to and value for KOut will be a list of all VOut values. We
essentially collect all VOut values under each KOut for all executed
MapCombineCommands.
At this point MapCombineCommand has finished its execution; list of KOut
keys is returned to a master node and its MapReduceTask. We do not
return VOut values as we do not need them at master task node.
MapReduceTask is ready to start with reduce phase.
Reduce phase
MapReduceTask initializes ReduceCommand with a user specified Reducer.
For each key KOut collected from a map phase we group them by execution
node N they are hashed to. For each node N and its grouped input KOut
keys MapReduceTask creates a ReduceCommand and sends it to a node N
where KOut keys are hashed. Once loaded on target execution node,
ReduceCommand for each KOut key grabs list of values VOut and invokes:
VOut reduce(KOut reducedKey, Iterator<VOut> iter).
A result of ReduceCommand is a map M where each key is KOut and value is
VOut. Each Infinispan execution node N returns one map M where each key
KOut is hashed to N and each VOut is KOut's reduced value.
When all ReduceCommands return to a calling node, MapReduceTask simply
combines all these M maps and returns final Map<KOut, VOut> as a result
of MapReduceTask. All intermediate KOut->List<VOut> maps left on
Infinispan cluster are then cleaned up.
[1] See section 4.3 of http://research.google.com/archive/mapreduce.html
12 years, 6 months
Time for a new NBST thread
by Dan Berindei
Hi guys
I published a new version of the non-blocking state transfer design
document here:
https://community.jboss.org/wiki/Non-blockingStateTransfer
This time I focused a bit on the "other" cache configuration (other
than dist-sync, that is). This revealed some new problems, these are
the most interesting ones:
* With async-commit phase or async-1pc, we can get stale locks on
joiners. I have an Infinispan-only approach that should work, but it
may be easier to just require FLUSH.
* Async-commit + recovery enabled: a node sends a commit command to 2
owners, one node loses the message, and then the originator dies
before retransmitting. How do we let the TM know that the transaction
is in-doubt, since to the TM it looks like it has been committed?
Please try to read the document and let me know if anything sounds too
opaque or doesn't make sense. I think it's good enough for starting
the implementation, although I've removed the actual implementation
details from the document for now.
Cheers
Dan
12 years, 6 months
Let's stop with pull requests
by Sanne Grinovero
guys, please don't take me as the one who is again complaining about
failing tests; I'm having doubts about the development process and the
amount of time this is wasting on all of us.
We're all humans and do mistakes, still it happens so extremely often
that this is getting systemic, and discipline could definitely be
improved: people regularly send pull requests with failing tests or
broken code, and very regularly this is just merged in master.
I did it myself a couple of days ago: didn't notice a failure, all
looked good, sent a pull, it was merged with no complaints. Three days
later, I resume my work and am appalled to see that it was broken. Now
fixing it, but I'll have to send another pull and wait for it - which
feels very pointless, as I'm pretty sure nobody is checking anyway.
It looks like as the pull request procedure is having this effect:
# patch writer is not as carefull as he used to be: "someone else
will check if it's fine or not. I have no time to run the tests
again..".
# reviewer has as quick look. "Looks good - in fact I don't care
much, it's not my code and need to return to my own issues.. worst
case someone else will fix it blaming the original author"
And then again some incomplete test makes it to master, or a patch
which doesn't even compile is integrated.
This pull request process is being a big failure. Shall we stop
wasting time on it and just push on master?
Which doesn't mean I'm suggesting "let's make it worse" | "unleash
hell": we should all take responsibility on any change very seriously.
Again, I'm not enjoying the role of "whom who complains on the
testsuite again". Just stating a fact, and trying to propose something
to make it work better. We have great individuals on this team, but we
need to admit that team work isn't working and we should deal with it
at it's best; denying it won't help.
Cheers,
Sanne
12 years, 6 months
Prepare and Rollback messages order
by Sebastiano Peluso
Hi all,
I have a question about the messages order guaranteed in the execution
of the 2PC for committing a transaction in Infinispan (all versions).
In particular I have noticed that both in replication and distribution
mode the RollbackCommand command is multicast by using a priority queue.
This means that:
1) In replication mode, the visitRollbackCommand method of the
ReplicationInterceptor class invokes the broadcastRpcCommand method on
the RpcManager by using a true value for the usePriorityQueue parameter;
2) In distribution mode, the visitRollbackCommand method of the
DistributionInterceptor class invokes the invokeRemotely method on the
RpcManager by using a true value for the usePriorityQueue parameter.
Therefore the JGroups' packets containing the RollbackCommand commands
will be marked with the org.jgroups.Message.OOB and will be sent and
received by means of the Jgroups' out-of-band (OOB) thread pool.
If I'm not wrong until now, I deduce that since OBB threads don't
guarantee ordering, then it can happen that a message M2 multicast by
node N1 after it has multicast message M1, can be delivered to a node N2
before message M1. So the sending order is M1 -> M2 and the delivery
order is M2 -> M1 [1].
If this is true as well, I can have runs in which the PreparedCommand
command of a transaction T involved in the execution of a 2PC, can be
received by a 2PC's participant node after a RollbackCommand command for
T, even if the JGroups' messages containing the PrepareCommand commands
are actually managed by means of the JGroups's regular thread pool.
For instance, we can consider a scenario in which a transaction T
requests a commit on node N0 and N0 multicasts the T's prepare message
on a large set of nodes (e.g. 100 nodes) S={N1,...,N100}. If the lock
acquisition fails on at least one of the 100 nodes, N0 can multicasts a
rollback message without waiting for the reply of the prepare from all
the nodes in S. This means that the last rollback message can be
delivered to a node in S, say N100, before the T's prepare message and
that the locks successfully acquired by N100 for T will not be released
anymore.
Am I missing anything?
Thanks.
Best Regards,
Sebastiano Peluso
[1] http://www.jgroups.org/manual/html/user-advanced.html
--
Sebastiano Peluso, PhD student
Department of Computer, Control and Management Engineering
Sapienza University of Rome
Via Ariosto 25, 00185, Rome, Italy
Tel. +39 06 77274108
Webpage http://www.dis.uniroma1.it/~peluso
Distributed Systems Group, INESC-ID
Instituto Superior Técnico
Rua Alves Redol 9, 1000-029, Lisbon, Portugal
Webpage http://www.gsd.inesc-id.pt/~peluso
12 years, 6 months
Welcome (back) Israel
by Sanne Grinovero
Hi All,
Israel Lacerra contributed the super-interesting Distributed Query
functionality to Infinispan last year during is graduation work; he
now returned with some very nice patches to improve his initial
contribution.
His code was already quite good and properly tested, but he also
quickly addressed all my detailed niggling and style requests during
the pull review: great job!
I've added him to the "Developers" group of JIRA, so we can now assign
him more issues as a reward for his cool contributions :-P
Israel, would you like to take another issue related to your field of
expertise? I was thinking about ISPN-1588: if you would like to look
at it, please assign the issue to yourself. I think you might already
have resolved it while doing ISPN-1980.
Cheers,
Sanne
12 years, 7 months
Map/Reduce or other batch processing on CacheLoader stored entries
by Sanne Grinovero
Hi all,
ideally I would like to build the MassIndexer for Infinispan Query on
top of Map/Reduce as it seems it should provide all what I need "out
of the box", but technically I'd need it to support some more
features:
1) Some resource injection in the Mapper / Reducer. I know Vladimir
has been working on a CDI approach, is there some similar approach
which could work for me when not having CDI in the environment?
Having a reference to the Cache would be fine - I remember this was
discussed a while back but forgot if some work was done on it.
2) Mapper and Reducer should work taking advantage of multiple cores
even on the same node .. so not just divide & conquer across multiple
nodes but also locally. Was this done already?
3) I need the Map/Reduce to execute also on all entries stored in the
CacheLoader entries. I don't believe that's the case today.. and even
if I wanted to use just the Executor, I believe the CacheLoader API
needs to provide some option to load all keys in a stream form
appropriate for batch processing: using loadAll will likely get me
into an OOM.
Even if we decide to perform such jobs without using Map/Reduce but
any other form, we need to support some form of "process on all
entries", what should we suggest as best way to iterate and perform
modifications to all entries in the data container?
Cheers,
Sanne
12 years, 7 months
