Re: [infinispan-dev] Removing Infinispan dependency on the Hibernate-Infinispan module in 4.x
by Galder Zamarreño
Scott, what do you suggest doing instead then? Without the commands, evictAll invalidation won't work.
Are you suggesting that I revert back to using the cache as a notification bus so that regions are invalidated?
On Feb 8, 2012, at 4:13 PM, Scott Marlow wrote:
> http://lists.jboss.org/pipermail/infinispan-dev/2012-February/010125.html has more context.
>
> Since there are no easy/quick fixes that can be applied at this time, to remove the AS7 Infinispan dependency on the Hibernate-Infinispan module, I think we should avoid depending on the service loader way to supply the custom commands (in the Hibernate-Infinispan module), at least until this can be addressed elsewhere.
>
> I propose that the Hibernate-Infinispan second level cache should not use the Service Loader to pass custom commands into Infinispan. If we agree, I'll create a jira for this.
>
> Scott
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
12 years, 5 months
Map Reduce 2.0
by Vladimir Blagojevic
Hey guys,
Before moving forward with next iteration of map reduce I wanted to hear
your thoughts about the following proposal. After we agree on the
general direction I will transcribe the agreed design on a wiki page and
start implementation.
Shortcoming of current map reduce implementation
While our current map reduce implementation is more than a proof of a
concept there are several drawbacks preventing it from being an
industrial grade map reduce solution. The main drawback is the inability
of the current solution to deal with a large data (in GB/TB) map reduce
problems. This shortcoming is mainly around our reduce phase execution.
Reduce phase, as you might know, is currently done on a single
Infinispan master task node; reduce phase of map reduce problems we can
support (data size wise) is therefore shrunk to a working memory of a
single node.
Proposed solution
The proposed solution involves distributing execution of reduce phase
tasks across the cluster thus effectively achieving higher reduce task
parallelization and at the same time removing the above mentioned reduce
phase restriction. Effectively leveraging our consistent hashing
solution even further we can parallelize reduce phase and elevate our
map reduce solution to an industrial level. Here is how we can achieve that.
Map phase
MapReduceTask, as it currently does, will hash task input keys and group
them by execution node N they are hashed to. For each node N and its
grouped input KIn keys MapReduceTask creates a MapCombineCommand which
is migrated to an execution target node N. MapCombineCommand is similar
to current MapReduceCommand. MapCombineCommand takes an instance of a
Mapper and an instance of a Reducer, which is a combiner [1].
Once loaded into target execution node MapCombineCommand takes each
local KIn key and executes Mapper method void map(KIn key, VIn value,
Collector<KOut, VOut> collector). Results are collected to a common
Collector<KOut, VOut> collector and combine phase is initiated. A
Combiner, if specified, takes KOut keys and imediatelly invokes reduce
phase on keys. The result of mapping phase executed on each node is
<KOut, VOut> map M. There will be one resulting M map per execution node N.
At the end of combine phase instead of returning map M to the master
task node (as we currently do), we now hash each KOut in map M and group
KOut keys by the execution node N they are hashed to. Each group of KOut
keys and its VOut values, hashed to the same node, is wrapped with a new
command Migrate. Command Migrate, which is very similar to
PutKeyValueCommand,executed on Infinispan target node N esentially
maintains KOut K -> List<VOut> mapping, i.e all KOut/VOut pairs from all
executed MapCombineCommands will be collocated on a node N where KOut is
hashed to and value for KOut will be a list of all VOut values. We
essentially collect all VOut values under each KOut for all executed
MapCombineCommands.
At this point MapCombineCommand has finished its execution; list of KOut
keys is returned to a master node and its MapReduceTask. We do not
return VOut values as we do not need them at master task node.
MapReduceTask is ready to start with reduce phase.
Reduce phase
MapReduceTask initializes ReduceCommand with a user specified Reducer.
For each key KOut collected from a map phase we group them by execution
node N they are hashed to. For each node N and its grouped input KOut
keys MapReduceTask creates a ReduceCommand and sends it to a node N
where KOut keys are hashed. Once loaded on target execution node,
ReduceCommand for each KOut key grabs list of values VOut and invokes:
VOut reduce(KOut reducedKey, Iterator<VOut> iter).
A result of ReduceCommand is a map M where each key is KOut and value is
VOut. Each Infinispan execution node N returns one map M where each key
KOut is hashed to N and each VOut is KOut's reduced value.
When all ReduceCommands return to a calling node, MapReduceTask simply
combines all these M maps and returns final Map<KOut, VOut> as a result
of MapReduceTask. All intermediate KOut->List<VOut> maps left on
Infinispan cluster are then cleaned up.
[1] See section 4.3 of http://research.google.com/archive/mapreduce.html
13 years, 5 months
Help on ticket allocator project
by shashank sharma
As per the infinispan web page, there is a lab exercise of ticket allocator
given.Can you please tell me whether er have to develop over that
infrastructure and build our application using that.
13 years, 7 months
Undesired behavior
by Pedro Ruivo
Hi everybody,
The problem is the following: I have some cases where the
RollbackCommand is processed before the correspondent PrepareCommand,
leading to locks to be acquired and never unlocked again.
I am in a scenario where the dataset of each node is disjoint (no
contention) but I have large transactions. When a transaction aborts, I
execute it again (so it has the same keys modified) and try to commit
it. Initially, the transaction aborts due to timeout waiting responses
(that is 2 min and 30 sec), but after, I see aborts due to lock
acquisition timeout. The final result is that all the nodes in the
system finishes except this one.
Is there anyway to release the locks for previously finished transactions?
Cheers,
Pedro Ruivo
13 years, 7 months
Re: [infinispan-dev] Any CDI experts?
by Vladimir Blagojevic
Thanks Stuart and Bruno! Stuart, your suggestion works indeed!! Pete had
some recommendations as well and it now seems like this issue has been
resolved!
Much appreciated everyone,
Vladimir
On 12-03-28 9:48 PM, Stuart Douglas wrote:
> This issue is in org.infinispan.cdi.CDIDistributedTaskLifecycle
>
> This class is re-used for both tests, however it only gets the current
> bean manager once at creation time, so the second test is run against
> the first tests (closed) bean manager.
>
> Using InfinispanExtension.getExtension().getRegisteredBeanManager() to
> get the bean manager each time allows the tests to pass.
>
> Stuart
>
>
>
> On 29/03/2012, at 7:56 AM, Bruno Georges wrote:
>
>> adding Stuart
>> On 29 Mar 2012, at 08:23, Vladimir Blagojevic wrote:
>>
>>> Hi,
>>>
>>> I think I have a decent solution for testing of distributed
>>> execution and CDI in the place now. Galder helped out a lot!
>>> However, I still can not bridge the last remaining mystery: I have
>>> two tests WordCountMapReduceCDITest and DistributedExecutorCDITest.
>>> When both tests are enabled WordCountMapReduceCDITest fails as it is
>>> run second. If only either one of the tests is enabled - everything
>>> works as expected.
>>>
>>> Pick up pull request at
>>> https://github.com/infinispan/infinispan/pull/982 , first maven
>>> install core and then proceed to cdi/extension to give test suite a
>>> run. You can see the root of the failure in surefire reports. Weld
>>> somehow cannot resolve bean passed to InjectionTarget inject method.
>>>
>>> Much appreciated,
>>> Vladimir
>>>
>>> Caused by: org.jboss.weld.exceptions.IllegalArgumentException: WELD-001324 Argument bean must not be null
>>> at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:657)
>>> at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:118)
>>> at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:703)
>>> at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:712)
>>> at org.jboss.weld.manager.SimpleInjectionTarget$1.proceed(SimpleInjectionTarget.java:106)
>>> at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48)
>>> at org.jboss.weld.manager.SimpleInjectionTarget.inject(SimpleInjectionTarget.java:102)
>>> at org.infinispan.cdi.CDIDistributedTaskLifecycle.onPreExecute(CDIDistributedTaskLifecycle.java:46)
>>> at org.infinispan.distexec.spi.DistributedTaskLifecycleService.onPreExecute(DistributedTaskLifecycleService.java:80)
>>> at org.infinispan.commands.read.MapReduceCommand.perform(MapReduceCommand.java:144)
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>
13 years, 7 months
Any CDI experts?
by Vladimir Blagojevic
Hi,
I think I have a decent solution for testing of distributed execution
and CDI in the place now. Galder helped out a lot! However, I still can
not bridge the last remaining mystery: I have two tests
WordCountMapReduceCDITest and DistributedExecutorCDITest. When both
tests are enabled WordCountMapReduceCDITest fails as it is run second.
If only either one of the tests is enabled - everything works as expected.
Pick up pull request at
https://github.com/infinispan/infinispan/pull/982 , first maven install
core and then proceed to cdi/extension to give test suite a run. You can
see the root of the failure in surefire reports. Weld somehow cannot
resolve bean passed to InjectionTarget inject method.
Much appreciated,
Vladimir
Caused by: org.jboss.weld.exceptions.IllegalArgumentException: WELD-001324 Argument bean must not be null
at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:657)
at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:118)
at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:703)
at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:712)
at org.jboss.weld.manager.SimpleInjectionTarget$1.proceed(SimpleInjectionTarget.java:106)
at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48)
at org.jboss.weld.manager.SimpleInjectionTarget.inject(SimpleInjectionTarget.java:102)
at org.infinispan.cdi.CDIDistributedTaskLifecycle.onPreExecute(CDIDistributedTaskLifecycle.java:46)
at org.infinispan.distexec.spi.DistributedTaskLifecycleService.onPreExecute(DistributedTaskLifecycleService.java:80)
at org.infinispan.commands.read.MapReduceCommand.perform(MapReduceCommand.java:144)
13 years, 7 months
