Are user classes/objects stored as MarshalledValues in AS?
by Vladimir Blagojevic
Galder and others,
Need some help! I am looking at
https://issues.jboss.org/browse/ISPN-2138 and trying to understand what
is going on! I tried example in ISPN-2138 and I confirmed the findings.
It must be that TestEntry instances put in cache are stored as
MarshalledValues! If I try the same example with Integer - Map/Reduce in
AS works fine. Mind you this is release 5.1.2 and MapReduceCommand used
to be executed at end of interceptor chain loading key/value pairs like
this:
for (Object key : keys) {
GetKeyValueCommand command =
commandsFactory.buildGetKeyValueCommand(key,
ctx.getFlags());
command.setReturnCacheEntry(false);
Object value = invoker.invoke(ctx, command);
mapper.map(key, value, collector);
}
then I am suspecting MarshalledValue is not unwrapped somehow to user
class TestEntry before being passed to Mapper! This is not how we do
things in 5.2.0Alpha2 - we use cache directly but I am unable to test
that yet in latest AS - still fighting with a build.
Regards,
Vladimir
11 years, 9 months
Infinispan 5.2.0.ALPHA2 is out!
by Galder Zamarreño
This was released last Friday but due to some glitches, I was only about to get around to blogging about it now: http://goo.gl/lX4Tq
Thanks to everyone in their involvement, particularly Vladimir and Adrian :)
Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
11 years, 9 months
Re: [infinispan-dev] Master broken :(
by Vladimir Blagojevic
Master is broken. Wait for Galder to integrate
https://github.com/infinispan/infinispan/pull/1219 before integrating
anything else.
On 12-07-19 1:55 PM, Galder Zamarreño wrote:
> Hey,
>
> Master is broken:
>
> [ERROR] /Users/g/Go/upstream/infinispan.git/cdi/extension/src/test/java/org/infinispan/cdi/test/distexec/WordCountMapReduceCDITest.java:[71,68] cannot find symbol
> [ERROR] symbol : method invokeMapReduce(java.lang.String[],org.infinispan.cdi.test.distexec.WordCountMapReduceCDITest.WordCountMapper,org.infinispan.cdi.test.distexec.WordCountMapReduceCDITest.WordCountReducer)
> [ERROR] location: class org.infinispan.distexec.mapreduce.BaseWordCountMapReduceTest
>
> I've tried to patch WordCountMapReduceCDITest so that it'd call with useCombiner=false or useCombiner=true, but the test fails.
>
> Can you please sort it out asap?
>
> Cheers,
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
>
11 years, 9 months
Map/Reduce enhancements with ISPN-2037
by Vladimir Blagojevic
Guys,
I believe I finished ISPN-2037 that Sanne needed for his queries.
However, adding ISPN-2037 commit to current master involves injection of
CacheLoaderManager into MapReduceCommand. That solution is pretty ugly
and we are switching soon to a new map/reduce implementation anyway.
Going in that direction is a waste IMHO.
I already talked to Mircea and Manik about using a Manager-like service
much like RecoveryManager or EvictionManager to manage incoming
map/reduce invocations on Infinispan node. MapReduceManager is per cache
component. It turned out to be an excellent fit! I converted our current
solution where map is done in distributed fashion while reduction is
done on master task node to this new approach. I made it a default just
as it is now. Then I added all the previous work on new map/reduce
reviewed by Dan and Galder and adjusted it to this new Manager paradigm.
Users can choose if they want reduce part done in distributed approach
simply by using a different MapReduceTask constructor. And finally I
added ISPN-2037 as a separate commit on top of this rework.
This proposal is at
https://github.com/vblagoje/infinispan/commits/t_mr_final
I would like to hear your thoughts and comments before proceeding forward.
Regards,
Vladimir
11 years, 9 months
AbstractGlobalConfigurationBuilder.read(template) - is it needed?
by Mircea Markus
Hi Pete,
Whilst the 'read(template)' method does make sense in the case of cache level configurations(AbstractConfigurationChildBuilder) in order to override the values from the default cache, is 'read(template)' in the global configuration also needed(AbstractGlobalConfigurationBuilder)? We don't use any override mechanism for global configuration; it also doesn't seem to be used anywhere in the code.
Cheers,
Mircea
11 years, 9 months
Fwd: [Infinispan] - Re: Global/Normalized timestamp generator
by Galder Zamarreño
Do we really need this? Maybe a nice to have (and to tick all boxes), but have not really seen demand for this.
Begin forwarded message:
> From: Guillermo GARCIA OCHOA <do-not-reply(a)jboss.com>
> Subject: [Infinispan] - Re: Global/Normalized timestamp generator
> Date: July 11, 2012 10:57:38 AM GMT+03:00
> To: Galder Zamarreño <galder.zamarreno(a)redhat.com>
>
> JBoss Community
> Re: Global/Normalized timestamp generator
> created by Guillermo GARCIA OCHOA in Infinispan - View the full discussion
> I strongly disagree with you ...
>
>
> A cluster should have a normalized timestamps generator for many use cases.
>
>
> One example
>
> Message producer/consumer: Almost all the implementations of this kind of patterns needs a normalized timestamps to know the order of processing of the distributed queue (this means several share-nothing queues).
>
>
> One real case scenario:
>
> Transaction processor: you have several servers handling business transactions, but to be faster do you not centralize all these transaction in the same queue, but in several ones. Then you can consume those queues to really process this sorted by 'creation timestamps'.
>
>
> I know there is a lot of way to do this, but timestamps are widely used in distributed environments. Please check this:
>
>
> A plugin for ActiveMQ http://activemq.apache.org/timestampplugin.html
> In Oracle Coherence you can get this normalized timestamps with CacheFactory.ensureCluster().getTimeMillis()
>
> Dont you think that this is important enough to be added to the roadmap? Is not blocker, but I think that this is a 'must have' feature.
>
> Reply to this message by going to Community
> Start a new discussion in Infinispan at Community
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
11 years, 9 months
Providing a context for object de-serialization
by Sanne Grinovero
Imagine I have a value object which needs to be stored in Infinispan:
class Person {
final String nationality = ...
final String fullName = ...
[constructor]
}
And now let's assume that - as you could expect - most Person
instances have the same value for the nationality String, but a
different name.
I want to define a custom Externalizer for my type, but the current
Externalizer API doesn't allow to refer to some common application
context, which might be extremely useful to deserialize this Person
instance:
we could avoid filling the memory of my Grid by having multiple copies
of the nationality String repeated all over, when a String [1] could
be reused.
Would it be a good idea to have the Externalizer instances have an
initialization phase receiving a ComponentRegistry, so I could look up
some custom service to de-duplicate or otherwise optimize my in-memory
data representation?
Personally I'd prefer to receive it injected via the constructor so
that I could use a final field when my custom Externalizer is
constructed.
This is OGM related.
Cheers,
Sanne
1 - or any immutable object: I'm using String as an example so let's
forget about the static String pool optimizations the JVM might
enable..
11 years, 9 months
On HotRod .Net Client packaging
by Mircea Markus
Hi guys,
Sunimal is about to finish a level 1 C# client implementation so we're thinking to release a first ALPHA to gather some feedback.
After doing a bit of research and asking around here are some thoughts about the release:
- the client is released into a .zip file uploaded in github, using the same pattern as the python client[1]
- the zip contains
- bin (dir): hotrodclient.dll(compiled in release mode), nlog.dll (logger dependency) and hotrodclient.pdb[2] (needed for debugging)
- config(dir): hotrodclient.xml (sample/default configuration file), NLog.xml (logger config) [3]
- apidocs: xml file(s) containing API docs[4]
- sample: sample application using the client (sources) + sample.html (describes the sample application)
- readme.txt: describes the required .NET version, link to the user-guide page[5], and to the source code/github
Nice to have/ consider for next releases:
- nuget[6] a sort of maven/dependency manager for Visual Studio
- an MS installer[7] instead of the zip would be nicer but I think a zip would do for now
Any suggestion welcomed! I know Dan has been on the dark side, does this look okay to you ? :-)
Cheers,
Mircea
[1] https://github.com/infinispan/python-client/archives/1.0.0b1
[2] http://msdn.microsoft.com/en-us/library/yd4f8bd1(v=vs.71).aspx
[3] in the next version we'll probably switch to Common.Logging (abstract logging interface): http://netcommon.sourceforge.net/
[4] note for Sunimal: this can be generated in Visual Studio 2010 by checking "xml documentation" in "Project properties"
[5] new page/entry added here: https://docs.jboss.org/author/display/ISPN/User+Guide
[6] http://visualstudiogallery.msdn.microsoft.com/27077b70-9dad-4c64-adcf-c7c...
[7] http://en.wikipedia.org/wiki/Windows_Installer
11 years, 9 months
Re: [infinispan-dev] Optimizing tx for DIST from user perspective
by Mircea Markus
On 06/07/2012 17:12, Vladimir Blagojevic wrote:
> Mircea,
>
> Privately to you as I am not sure this makes sense for wider
> distribution - yet.
Adding infinispan-dev to this as I think it's interesting for a wider
audience.
>
> Say user has bunch of keys/values to insert into cache. He could do it
> one key/value at the time, all in one tx or in tx batches. If he wants
> to do it in batches of transactions then it would make sense to group
> keys by the primary Address assigned on hashwheel.
Very interesting point. Besides the locking, grouping keys has another
significant advantage : during the prepare phase each node receives the
complete list of modifications in that transaction and not only the
modification pertaining to it.
E.g. say we have the following key->node mapping:
k1 -> A
k2 -> B
k3 -> C
Where k1, k2 and k3 are keys; A, B and C are nodes.
If Tx1 writes (k1,k2,k3) then during the prepare A,B and C will receive
the the same package containing all the modification - namely (k1,
k2,k3). There are several reasons for doing this (apparently)
unoptimized approach: serialize the prepare only once, better handling
of recovery information.
Now if you group transactions/batches base on key distribution, as you
suggested, the amount of redundant traffic is significantly reduced -
and that translates in better performance especially when the datasets
you're inserting is quite high.
> Therefore each tx batch would lock keys only on primary node and
> nowhere else - call it tx node pinning if you want! Now imagine a
> cluster with bunch of concurrent txs initiated from all nodes. If I am
> not mistaken this tx pinning algorithm would not only increase
> throughput but also minimize deadlocks.
yes. With optimistic tx caches, the only possibility for deadlocks is
between transactions touching multiple nodes[1]. As long as your
transactions only write to the same node, even if they do it on the same
key-set, the possibility of deadlock is (almost[2]) zero.
> Does this make sense? If so, why not support it somehow on API level
> or do we already? ;-)
We don't have a service like this for now. I think your best option is
to fetch the CH from the advanced cache
(cache.getAdvancedCache().getDistributionManager().getConistentHahs())
and use it to group the inserts.
Thinking about it, might be worth having an blog entry describing this
as it can really boost performance when you need to load an initial
large set of data in Infinispan.
> Regards,
> Vladimir
[1] this DLD situation will be fixed once we have incremental locking in
place: https://issues.jboss.org/browse/ISPN-1219
[2] we use key's CH value to induce an order over the keys written in a
transaction - that's in order to avoid deadlocks. If there are
collisions between these values then there's still a chance for deadlock.
11 years, 9 months
