Write Skew issue (versioning)
by Pedro Ruivo
Hi,
I think I have spotted a problem with the write skew check
implementation based on versioning.
I've made this test to confirm:
I have a global counter that is incremented concurrently by two
different nodes, running ISPN with Repeatable Read with write skew
enabled. I expected that each successfully transaction will commit a
different value.
In detail, each node do the following:
beginTx
Integer count = cache.get("counter");
count = count + 1;
cache.put("counter", count)
commitTx
To avoid errors, I've run this test on two ISPN versions: 5.1.0.CR4 and
5.0.1.Final. In 5.0.1.Final, it works as expected. However, on 5.1.0.CR4
I have a lot of repeated values. After a first check at the code, I've
the impression that the problem may be due to that the version numbers
of the keys for which the write skew check should be run is not sent
with the prepare command.
Cheers,
Pedro Ruivo
12 years, 11 months
very interesting performance results
by Mircea Markus
Hi,
As discussed on another thread, I've hacked the remote get to only go to a single node. The performance of put almost doubled, whilst the one of gets decreased by about 10 percent.
The performance decrease for gets is understandable, but I can't really understand why the write got so very fast. The OOB thread pool is not shared between the writes and the reads...
That makes me think that there might be a contention point between OOB and regular threads?
Cheers,
Mircea
12 years, 11 months
Re: [infinispan-dev] Looking into OProfile logs
by Manik Surtani
Also, lets please move this discussion to infinispan-dev…
On 25 Jan 2012, at 17:57, Dan Berindei wrote:
> On Wed, Jan 25, 2012 at 6:32 PM, Sanne Grinovero <sanne(a)infinispan.org> wrote:
>> On 25 January 2012 15:56, Dan Berindei <dan.berindei(a)gmail.com> wrote:
>>> On Wed, Jan 25, 2012 at 3:16 PM, Sanne Grinovero <sanne(a)infinispan.org> wrote:
>>>>
>>>> - ComponentRegistry.getComponent # can't we make sure this is not
>>>> needed at runtime, or create direct accessors for the hottest ones,
>>>> like Configuration.class ? I'll make a proposal and measure it.
>>>>
>>>
>>> I had an idea of registering lots of CacheInboundInvocationHandlers in
>>> CommandAwareRpcDispatcher instead of a single global
>>> InboundInvocationHandler but I never implemented it. Are you thinking
>>> along the same lines?
>>
>>
>> No I've been patching CacheRpcCommandExternalizer instead. But please
>> change that one if you have an idea.
>>
>
> Ok, I'll do that.
>
>>
>>>> - DefaultConsistentHash.isKeyLocalToAddress # Should be possible to
>>>> speed up this one
>>>>
>>>
>>> I didn't think of any optimization specific for isKeyLocalToAddress,
>>> but we could precompute the list of owners for each hash wheel
>>> "segment" and store that in the positionValues array instead of a
>>> single address. It would get kind of expensive with huge numbers of
>>> virtual nodes, so it would be nice if we could prevent the users from
>>> using thousands of virtual nodes.
>>>
>>> Address interning could help us somewhat, if we could eliminate the
>>> equals() calls with reference equality checks.
>>
>> Right, but it means that all Address should be created via the same
>> service, including unmarshalled ones.
>> Would be nice doing it, but sounds like dangerous if not doing an
>> extensive refactoring.
>> I'd try something like this by introducing a new type, mandate the
>> type on the API, and do this possibly after changing the Address
>> collections to an ad-hoc Collection as suggested last week; not sure
>> yet how it would look like, but let's evaluate options after the
>> custom collections is in place.
>>
>
> I was actually thinking knowing that a1 != a2 => !a1.equals(a2) would
> enable us to use even more efficient custom collections.
> But I agree that replacing all addresses with interned ones is not an easy task.
>
>>>> - boolean org.infinispan.transaction.xa.GlobalTransaction.equals(java.lang.Object)
>>>> # let's see if we can do something about this.
>>>>
>>>
>>> Checking the address is more expensive than checking the id, we should
>>> check the id first.
>>> Other than that, the only thing we can do is call it less often :)
>>
>> And idea on "less often" ?
>>
>
> Nope, no idea I'm afraid.
>
>>>
>>>> - jni_GetObjectField # would like to know where this is coming from
>>>>
>>>
>>> It looks like it's PlainDatagramSocketImpl.send and receive:
>>>
>>> 6184 0.2442 libnet.so libnet.so
>>> Java_java_net_PlainDatagramSocketImpl_send
>>> 7483 34.3556 libjvm.so libnet.so
>>> jni_GetObjectField
>>>
>>> 3849 0.1520 libnet.so libnet.so
>>> Java_java_net_PlainDatagramSocketImpl_receive0
>>> 8221 34.7773 libjvm.so libnet.so
>>> jni_GetObjectField
>>
>> Right, that's likely. Would like to make sure.
>>
>
> This is certainly a big part of where it's coming from - but perhaps
> there are other places as well.
>
>>
>>> I also have a question, are you using virtual nodes? We should enable
>>> it in our perf tests (say with numVirtualNodes = 10), I suspect it
>>> will make DCH.locate and DCH.isKeyLocalToAddress even slower.
>>
>> We've discussed VNodes a lot on IRC, you should join us.
>> [My tests where without, but have already applied the patch to enable it]
>>
>
> I'm also going to update VNodesCHPerfTest to look closer at key distribution.
>
> Cheers
> Dan
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
12 years, 11 months
State transfer improvements for 5.2?
by Manik Surtani
Dan,
Looking through what we currently have, given the blocking nature of state transfer, I wonder if we can improve on it by pausing the state transfer from time to time to flush waiting transactions? I.e., if state transfer is in progress and a write command or transaction is waiting for it to finish, and since state transfer is chunked anyway, perhaps to let a few transactions through in between transferring chunks?
Cheers
Manik
--
Manik Surtani
manik(a)jboss.org
twitter.com/maniksurtani
Lead, Infinispan
http://www.infinispan.org
12 years, 11 months
What's master?
by Sanne Grinovero
Hello all,
I couldn't find a 5.1 branch, and the current master builds produce
snapshot versions of 5.1.
Should we change this to 5.2.0-SNAPSHOT or are we expecting some work
as 5.1.1-SNAPSHOT ?
12 years, 11 months
Infinispan 5.1.0.FINAL is out!
by Galder Zamarreño
Hi all,
Infinispan 5.1.0.FINAL is finally out! You can read all about it in http://goo.gl/pMoix
A very big thank you to everyone that has been involved during the 5.1 'Brahma' series!
Btw, let me know if I've forgotten something in the blog post.
Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
12 years, 11 months
Re: [infinispan-dev] [jgroups-dev] experiments with NAKACK2
by Bela Ban
Hi Sanne,
(redirected back to infinispan-dev)
> Hello,
> I've run the same Infinispan benchmark mentioned today on the
> Infinispan mailing list, but having the goal to test NAKACK2
> development.
>
> Infinispan 5.1.0 at 2d7c65e with JGroups 3.0.2.Final :
>
> Done 844,952,883 transactional operations in 22.08 minutes using 5.1.0-SNAPSHOT
> 839,810,425 reads and 5,142,458 writes
> Reads / second: 634,028
> Writes/ second: 3,882
>
> Same Infinispan, with JGroups b294965 (and reconfigured for NAKACK2):
>
> Done 807,220,989 transactional operations in 18.15 minutes using 5.1.0-SNAPSHOT
> 804,162,247 reads and 3,058,742 writes
> Reads / second: 738,454
> Writes/ second: 2,808
>
> same versions and configuration, run it again as I was too surprised:
>
> Done 490,928,700 transactional operations in 10.94 minutes using 5.1.0-SNAPSHOT
> 489,488,339 reads and 1,440,361 writes
> Reads / second: 745,521
> Writes/ second: 2,193
>
> So the figures aren't very stable, I might need to run longer tests,
> but there seems to be a trend of this new protocol speeding up Read
> operations at the cost of writes.
This is really strange !
In my own tests with 2 members on the same box (using MPerf), I found
that the blockings on Table.add() and Table.removeMany() were much
smaller than in the previous tests, and now the
TP.TransferQueueBundler.send() method was the culprit #1 by far ! Of
course, still being much smaller than the previous highest blockings !
I'll run Transactional on my own box today, to see the diffs between
various versions of JGroups.
Can you send me your bench.sh ? If I don't change the values, the test
takes forever !
--
Bela Ban
Lead JGroups (http://www.jgroups.org)
JBoss / Red Hat
12 years, 11 months
compiler warning: code cache is full
by Sanne Grinovero
Hi,
I just noticed this warning in my build logs; I don't read them often
so it might have been there for a while:
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @
infinispan-spring ---
[INFO] Compiling 22 source files to
/home/sanne/workspaces/infinispan/infinispan/spring/target/classes
OpenJDK 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
OpenJDK 64-Bit Server VM warning: Try increasing the code cache size
using -XX:ReservedCodeCacheSize=
Code Cache [0x00007fb8da000000, 0x00007fb8dff90000, 0x00007fb8e0000000)
total_blobs=34047 nmethods=33597 adapters=367 free_code_cache=1036Kb
largest_free_block=511552
[INFO]
[INFO] >>> exec-maven-plugin:1.2.1:java (serialize_component_metadata)
@ infinispan-spring >>>
I've never seen such a warning before, anything to be concerned about?
Cheers,
Sanne
12 years, 11 months
moving asyncMarshalling to jgroups
by Mircea Markus
Hi Bela,
ATM when asyncMarshalling is enabled[1], we use our own thread pool for a) serializing the object and then b) pass it to the jgroups transport to send it async.
As per [1], this has a major limitation: requests might be re-ordered at infinispan's level, in the thread pool that handles serialization.
A possible way to improve this is by sending both the marshaller and the object to the jgroups transport which would serialize it on the same thread used for sending it. This way we would avoid re-ordering and potentially reduce thread's context switching.
Wdyt?
[1] http://bit.ly/yNk6In
12 years, 11 months
