[JBoss JIRA] (ISPN-4993) Expose node and clustered cache statistics to DMR
by Vladimir Blagojevic (JIRA)
Vladimir Blagojevic created ISPN-4993:
-----------------------------------------
Summary: Expose node and clustered cache statistics to DMR
Key: ISPN-4993
URL: https://issues.jboss.org/browse/ISPN-4993
Project: Infinispan
Issue Type: Task
Components: JMX, reporting and management
Reporter: Vladimir Blagojevic
Assignee: Vladimir Blagojevic
As we implement ISPN-4991 and ISPN-4992 we should also expose these statistics to DMR so they can be used in Infinispan admin console.
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 4 months
[JBoss JIRA] (ISPN-4992) Implement node statistics
by Vladimir Blagojevic (JIRA)
Vladimir Blagojevic created ISPN-4992:
-----------------------------------------
Summary: Implement node statistics
Key: ISPN-4992
URL: https://issues.jboss.org/browse/ISPN-4992
Project: Infinispan
Issue Type: Task
Components: JMX, reporting and management
Reporter: Vladimir Blagojevic
Assignee: Vladimir Blagojevic
We need to implement node statistics aggregating statistics of all caches on that particular node. Implementing class should be an MBean as well.
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 4 months
[JBoss JIRA] (ISPN-4991) Implement clustered cache statistics
by Vladimir Blagojevic (JIRA)
Vladimir Blagojevic created ISPN-4991:
-----------------------------------------
Summary: Implement clustered cache statistics
Key: ISPN-4991
URL: https://issues.jboss.org/browse/ISPN-4991
Project: Infinispan
Issue Type: Task
Components: JMX, reporting and management
Reporter: Vladimir Blagojevic
Assignee: Vladimir Blagojevic
As of 7.0.0 release we implement cache statistics on a per node cache level. For Infinispan admin console we need to implement aggregate statistics for each cache across all nodes in the cluster. The implementing class should be a registered MBean and should implement similar cache statistics currently implemented by org.infinispan.interceptors.CacheMgmtInterceptor
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 4 months
[JBoss JIRA] (ISPN-4887) Stale locks in non-tx cache after merge
by Alexandre Nikolov (JIRA)
[ https://issues.jboss.org/browse/ISPN-4887?page=com.atlassian.jira.plugin.... ]
Alexandre Nikolov commented on ISPN-4887:
-----------------------------------------
I can reproduce the same exception in the following situation:
First thread calls cache.addAll(map) where map contain about 20-30 000 entries.
At the same time another thread tries to perform Map/Reduce on the same cache. As a result the Map/Reduce operation hangs infinitely and the execution thread consumes a lot of CPU. I have logged this as a separate issue, but probably it is the same thing: ISPN-4964
> Stale locks in non-tx cache after merge
> ---------------------------------------
>
> Key: ISPN-4887
> URL: https://issues.jboss.org/browse/ISPN-4887
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 7.0.0.CR2
> Reporter: Dan Berindei
> Priority: Blocker
> Labels: testsuite_stability
> Fix For: 7.1.0.Alpha1
>
>
> It appears to cause random failures in {{ThreeNodesReplicatedSplitAndMergeTest.testSplitAndMerge0}}:
> {noformat}
> org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 10 seconds for key MagicKey#null{74043ff7@ThreeNodesReplicatedSplitAndMergeTest-NodeC-12710/9} and requestor Thread[testng-ThreeNodesReplicatedSplitAndMergeTest,5,main]. Lock is held by Thread[remote-thread-ThreeNodesReplicatedSplitAndMergeTest-NodeC-p5654-t5,5,main], while request came from null
> at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:198)
> at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:181)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:127)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:123)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:47)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:172)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:95)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:148)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:134)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.partionhandling.impl.PartitionHandlingInterceptor.visitPutKeyValueCommand(PartitionHandlingInterceptor.java:34)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:33)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1512)
> at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:990)
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:982)
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1582)
> at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:235)
> at org.infinispan.partitionhandling.ThreeNodesReplicatedSplitAndMergeTest.testSplitAndMerge(ThreeNodesReplicatedSplitAndMergeTest.java:132)
> at org.infinispan.partitionhandling.ThreeNodesReplicatedSplitAndMergeTest.testSplitAndMerge0(ThreeNodesReplicatedSplitAndMergeTest.java:27)
> {noformat}
> http://ci.infinispan.org/viewLog.html?buildId=13499&tab=buildResultsDiv&b...
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 4 months
[JBoss JIRA] (ISPN-4979) CacheStatusResponse map uses too much memory
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-4979?page=com.atlassian.jira.plugin.... ]
William Burns edited comment on ISPN-4979 at 11/18/14 12:38 PM:
----------------------------------------------------------------
-Looking closer the simple serialization change won't actually work it appears that the consistent hashes stored all have different instances of JGroupsAddress so they don't have instance equality which is needed for serialization reuse checks. I will also need to check why these are not the same.-
This was caused by the UPDATE_CH command sending alone the CH and it wasn't properly serializing the addresses, so that just needed a tweak as well and it has reduced my very simple case to half usage of memory. Note that this is not indicative of each case as the number of caches, segments & owners increase the effect will become larger and larger. If I can find a way to merge the Response info between then this would also scale better with the number of nodes.
was (Author: william.burns):
Looking closer the simple serialization change won't actually work it appears that the consistent hashes stored all have different instances of JGroupsAddress so they don't have instance equality which is needed for serialization reuse checks. I will also need to check why these are not the same.
> CacheStatusResponse map uses too much memory
> --------------------------------------------
>
> Key: ISPN-4979
> URL: https://issues.jboss.org/browse/ISPN-4979
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.0.0.Final
> Reporter: Dan Berindei
> Assignee: William Burns
> Priority: Critical
> Fix For: 7.1.0.Final
>
>
> When the cluster is large and there are a log of caches, the {{CacheStatusResponse}} map on the new coordinator can get quite large. One of the problems that seems to be that the addresses in {{DefaultConsistentHash}} are duplicated on serialization, so the deserialized version occupies more memory.
> We need to investigate why the objects are not "shared" by the River marshaller, and maybe work around the problem by de-duplicating the addresses in the externalizer.
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 4 months
[JBoss JIRA] (ISPN-4444) After state transfer, a node is able to read keys it no longer owns from its data container
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4444?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-4444:
------------------------------------
Yes, we want to fix both cases.
> After state transfer, a node is able to read keys it no longer owns from its data container
> -------------------------------------------------------------------------------------------
>
> Key: ISPN-4444
> URL: https://issues.jboss.org/browse/ISPN-4444
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.0.0.Alpha4
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Critical
> Fix For: 7.1.0.Alpha1
>
>
> When state transfer ends and each node receives a CH_UPDATE command from the coordinator, it first installs the new topology and then it starts invalidating entries it no longer owns.
> However, there are two cases when the node can still read its stale values:
> 1. If L1 is enabled, it will look in the local DataContainer first, regardless of the key's location.
> 2. If L1 is disabled, but the key was removed on the new owners, the node will still look up the key in the local DataContainer after receiving a null response.
> The problem can be reproduced with {{TxReadAfterLosingOwnershipTest}} and its subclasses, by replacing the {{operation.update(cache(1));}} line with {{operation.update(cache(0));}}
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 4 months
[JBoss JIRA] (ISPN-4979) CacheStatusResponse map uses too much memory
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-4979?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-4979:
------------------------------------
[~sannegrinovero] you mean in the last paragraph? That only covers single {{JGroupsAddress}} instances AFAICT, not address collections.
> CacheStatusResponse map uses too much memory
> --------------------------------------------
>
> Key: ISPN-4979
> URL: https://issues.jboss.org/browse/ISPN-4979
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.0.0.Final
> Reporter: Dan Berindei
> Assignee: William Burns
> Priority: Critical
> Fix For: 7.1.0.Final
>
>
> When the cluster is large and there are a log of caches, the {{CacheStatusResponse}} map on the new coordinator can get quite large. One of the problems that seems to be that the addresses in {{DefaultConsistentHash}} are duplicated on serialization, so the deserialized version occupies more memory.
> We need to investigate why the objects are not "shared" by the River marshaller, and maybe work around the problem by de-duplicating the addresses in the externalizer.
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 4 months
[JBoss JIRA] (ISPN-4979) CacheStatusResponse map uses too much memory
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-4979?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero commented on ISPN-4979:
---------------------------------------
[~dan.berindei] yes I remember we discussed that. It doesn't make the solution much more complex though, in fact I covered the case in the above recap.
> CacheStatusResponse map uses too much memory
> --------------------------------------------
>
> Key: ISPN-4979
> URL: https://issues.jboss.org/browse/ISPN-4979
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.0.0.Final
> Reporter: Dan Berindei
> Assignee: William Burns
> Priority: Critical
> Fix For: 7.1.0.Final
>
>
> When the cluster is large and there are a log of caches, the {{CacheStatusResponse}} map on the new coordinator can get quite large. One of the problems that seems to be that the addresses in {{DefaultConsistentHash}} are duplicated on serialization, so the deserialized version occupies more memory.
> We need to investigate why the objects are not "shared" by the River marshaller, and maybe work around the problem by de-duplicating the addresses in the externalizer.
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 4 months
[JBoss JIRA] (ISPN-4979) CacheStatusResponse map uses too much memory
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-4979?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-4979:
-------------------------------------
Looking closer the simple serialization change won't actually work it appears that the consistent hashes stored all have different instances of JGroupsAddress so they don't have instance equality which is needed for serialization reuse checks. I will also need to check why these are not the same.
> CacheStatusResponse map uses too much memory
> --------------------------------------------
>
> Key: ISPN-4979
> URL: https://issues.jboss.org/browse/ISPN-4979
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.0.0.Final
> Reporter: Dan Berindei
> Assignee: William Burns
> Priority: Critical
> Fix For: 7.1.0.Final
>
>
> When the cluster is large and there are a log of caches, the {{CacheStatusResponse}} map on the new coordinator can get quite large. One of the problems that seems to be that the addresses in {{DefaultConsistentHash}} are duplicated on serialization, so the deserialized version occupies more memory.
> We need to investigate why the objects are not "shared" by the River marshaller, and maybe work around the problem by de-duplicating the addresses in the externalizer.
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 4 months
[JBoss JIRA] (ISPN-4444) After state transfer, a node is able to read keys it no longer owns from its data container
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-4444?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo commented on ISPN-4444:
-----------------------------------
[~dan.berindei] I checked the code and I found out that it is checking the DataContainer even without check if it is an owner or not. This solves the issue when L1 is disabled.
However, when L1 is enabled, it can fetch the stale value while the invalidation is in progress. If I understand correctly, we want to avoid this case, right?
> After state transfer, a node is able to read keys it no longer owns from its data container
> -------------------------------------------------------------------------------------------
>
> Key: ISPN-4444
> URL: https://issues.jboss.org/browse/ISPN-4444
> Project: Infinispan
> Issue Type: Bug
> Components: Core, State Transfer
> Affects Versions: 7.0.0.Alpha4
> Reporter: Dan Berindei
> Assignee: Pedro Ruivo
> Priority: Critical
> Fix For: 7.1.0.Alpha1
>
>
> When state transfer ends and each node receives a CH_UPDATE command from the coordinator, it first installs the new topology and then it starts invalidating entries it no longer owns.
> However, there are two cases when the node can still read its stale values:
> 1. If L1 is enabled, it will look in the local DataContainer first, regardless of the key's location.
> 2. If L1 is disabled, but the key was removed on the new owners, the node will still look up the key in the local DataContainer after receiving a null response.
> The problem can be reproduced with {{TxReadAfterLosingOwnershipTest}} and its subclasses, by replacing the {{operation.update(cache(1));}} line with {{operation.update(cache(0));}}
--
This message was sent by Atlassian JIRA
(v6.3.8#6338)
11 years, 4 months