[JBoss JIRA] (ISPN-1769) Rehashing hang ups leading to nodes starting as sole member in cluster
by Galder Zamarreño (JIRA)
Galder Zamarreño created ISPN-1769:
--------------------------------------
Summary: Rehashing hang ups leading to nodes starting as sole member in cluster
Key: ISPN-1769
URL: https://issues.jboss.org/browse/ISPN-1769
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache
Affects Versions: 5.1.0.CR4
Reporter: Galder Zamarreño
Assignee: Dan Berindei
Priority: Blocker
Fix For: 5.1.0.FINAL
Attachments: join-timeout-dan.zip, join-timeout-threaddump.zip
It seems we still have rehashing issues starting nodes, see email below:
{quote}
Hey Dan,
I'm trying to run the old Infinispan Lab where I start 4 nodes, each of which starts a DIST_SYNC cache (Infinispan 5.1.0.CR4) with these configurations:
new ConfigurationBuilder()
.clustering()
.cacheMode(CacheMode.DIST_SYNC)
.l1().disable()
.jmxStatistics()
.build();
new DefaultCacheManager(
GlobalConfigurationBuilder.defaultClusteredBuilder()
.transport()
.addProperty("configurationFile", "jgroups.xml")
.build()
);
And I got a hang in one of the nodes that ended up starting on its own. This is very similar to the issues we had back in November.
I don't have TRACE logs yet but I have a thread dump of all the nodes which you can find attached.
It's run in AS7 domain model so the output of all processes is mixed up. The node that doesn't start in time is 'Server:server-four', so you can grep by that.
There's barely 7 entries in memory and should not have up like this.
I'm gonna try to get some TRACE logs.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 7 months
[JBoss JIRA] (ISPN-1791) Change of numOwners during runtime
by Thomas Fromm (JIRA)
Thomas Fromm created ISPN-1791:
----------------------------------
Summary: Change of numOwners during runtime
Key: ISPN-1791
URL: https://issues.jboss.org/browse/ISPN-1791
Project: Infinispan
Issue Type: Feature Request
Components: Distributed Cache
Affects Versions: 5.1.0.FINAL
Reporter: Thomas Fromm
Assignee: Manik Surtani
When the number of nodes increases, I want to increase/decrease the number of numOwners at DIST too.
For me one or both of the following options would be nice:
1) Additional to numOwners an attribute "numOwnersRatio". When numOwners is set, the numOwners value is used as minimum number of owners. If the ratio results into a higher number, this value is used.
e.g. numOwners = 2, numOwnersRatio = 0.5; In case of 3 nodes inside the cluster, the numOwners value of 2 is used. 5 nodes => 3 owner, 8 nodes => 4 owners...
2) Simply change the numOwners setting programmatic via API
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 7 months
[JBoss JIRA] (ISPN-1782) Could not lock to invalidate L1 WARN messages
by Galder Zamarreño (JIRA)
Galder Zamarreño created ISPN-1782:
--------------------------------------
Summary: Could not lock to invalidate L1 WARN messages
Key: ISPN-1782
URL: https://issues.jboss.org/browse/ISPN-1782
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.1.0.CR4
Reporter: Galder Zamarreño
Assignee: Manik Surtani
Priority: Blocker
Fix For: 5.1.0.FINAL
While executing the labs and playing with retrieving keys from other nodes, I see these in the logs:
{code}
[Server:server-three] 20:02:46,847 WARN [org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor] (OOB-3,ISPN,z-63185) ISPN000135: Could not lock key sanne-Devoxx presents... in order to invalidate from L1 at node z-63185, skipping....
[Server:server-three] 20:02:46,857 WARN [org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor] (OOB-2,ISPN,z-63185) ISPN000135: Could not lock key manik-Best of Abba in order to invalidate from L1 at node z-63185, skipping....
[Server:server-three] 20:02:46,867 WARN [org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor] (OOB-2,ISPN,z-63185) ISPN000135: Could not lock key dan-Cher's greatest hits in order to invalidate from L1 at node z-63185, skipping....
...
[Server:server-four] 20:03:05,500 WARN [org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor] (OOB-4,ISPN,z-25294) ISPN000135: Could not lock key galder-M83 live in order to invalidate from L1 at node z-25294, skipping....{code}
I'll attach TRACE logs asap if I can replicate it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 7 months
[JBoss JIRA] (ISPN-1750) OptimisticLockingInterceptor fails due to an IllegalArgumentException
by Nicolas Filotto (JIRA)
Nicolas Filotto created ISPN-1750:
-------------------------------------
Summary: OptimisticLockingInterceptor fails due to an IllegalArgumentException
Key: ISPN-1750
URL: https://issues.jboss.org/browse/ISPN-1750
Project: Infinispan
Issue Type: Bug
Components: Core API
Affects Versions: 5.1.0.CR4
Reporter: Nicolas Filotto
Assignee: Manik Surtani
Priority: Blocker
With ISPN 5.1.0.CR4, I face exception of the following type:
{code}
17.01.2012 11:56:53 *ERROR* [main] InvocationContextInterceptor: ISPN000136: Execution error (InvocationContextInterceptor.java, line 123)
java.lang.IllegalArgumentException: Comparison method violates its general contract!
at org.infinispan.util.TimSort.mergeHi(TimSort.java:872)
at org.infinispan.util.TimSort.mergeAt(TimSort.java:489)
at org.infinispan.util.TimSort.mergeForceCollapse(TimSort.java:430)
at org.infinispan.util.TimSort.sort(TimSort.java:227)
at org.infinispan.util.TimSort.sort(TimSort.java:177)
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.sort(OptimisticLockingInterceptor.java:252)
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:94)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:106)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:116)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:129)
at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:111)
at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:122)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:811)
at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2656)
at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1784)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:94)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1423)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:137)
{code}
After a deeper investigation, it seems to be due to the implementation of {{keyComparator}} used to sort the keys based on {{MurmurHash3}}. Please find as attached file a test case that shows the issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 7 months
[JBoss JIRA] (ISPN-1796) Out-of-memory adding a lot of elements in cache with AsyncStore
by Andrew Pushkin (JIRA)
Andrew Pushkin created ISPN-1796:
------------------------------------
Summary: Out-of-memory adding a lot of elements in cache with AsyncStore
Key: ISPN-1796
URL: https://issues.jboss.org/browse/ISPN-1796
Project: Infinispan
Issue Type: Bug
Components: Loaders and Stores, Locking and Concurrency
Affects Versions: 5.1.0.CR3
Environment: We plan to use Infinispan as a large distributed write-behind cache of terabytes of data, with a little fraction cached in RAM, so OOM is real threat for us.
Reporter: Andrew Pushkin
Assignee: Manik Surtani
OOM occure on peaks of putting objects in cache configured to use AsyncStore.
See Steps to Reproduce.
Profiling shows that the gc path is through AsyncStore.state field.
The AsyncStore.executor initialized to ThreadPoolExecutor with DiscardPolicy to silently discard tasks if the queue is full, which delays async processing of entries in *state* map, which continues to grow.
Suggested solution.
Instead of DiscardPolicy use customized behavior, which is to estimate accumulated state size and (probably comparing it with modificationQueueSize) decide to discard or to block until it is processed.
The downside of suggested solution is the necessity to lock to estimate state size every time the task is rejected. Possibly it can be alleviated by increasing workingQueue size, so that it survive peaks without rejection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12 years, 7 months