[JBoss JIRA] (ISPN-2982) CLONE - State transfer does not end because some segments are erroneously reported as unreceived
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2982?page=com.atlassian.jira.plugin.... ]
Mircea Markus reassigned ISPN-2982:
-----------------------------------
Assignee: Mircea Markus (was: Adrian Nistor)
> CLONE - State transfer does not end because some segments are erroneously reported as unreceived
> ------------------------------------------------------------------------------------------------
>
> Key: ISPN-2982
> URL: https://issues.jboss.org/browse/ISPN-2982
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.1.Final
> Reporter: Ittay Dror
> Assignee: Mircea Markus
> Priority: Critical
> Attachments: jgroups.xml
>
>
> Hard to reproduce. I lost the last log where this was visible but still have a stack trace:
> org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:879)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:650)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:639)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:542)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:197)
> at org.infinispan.CacheImpl.start(CacheImpl.java:517)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:689)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:652)
> at org.infinispan.manager.DefaultCacheManager.access$100(DefaultCacheManager.java:126)
> at org.infinispan.manager.DefaultCacheManager$1.run(DefaultCacheManager.java:574)
> Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache LuceneIndexesMetadata on PersistentStateTransferQueryDistributedIndexTest-NodeC-6067
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:199)
> at sun.reflect.GeneratedMethodAccessor139.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
> ... 10 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-2572) "CacheException: Initial state transfer timed out for cache" reliably on AS7 testsuite
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2572?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2572:
--------------------------------
Assignee: Dan Berindei (was: Mircea Markus)
> "CacheException: Initial state transfer timed out for cache" reliably on AS7 testsuite
> --------------------------------------------------------------------------------------
>
> Key: ISPN-2572
> URL: https://issues.jboss.org/browse/ISPN-2572
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.0.Beta4
> Reporter: Radoslav Husar
> Assignee: Dan Berindei
> Priority: Blocker
> Fix For: 5.2.0.Beta6
>
>
> While running AS7 testsuite with speedups implemented in my branch (https://github.com/jbossas/jboss-as/pull/3381) we are contantly seeing (log below) on Windows 2008.
> Run:
> http://teamcity.cafe-babe.org/viewLog.html?buildId=1689&tab=buildResultsD...
> {code}
> 16:34:46,092 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 13) MSC00001: Failed to start service jboss.infinispan.ejb.remote-connector-client-mappings: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.remote-connector-client-mappings: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_32]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_32]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_32]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]
> Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:199)
> at org.infinispan.CacheImpl.start(CacheImpl.java:520)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:690)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:653)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:107)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:98)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
> ... 4 more
> Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache remote-connector-client-mappings on node-1/ejb
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:209)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_32]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_32]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_32]
> at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_32]
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
> ... 18 more
> {code}
> Affected version -- current master (say 7dc531002539b078e429418d8ef204e401beafd1).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-2572) "CacheException: Initial state transfer timed out for cache" reliably on AS7 testsuite
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2572?page=com.atlassian.jira.plugin.... ]
Mircea Markus reassigned ISPN-2572:
-----------------------------------
Assignee: Mircea Markus (was: Dan Berindei)
> "CacheException: Initial state transfer timed out for cache" reliably on AS7 testsuite
> --------------------------------------------------------------------------------------
>
> Key: ISPN-2572
> URL: https://issues.jboss.org/browse/ISPN-2572
> Project: Infinispan
> Issue Type: Bug
> Components: State transfer
> Affects Versions: 5.2.0.Beta4
> Reporter: Radoslav Husar
> Assignee: Mircea Markus
> Priority: Blocker
> Fix For: 5.2.0.Beta6
>
>
> While running AS7 testsuite with speedups implemented in my branch (https://github.com/jbossas/jboss-as/pull/3381) we are contantly seeing (log below) on Windows 2008.
> Run:
> http://teamcity.cafe-babe.org/viewLog.html?buildId=1689&tab=buildResultsD...
> {code}
> 16:34:46,092 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 13) MSC00001: Failed to start service jboss.infinispan.ejb.remote-connector-client-mappings: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.remote-connector-client-mappings: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_32]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_32]
> at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_32]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]
> Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:199)
> at org.infinispan.CacheImpl.start(CacheImpl.java:520)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:690)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:653)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:107)
> at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:98)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
> ... 4 more
> Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache remote-connector-client-mappings on node-1/ejb
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:209)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_32]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_32]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_32]
> at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_32]
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
> ... 18 more
> {code}
> Affected version -- current master (say 7dc531002539b078e429418d8ef204e401beafd1).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-3055) Unreleased lock after node restart
by Deyan Pandulev (JIRA)
[ https://issues.jboss.org/browse/ISPN-3055?page=com.atlassian.jira.plugin.... ]
Deyan Pandulev commented on ISPN-3055:
--------------------------------------
I've tested with the suggested 5.2.6 Final version of Infinispan and there is no change in the behavior described in the bug.
> Unreleased lock after node restart
> ----------------------------------
>
> Key: ISPN-3055
> URL: https://issues.jboss.org/browse/ISPN-3055
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.2.1.Final
> Environment: Debian Squeeze Amd 64, JDK 1.6.0_32
> Reporter: Deyan Pandulev
> Assignee: Pedro Ruivo
> Fix For: 6.0.0.Beta1, 6.0.0.Final
>
> Attachments: SimpleClusterRead.java, SimpleClusterWriter.java, simple_cluster_node1.xml, simple_cluster_node2.xml
>
>
> Using dummy transaction manager I was able to produce locks that are not released. The same behavior can be reproduced with Glassfish AS 3.1.1 using its own TM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-3032) Use both context and class' class loader when reading component metadata
by Bert Jacobs (JIRA)
[ https://issues.jboss.org/browse/ISPN-3032?page=com.atlassian.jira.plugin.... ]
Bert Jacobs commented on ISPN-3032:
-----------------------------------
I've ran into this problem using JBoss EAP 5 and a private classloader to get Modeshape running, but I'd think you would also see it if Infinispan is ever ran in OSGi.
Either you could try to query a number of ClassLoaders or you could even pass Class objects on which you can call .getResource("/file.dat"), as suggested in [http://njbartlett.name/2012/10/23/dreaded-thread-context-classloader.html]. Of course, this assumes you know that a given class file is always packaged in the same JAR file as the dat file.
> Use both context and class' class loader when reading component metadata
> ------------------------------------------------------------------------
>
> Key: ISPN-3032
> URL: https://issues.jboss.org/browse/ISPN-3032
> Project: Infinispan
> Issue Type: Enhancement
> Reporter: Mircea Markus
> Assignee: Mircea Markus
> Priority: Minor
>
> {quote}
> Caused by: org.hibernate.search.SearchException: Unable to initialize
> directory provider:
> org.hibernate.search.test.integration.jbossas7.model.Member
> at org.hibernate.search.store.impl.DirectoryProviderFactory.createDirectoryProvider(DirectoryProviderFactory.java:87)
> at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.createDirectoryProvider(DirectoryBasedIndexManager.java:232)
> at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.initialize(DirectoryBasedIndexManager.java:100)
> at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:227)
> ... 19 more
> Caused by: org.infinispan.config.ConfigurationException:
> org.infinispan.CacheException: Unable to load component metadata!
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:386)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:341)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:328)
> at org.hibernate.search.infinispan.CacheManagerServiceProvider.start(CacheManagerServiceProvider.java:93)
> at org.hibernate.search.engine.impl.StandardServiceManager$ServiceProviderWrapper.startVirtual(StandardServiceManager.java:178)
> at org.hibernate.search.engine.impl.StandardServiceManager.requestService(StandardServiceManager.java:124)
> at org.hibernate.search.infinispan.impl.InfinispanDirectoryProvider.initialize(InfinispanDirectoryProvider.java:86)
> at org.hibernate.search.store.impl.DirectoryProviderFactory.createDirectoryProvider(DirectoryProviderFactory.java:84)
> ... 22 more
> Caused by: org.infinispan.CacheException: Unable to load component metadata!
> at org.infinispan.factories.components.ComponentMetadataRepo.initialize(ComponentMetadataRepo.java:131)
> at org.infinispan.factories.GlobalComponentRegistry.<init>(GlobalComponentRegistry.java:103)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:381)
> ... 29 more
> Caused by: java.lang.NullPointerException
> at org.infinispan.factories.components.ComponentMetadataRepo.readMetadata(ComponentMetadataRepo.java:53)
> at org.infinispan.factories.components.ComponentMetadataRepo.initialize(ComponentMetadataRepo.java:129)
> ... 31 more
> {quote}
> The exception above is caused by Infinispan using (by default) invocation context's ClassLoader when trying to read the component metadata. In a modularized environment such as AS7 this will cause the metadata not to be found (metadata is packaged in the same jar as the infinispan class that reads it). In order to overcome this problem we should try to use both class' ClassLoader and InvocationContxt's class loader when trying to read the metadata.
> Workaround: set the class loader on the GlobalConfiguration object:
> {code:java}
> GlobalConfiguration.classLoader( GlobalConfiguration.class.getClassLoder())
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month
[JBoss JIRA] (ISPN-2240) Per-key lock container leads to superfluous TimeoutExceptions on concurrent access to same key
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-2240?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-2240:
--------------------------------
Priority: Major (was: Critical)
> Per-key lock container leads to superfluous TimeoutExceptions on concurrent access to same key
> ----------------------------------------------------------------------------------------------
>
> Key: ISPN-2240
> URL: https://issues.jboss.org/browse/ISPN-2240
> Project: Infinispan
> Issue Type: Bug
> Components: Locking and Concurrency
> Affects Versions: 5.1.6.FINAL, 5.1.x
> Reporter: Robert Stupp
> Assignee: Mircea Markus
> Fix For: 5.3.0.Final
>
> Attachments: ISPN-2240_fix_TimeoutExceptions.patch, somehow.zip
>
>
> Hi,
> I've encountered a lot of TimeoutExceptions just running a load test against an infinispan cluster.
> I tracked down the reason and found out, that the code in org.infinispan.util.concurrent.locks.containers.AbstractPerEntryLockContainer#releaseLock() causes these superfluous TimeoutExceptions.
> A small test case (which just prints out timeouts, too late timeouts and "paints" a lot of dots to the console - more dots/second on the console means better throughput ;-)
> In a short test I extended the class ReentrantPerEntryLockContainer and changed the implementation of releaseLock() as follows:
> {noformat}
> public void releaseLock(Object lockOwner, Object key) {
> ReentrantLock l = locks.get(key);
> if (l != null) {
> if (!l.isHeldByCurrentThread())
> throw new IllegalStateException("Lock for [" + key + "] not held by current thread " + Thread.currentThread());
> while (l.isHeldByCurrentThread())
> unlock(l, lockOwner);
> if (!l.hasQueuedThreads())
> locks.remove(key);
> }
> else
> throw new IllegalStateException("No lock for [" + key + ']');
> }
> {noformat}
> The main improvement is that locks are not removed from the concurrent map as long as other threads are waiting on that lock.
> If the lock is removed from the map while other threads are waiting for it, they may run into timeouts and force TimeoutExceptions to the client.
> The above methods "paints more dots per second" - means: it gives a better throughput for concurrent accesses to the same key.
> The re-implemented method should also fix some replication timeout exceptions.
> Please, please add this to 5.1.7, if possible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 1 month