[JBoss JIRA] (ISPN-6387) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: Replication timeout for X
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6387?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-6387:
------------------------------------
[~rhusar] could this be a duplicate of ISPN-6357?
> ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: Replication timeout for X
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-6387
> URL: https://issues.jboss.org/browse/ISPN-6387
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.1.2.Final
> Reporter: Radoslav Husar
> Assignee: Radoslav Husar
>
> Booting WF with starting caches yields after 1 minute:
> The problematic call originates in Infinispan's org.infinispan.topology.ClusterTopologyManagerImpl#confirmMembersAvailable heartbeat command.
> {noformat}
> 00:20:51,646 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (timeout-thread--p10-t1) Response: sender=node2, received=false, suspected=false
> 00:20:51,647 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p13-t2) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node2
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:599)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
> at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6387) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: Replication timeout for X
by Radoslav Husar (JIRA)
Radoslav Husar created ISPN-6387:
------------------------------------
Summary: ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: Replication timeout for X
Key: ISPN-6387
URL: https://issues.jboss.org/browse/ISPN-6387
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 8.1.2.Final
Reporter: Radoslav Husar
Assignee: Radoslav Husar
Booting WF with starting caches yields after 1 minute:
The problematic call originates in Infinispan's org.infinispan.topology.ClusterTopologyManagerImpl#confirmMembersAvailable heartbeat command.
{noformat}
00:20:51,646 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (timeout-thread--p10-t1) Response: sender=node2, received=false, suspected=false
00:20:51,647 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p13-t2) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node2
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:599)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6275) Double invalidate of invalid Hot Rod connections
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-6275?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-6275:
----------------------------------
Status: Open (was: New)
> Double invalidate of invalid Hot Rod connections
> ------------------------------------------------
>
> Key: ISPN-6275
> URL: https://issues.jboss.org/browse/ISPN-6275
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Protocols
> Affects Versions: 6.0.2.Final
> Reporter: Dennis Reed
> Assignee: Galder Zamarreño
>
> When there's a problem with a Hot Rod operation, RetryOnFailureOperation invalidates the connection twice (once in a catch block, and once in a finally block).
> This causes the GenericKeyedObjectPool counts to get off, and anything relying on that count (such as the maxTotal configuration for the pool) to break.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6353) REST service fails to start during remote query server integration tests
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-6353?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec updated ISPN-6353:
--------------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> REST service fails to start during remote query server integration tests
> ------------------------------------------------------------------------
>
> Key: ISPN-6353
> URL: https://issues.jboss.org/browse/ISPN-6353
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Querying
> Affects Versions: 8.1.0.Final
> Reporter: Adrian Nistor
> Assignee: Adrian Nistor
> Fix For: 9.0.0.Final, 9.0.0.Alpha1, 8.1.3.Final, 8.2.1.Final
>
>
> Errors are logged, REST service fails to start due to classloading problems of InfinispanIndexManager. This happens because the rest cache is not defined in the configuration so it gets created automatically based on the default config which happens to be an indexed cache, using InfinispanIndexManager, which is not normally available to the REST service. The tests do not fail.
> {code}
> 8:58:51,182 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.datagrid-infinispan-endpoint.rest.rest-connector: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan-endpoint.rest.rest-connector: DGENDPT10015: Could not create the web context for the REST Server
> at org.infinispan.server.endpoint.subsystem.RestService.start(RestService.java:103)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.hibernate.search.engine.service.classloading.spi.ClassLoadingException: Unable to load class [org.infinispan.query.indexmanager.InfinispanIndexManager]
> at org.hibernate.search.engine.service.classloading.impl.DefaultClassLoaderService.classForName(DefaultClassLoaderService.java:64)
> at org.hibernate.search.util.impl.ClassLoaderHelper.classForName(ClassLoaderHelper.java:320)
> at org.hibernate.search.engine.impl.DefaultIndexManagerFactory.createIndexManagerByName(DefaultIndexManagerFactory.java:54)
> at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:247)
> at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:513)
> at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManagers(IndexManagerHolder.java:482)
> at org.hibernate.search.indexes.impl.IndexManagerHolder.buildEntityIndexBinding(IndexManagerHolder.java:91)
> at org.hibernate.search.spi.SearchIntegratorBuilder.initDocumentBuilders(SearchIntegratorBuilder.java:358)
> at org.hibernate.search.spi.SearchIntegratorBuilder.buildNewSearchFactory(SearchIntegratorBuilder.java:199)
> at org.hibernate.search.spi.SearchIntegratorBuilder.buildSearchIntegrator(SearchIntegratorBuilder.java:117)
> at org.infinispan.query.impl.LifecycleManager.getSearchFactory(LifecycleManager.java:300)
> at org.infinispan.query.impl.LifecycleManager.cacheStarting(LifecycleManager.java:112)
> at org.infinispan.factories.ComponentRegistry.notifyCacheStarting(ComponentRegistry.java:247)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:236)
> at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:849)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:635)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:585)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:451)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:470)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:94)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:85)
> at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:26)
> at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:14)
> at org.infinispan.security.Security.doPrivileged(Security.java:76)
> at org.infinispan.rest.SecurityActions.doPrivileged(SecurityActions.java:24)
> at org.infinispan.rest.SecurityActions.getCache(SecurityActions.java:31)
> at org.infinispan.rest.NettyRestServer$$anonfun$startCaches$1.apply(NettyRestServer.scala:77)
> at org.infinispan.rest.NettyRestServer$$anonfun$startCaches$1.apply(NettyRestServer.scala:77)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at org.infinispan.rest.NettyRestServer$.startCaches(NettyRestServer.scala:77)
> at org.infinispan.rest.NettyRestServer$.apply(NettyRestServer.scala:52)
> at org.infinispan.rest.NettyRestServer$.apply(NettyRestServer.scala:46)
> at org.infinispan.rest.NettyRestServer.apply(NettyRestServer.scala)
> at org.infinispan.server.endpoint.subsystem.RestService.start(RestService.java:101)
> ... 5 more
> Caused by: java.lang.ClassNotFoundException: Could not load requested class : org.infinispan.query.indexmanager.InfinispanIndexManager
> at org.hibernate.search.util.impl.AggregatedClassLoader.findClass(AggregatedClassLoader.java:75)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.hibernate.search.engine.service.classloading.impl.DefaultClassLoaderService.classForName(DefaultClassLoaderService.java:61)
> ... 42 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-6239) InitialClusterSizeTest.testInitialClusterSizeFail random failures
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-6239?page=com.atlassian.jira.plugin.... ]
Dan Berindei reopened ISPN-6239:
--------------------------------
The test is still failing randomly:
{noformat}
10:17:52,124 DEBUG (ForkThread-1,InitialClusterSizeTest) [GMS] InitialClusterSizeTest-NodeF-44973: installing view [InitialClusterSizeTest-NodeF-44973|0] (1) [InitialClusterSizeTest-NodeF-44973]
10:17:52,124 INFO (ForkThread-1,InitialClusterSizeTest) [JGroupsTransport] ISPN000094: Received new cluster view for channel ISPN: [InitialClusterSizeTest-NodeF-44973|0] (1) [InitialClusterSizeTest-NodeF-44973]
10:17:52,126 DEBUG (ForkThread-1,InitialClusterSizeTest) [GMS] InitialClusterSizeTest-NodeF-44973: created cluster (first member). My view is [InitialClusterSizeTest-NodeF-44973|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl
10:17:52,215 DEBUG (ForkThread-4,InitialClusterSizeTest) [GMS] InitialClusterSizeTest-NodeE-6608: sending JOIN(InitialClusterSizeTest-NodeE-6608) to InitialClusterSizeTest-NodeF-44973
10:17:52,215 DEBUG (ForkThread-2,InitialClusterSizeTest) [GMS] InitialClusterSizeTest-NodeG-58252: sending JOIN(InitialClusterSizeTest-NodeG-58252) to InitialClusterSizeTest-NodeF-44973
10:17:52,762 DEBUG (Incoming-1,InitialClusterSizeTest-NodeF-44973) [GMS] InitialClusterSizeTest-NodeF-44973: installing view [InitialClusterSizeTest-NodeF-44973|1] (3) [InitialClusterSizeTest-NodeF-44973, InitialClusterSizeTest-NodeE-6608, InitialClusterSizeTest-NodeG-58252]
10:17:53,261 DEBUG (ForkThread-4,InitialClusterSizeTest) [GMS] InitialClusterSizeTest-NodeE-6608: installing view [InitialClusterSizeTest-NodeF-44973|1] (3) [InitialClusterSizeTest-NodeF-44973, InitialClusterSizeTest-NodeE-6608, InitialClusterSizeTest-NodeG-58252]
10:17:53,355 DEBUG (ForkThread-2,InitialClusterSizeTest) [GMS] InitialClusterSizeTest-NodeG-58252: installing view [InitialClusterSizeTest-NodeF-44973|1] (3) [InitialClusterSizeTest-NodeF-44973, InitialClusterSizeTest-NodeE-6608, InitialClusterSizeTest-NodeG-58252]
10:17:57,736 ERROR (testng-InitialClusterSizeTest) [UnitTestTestNGListener] Test testInitialClusterSizeFail(org.infinispan.remoting.transport.InitialClusterSizeTest) failed.
org.testng.TestException:
Expected exception org.infinispan.commons.CacheException but got java.util.concurrent.TimeoutException
{noformat}
Here {{TEST_PING}} seems to be working fine, but it still takes > 1 second for all the nodes to receive their initial view. Because the {{initialClusterTimeout}} timeout starts after the initial view, some nodes will time out 1 second later, and because the test expects all the nodes to time out in {{initialClusterTimeout + 1s}}, it will fail.
> InitialClusterSizeTest.testInitialClusterSizeFail random failures
> -----------------------------------------------------------------
>
> Key: ISPN-6239
> URL: https://issues.jboss.org/browse/ISPN-6239
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 8.2.0.Beta2
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: testsuite_failure
> Fix For: 8.2.0.CR1, 8.2.0.Final
>
>
> The test starts 3 nodes concurrently, but configures Infinispan to wait for a cluster of 4 nodes, and expects that the nodes fail to start in {{initialClusterTimeout}} + 1 second.
> However, because of a bug in {{TEST_PING}}, the first 2 nodes see each other as coordinator and send a {{JOIN}} request to each other, and it takes 3 seconds to recover and start the cluster properly.
> The bug in {{TEST_PING}} is actually a hack introduced for {{ISPN-5106}}. The problem was that the first node (A) to start would install a view with itself as the single node, but the second node to start (B) would start immediately, and the discovery request from B would reach B's {{TEST_PING}} before it saw the view. That way, B could choose itself as the coordinator based on the order of A's and B's UUIDs, and the cluster would start as 2 partitions. Since most of our tests actually remove {{MERGE3}} from the protocol stack, the partitions would never merge and the test would fail with a timeout.
> I fixed this in {{TEST_PING}} by assuming that the sender of the first discovery response is a coordinator, when there is a single response. This worked because all but a few tests start their managers sequentially, however it sometimes introduces this 3 seconds delay when nodes start in parallel.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-5953) Extract interface from RemoteCacheManager
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-5953?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño updated ISPN-5953:
-----------------------------------
Status: Open (was: New)
> Extract interface from RemoteCacheManager
> -----------------------------------------
>
> Key: ISPN-5953
> URL: https://issues.jboss.org/browse/ISPN-5953
> Project: Infinispan
> Issue Type: Enhancement
> Components: CDI Integration, Integration
> Reporter: Sebastian Łaskawiec
> Priority: Minor
> Fix For: 9.0.0.Final
>
>
> Currently RemoteCacheManager is a concrete class which is problematic when using it in CDI (CDI will register it as a bean). This is very inconvenient because this instance should be created by InfinispanEmbeddedExtension.
> Since this change is not backwards compatible - we need to implement it in ISPN 9.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ISPN-5953) Extract interface from RemoteCacheManager
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-5953?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño reassigned ISPN-5953:
--------------------------------------
Assignee: Galder Zamarreño
> Extract interface from RemoteCacheManager
> -----------------------------------------
>
> Key: ISPN-5953
> URL: https://issues.jboss.org/browse/ISPN-5953
> Project: Infinispan
> Issue Type: Enhancement
> Components: CDI Integration, Integration
> Reporter: Sebastian Łaskawiec
> Assignee: Galder Zamarreño
> Priority: Minor
> Fix For: 9.0.0.Final
>
>
> Currently RemoteCacheManager is a concrete class which is problematic when using it in CDI (CDI will register it as a bean). This is very inconvenient because this instance should be created by InfinispanEmbeddedExtension.
> Since this change is not backwards compatible - we need to implement it in ISPN 9.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years