[JBoss JIRA] (ISPN-7092) Failed to broadcast asynchronous command
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-7092?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-7092:
----------------------------------
Fix Version/s: 10.0.0.CR3
(was: 10.0.0.CR2)
> Failed to broadcast asynchronous command
> ----------------------------------------
>
> Key: ISPN-7092
> URL: https://issues.jboss.org/browse/ISPN-7092
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.4.Final
> Reporter: Radoslav Husar
> Assignee: Dan Berindei
> Priority: Minor
> Labels: infinispan_clean_shutdown
> Fix For: 10.0.0.CR3
>
>
> Occured on server at the end of server stopping.
> Scenarios affected:
> eap-7x-failover-ejb-ejbremote-jvmkill-dist-sync
> eap-7x-failover-ejb-ejbremote-jvmkill-repl-async
> eap-7x-failover-ejb-ejbremote-jvmkill-repl-sync
> eap-7x-failover-ejb-ejbremote-shutdown-dist-async
> eap-7x-failover-ejb-ejbremote-shutdown-dist-sync-tcpStack
> eap-7x-failover-ejb-ejbremote-shutdown-repl-sync
> eap-7x-failover-ejb-ejbremote-undeploy-repl-sync
> eap-7x-failover-ejb-ejbservlet-shutdown-repl-async
> eap-7x-failover-ejb-ejbstateless-jvmkill-repl-async
> eap-7x-failover-http-granular-shutdown-repl-sync
> eap-7x-failover-http-session-shutdown-repl-async
> eap-7x-failover-http-session-shutdown-repl-async-tcpStack
> eap-7x-failover-http-session-shutdown-repl-sync
> eap-7x-failover-jvmkill-ha-ss
> WARN message has two variants.
> *Variant 1*:
> {code}
> 15:01:15,157 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel web
> [JBossINF] [0m[0m15:01:15,157 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel web
> [JBossINF] [0m[0m15:01:15,158 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [JBossINF] [0m[33m15:01:15,158 WARN [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread--p7-t13) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=REBALANCE_CONFIRM, sender=dev214, joinInfo=null, topologyId=35, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=12}: org.infinispan.commons.CacheException: Failed to broadcast asynchronous command: CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=STABLE_TOPOLOGY_UPDATE, sender=dev213, joinInfo=null, topologyId=36, rebalanceId=17, currentCH=DefaultConsistentHash{ns=256, owners = (2)[dev214: 124+132, dev215: 132+124]}, pendingCH=null, availabilityMode=null, actualMembers=[dev214, dev215], throwable=null, viewId=12}
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:595)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastStableTopologyUpdate(ClusterTopologyManagerImpl.java:615)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.startQueuedRebalance(ClusterCacheStatus.java:633)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.endRebalance(ClusterCacheStatus.java:373)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.doConfirmRebalance(ClusterCacheStatus.java:310)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:263)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:176)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:153)
> [JBossINF] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:159)
> [JBossINF] at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:199)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: java.lang.IllegalStateException: channel is not connected
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:158)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:138)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:578)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:532)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:592)
> [JBossINF] ... 13 more
> [JBossINF] Caused by: java.lang.IllegalStateException: channel is not connected
> [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:682)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:172)
> [JBossINF] at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:325)
> [JBossINF] at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:76)
> [JBossINF] at org.jgroups.blocks.Request.execute(Request.java:67)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:373)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:386)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:407)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:133)
> [JBossINF] ... 16 more
> {code}
> *Variant 2*:
> {code}
> 15:53:27,821 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t19) ISPN000336: Finished cluster-wide rebalance for cache routing, topology id = 33
> [JBossINF] [0m[0m15:53:27,829 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> [JBossINF] [0m[0m15:53:27,830 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel web
> [JBossINF] [0m[0m15:53:27,830 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel web
> [JBossINF] [0m[0m15:53:27,821 INFO [org.infinispan.CLUSTER] (remote-thread--p6-t14) ISPN000336: Finished cluster-wide rebalance for cache clusterbench-ee7.ear.clusterbench-ee7-web-default.war, topology id = 39
> [JBossINF] [0m[0m15:53:27,834 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0019: Stopped Driver service with driver-name = h2
> [JBossINF] [0m[33m15:53:27,832 WARN [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread--p6-t14) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=REBALANCE_CONFIRM, sender=dev215, joinInfo=null, topologyId=39, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=12}: org.infinispan.commons.CacheException: Failed to broadcast asynchronous command: CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=CH_UPDATE, sender=dev213, joinInfo=null, topologyId=40, rebalanceId=21, currentCH=DefaultConsistentHash{ns=256, owners = (2)[dev214: 129+127, dev215: 127+129]}, pendingCH=null, availabilityMode=null, actualMembers=[dev214, dev215], throwable=null, viewId=12}
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:595)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastTopologyUpdate(ClusterTopologyManagerImpl.java:606)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.endRebalance(ClusterCacheStatus.java:369)
> [JBossINF] at org.infinispan.topology.ClusterCacheStatus.doConfirmRebalance(ClusterCacheStatus.java:310)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:263)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:176)
> [JBossINF] at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:153)
> [JBossINF] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:159)
> [JBossINF] at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:199)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: java.lang.NullPointerException
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:611)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:532)
> [JBossINF] at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:592)
> [JBossINF] ... 12 more
> {code}
> Link to server log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 7 months
[JBoss JIRA] (ISPN-8329) ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus2 random failures with scattered cache
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-8329?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-8329:
----------------------------------
Fix Version/s: 10.0.0.CR3
(was: 10.0.0.CR2)
> ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus2 random failures with scattered cache
> ----------------------------------------------------------------------------------------------
>
> Key: ISPN-8329
> URL: https://issues.jboss.org/browse/ISPN-8329
> Project: Infinispan
> Issue Type: Enhancement
> Components: Core
> Reporter: Tristan Tarrant
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.0.0.CR3, 9.4.17.Final
>
>
> Error Message
> Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false]-NodeB-47665, current topology is CacheTopology{id=8, rebalanceId=4, currentCH=PartitionerConsistentHash:ScatteredConsistentHash{ns=256, rebalanced=false, owners = (1)[ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false]-NodeB-47665: 85]}, pendingCH=null, unionCH=null, phase=NO_REBALANCE, actualMembers=[ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false]-NodeB-47665], persistentUUIDs=[63b3a997-f229-475b-a14c-9c892f608ba0]}. rebalanceInProgress=false, currentChIsBalanced=false
> Stacktrace
> java.lang.RuntimeException: Timed out waiting for rebalancing to complete on node ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false]-NodeB-47665, current topology is CacheTopology{id=8, rebalanceId=4, currentCH=PartitionerConsistentHash:ScatteredConsistentHash{ns=256, rebalanced=false, owners = (1)[ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false]-NodeB-47665: 85]}, pendingCH=null, unionCH=null, phase=NO_REBALANCE, actualMembers=[ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false]-NodeB-47665], persistentUUIDs=[63b3a997-f229-475b-a14c-9c892f608ba0]}. rebalanceInProgress=false, currentChIsBalanced=false
> at org.infinispan.test.TestingUtil.waitForNoRebalance(TestingUtil.java:386)
> at org.infinispan.statetransfer.ClusterTopologyManagerTest.testAbruptLeaveAfterGetStatus2(ClusterTopologyManagerTest.java:430)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> ... Removed 16 stack frames
>
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 7 months
[JBoss JIRA] (ISPN-9541) Module initialization is not thread-safe
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9541?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9541:
----------------------------------
Fix Version/s: 10.0.0.CR3
(was: 10.0.0.CR2)
> Module initialization is not thread-safe
> ----------------------------------------
>
> Key: ISPN-9541
> URL: https://issues.jboss.org/browse/ISPN-9541
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Server
> Affects Versions: 9.4.0.CR3
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.CR3, 10.0.0.Final
>
>
> In my ISPN-9127 fix I created a {{BasicComponentRegistry}} interface that represents a mostly-read-only collection of components. It has {{replaceComponent()}} method and a {{rewire()}} method for testing purposes, but it turns out modules were also relying on the ability to replace existing components in order to work.
> Replacing global components is normally safe during the {{ModuleLifecycle.cacheManagerStarting()}}, because none of the components are started yet, so when a component starts later we can still start its dependencies first. But because some modules starts some global components, e.g. by calling {{manager.getCache(name)}}, that assumption breaks.
> The {{infinispan-server-event-logger}} module is a bit more sneaky: it doesn't replace a component, instead it replaces the actual implementation of the event logger in the {{EventLogManager}} component. Events that happen before the module's {{cacheManagerStarting()}} or after {{cacheManagerStopping()}} will be silently dropped from the persistent event log.
> I am investigating making the module a factory of factories. Instead of having a monolitic {{cacheManagerStarting()}} method, it could define a set of components that it can create, and a set of components that should be started before the cache manager is "running". We probably need a way to depend on other modules as well, maybe reusing the {{@Inject}} and {{@ComponentName}} annotations.
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 7 months
[JBoss JIRA] (ISPN-9620) Rolling Upgrade Marshaller Changes
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9620?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9620:
----------------------------------
Fix Version/s: 10.0.0.CR3
(was: 10.0.0.CR2)
> Rolling Upgrade Marshaller Changes
> ----------------------------------
>
> Key: ISPN-9620
> URL: https://issues.jboss.org/browse/ISPN-9620
> Project: Infinispan
> Issue Type: Enhancement
> Components: Core, Loaders and Stores
> Affects Versions: 9.4.0.Final
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
> Fix For: 10.0.0.CR3
>
>
> In order to allow for compatibility between infinispan versions it is necessary for us to utilise a marshalling implementation at both the cluster (internal node-to-node communication) and persistence layer that is strictly defined but allows for future changes. This is necessary in order to facilitate both rolling and start/stop upgrades. Protocol buffers should be utilised as the wire/storage format, with protostream providing the implementation.
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 7 months