[jboss-jira] [JBoss JIRA] (JGRP-1784) could not flush the cluster for state retrieval

could not flush the cluster for state retrieval ----------------------------------------------- Key: JGRP-1784 URL: https://issues.jboss.org/browse/JGRP-1784 Project: JGroups Issue Type: Bug Affects Versions: 2.6.10 Environment: production Reporter: scott bilyeu Assignee: Bela Ban My production environment cluster was working and then stopped for some reason now all I get no matter what I try is 2014-01-25 09:20:06,521 WARN [org.jgroups.protocols.UDP] (JBoss System Threads(1)-3) failed to join /224.0.0.75:7500 on net3: java.net.SocketException: Unrecognized Windows Sockets error: 0: no Inet4Address associated with interface 2014-01-25 09:20:06,521 WARN [org.jgroups.protocols.UDP] (JBoss System Threads(1)-3) failed to join /224.0.0.75:7500 on net4: java.net.SocketException: Unrecognized Windows Sockets error: 0: no Inet4Address associated with interface 2014-01-25 09:20:06,521 INFO [STDOUT] (JBoss System Threads(1)-3) --------------------------------------------------------- GMS: address is 10.96.100.142:51826 (cluster=IdmPartition) --------------------------------------------------------- 2014-01-25 09:20:06,662 INFO [org.jboss.cache.jmx.PlatformMBeanServerRegistration] (main) JBossCache MBeans were successfully registered to the platform mbean server. 2014-01-25 09:20:06,709 INFO [org.jboss.ha.framework.interfaces.HAPartition.IdmPartition] (JBoss System Threads(1)-3) Number of cluster members: 2 2014-01-25 09:20:06,709 INFO [org.jboss.ha.framework.interfaces.HAPartition.IdmPartition] (JBoss System Threads(1)-3) Other members: 1 2014-01-25 09:20:06,756 INFO [STDOUT] (main) --------------------------------------------------------- GMS: address is 10.96.100.142:51826 (cluster=IdmPartition-HAPartitionCache) --------------------------------------------------------- 2014-01-25 09:20:08,771 INFO [org.jboss.cache.RPCManagerImpl] (main) Received new cluster view: [10.96.100.142:51826|0] [10.96.100.142:51826] 2014-01-25 09:20:08,771 INFO [org.jboss.cache.RPCManagerImpl] (main) Cache local address is 10.96.100.142:51826 2014-01-25 09:20:08,787 INFO [org.jboss.cache.RPCManagerImpl] (main) state was retrieved successfully (in 2.03 seconds) 2014-01-25 09:20:08,803 INFO [org.jboss.cache.factories.ComponentRegistry] (main) JBoss Cache version: JBossCache 'Cascabel' 3.1.0.GA 2014-01-25 09:20:08,803 INFO [org.jboss.ha.framework.interfaces.HAPartition.IdmPartition] (main) Fetching serviceState (will wait for 30000 milliseconds): 2014-01-25 09:20:25,092 ERROR [org.jboss.kernel.plugins.dependency.AbstractKernelController] (main) Error installing to Start: name=HAPartition state=Create java.lang.IllegalStateException: Node 10.96.100.142:51826 could not flush the cluster for state retrieval at org.jgroups.JChannel.getState(JChannel.java:1109) at org.jgroups.JChannel.getState(JChannel.java:1034) at org.jgroups.JChannel.getState(JChannel.java:978) at org.jgroups.JChannel.getState(JChannel.java:932) at org.jboss.ha.framework.server.ClusterPartition.fetchState(ClusterPartition.java:616) at org.jboss.ha.framework.server.ClusterPartition.startService(ClusterPartition.java:422) at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376) at org.jboss.system.ServiceMBeanSupport.pojoStart(ServiceMBeanSupport.java:216) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:59) at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:150) at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:241) at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:47) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:109) at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:70) at org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:221) at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54) at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42) at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62) at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71) at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:774) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:540) at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:121) at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:51) at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62) at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50) at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:171) at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1439) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1157) at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1178) at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1098) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553) at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:781) at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:702) at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117) at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70) at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53) at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:361) at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822) at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553) at org.jboss.system.server.profileservice.repository.AbstractProfileService.activateProfile(AbstractProfileService.java:306) at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:271) at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461) at org.jboss.Main.boot(Main.java:221) at org.jboss.Main$1.run(Main.java:556) at java.lang.Thread.run(Thread.java:662) 2014-01-25 09:20:25,092 INFO [org.jboss.ha.framework.interfaces.HAPartition.IdmPartition] (main) Partition IdmPartition destroyed. 2014-01-25 09:20:25,921 INFO [org.jboss.invocation.unified.server.UnifiedInvokerHA] (main) Service name is jboss:service=invoker,type=unifiedha 2014-01-25 09:20:51,656 WARN [org.jgroups.protocols.pbcast.NAKACK] (Incoming-11,10.96.100.142:51826) 10.96.100.142:51826] discarded message from non-member 10.96.100.141:58840, my view is [10.96.100.142:51826|0] [10.96.100.142:51826] 2014-01-25 09:20:51,672 WARN [org.jgroups.protocols.pbcast.NAKACK] (Incoming-16,10.96.100.142:51826) 10.96.100.142:51826] discarded message from non-member 10.96.100.141:58840, my view is [10.96.100.142:51826|0] [10.96.100.142:51826] 2014-01-25 09:20:51,672 ERROR [org.jgroups.protocols.pbcast.NAKACK] (Incoming-17,10.96.100.142:51826) sender 10.96.100.141:58840 not found in xmit_table 2014-01-25 09:20:51,672 ERROR [org.jgroups.protocols.pbcast.NAKACK] (Incoming-17,10.96.100.142:51826) range is null 2014-01-25 09:20:51,672 INFO [org.jboss.cache.RPCManagerImpl] (Incoming-17,10.96.100.142:51826) Received new cluster view: MergeView::[10.96.100.141:58840|3] [10.96.100.141:58840, 10.96.100.142:51826], subgroups=[[10.96.100.141:58840|2] [10.96.100.141:58840], [10.96.100.142:51826|0] [10.96.100.142:51826]] 2014-01-25 09:21:04,422 WARN [org.jboss.annotation.factory.AnnotationCreator] (main) No ClassLoader provided, using TCCL: org.jboss.managed.api.annotation.ManagementComponent 2014-01-25 09:21:04,547 WARN [org.jboss.annotation.factory.AnnotationCreator] (main) No ClassLoader provided, using TCCL: org.jboss.managed.api.annotation.ManagementComponent