exception in stopping a component resulted in failing to stop cache manager --------------------------------------------------------------------------- Key: ISPN-2056 URL: https://issues.jboss.org/browse/ISPN-2056 Project: Infinispan Issue Type: Bug Components: Core API Affects Versions: 5.1.4.FINAL Environment: ISPN 5.1.4.Final 2 node cluster with jgroup-tcp.xml and replication mode Reporter: dex chen Assignee: Galder Zamarreño Priority: Critical Fix For: 5.1.5.FINAL If cacheManager.stop() got any exception, the cache manger will not be able stopped by retrying stop(). This will resulted in following new cache manager instance with same configure fail to start with following exception: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:238) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:885) at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:639) at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:226) at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:152) at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175) at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291) at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246) at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83) at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792) at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220) at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154) at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175) at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291) at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246) at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83) at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792) at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220) at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154) at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175) at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291) at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246) at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83) at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792) at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220) at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154) at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175) at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:167) at org.infinispan.factories.InterceptorChainFactory.register(InterceptorChainFactory.java:73) at org.infinispan.factories.InterceptorChainFactory.createInterceptor(InterceptorChainFactory.java:62) at org.infinispan.factories.InterceptorChainFactory.buildInterceptorChain(InterceptorChainFactory.java:95) at org.infinispan.factories.InterceptorChainFactory.construct(InterceptorChainFactory.java:272) at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:286) at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246) at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83) at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792) at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220) at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154) at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175) at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291) at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246) at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83) at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792) at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220) at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154) at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175) at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291) at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246) at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83) at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792) at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220) at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154) at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175) at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291) at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246) at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83) at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792) at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220) at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154) at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175) at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291) at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246) at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83) at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792) at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220) at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154) at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175) at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291) at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246) at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83) at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792) at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220) at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154) at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175) at org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291) at org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246) at org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83) at org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792) at org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220) at org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154) at org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175) at org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory.java:102) at org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFactory.java:78) at org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFactory.java:62) at org.infinispan.manager.DefaultCacheManager.wireCache(DefaultCacheManager.java:692) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549) at .....StatusUpdater.poll..... .... Caused by: org.infinispan.CacheException: Unable to start JGroups Channel at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:186) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:175) at sun.reflect.GeneratedMethodAccessor104.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:236) ... 96 more Caused by: java.lang.IllegalStateException: cluster 'HA_Customer_9.3' is already connected to singleton transport: [dummy-1336085803144, dummy-1336085899401, dummy-1336085785840, dummy-1336085828476, dummy-1336085808197, HA_Customer_9.3, dummy-1336085874063, dummy-1336085868964, dummy-1336085785779, dummy-1336085818352, dummy-1336085792988, dummy-1336085879152, dummy-1336085823409, dummy-1336085785693, dummy-1336085848698, dummy-1336085785734, dummy-1336085813295, dummy-1336085889257, dummy-1336085853769, dummy-1336085894308, dummy-1336085843643, dummy-1336085833534, dummy-1336085787888, dummy-1336085798048, dummy-1336085785653, dummy-1336085884206, dummy-1336085863914, dummy-1336085785603, dummy-1336085838588, dummy-1336085858821] at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:909) at org.jgroups.JChannel.startStack(JChannel.java:841) at org.jgroups.JChannel.connect(JChannel.java:277) at org.jgroups.JChannel.connect(JChannel.java:261) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:184) This specific case is caused by cache manager can not terminate the first cache manager instance becaue of SyncFailedException: sync failed as shown: 2012-05-17/10:11:32.539/MDT [Thread-23] DEBUG com.symplified.service.shared.platform.cluster.InfinispanClusterCacheStoreServiceImpl[183] - exception caught in stop existing cache manager org.infinispan.CacheException: Unable to invoke method public void org.infinispan.loaders.CacheLoaderManagerImpl.stop() on object of type CacheLoaderManagerImpl at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:238) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:885) at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:674) at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:553) at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:199) at org.infinispan.CacheImpl.stop(CacheImpl.java:539) at org.infinispan.CacheImpl.stop(CacheImpl.java:531) at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:742) ..... Caused by: org.infinispan.CacheException: java.io.SyncFailedException: sync failed at org.infinispan.loaders.jdbm.JdbmCacheStore.stop(JdbmCacheStore.java:246) at org.infinispan.loaders.decorators.AbstractDelegatingStore.stop(AbstractDelegatingStore.java:155) at org.infinispan.loaders.decorators.AsyncStore.stop(AsyncStore.java:237) at org.infinispan.loaders.CacheLoaderManagerImpl.stop(CacheLoaderManagerImpl.java:221) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:236) ... 14 more Caused by: java.io.SyncFailedException: sync failed at java.io.FileDescriptor.sync(Native Method) at jdbm.recman.TransactionManager.sync(TransactionManager.java:331) at jdbm.recman.TransactionManager.close(TransactionManager.java:347) at jdbm.recman.TransactionManager.synchronizeLogFromMemory(TransactionManager.java:156) at jdbm.recman.TransactionManager.shutdown(TransactionManager.java:339) at jdbm.recman.RecordFile.close(RecordFile.java:346) at jdbm.recman.BaseRecordManager.close(BaseRecordManager.java:168) I do not see there is any way to froce stop the cache manager.