Singleton and/or deployer is too brittle ---------------------------------------- Key: JBAS-73 URL: https://jira.jboss.org/jira/browse/JBAS-73 Project: JBoss Application Server Issue Type: Bug Security Level: Public(Everyone can see) Components: Clustering Affects Versions: JBossAS-3.2.6 Final Reporter: Adrian Brock Assignee: Brian Stansberry Fix For: JBossAS-5.1.0.CR1 Trying to bootstrap two JBoss servers, but the cluster join failed because the message took too long (long garbage collection/paging on my computer?) Server 2: GMS: address is htimes2:32925 (additional data: 14 bytes) ------------------------------------------------------- 2004-12-21 15:14:30,909 DEBUG [org.jboss.ha.framework.server.ClusterPartition] Starting channel 2004-12-21 15:14:30,910 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] get nodeName 2004-12-21 15:14:30,990 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Get current members 2004-12-21 15:14:30,991 INFO [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Number of cluster members: 1 2004-12-21 15:14:30,992 INFO [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Other members: 0 2004-12-21 15:14:30,992 INFO [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Fetching state (will wait for 60000 milliseconds): 2004-12-21 15:14:30,999 INFO [org.jboss.ha.framework.interfaces.HAPartition.lifecycle.DefaultPartition] New cluster view for partition DefaultPartition (id: 0, delta: 0) : [127.0.0.1:1199] 2004-12-21 15:14:31,008 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] membership changed from 1 to 1 2004-12-21 15:14:31,009 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Begin notifyListeners, viewID: 0 2004-12-21 15:14:31,009 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] I am (null) received membershipChanged event: 2004-12-21 15:14:31,010 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] Dead members: 0 ([]) 2004-12-21 15:14:31,010 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] New Members : 0 ([]) 2004-12-21 15:14:31,010 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] All Members : 1 ([127.0.0.1:1199]) 2004-12-21 15:14:31,011 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] End notifyListeners, viewID: 0 2004-12-21 15:14:31,058 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] State could not be retrieved, (must be first member of group) 2004-12-21 15:14:31,137 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] setState called 2004-12-21 15:14:31,138 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] state is null 2004-12-21 15:14:31,139 DEBUG [org.jboss.ha.framework.server.ClusterPartition] Started ClusterPartition: DefaultPartition The message was rejected by the other server (too late?) but the cluster eventually joined together Server 1: 15:13:05,825 INFO [Server] JBoss (MX MicroKernel) [3.2.7RC2 (build: CVSTag=Branch_3_2 date=200411160220)] Started in 4m:43s:249ms 15:15:39,184 WARN [NAKACK] [htimes2:32920 (additional data: 14 bytes)] discarded message from non-member htimes2:32925 (additional data: 14 bytes) 15:15:41,080 WARN [NAKACK] [htimes2:32920 (additional data: 14 bytes)] discarded message from non-member htimes2:32925 (additional data: 14 bytes) 15:15:55,106 INFO [DefaultPartition] New cluster view for partition DefaultPartition (id: 1, delta: 1) : [127.0.0.1:1099, 127.0.0.1:1199] 15:15:55,107 INFO [DefaultPartition] Merging partitions... 15:15:55,107 INFO [DefaultPartition] Dead members: 0 This meant that server2 tried to bootstrap the deploy-hasingleton but part way through doing that (when it eventually joined the cluster) it was told to undeploy it. Since the original deploy hadn't finished, this caused havoc in particular trying to use a classloader that no longer existed: 2004-12-21 15:16:28,614 ERROR [org.jboss.mq.sm.jdbc.JDBCStateManager] Starting failed JDBCStateManager org.jboss.mq.SpyJMSException: Error creating connection to the database.; - nested throwable: (java.lang.NullPointerException) at org.jboss.mq.sm.jdbc.JDBCStateManager$JDBCSession.<init>(JDBCStateManager.java:542) at org.jboss.mq.sm.jdbc.JDBCStateManager.initDB(JDBCStateManager.java:432) at org.jboss.mq.sm.jdbc.JDBCStateManager.startService(JDBCStateManager.java:399) at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:271) at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:221) at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54) at org.jboss.mx.server.Invocation.invoke(Invocation.java:82) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473) at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:884) at $Proxy18.start(Unknown Source) at org.jboss.system.ServiceController.start(ServiceController.java:414) at org.jboss.system.ServiceController.start(ServiceController.java:436) at org.jboss.system.ServiceController.start(ServiceController.java:436) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54) at org.jboss.mx.server.Invocation.invoke(Invocation.java:82) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176) at $Proxy4.start(Unknown Source) at org.jboss.deployment.SARDeployer.start(SARDeployer.java:251) at org.jboss.deployment.MainDeployer.start(MainDeployer.java:829) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:641) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:604) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54) at org.jboss.mx.server.Invocation.invoke(Invocation.java:82) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176) at $Proxy8.deploy(Unknown Source) at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:304) at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:478) at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:201) at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:274) at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:271) at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:221) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54) at org.jboss.mx.server.Invocation.invoke(Invocation.java:82) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473) at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:884) at $Proxy0.start(Unknown Source) at org.jboss.system.ServiceController.start(ServiceController.java:414) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54) at org.jboss.mx.server.Invocation.invoke(Invocation.java:82) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176) at $Proxy4.start(Unknown Source) at org.jboss.deployment.SARDeployer.start(SARDeployer.java:251) at org.jboss.deployment.MainDeployer.start(MainDeployer.java:829) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:641) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:604) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:588) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324) at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54) at org.jboss.mx.server.Invocation.invoke(Invocation.java:82) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176) at $Proxy5.deploy(Unknown Source) at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:407) at org.jboss.system.server.ServerImpl.start(ServerImpl.java:311) at org.jboss.Main.boot(Main.java:147) at org.jboss.Main$1.run(Main.java:422) at java.lang.Thread.run(Thread.java:534) Caused by: java.lang.NullPointerException at org.jboss.mx.loading.LoadMgr3.beginLoadTask(LoadMgr3.java:143) at org.jboss.mx.loading.UnifiedClassLoader3.loadClassImpl(UnifiedClassLoader3.java:186) at org.jboss.mx.loading.UnifiedClassLoader3.loadClass(UnifiedClassLoader3.java:136) at java.lang.ClassLoader.loadClass(ClassLoader.java:235) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:302) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:219) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:637) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:129) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:610) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokeModule(LoginContext.java:607) at javax.security.auth.login.LoginContext.login(LoginContext.java:534) at org.jboss.security.plugins.JaasSecurityManager.defaultLogin(JaasSecurityManager.java:480) at org.jboss.security.plugins.JaasSecurityManager.authenticate(JaasSecurityManager.java:420) at org.jboss.security.plugins.JaasSecurityManager.isValid(JaasSecurityManager.java:237) at org.jboss.resource.connectionmanager.BaseConnectionManager2.getSubject(BaseConnectionManager2.java:665) at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496) at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:888) at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102) at org.jboss.mq.sm.jdbc.JDBCStateManager$JDBCSession.<init>(JDBCStateManager.java:518) ... 95 more I would guess a similar race condition exists on all the HASingleton processing and with all attempts to multi-threadedly deploy/undeploy the same url?