[jboss-jira] [JBoss JIRA] (WFLY-2458) Re-deployment of a clustered application fail due to race-condition with infinispan
RH Bugzilla Integration (JIRA)
jira-events at lists.jboss.org
Tue Nov 12 02:41:05 EST 2013
[ https://issues.jboss.org/browse/WFLY-2458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922615#comment-12922615 ]
RH Bugzilla Integration commented on WFLY-2458:
-----------------------------------------------
Ladislav Thon <lthon at redhat.com> made a comment on [bug 1027738|https://bugzilla.redhat.com/show_bug.cgi?id=1027738]
I just built the GitHub EAP 6.x branch, which contains Paul's fix from the pull request, and it's no good. When replaying the reproducer I described in comment 5, I no longer get the "java.lang.IllegalStateException: channel is closed" exception. Instead, I see a series of lovely NPEs, looking like this:
08:28:58,082 WARN [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-19,shared=udp) Problems unmarshalling remote command from byte buffer: java.lang.NullPointerException
at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:45)
at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163)
...
08:28:58,083 ERROR [org.jgroups.blocks.RequestCorrelator] (OOB-20,shared=udp) failed unmarshalling buffer into return value: java.lang.NullPointerException
at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:45)
at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163)
...
And there's even this:
08:28:57,806 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 57) MSC000001: Failed to start service jboss.infinispan.ejb.repl: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.repl: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [jboss-as-clustering-common-7.3.0.Final-redhat-SNAPSHOT.jar:7.3.0.Final-redhat-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final-redhat-1.jar:2.1.1.Final-redhat-1]
Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:646)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
...
Caused by: org.infinispan.CacheException: java.lang.NullPointerException
at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:293)
at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:107)
...
Caused by: java.lang.NullPointerException
at org.infinispan.marshall.jboss.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:164)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:355)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:213)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:45)
at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163)
...
But it's all essentially the same.
Note that there is a LOT of these messages, they are being reported for 1 minute, after that, the log shows:
08:28:58,090 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting and closing JGroups Channel
08:28:58,455 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher
Please, take this as a preliminary feedback. Once we get an official build, we'll provide official QA statement.
> Re-deployment of a clustered application fail due to race-condition with infinispan
> -----------------------------------------------------------------------------------
>
> Key: WFLY-2458
> URL: https://issues.jboss.org/browse/WFLY-2458
> Project: WildFly
> Issue Type: Bug
> Security Level: Public(Everyone can see)
> Components: Clustering, Server
> Affects Versions: 8.0.0.Beta1
> Environment: Clustered Standalone server with one clustered EAR
> Reporter: Wolf-Dieter Fink
> Assignee: Paul Ferraro
> Labels: deployer
> Attachments: appone.ear
>
>
> If a clustered application should be redeployed it looks like that infinispan is not able to stop complete until the deployer finish undeployment and start to deploy the new application. See the Exception below.
> The behaviour is the same for managed or unmanged deployments.
> If it is unmanaged the .failed marker file exists.
> After the failure the application is shown in the mgmt console without deployed Beans.
> A second deploy attempt will be successfull
> 11:22:36,397 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,shared=udp) ISPN000094: Received new cluster view: [home/ejb|1] (2) [home/ejb, node2/ejb]
> 11:22:52,746 INFO [org.jboss.weld.deployer] (MSC service thread 1-11) JBAS016009: Stopping weld service for deployment appone.ear
> 11:22:52,751 INFO [org.infinispan.eviction.PassivationManagerImpl] (ServerService Thread Pool -- 58) ISPN000029: Passivating all entries to disk
> 11:22:52,752 INFO [org.infinispan.eviction.PassivationManagerImpl] (ServerService Thread Pool -- 58) ISPN000030: Passivated 0 entries in 0 milliseconds
> 11:22:52,759 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 58) JBAS010282: Stopped repl cache from ejb container
> 11:22:52,765 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 60) JBAS010282: Stopped remote-connector-client-mappings cache from ejb container
> 11:22:52,768 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000080: Disconnecting and closing JGroups Channel
> 11:22:52,771 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015877: Stopped deployment null (runtime-name: ejb.jar) in 36ms
> 11:22:52,772 INFO [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015877: Stopped deployment appone.ear (runtime-name: appone.ear) in 39ms
> 11:22:52,773 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) JBAS015876: Starting deployment of "appone.ear" (runtime-name: "appone.ear")
> 11:22:52,778 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "null" (runtime-name: "ejb.jar")
> 11:22:52,792 INFO [org.jboss.weld.deployer] (MSC service thread 1-12) JBAS016002: Processing weld deployment appone.ear
> 11:22:52,803 INFO [org.jboss.weld.deployer] (MSC service thread 1-5) JBAS016002: Processing weld deployment ejb.jar
> 11:22:52,805 INFO [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-5) JNDI bindings for session bean named AppOneBean in deployment unit subdeployment "ejb.jar" of deployment "appone.ear" are as follows:
> java:global/appone/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne
> java:app/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne
> java:module/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne
> java:jboss/exported/appone/ejb/AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne
> java:global/appone/ejb/AppOneBean
> java:app/ejb/AppOneBean
> java:module/AppOneBean
> 11:22:52,807 INFO [org.jboss.weld.deployer] (MSC service thread 1-11) JBAS016005: Starting Services for CDI deployment: appone.ear
> 11:22:52,811 INFO [org.jboss.weld.deployer] (MSC service thread 1-3) JBAS016008: Starting weld service for deployment appone.ear
> 11:22:53,087 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000082: Stopping the RpcDispatcher
> 11:22:53,118 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 60) ISPN000078: Starting JGroups Channel
> 11:22:53,119 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 60) MSC000001: Failed to start service jboss.infinispan.ejb.global-component-registry: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.global-component-registry: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
> at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.1.Final.jar:2.1.1.Final]
> Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:241)
> at org.jboss.as.clustering.infinispan.subsystem.GlobalComponentRegistryService.start(GlobalComponentRegistryService.java:33)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [wildfly-clustering-common-8.0.0.Beta2-SNAPSHOT.jar:8.0.0.Beta2-SNAPSHOT]
> ... 4 more
> Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on object of type JGroupsTransport
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:219)
> ... 6 more
> Caused by: org.infinispan.commons.CacheException: Unable to start JGroups Channel
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:196)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:185)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
> at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)
> ... 11 more
> Caused by: java.lang.IllegalStateException: channel is closed
> at org.jgroups.JChannel.checkClosed(JChannel.java:902)
> at org.jgroups.JChannel._preConnect(JChannel.java:522)
> at org.jgroups.JChannel.connect(JChannel.java:284)
> at org.jgroups.JChannel.connect(JChannel.java:275)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:194)
> ... 17 more
> 11:22:53,128 INFO [org.jboss.weld.deployer] (MSC service thread 1-7) JBAS016009: Stopping weld service for deployment appone.ear
> 11:22:53,137 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) JBAS015877: Stopped deployment null (runtime-name: ejb.jar) in 11ms
> 11:22:53,138 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015877: Stopped deployment appone.ear (runtime-name: appone.ear) in 12ms
> 11:22:53,143 INFO [org.jboss.as.controller] (DeploymentScanner-threads - 1) JBAS014774: Service status report
> JBAS014775: New missing/unsatisfied dependencies:
> service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.CREATE (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".moduleDeploymentRuntimeInformation, service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.START, service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.VIEW."org.jboss.as.quickstarts.ejb.multi.server.app.AppOne".REMOTE]
> service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".jndiDependencyService]
> service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.START (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".deploymentCompleteService, service jboss.deployment.subunit."appone.ear"."ejb.jar".moduleDeploymentRuntimeInformationStart]
> service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.VIEW."org.jboss.as.quickstarts.ejb.multi.server.app.AppOne".REMOTE (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".moduleDeploymentRuntimeInformation, service jboss.naming.context.java.module.appone.ejb."AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne", service jboss.naming.context.java.global.appone.ejb.AppOneBean, service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.START, JBAS014799: ... and 5 more ]
> service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.WeldInstantiator (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.START]
> service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.ejb.non-functional-timerservice (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.START]
> service jboss.deployment.subunit."appone.ear"."ejb.jar".deploymentCompleteService (missing) dependents: [service jboss.deployment.unit."appone.ear".deploymentCompleteService]
> service jboss.deployment.subunit."appone.ear"."ejb.jar".jndiDependencyService (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.START]
> service jboss.deployment.subunit."appone.ear"."ejb.jar".moduleDeploymentRuntimeInformation (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.START, service jboss.deployment.subunit."appone.ear"."ejb.jar".moduleDeploymentRuntimeInformationStart]
> service jboss.naming.context.java.app.appone.ejb.AppOneBean (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.app.appone.ejb."AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne" (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.comp.appone.ejb.AppOneBean (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.START]
> service jboss.naming.context.java.comp.appone.ejb.AppOneBean.BeanManager (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".jndiDependencyService]
> service jboss.naming.context.java.comp.appone.ejb.AppOneBean.DefaultContextService (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.comp.appone.ejb.AppOneBean.DefaultDataSource (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.comp.appone.ejb.AppOneBean.DefaultManagedExecutorService (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.comp.appone.ejb.AppOneBean.DefaultManagedScheduledExecutorService (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.comp.appone.ejb.AppOneBean.DefaultManagedThreadFactory (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.comp.appone.ejb.AppOneBean.EJBContext (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.comp.appone.ejb.AppOneBean.TimerService (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.comp.appone.ejb.AppOneBean.TransactionSynchronizationRegistry (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".jndiDependencyService]
> service jboss.naming.context.java.comp.appone.ejb.AppOneBean.UserTransaction (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".jndiDependencyService]
> service jboss.naming.context.java.comp.appone.ejb.AppOneBean.env (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.comp.appone.ejb.AppOneBean.env."org.jboss.as.quickstarts.ejb.multi.server.app.AppOneBean".context (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService, service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.START]
> service jboss.naming.context.java.global.appone.ejb.AppOneBean (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.global.appone.ejb."AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne" (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.module.appone.ejb.AppOneBean (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.module.appone.ejb."AppOneBean!org.jboss.as.quickstarts.ejb.multi.server.app.AppOne" (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".component.AppOneBean.JndiBindingsService]
> service jboss.naming.context.java.module.appone.ejb.BeanManager (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".jndiDependencyService]
> service jboss.naming.context.java.module.appone.ejb.env (missing) dependents: [service jboss.deployment.subunit."appone.ear"."ejb.jar".jndiDependencyService]
> JBAS014777: Services which failed to start: service jboss.infinispan.ejb.global-component-registry
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
More information about the jboss-jira
mailing list