[JBoss JIRA] (WFLY-5812) HA singleton service shortly unreachable after being started on a node
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-5812?page=com.atlassian.jira.plugin.... ]
Radoslav Husar edited comment on WFLY-5812 at 12/14/15 10:47 AM:
-----------------------------------------------------------------
The ISE is thrown by the user code, so the user code should to handle it too:
https://github.com/Ladicek/SingletonServiceTest-ejb/blob/master/src/main/...
In either case the client needs to be resilient and deal with times when the service is not available, such as:
* quorum is not reached
* elected node crashed and the new one owner is not yet started
was (Author: rhusar):
The exceptions is thrown by the user code, so the user code should to handle it too:
https://github.com/Ladicek/SingletonServiceTest-ejb/blob/master/src/main/...
In either case the client needs to be resilient and deal with times when the service is not available, such as:
* quorum is not reached
* elected node crashed and the new one owner is not yet started
> HA singleton service shortly unreachable after being started on a node
> ----------------------------------------------------------------------
>
> Key: WFLY-5812
> URL: https://issues.jboss.org/browse/WFLY-5812
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.CR5
> Reporter: Michal Vinkler
> Assignee: Radoslav Husar
>
> Seen in our failover tests, jvmkill scenario with HA singleton service
> Timeline:
> - perf21 goes down (jvmkill)
> - perf18 receives new cluster view
> - perf18 is elected as the singleton provider of the service
> - the singleton service is started on perf18
> - EJB Invocation fails few times - see the stacktrace below
> Can you please check if the issue is in the [TestingSingletonService|https://github.com/Ladicek/SingletonServiceTest-e...] implementation or in the EAP server?
> {code}
> [JBossINF] [0m[0m22:28:23,791 INFO [org.wildfly.clustering.server] (notification-thread--p1-t1) WFLYCLSV0003: perf18 elected as the singleton provider of the jboss.test.myservice service
> [JBossINF] [0m[0m22:28:23,794 INFO [org.wildfly.clustering.server] (notification-thread--p1-t1) WFLYCLSV0001: This node will now operate as the singleton provider of the jboss.test.myservice service
> [JBossINF] [0m[0m22:28:23,797 INFO [org.jboss.test.singletonservice.TestingSingletonService] (MSC service thread 1-7) Start service 'org.jboss.test.singletonservice.TestingSingletonService'
> [JBossINF] [0m[0m22:28:23,798 INFO [org.jboss.test.singletonservice.TestingSingletonService] (MSC service thread 1-7) Service started on perf18
> [JBossINF] [0m[31m22:28:23,824 ERROR [org.jboss.as.ejb3.invocation] (EJB default - 2) WFLYEJB0034: EJB Invocation failed on component ServiceAccessBean for method public abstract java.lang.String org.jboss.test.singletonservice.ServiceAccess.getNodeNameOfService(): javax.ejb.EJBException: java.lang.IllegalStateException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> [JBossINF] at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> [JBossINF] at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> [JBossINF] at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> [JBossINF] at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
> [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:318)
> [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:68)
> [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> [JBossINF] Caused by: java.lang.IllegalStateException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
> [JBossINF] at org.wildfly.clustering.server.singleton.CacheSingletonServiceBuilder.getRemoteValueRef(CacheSingletonServiceBuilder.java:329)
> [JBossINF] at org.wildfly.clustering.server.singleton.CacheSingletonServiceBuilder.getValue(CacheSingletonServiceBuilder.java:281)
> [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder.getValue(AsynchronousServiceBuilder.java:92)
> [JBossINF] at org.jboss.msc.service.ServiceControllerImpl.getValue(ServiceControllerImpl.java:1158)
> [JBossINF] at org.jboss.test.singletonservice.ServiceAccessBean.getNodeNameOfService(ServiceAccessBean.java:20)
> [JBossINF] at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:497)
> [JBossINF] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> [JBossINF] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:82)
> [JBossINF] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
> [JBossINF] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> [JBossINF] at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
> [JBossINF] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> [JBossINF] at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
> [JBossINF] ... 41 more
> [JBossINF] Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
> [JBossINF] at org.wildfly.clustering.server.dispatcher.SimpleCommandResponse.<init>(SimpleCommandResponse.java:43)
> [JBossINF] at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcher.createCommandResponse(ChannelCommandDispatcher.java:206)
> [JBossINF] at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcher.executeOnCluster(ChannelCommandDispatcher.java:94)
> [JBossINF] at org.wildfly.clustering.server.singleton.CacheSingletonServiceBuilder.getRemoteValueRef(CacheSingletonServiceBuilder.java:298)
> [JBossINF] ... 73 more
> [JBossINF] Caused by: java.lang.reflect.InvocationTargetException
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:464)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684)
> [JBossINF] at org.jgroups.JChannel.up(JChannel.java:738)
> [JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
> [JBossINF] at org.jgroups.protocols.FORK.up(FORK.java:133)
> [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:383)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
> [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1045)
> [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> [JBossINF] at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:435)
> [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.deliver(NAKACK2.java:961)
> [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:840)
> [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618)
> [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> [JBossINF] at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)
> [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> [JBossINF] at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:295)
> [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
> [JBossINF] at org.jgroups.protocols.TP$3.run(TP.java:1511)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: java.lang.IllegalStateException
> [JBossINF] at org.jboss.test.singletonservice.TestingSingletonService.getValue(TestingSingletonService.java:30)
> [JBossINF] at org.jboss.test.singletonservice.TestingSingletonService.getValue(TestingSingletonService.java:17)
> [JBossINF] at org.wildfly.clustering.server.singleton.CacheSingletonServiceBuilder.getValueRef(CacheSingletonServiceBuilder.java:288)
> [JBossINF] at org.wildfly.clustering.server.singleton.SingletonValueCommand.execute(SingletonValueCommand.java:31)
> [JBossINF] at org.wildfly.clustering.server.singleton.SingletonValueCommand.execute(SingletonValueCommand.java:26)
> [JBossINF] at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcherFactory.handle(ChannelCommandDispatcherFactory.java:120)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.handle(MessageDispatcher.java:502)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
> [JBossINF] ... 25 more
> {code}
> Link:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-jvmki...
> Link to source of TestingSingletonService:
> https://github.com/Ladicek/SingletonServiceTest-ejb/blob/master/src/main/...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 5 months
[JBoss JIRA] (WFLY-5812) HA singleton service shortly unreachable after being started on a node
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-5812?page=com.atlassian.jira.plugin.... ]
Radoslav Husar resolved WFLY-5812.
----------------------------------
Resolution: Rejected
The exceptions is thrown by the user code, so the user code should to handle it too:
https://github.com/Ladicek/SingletonServiceTest-ejb/blob/master/src/main/...
In either case the client needs to be resilient and deal with times when the service is not available, such as:
* quorum is not reached
* elected node crashed and the new one owner is not yet started
> HA singleton service shortly unreachable after being started on a node
> ----------------------------------------------------------------------
>
> Key: WFLY-5812
> URL: https://issues.jboss.org/browse/WFLY-5812
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.CR5
> Reporter: Michal Vinkler
> Assignee: Radoslav Husar
>
> Seen in our failover tests, jvmkill scenario with HA singleton service
> Timeline:
> - perf21 goes down (jvmkill)
> - perf18 receives new cluster view
> - perf18 is elected as the singleton provider of the service
> - the singleton service is started on perf18
> - EJB Invocation fails few times - see the stacktrace below
> Can you please check if the issue is in the [TestingSingletonService|https://github.com/Ladicek/SingletonServiceTest-e...] implementation or in the EAP server?
> {code}
> [JBossINF] [0m[0m22:28:23,791 INFO [org.wildfly.clustering.server] (notification-thread--p1-t1) WFLYCLSV0003: perf18 elected as the singleton provider of the jboss.test.myservice service
> [JBossINF] [0m[0m22:28:23,794 INFO [org.wildfly.clustering.server] (notification-thread--p1-t1) WFLYCLSV0001: This node will now operate as the singleton provider of the jboss.test.myservice service
> [JBossINF] [0m[0m22:28:23,797 INFO [org.jboss.test.singletonservice.TestingSingletonService] (MSC service thread 1-7) Start service 'org.jboss.test.singletonservice.TestingSingletonService'
> [JBossINF] [0m[0m22:28:23,798 INFO [org.jboss.test.singletonservice.TestingSingletonService] (MSC service thread 1-7) Service started on perf18
> [JBossINF] [0m[31m22:28:23,824 ERROR [org.jboss.as.ejb3.invocation] (EJB default - 2) WFLYEJB0034: EJB Invocation failed on component ServiceAccessBean for method public abstract java.lang.String org.jboss.test.singletonservice.ServiceAccess.getNodeNameOfService(): javax.ejb.EJBException: java.lang.IllegalStateException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> [JBossINF] at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> [JBossINF] at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> [JBossINF] at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> [JBossINF] at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
> [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:318)
> [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:68)
> [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> [JBossINF] Caused by: java.lang.IllegalStateException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
> [JBossINF] at org.wildfly.clustering.server.singleton.CacheSingletonServiceBuilder.getRemoteValueRef(CacheSingletonServiceBuilder.java:329)
> [JBossINF] at org.wildfly.clustering.server.singleton.CacheSingletonServiceBuilder.getValue(CacheSingletonServiceBuilder.java:281)
> [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder.getValue(AsynchronousServiceBuilder.java:92)
> [JBossINF] at org.jboss.msc.service.ServiceControllerImpl.getValue(ServiceControllerImpl.java:1158)
> [JBossINF] at org.jboss.test.singletonservice.ServiceAccessBean.getNodeNameOfService(ServiceAccessBean.java:20)
> [JBossINF] at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:497)
> [JBossINF] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> [JBossINF] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:82)
> [JBossINF] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
> [JBossINF] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> [JBossINF] at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
> [JBossINF] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> [JBossINF] at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
> [JBossINF] ... 41 more
> [JBossINF] Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
> [JBossINF] at org.wildfly.clustering.server.dispatcher.SimpleCommandResponse.<init>(SimpleCommandResponse.java:43)
> [JBossINF] at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcher.createCommandResponse(ChannelCommandDispatcher.java:206)
> [JBossINF] at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcher.executeOnCluster(ChannelCommandDispatcher.java:94)
> [JBossINF] at org.wildfly.clustering.server.singleton.CacheSingletonServiceBuilder.getRemoteValueRef(CacheSingletonServiceBuilder.java:298)
> [JBossINF] ... 73 more
> [JBossINF] Caused by: java.lang.reflect.InvocationTargetException
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:464)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684)
> [JBossINF] at org.jgroups.JChannel.up(JChannel.java:738)
> [JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
> [JBossINF] at org.jgroups.protocols.FORK.up(FORK.java:133)
> [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:383)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
> [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1045)
> [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> [JBossINF] at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:435)
> [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.deliver(NAKACK2.java:961)
> [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:840)
> [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618)
> [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> [JBossINF] at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)
> [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> [JBossINF] at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:295)
> [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
> [JBossINF] at org.jgroups.protocols.TP$3.run(TP.java:1511)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: java.lang.IllegalStateException
> [JBossINF] at org.jboss.test.singletonservice.TestingSingletonService.getValue(TestingSingletonService.java:30)
> [JBossINF] at org.jboss.test.singletonservice.TestingSingletonService.getValue(TestingSingletonService.java:17)
> [JBossINF] at org.wildfly.clustering.server.singleton.CacheSingletonServiceBuilder.getValueRef(CacheSingletonServiceBuilder.java:288)
> [JBossINF] at org.wildfly.clustering.server.singleton.SingletonValueCommand.execute(SingletonValueCommand.java:31)
> [JBossINF] at org.wildfly.clustering.server.singleton.SingletonValueCommand.execute(SingletonValueCommand.java:26)
> [JBossINF] at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcherFactory.handle(ChannelCommandDispatcherFactory.java:120)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.handle(MessageDispatcher.java:502)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
> [JBossINF] ... 25 more
> {code}
> Link:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-jvmki...
> Link to source of TestingSingletonService:
> https://github.com/Ladicek/SingletonServiceTest-ejb/blob/master/src/main/...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 5 months
[JBoss JIRA] (WFCORE-1218) Connecting to controller via new CommandContext instance leak threads
by Alexey Loubyansky (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1218?page=com.atlassian.jira.plugi... ]
Alexey Loubyansky moved JBEAP-2395 to WFCORE-1218:
--------------------------------------------------
Project: WildFly Core (was: JBoss Enterprise Application Platform)
Key: WFCORE-1218 (was: JBEAP-2395)
Workflow: GIT Pull Request workflow (was: CDW v1)
Component/s: CLI
(was: CLI)
Target Release: (was: 7.0.0.GA)
Affects Version/s: (was: 7.0.0.ER2 (Beta))
Affects Testing: (was: Regression)
> Connecting to controller via new CommandContext instance leak threads
> ---------------------------------------------------------------------
>
> Key: WFCORE-1218
> URL: https://issues.jboss.org/browse/WFCORE-1218
> Project: WildFly Core
> Issue Type: Bug
> Components: CLI
> Reporter: Alexey Loubyansky
> Assignee: Alexey Loubyansky
> Priority: Blocker
>
> Connecting to a controller via a new CommandContext always [init a new console|https://github.com/jbossas/wildfly-core-eap/commit/9083ca79b4e30a...] which could lead into "OutOfMemoryError: unable to create new native thread" as initBasicConsole() creates a pair of Aesh thread which are not cleaned up with terminateSession().
> {code:java}
> import org.jboss.as.cli.scriptsupport.CLI;
> CLI cli = CLI.newInstance();
> for (int i = 1; i <= tries; i++) {
> cli.connect();
> cli.disconnect();
> }
> {code}
> Reproducer waits for 10s after every 50 connections, that's where the steps on the graph comes from:
> !7.0.0.er2.png|thumbnail!
> Regression against 7.0.0.ER1 and 6.4.x
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 5 months
[JBoss JIRA] (WFLY-5520) Manually migrated EAP 6.x profiles fail to boot due to 'default-clustered-sfsb-cache' used in runtime
by Ladislav Thon (JIRA)
[ https://issues.jboss.org/browse/WFLY-5520?page=com.atlassian.jira.plugin.... ]
Ladislav Thon updated WFLY-5520:
--------------------------------
Workaround Description:
Run this batch on manually migrated EAP6 profile:
{code}
batch
/subsystem=ejb3:undefine-attribute(name=default-clustered-sfsb-cache)
/subsystem=ejb3:write-attribute(name=default-sfsb-cache,value=clustered)
/subsystem=ejb3:write-attribute(name=default-sfsb-passivation-disabled-cache,value=simple)
run-batch
{code}
was:
Run this batch on manually curated EAP6 profile:
{code}
batch
/subsystem=ejb3:undefine-attribute(name=default-clustered-sfsb-cache)
/subsystem=ejb3:write-attribute(name=default-sfsb-cache,value=clustered)
/subsystem=ejb3:write-attribute(name=default-sfsb-passivation-disabled-cache,value=simple)
run-batch
{code}
> Manually migrated EAP 6.x profiles fail to boot due to 'default-clustered-sfsb-cache' used in runtime
> -----------------------------------------------------------------------------------------------------
>
> Key: WFLY-5520
> URL: https://issues.jboss.org/browse/WFLY-5520
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 10.0.0.CR2
> Reporter: Eduardo Martins
> Assignee: Radoslav Husar
> Priority: Critical
> Attachments: eap-6.1-migrated-configs.zip, eap-6.4-migrated-configs.zip
>
>
> If you try (curated) EAP 6 standalone-ha.xml configuration on WildFly the server will fail to boot due to an EJB3 subsystem attribute.
> Server boot console log:
> mbp:migration-tests emmartins$ ./default-config-manual-copy/wildfly-10.0.0.CR3-SNAPSHOT/bin/standalone.sh -c standalone-ha.xml
> =========================================================================
> JBoss Bootstrap Environment
> JBOSS_HOME: /Users/emmartins/wildfly/migration-tests/default-config-manual-copy/wildfly-10.0.0.CR3-SNAPSHOT
> JAVA: /Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/bin/java
> JAVA_OPTS: -server -Xms64m -Xmx512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
> =========================================================================
> 11:37:25,583 INFO [org.jboss.modules] (main) JBoss Modules version 1.4.4.Final
> 11:37:25,896 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
> 11:37:25,985 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0049: WildFly Full 10.0.0.CR3-SNAPSHOT (WildFly Core 2.0.0.CR6) starting
> 11:37:27,307 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 7) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
> 11:37:27,307 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 15) WFLYCTL0028: Attribute 'default-clustered-sfsb-cache' in the resource at address '/subsystem=ejb3' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
> 11:37:27,385 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 15) WFLYCTL0013: Operation ("add") failed - address: ([("subsystem" => "ejb3")]) - failure description: "WFLYEJB0451: Attribute 'default-clustered-sfsb-cache' is not supported on current version servers; it is only allowed if its value matches 'default-sfsb-cache'"
> 11:37:27,586 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) "WFLYCTL0193: Failed executing subsystem ejb3 boot operations"
> 11:37:27,588 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("parallel-subsystem-boot") failed - address: ([]) - failure description: "\"WFLYCTL0193: Failed executing subsystem ejb3 boot operations\""
> 11:37:27,591 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
> 11:37:27,594 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested.
> 11:37:27,603 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0050: WildFly Full 10.0.0.CR3-SNAPSHOT (WildFly Core 2.0.0.CR6) stopped in 6ms
> Attached is the standalone-ha.xml config, with just threads subsystem removed, and web subsystem migrated to undertow, that should be able to boot the server.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 5 months
[JBoss JIRA] (WFLY-5812) HA singleton service shortly unreachable after being started on a node
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-5812?page=com.atlassian.jira.plugin.... ]
Radoslav Husar reassigned WFLY-5812:
------------------------------------
Assignee: Radoslav Husar (was: Paul Ferraro)
> HA singleton service shortly unreachable after being started on a node
> ----------------------------------------------------------------------
>
> Key: WFLY-5812
> URL: https://issues.jboss.org/browse/WFLY-5812
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.CR5
> Reporter: Michal Vinkler
> Assignee: Radoslav Husar
>
> Seen in our failover tests, jvmkill scenario with HA singleton service
> Timeline:
> - perf21 goes down (jvmkill)
> - perf18 receives new cluster view
> - perf18 is elected as the singleton provider of the service
> - the singleton service is started on perf18
> - EJB Invocation fails few times - see the stacktrace below
> Can you please check if the issue is in the [TestingSingletonService|https://github.com/Ladicek/SingletonServiceTest-e...] implementation or in the EAP server?
> {code}
> [JBossINF] [0m[0m22:28:23,791 INFO [org.wildfly.clustering.server] (notification-thread--p1-t1) WFLYCLSV0003: perf18 elected as the singleton provider of the jboss.test.myservice service
> [JBossINF] [0m[0m22:28:23,794 INFO [org.wildfly.clustering.server] (notification-thread--p1-t1) WFLYCLSV0001: This node will now operate as the singleton provider of the jboss.test.myservice service
> [JBossINF] [0m[0m22:28:23,797 INFO [org.jboss.test.singletonservice.TestingSingletonService] (MSC service thread 1-7) Start service 'org.jboss.test.singletonservice.TestingSingletonService'
> [JBossINF] [0m[0m22:28:23,798 INFO [org.jboss.test.singletonservice.TestingSingletonService] (MSC service thread 1-7) Service started on perf18
> [JBossINF] [0m[31m22:28:23,824 ERROR [org.jboss.as.ejb3.invocation] (EJB default - 2) WFLYEJB0034: EJB Invocation failed on component ServiceAccessBean for method public abstract java.lang.String org.jboss.test.singletonservice.ServiceAccess.getNodeNameOfService(): javax.ejb.EJBException: java.lang.IllegalStateException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> [JBossINF] at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> [JBossINF] at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> [JBossINF] at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> [JBossINF] at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
> [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:318)
> [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:68)
> [JBossINF] at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:201)
> [JBossINF] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> [JBossINF] Caused by: java.lang.IllegalStateException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
> [JBossINF] at org.wildfly.clustering.server.singleton.CacheSingletonServiceBuilder.getRemoteValueRef(CacheSingletonServiceBuilder.java:329)
> [JBossINF] at org.wildfly.clustering.server.singleton.CacheSingletonServiceBuilder.getValue(CacheSingletonServiceBuilder.java:281)
> [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder.getValue(AsynchronousServiceBuilder.java:92)
> [JBossINF] at org.jboss.msc.service.ServiceControllerImpl.getValue(ServiceControllerImpl.java:1158)
> [JBossINF] at org.jboss.test.singletonservice.ServiceAccessBean.getNodeNameOfService(ServiceAccessBean.java:20)
> [JBossINF] at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:497)
> [JBossINF] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> [JBossINF] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:82)
> [JBossINF] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93)
> [JBossINF] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
> [JBossINF] at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
> [JBossINF] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> [JBossINF] at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
> [JBossINF] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> [JBossINF] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
> [JBossINF] ... 41 more
> [JBossINF] Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
> [JBossINF] at org.wildfly.clustering.server.dispatcher.SimpleCommandResponse.<init>(SimpleCommandResponse.java:43)
> [JBossINF] at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcher.createCommandResponse(ChannelCommandDispatcher.java:206)
> [JBossINF] at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcher.executeOnCluster(ChannelCommandDispatcher.java:94)
> [JBossINF] at org.wildfly.clustering.server.singleton.CacheSingletonServiceBuilder.getRemoteValueRef(CacheSingletonServiceBuilder.java:298)
> [JBossINF] ... 73 more
> [JBossINF] Caused by: java.lang.reflect.InvocationTargetException
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:464)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684)
> [JBossINF] at org.jgroups.JChannel.up(JChannel.java:738)
> [JBossINF] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
> [JBossINF] at org.jgroups.protocols.FORK.up(FORK.java:133)
> [JBossINF] at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:383)
> [JBossINF] at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
> [JBossINF] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1045)
> [JBossINF] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
> [JBossINF] at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:435)
> [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.deliver(NAKACK2.java:961)
> [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:840)
> [JBossINF] at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:618)
> [JBossINF] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
> [JBossINF] at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)
> [JBossINF] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
> [JBossINF] at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
> [JBossINF] at org.jgroups.protocols.Discovery.up(Discovery.java:295)
> [JBossINF] at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
> [JBossINF] at org.jgroups.protocols.TP$3.run(TP.java:1511)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] Caused by: java.lang.IllegalStateException
> [JBossINF] at org.jboss.test.singletonservice.TestingSingletonService.getValue(TestingSingletonService.java:30)
> [JBossINF] at org.jboss.test.singletonservice.TestingSingletonService.getValue(TestingSingletonService.java:17)
> [JBossINF] at org.wildfly.clustering.server.singleton.CacheSingletonServiceBuilder.getValueRef(CacheSingletonServiceBuilder.java:288)
> [JBossINF] at org.wildfly.clustering.server.singleton.SingletonValueCommand.execute(SingletonValueCommand.java:31)
> [JBossINF] at org.wildfly.clustering.server.singleton.SingletonValueCommand.execute(SingletonValueCommand.java:26)
> [JBossINF] at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcherFactory.handle(ChannelCommandDispatcherFactory.java:120)
> [JBossINF] at org.jgroups.blocks.MessageDispatcher.handle(MessageDispatcher.java:502)
> [JBossINF] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
> [JBossINF] ... 25 more
> {code}
> Link:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-jvmki...
> Link to source of TestingSingletonService:
> https://github.com/Ladicek/SingletonServiceTest-ejb/blob/master/src/main/...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 5 months
[JBoss JIRA] (WFLY-5520) Manually migrated EAP 6.x profiles fail to boot due to 'default-clustered-sfsb-cache' used in runtime
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-5520?page=com.atlassian.jira.plugin.... ]
Radoslav Husar updated WFLY-5520:
---------------------------------
Workaround Description:
Run this batch on manually curated EAP6 profile:
{code}
batch
/subsystem=ejb3:undefine-attribute(name=default-clustered-sfsb-cache)
/subsystem=ejb3:write-attribute(name=default-sfsb-cache,value=clustered)
/subsystem=ejb3:write-attribute(name=default-sfsb-passivation-disabled-cache,value=simple)
run-batch
{code}
was:
Run this batch on manually curated EAP6 profile:
{code}
batch
/subsystem=ejb3:undefine-attribute(name=default-clustered-sfsb-cache)
/subsystem=ejb3:write-attribute(name=default-sfsb-cache,value=distributable)
/subsystem=ejb3:write-attribute(name=default-sfsb-passivation-disabled-cache,value=simple)
run-batch
{code}
> Manually migrated EAP 6.x profiles fail to boot due to 'default-clustered-sfsb-cache' used in runtime
> -----------------------------------------------------------------------------------------------------
>
> Key: WFLY-5520
> URL: https://issues.jboss.org/browse/WFLY-5520
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 10.0.0.CR2
> Reporter: Eduardo Martins
> Assignee: Radoslav Husar
> Priority: Critical
> Attachments: eap-6.1-migrated-configs.zip, eap-6.4-migrated-configs.zip
>
>
> If you try (curated) EAP 6 standalone-ha.xml configuration on WildFly the server will fail to boot due to an EJB3 subsystem attribute.
> Server boot console log:
> mbp:migration-tests emmartins$ ./default-config-manual-copy/wildfly-10.0.0.CR3-SNAPSHOT/bin/standalone.sh -c standalone-ha.xml
> =========================================================================
> JBoss Bootstrap Environment
> JBOSS_HOME: /Users/emmartins/wildfly/migration-tests/default-config-manual-copy/wildfly-10.0.0.CR3-SNAPSHOT
> JAVA: /Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/bin/java
> JAVA_OPTS: -server -Xms64m -Xmx512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
> =========================================================================
> 11:37:25,583 INFO [org.jboss.modules] (main) JBoss Modules version 1.4.4.Final
> 11:37:25,896 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
> 11:37:25,985 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0049: WildFly Full 10.0.0.CR3-SNAPSHOT (WildFly Core 2.0.0.CR6) starting
> 11:37:27,307 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 7) WFLYCTL0028: Attribute 'default-stack' in the resource at address '/subsystem=jgroups' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
> 11:37:27,307 INFO [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 15) WFLYCTL0028: Attribute 'default-clustered-sfsb-cache' in the resource at address '/subsystem=ejb3' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
> 11:37:27,385 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 15) WFLYCTL0013: Operation ("add") failed - address: ([("subsystem" => "ejb3")]) - failure description: "WFLYEJB0451: Attribute 'default-clustered-sfsb-cache' is not supported on current version servers; it is only allowed if its value matches 'default-sfsb-cache'"
> 11:37:27,586 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) "WFLYCTL0193: Failed executing subsystem ejb3 boot operations"
> 11:37:27,588 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("parallel-subsystem-boot") failed - address: ([]) - failure description: "\"WFLYCTL0193: Failed executing subsystem ejb3 boot operations\""
> 11:37:27,591 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
> 11:37:27,594 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested.
> 11:37:27,603 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0050: WildFly Full 10.0.0.CR3-SNAPSHOT (WildFly Core 2.0.0.CR6) stopped in 6ms
> Attached is the standalone-ha.xml config, with just threads subsystem removed, and web subsystem migrated to undertow, that should be able to boot the server.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 5 months
[JBoss JIRA] (WFLY-5833) WFLYCLSV0006: Failed to reach quorum should not be logged as ERROR
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-5833?page=com.atlassian.jira.plugin.... ]
Radoslav Husar closed WFLY-5833.
--------------------------------
Resolution: Rejected
This is intentional.
The user expects the service to be run, uses quorum to mitigate split-brains and if the service is not run, this is not a desired state and is considered an error state.
> WFLYCLSV0006: Failed to reach quorum should not be logged as ERROR
> ------------------------------------------------------------------
>
> Key: WFLY-5833
> URL: https://issues.jboss.org/browse/WFLY-5833
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.CR4
> Reporter: Michal Vinkler
> Assignee: Radoslav Husar
> Priority: Minor
>
> When using a singleton deployment and specifying quorum, the servers log ERROR message when they fail to reach quorum:
> {code}
> [JBossINF] [0m[31m06:09:25,064 ERROR [org.wildfly.clustering.server] (notification-thread--p29-t1) WFLYCLSV0006: Failed to reach quorum of 4 for jboss.deployment.unit."clusterbench-ee7-singleton-descriptor.ear".FIRST_MODULE_USE service. No singleton master will be elected.
> {code}
> On the other hand, when reaching required quorum, WARNING message is logged:
> {code}
> [JBossINF] [0m[33m06:11:26,674 WARN [org.wildfly.clustering.server] (notification-thread--p29-t1) WFLYCLSV0007: Just reached required quorum of 4 for jboss.deployment.unit."clusterbench-ee7-singleton-descriptor.ear".FIRST_MODULE_USE service. If this cluster loses another member, no node will be chosen to provide this service.
> {code}
> As this is expected behavior not to elect new singleton provider when the cluster fails to reach the quorum, this message imho should be logged as a WARN.
> Link:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-singl...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 5 months
[JBoss JIRA] (WFLY-5833) WFLYCLSV0006: Failed to reach quorum should not be logged as ERROR
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-5833?page=com.atlassian.jira.plugin.... ]
Radoslav Husar reassigned WFLY-5833:
------------------------------------
Assignee: Radoslav Husar (was: Paul Ferraro)
> WFLYCLSV0006: Failed to reach quorum should not be logged as ERROR
> ------------------------------------------------------------------
>
> Key: WFLY-5833
> URL: https://issues.jboss.org/browse/WFLY-5833
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.CR4
> Reporter: Michal Vinkler
> Assignee: Radoslav Husar
> Priority: Minor
>
> When using a singleton deployment and specifying quorum, the servers log ERROR message when they fail to reach quorum:
> {code}
> [JBossINF] [0m[31m06:09:25,064 ERROR [org.wildfly.clustering.server] (notification-thread--p29-t1) WFLYCLSV0006: Failed to reach quorum of 4 for jboss.deployment.unit."clusterbench-ee7-singleton-descriptor.ear".FIRST_MODULE_USE service. No singleton master will be elected.
> {code}
> On the other hand, when reaching required quorum, WARNING message is logged:
> {code}
> [JBossINF] [0m[33m06:11:26,674 WARN [org.wildfly.clustering.server] (notification-thread--p29-t1) WFLYCLSV0007: Just reached required quorum of 4 for jboss.deployment.unit."clusterbench-ee7-singleton-descriptor.ear".FIRST_MODULE_USE service. If this cluster loses another member, no node will be chosen to provide this service.
> {code}
> As this is expected behavior not to elect new singleton provider when the cluster fails to reach the quorum, this message imho should be logged as a WARN.
> Link:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-singl...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 5 months
[JBoss JIRA] (JGRP-1989) Bundlers: reuse send buffer when transport == sync.
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/JGRP-1989?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero commented on JGRP-1989:
---------------------------------------
{quote}Even the most light-weight impl adds ~16 bytes to every message, and I'm not willing to add that cost.{quote}
Let me clarify why I returned to that idea, and why I think you should take a second look at it too, especially after this interesting reuse opportunities which you have found.
By abstracting the "buffer type" into some more abstract JGroups specific contract, one would be able to replace these reusable buffers with an off-heap pool. (Yea, I know we talked about that, and bear with me about the details of implementing such an efficient pool for now, let's simply agree that while it might be non trivial, it surely can be done. and it's not that hard considering you have a rather "stable" buffer size).
So with the {{byte[]}} solution the memory cost of JVM heap usage would be
* *A* {{(buffer_size + array_headers) * N}}
With a simple wrapper option like I am re-proposing now this would result into (what you say is unacceptable):
* *B* {{(~16 bytes + buffer_size + array_headers) * N}}
My guess is you consider this unacceptable because N is often large? But then we'll agree that the {{buffer_size}} is more concerning than the ~16 bytes.
But the ultimate solution looks like this:
* *C* {{(~16 bytes + cost_of_pointers_to_mmapped_memory) * N}}
Which imo looks much better as the overall cost is better predictable as it's not depending on the size of the objects being transmitted. Predictability is good, and {{N}} is probably not unbounded (I guess?) so that would allow people configuring it to make a clear decision on how much memory mapped space one wants to dedicate for this purpose.
So while we can certainly agree that *B* is less efficient than *A*, the reason I'd change the interface nevertheless is that it can then allow us to move on to solution *C* at due time. And really, those 16bytes, I have a hard time believing they make all the difference: compared to the buffer size itself I suspect that's a negligible overhead.
Not least, I believe the abstraction would make for a better internal API and integration points to either express reusage of buffers explicitly, or be able to perform some (optional?) validation for correct behaviour.
> Bundlers: reuse send buffer when transport == sync.
> ---------------------------------------------------
>
> Key: JGRP-1989
> URL: https://issues.jboss.org/browse/JGRP-1989
> Project: JGroups
> Issue Type: Enhancement
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 3.6.7
>
>
> With the addition of {{TCP_NIO2}}, all bundlers now create new send buffers for every message (or message list). This generates a lot of memory allocations, perhaps it is better to revert this change for *synchronous transports* such as {{UDP}} and {{TCP}}, and still create new buffers for *asynchronous transports* such as {{TCP_NIO2}}.
> Synchronous transports guarantee a message has been put on the wire when {{TP.send()}} returns, whereas asynchronous transports may only have completed a partial write (so we cannot reuse the buffer).
> The code in the bundler should check for this, and copy if async or not copy if sync.
> Whether or not a transport is sync is determined by a new abstract method that needs to be overridden by every transport.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 5 months