[jboss-jira] [JBoss JIRA] (WFLY-11489) SFSB not sticky on a single cluster node when clustering of the bean is disabled
Richard Achmatowicz (Jira)
issues at jboss.org
Fri Jan 11 13:41:00 EST 2019
[ https://issues.jboss.org/browse/WFLY-11489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13681107#comment-13681107 ]
Richard Achmatowicz edited comment on WFLY-11489 at 1/11/19 1:40 PM:
---------------------------------------------------------------------
I've now had a look in detail at the affinity processing here and have an idea for a fix. There are three problems to overcome.
The first is to correct the assignment of strong and weak affinity for the SFSB with passivationEnabled == false. If the initial affinity of the proxy before session creation is (strong affinity, weak affinity) = (NONE, NONE), which is the default, then the affinity assignment we expect is either (NODE, NONE) or (NONE, NODE) which are equivalent. In other words, invocations should be tied to the node that the session is created on. This will require simple changes to the way the affinity is calculated - namely using the session cache do to the calculations.
The second problem is that the v3 protocol will not allow sending arbitrary affinity values back to the client: it is assumed that:
* updates to strong affinity are always of type ClusterAffinity
* updates to weak affinity are always of type NodeAffinity
For the code, see for example https://github.com/wildfly/jboss-ejb-client/blob/master/src/main/java/org/jboss/ejb/protocol/remote/EJBServerChannel.java#L705
This means that we can't assign (NODE, NONE) to the SFSB with passivationEnabled=false, but we can assign (NONE, NODE) which is equivalent. This seems to be the only way around this particular issue.
The third problem is that, on the client side, when the reply from the server is unpacked, there are some adjustments made to the affinity values before the invocation completes. These adjustments seem to interpret Affinity.NONE as "the affinity is not set", and try to replace Affinity.NONE with the targetAffinity which was determined when doing discovery. For example: https://github.com/wildfly/jboss-ejb-client/blob/master/src/main/java/org/jboss/ejb/client/DiscoveryEJBClientInterceptor.java#L172. In fact, Affinity.NONE means "I don't care which node you send me to" and does not mean "the affinity is not set". So, for example, the affinity adjustment made in the DiscoveryEJBClientInterceptor above will transform (NONE, NODE) into (NODE, NODE) or possibly (URI, NODE) where URI is a URIAffinity assigment for the locator pointing to the server used to create the session. In this case, such a manipulation may not hurt, but it seems to be unnecessary.
So, i'll be putting together a fix which incorporates these three changes. It would have been much easier and cleaner to be able to assign to the strong or weak affinity any affinity type, but with the v3 protocol, that is currently not possible.
PS For those interested, targetAffinity is assigned as a side effect when we choose a target for the invocation. Doing discovery results in choosing a serviceURL to send the invocation to which satisfies the strong/weak affinity constraints. That serviceURL contains a location together with optional node, module and cluster attributes. The location tells us how to connect to the node, the attributes tell us what the node name is, which modules are deployed there and which clusters it belongs to. targetAffinity is NodeAffinity(<node name>) if the serviceURL does have a node attribute, and URIAffinity(<location>) if there is no node attribute. It seems to be used to make adjustments to affinity on the client based on where the last invocation attempt went. I'm really not seeing why this is needed; it may be related to the assumptions made in the v3 protocol.
was (Author: rachmato):
I've now had a look in detail at the affinity processing here and have an idea for a fix. There are three problems to overcome.
The first is to correct the assignment of strong and weak affinity for the SFSB with passivationEnabled == false. If the initial affinity of the proxy before session creation is (strong affinity, weak affinity) = (NONE, NONE), which is the default, then the affinity assignment we expect is either (NODE, NONE) or (NONE, NODE) which are equivalent. In other words, invocations should be tied to the node that the session is created on. This will require simple changes to the way the affinity is calculated - namely using the session cache do to the calculations.
The second problem is that the v3 protocol will not allow sending arbitrary affinity values back to the client: it is assumed that:
* updates to strong affinity are always of type ClusterAffinity
* updates to weak affinity are always of type NodeAffinity
For the code, see for example https://github.com/wildfly/jboss-ejb-client/blob/master/src/main/java/org/jboss/ejb/protocol/remote/EJBServerChannel.java#L705
This means that we can't assign (NODE, NONE) to the SFSB with passivationEnabled=false, but we can assign (NONE, NODE) which is equivalent. This seems to be the only way around this particular issue.
The third problem is that, on the client side, when the reply from the server is unpacked, there are some adjustments made to the affinity values before the invocation completes. These adjustments seem to interpret Affinity.NONE as "the affinity is not set", and try to replace Affinity.NONE with the targetAffinity which was determined when doing discovery. For example: https://github.com/wildfly/jboss-ejb-client/blob/master/src/main/java/org/jboss/ejb/client/DiscoveryEJBClientInterceptor.java#L172. In fact, Affinity.NONE means "I don't care which node you send me to" and does not mean "the affinity is not set". So, for example, the affinity adjustment made in the DiscoveryEJBClientInterceptor above will transform (NONE, NODE) into (NODE, NODE) or possibly (URI, NODE) where URI is a URIAffinity assigment for the locator pointing to the server used to create the session. In this case, such a manipulation may not hurt, but it seems to be unnecessary.
So, i'll be putting together a fix which incorporates these three changes. It would have been much easier and cleaner to be able to assign to the strong or weak affinity any affinity type, but with the v3 protocol, that is currently not possible.
PS For those interested, targetAffinity is assigned as a side effect when we choose a target for the invocation. Doing discovery results in choosing a serviceURL to send the invocation to which satisfies the strong/weak affinity constraints. That serviceURL contains a location together with optional node, module and cluster attributes. The location tells us how to connect to the node, the attributes tell us what the node name is, which modules are deployed there and which clusters it belongs to. targetAffinity is NodeAffinity(<node name>) if the serviceURL does have a node attribute, and URIAffinity(<location>) if there is no node attribute. It seems to be used to make adjustments to affinity on the client based on where the last invocation attempt went. I'm really not seeing why this is needed.
> SFSB not sticky on a single cluster node when clustering of the bean is disabled
> --------------------------------------------------------------------------------
>
> Key: WFLY-11489
> URL: https://issues.jboss.org/browse/WFLY-11489
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, Remoting
> Affects Versions: 15.0.0.Final
> Environment: A clustered environment with 2 nodes. Both nodes started with an unmodified {{standalone-ha.xml}} configuration (see _Steps to Reproduce_)
> This issue happens on 15.0.0.Final and was tested as well as on current head (sha: 372697282dccefd0b9df48e6aa4dcb69e1c4b40f).
> Reporter: Jörg Bäsner
> Assignee: Richard Achmatowicz
> Priority: Major
> Attachments: reproducer.zip
>
>
> In case a stateful session bean is annotated with:
> {{@Stateful(passivationCapable=false)}}
> then the serialization is *disabled* and thus the bean is only available on a single cluster node.
> When a client now calls two different methods on this stateful bean it intermittently ends up on different cluster nodes, resulting in the following Exception:
> {code}
> ERROR [org.jboss.as.ejb3.invocation] (default task-2) WFLYEJB0034: EJB Invocation failed on component TestSessionEJB for method public abstract void test.ITestSession.method2(java.lang.String,java.lang.String) throws javax.ejb.EJBException,java.rmi.RemoteException: javax.ejb.NoSuchEJBException: WFLYEJB0168: Could not find EJB with id UUIDSessionID [4b0f4a27-40ba-411b-8852-0108a5ec64f4]
> at org.jboss.as.ejb3.component.stateful.StatefulComponentInstanceInterceptor.processInvocation(StatefulComponentInstanceInterceptor.java:55)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
> at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81)
> at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
> at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:406)
> at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:565)
> at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:546)
> at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:197)
> at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
More information about the jboss-jira
mailing list