[JBoss JIRA] (WFLY-4697) EJBCLient access to SFSBs does not play well with clean shutdown
by Stuart Douglas (JIRA)
[ https://issues.jboss.org/browse/WFLY-4697?page=com.atlassian.jira.plugin.... ]
Stuart Douglas commented on WFLY-4697:
--------------------------------------
Sorry, I missed your earlier comment. I did not realise that receiving the module unavailable message would trigger the client to disconnect, if this is the case then the approach sounds fine. Otherwise because connections can be very long lived this approach would not be great, as in many situations graceful shutdown will end up timing out.
> EJBCLient access to SFSBs does not play well with clean shutdown
> ----------------------------------------------------------------
>
> Key: WFLY-4697
> URL: https://issues.jboss.org/browse/WFLY-4697
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB, Remoting
> Affects Versions: 10.0.0.Alpha1
> Reporter: Richard Achmatowicz
> Assignee: Richard Achmatowicz
>
> The clean shutdown mechanism allows EJB and web applications to make use of shutdown interceptors to allow the application server to refuse requests when a server is in the process of shutting down. These interceptors are tied to the processing of EJB invocations and web requests.
> In the case of EJBCLient invocations, they arrive at the remoting connector, and undergo some preliminary processing before being sent to the EJB interface in question. When a server is shutting down, EJBCLient invocations can arrive at the RemoteConnector and start processing, even when the EJB interface has been locked down, so to speak.
> I am seeing various types of exceptions arising from this preliminary processing (e.g. NPE on DeploymentRepository lookups) and these get returned to the client as exceptions on the SFSB invocation, before even reaching the SFSB interceptors.
> If the EJBCLient is running in a managed transaction context, these returned exceptions will case the SFSB to be discarded (as the SFSB invocation is considered failed with the exception returned) and the transaction will attempt to rollback. If the rollback processing fails (because the original node in the transaction is down the bean gets removed anyway. The SFSB session state is lost, even if there is another node in the cluster which can support the invocation.
> In short, it looks as though the clean shutdown mechanism needs to be used to also lock down the processing of EJBCLient invocations in some way.
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years
[JBoss JIRA] (WFLY-4826) Add service name alias for a data source resource name
by James Perkins (JIRA)
James Perkins created WFLY-4826:
-----------------------------------
Summary: Add service name alias for a data source resource name
Key: WFLY-4826
URL: https://issues.jboss.org/browse/WFLY-4826
Project: WildFly
Issue Type: Feature Request
Components: JCA
Reporter: James Perkins
Assignee: James Perkins
Currently the data-source service name is bound to the JNDI name. An alias should be added to the resource name as well. For example the default {{ExampleDS}} is bound to {{jboss.data-source.java:jboss/datasources/ExampleDS}}. There should also be a binding to {{jboss.data-source.ExampleDS}} so other subsystems can lookup the data source.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years
[JBoss JIRA] (JGRP-1914) S3_PING doesn't work with S3 buckets created in Frankfurt region
by Mark Morris (JIRA)
[ https://issues.jboss.org/browse/JGRP-1914?page=com.atlassian.jira.plugin.... ]
Mark Morris updated JGRP-1914:
------------------------------
I agree, after I complete this sprint, which captures and implements the core functionality, refactoring into a new class implementation will be easier. I’m creating TestNG @Tests to exercise the functionality. I’m not altering any existing code beyond overriding existing methods with the addition of one argument, the region name. This will facilitate all existing clients and new or upgraded clients to implement AWS Authentication Version 4; the only supported version for new regions and Frankfurt (version 4 only). So, version 2 is being phased out. This implementation will provide a smooth and easy migration.
-Mark
> S3_PING doesn't work with S3 buckets created in Frankfurt region
> -----------------------------------------------------------------
>
> Key: JGRP-1914
> URL: https://issues.jboss.org/browse/JGRP-1914
> Project: JGroups
> Issue Type: Bug
> Reporter: Gleb Leonov
> Assignee: Bela Ban
> Fix For: 3.6.4
>
>
> I tried to use S3_PING with access and secret key pair. However, I got 400 (Bad Request) error. After some investigation, I found that modern S3 buckets created in Frankfurt needs amazon v4 authentication, which needs hmac-sha256. But now S3_PING supports only hmac-sha1, so I see no way to use S3_PING with buckets in Frankfurt region.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years
[JBoss JIRA] (WFLY-4697) EJBCLient access to SFSBs does not play well with clean shutdown
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-4697?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz commented on WFLY-4697:
-------------------------------------------
In the case of a managed transaction context. there are two problems when an invocation arrives at a server and comes back to the client with a NoSuchEJBException:
(i) the invocation will be automatically retried by the EJBClient retry mechanism, which should not happen as there is no replicated transaction state on the other nodes in the cluster
(ii) the retry will not take into account the fact that the TransactionInterceptor on the client side has already set up an XAResource on the client side to represent the server receiving the invocation, and that XAResource will ultimately fail when it tries to commit or rollback as it points to the wrong node. In fact, because the NoSuchEJBException was raised, no work was done on the server side, as we did not reach the EJB interceptors.
We need to disable retry of invocations in a managed context, because of point (i).
Due to the way the EJBClient invocation mechanism works, as well as the way transactions are associated to invocation threads, this is tricky. It is possible to assign a RETRY_DISABLED_KEY attachment to the invocation context when we know that the invocation is in an EJBCLientmanagedTransactionContext, so that when we are processing a response in the NoSuchEJBExceptionResponseHandler, we can avoid doing a retry at the message send/receive level. Additionally, the TransactionInterceptor processing of the result is only called when we call EJBClientInvocationContext.getResult(). It would be possible at this point to somehow clean up the XAResource, perhaps without triggering a rollback?
> EJBCLient access to SFSBs does not play well with clean shutdown
> ----------------------------------------------------------------
>
> Key: WFLY-4697
> URL: https://issues.jboss.org/browse/WFLY-4697
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB, Remoting
> Affects Versions: 10.0.0.Alpha1
> Reporter: Richard Achmatowicz
> Assignee: Richard Achmatowicz
>
> The clean shutdown mechanism allows EJB and web applications to make use of shutdown interceptors to allow the application server to refuse requests when a server is in the process of shutting down. These interceptors are tied to the processing of EJB invocations and web requests.
> In the case of EJBCLient invocations, they arrive at the remoting connector, and undergo some preliminary processing before being sent to the EJB interface in question. When a server is shutting down, EJBCLient invocations can arrive at the RemoteConnector and start processing, even when the EJB interface has been locked down, so to speak.
> I am seeing various types of exceptions arising from this preliminary processing (e.g. NPE on DeploymentRepository lookups) and these get returned to the client as exceptions on the SFSB invocation, before even reaching the SFSB interceptors.
> If the EJBCLient is running in a managed transaction context, these returned exceptions will case the SFSB to be discarded (as the SFSB invocation is considered failed with the exception returned) and the transaction will attempt to rollback. If the rollback processing fails (because the original node in the transaction is down the bean gets removed anyway. The SFSB session state is lost, even if there is another node in the cluster which can support the invocation.
> In short, it looks as though the clean shutdown mechanism needs to be used to also lock down the processing of EJBCLient invocations in some way.
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years
[JBoss JIRA] (WFLY-4697) EJBCLient access to SFSBs does not play well with clean shutdown
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-4697?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz edited comment on WFLY-4697 at 6/23/15 2:48 PM:
--------------------------------------------------------------------
I have been experimenting with this fix which uses preSuspend() and resume() to send out moduleUnavailable and moduleAvailable messages to all connected EJBClient connections. It works well in communicating to clients that they need to start redirecting invocations to another node. Consequently, invocations are no longer getting directed to a node shutting down and the exceptions above are greatly reduced in number, in most cases down to zero.
This works quite well in the case of RemoteFailoverTestCase (which was failing for similar reasons) where the invocations are not part of a managed transaction context. The only other change required to fix RemoteFailoverTestCase was to adjust the default value of the cache stop timeout (see cache transaction configuration). The cache stop timeout is used by Infinispan to allow a server which is shutting down to delay shutdown by stop-timeout ms in order to wait for any remote transactions it is involved in to complete (the default is 30 seconds). This delays the generation of the new cache topology. If a transaction on another node in the cluster (for example, the node we have just redirected the invocation to) is trying to commit, it will wait for the cache topology to arrive (the default wait time is 17.5 seconds). YThis was resulting in the following exceptions:
{noformat}
[0m[31m14:05:14,181 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (EJB default - 5) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 7
at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:92)
at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:227)
at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:95)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:367)
at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:413)
at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:303)
at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:104)
at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:73)
at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:48)
at org.jboss.as.ejb3.cache.distributable.DistributableCache.release(DistributableCache.java:139)
at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.releaseInstance(StatefulSessionSynchronizationInterceptor.java:173)
at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor$StatefulSessionSynchronization.afterCompletion(StatefulSessionSynchronizationInterceptor.java:257)
at org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.afterCompletion(JCAOrderedLastSynchronizationList.java:147)
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:545)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:101)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:91)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:635)
at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:331)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:69)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
{noformat}
Reducing the stop-timeout from 30 seconds to 10 seconds prevented these exceptions from occurring.
was (Author: rachmato):
I have been experimenting with this fix which uses preSuspend() and resume() to send out moduleUnavailable and moduleAvailable messages to all connected EJBClient connections. It works well in communicating to clients that they need to start redirecting invocations to another node. Consequently, invocations are no longer getting directed to a node shutting down and the exceptions above are greatly reduced in number.
This works quite well in the case of RemoteFailoverTestCase (which was failing for similar reasons) where the invocations are not part of a managed transaction context. The only other change required to fix RemoteFailoverTestCase was to adjust the default value of the cache stop timeout (see cache transaction configuration). The cache stop timeout is used by Infinispan to allow a server which is shutting down to delay shutdown by stop-timeout ms in order to wait for any remote transactions it is involved in to complete (the default is 30 seconds). This delays the generation of the new cache topology. If a transaction on another node in the cluster (for example, the node we have just redirected the invocation to) is trying to commit, it will wait for the cache topology to arrive (the default wait time is 17.5 seconds). YThis was resulting in the following exceptions:
{noformat}
[0m[31m14:05:14,181 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (EJB default - 5) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 7
at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:92)
at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:227)
at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:95)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:367)
at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:413)
at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:303)
at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:104)
at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:73)
at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:48)
at org.jboss.as.ejb3.cache.distributable.DistributableCache.release(DistributableCache.java:139)
at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.releaseInstance(StatefulSessionSynchronizationInterceptor.java:173)
at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor$StatefulSessionSynchronization.afterCompletion(StatefulSessionSynchronizationInterceptor.java:257)
at org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.afterCompletion(JCAOrderedLastSynchronizationList.java:147)
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:545)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:101)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:91)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:635)
at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:331)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:69)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
{noformat}
Reducing the stop-timeout from 30 seconds to 10 seconds prevented these exceptions from occurring.
> EJBCLient access to SFSBs does not play well with clean shutdown
> ----------------------------------------------------------------
>
> Key: WFLY-4697
> URL: https://issues.jboss.org/browse/WFLY-4697
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB, Remoting
> Affects Versions: 10.0.0.Alpha1
> Reporter: Richard Achmatowicz
> Assignee: Richard Achmatowicz
>
> The clean shutdown mechanism allows EJB and web applications to make use of shutdown interceptors to allow the application server to refuse requests when a server is in the process of shutting down. These interceptors are tied to the processing of EJB invocations and web requests.
> In the case of EJBCLient invocations, they arrive at the remoting connector, and undergo some preliminary processing before being sent to the EJB interface in question. When a server is shutting down, EJBCLient invocations can arrive at the RemoteConnector and start processing, even when the EJB interface has been locked down, so to speak.
> I am seeing various types of exceptions arising from this preliminary processing (e.g. NPE on DeploymentRepository lookups) and these get returned to the client as exceptions on the SFSB invocation, before even reaching the SFSB interceptors.
> If the EJBCLient is running in a managed transaction context, these returned exceptions will case the SFSB to be discarded (as the SFSB invocation is considered failed with the exception returned) and the transaction will attempt to rollback. If the rollback processing fails (because the original node in the transaction is down the bean gets removed anyway. The SFSB session state is lost, even if there is another node in the cluster which can support the invocation.
> In short, it looks as though the clean shutdown mechanism needs to be used to also lock down the processing of EJBCLient invocations in some way.
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years
[JBoss JIRA] (WFLY-4697) EJBCLient access to SFSBs does not play well with clean shutdown
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-4697?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz commented on WFLY-4697:
-------------------------------------------
I have been experimenting with this fix which uses preSuspend() and resume() to send out moduleUnavailable and moduleAvailable messages to all connected EJBClient connections. It works well in communicating to clients that they need to start redirecting invocations to another node. Consequently, invocations are no longer getting directed to a node shutting down and the exceptions above are greatly reduced in number.
This works quite well in the case of RemoteFailoverTestCase (which was failing for similar reasons) where the invocations are not part of a managed transaction context. The only other change required to fix RemoteFailoverTestCase was to adjust the default value of the cache stop timeout (see cache transaction configuration). The cache stop timeout is used by Infinispan to allow a server which is shutting down to delay shutdown by stop-timeout ms in order to wait for any remote transactions it is involved in to complete (the default is 30 seconds). This delays the generation of the new cache topology. If a transaction on another node in the cluster (for example, the node we have just redirected the invocation to) is trying to commit, it will wait for the cache topology to arrive (the default wait time is 17.5 seconds). YThis was resulting in the following exceptions:
{noformat}
[0m[31m14:05:14,181 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (EJB default - 5) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 7
at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:92)
at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:227)
at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:95)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:367)
at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:413)
at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:303)
at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:104)
at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:73)
at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:48)
at org.jboss.as.ejb3.cache.distributable.DistributableCache.release(DistributableCache.java:139)
at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.releaseInstance(StatefulSessionSynchronizationInterceptor.java:173)
at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor$StatefulSessionSynchronization.afterCompletion(StatefulSessionSynchronizationInterceptor.java:257)
at org.jboss.as.txn.service.internal.tsr.JCAOrderedLastSynchronizationList.afterCompletion(JCAOrderedLastSynchronizationList.java:147)
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:545)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:101)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:91)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:635)
at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:331)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:69)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
{noformat}
Reducing the stop-timeout from 30 seconds to 10 seconds prevented these exceptions from occurring.
> EJBCLient access to SFSBs does not play well with clean shutdown
> ----------------------------------------------------------------
>
> Key: WFLY-4697
> URL: https://issues.jboss.org/browse/WFLY-4697
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB, Remoting
> Affects Versions: 10.0.0.Alpha1
> Reporter: Richard Achmatowicz
> Assignee: Richard Achmatowicz
>
> The clean shutdown mechanism allows EJB and web applications to make use of shutdown interceptors to allow the application server to refuse requests when a server is in the process of shutting down. These interceptors are tied to the processing of EJB invocations and web requests.
> In the case of EJBCLient invocations, they arrive at the remoting connector, and undergo some preliminary processing before being sent to the EJB interface in question. When a server is shutting down, EJBCLient invocations can arrive at the RemoteConnector and start processing, even when the EJB interface has been locked down, so to speak.
> I am seeing various types of exceptions arising from this preliminary processing (e.g. NPE on DeploymentRepository lookups) and these get returned to the client as exceptions on the SFSB invocation, before even reaching the SFSB interceptors.
> If the EJBCLient is running in a managed transaction context, these returned exceptions will case the SFSB to be discarded (as the SFSB invocation is considered failed with the exception returned) and the transaction will attempt to rollback. If the rollback processing fails (because the original node in the transaction is down the bean gets removed anyway. The SFSB session state is lost, even if there is another node in the cluster which can support the invocation.
> In short, it looks as though the clean shutdown mechanism needs to be used to also lock down the processing of EJBCLient invocations in some way.
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
11 years