[JBoss JIRA] (WFLY-6489) Distributable session may not exist after redirect to same node with optimistic locking.
by Gabriel Lavoie (JIRA)
[ https://issues.jboss.org/browse/WFLY-6489?page=com.atlassian.jira.plugin.... ]
Gabriel Lavoie commented on WFLY-6489:
--------------------------------------
We previously attempted to disable batching on WildFly 8 without success. I just tried it on WildFly 10.1.0 and I am getting a lot of NullPointerException from DistributableSession when it attempts to do operations on the "batch" field, making such configuration unusable for now.
I was able though to do some session attribute changes and verify the replication behavior. By disabling transactions, I see that a whole session replication occurs on every operation on it.
We certainly don't want to replicate the whole session every times it changes as it is currently too big and we had a lot of performance issues with it in the past. Attribute replication could be acceptable (network/performance wise) but would prove difficult as we would have to track every non-immutable attributes retrieval and force a setAttribute() at the end of the request to force an update on the other nodes, which is done transparently with the transaction enabled.
> Distributable session may not exist after redirect to same node with optimistic locking.
> ----------------------------------------------------------------------------------------
>
> Key: WFLY-6489
> URL: https://issues.jboss.org/browse/WFLY-6489
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 8.2.1.Final, 10.0.0.Final, 10.1.0.Final
> Reporter: Gabriel Lavoie
> Assignee: Paul Ferraro
> Priority: Critical
> Attachments: wfly-6489-showcase.zip, wildfly-10-session-issue.zip
>
>
> I'm currently working on porting an application running on EAP 6.1 to WildFly 10 and am encountering multiple session/authentication issues with clustering enabled. Our login flow currently starts from a servlet that accepts the credentials, creates the session, then redirect to the welcome page.
> The first time we execute this flow after the startup of a node, the welcome page can't see at all the session created previously.
> - request.getSession() creates yet another session and a new session cookie is returned.
> - request.getSession(false) returns "null"
> On the second attempt, the flow works as expected.
> The issue can be reproduced on both a single node or a two nodes cluster, as long as <distributable /> is enabled in web.xml.
> We are currently using the master build https://ci.jboss.org/hudson/job/WildFly-latest-master/2244/, but the problem has been noticed on 10.0.0-Final and also 8.2.1-Final.
> I attached a sample web application that I used to reproduce the issue. Our standalone.xml is also included with the clustering configuration we've been using for the web/session cache.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 1 month
[JBoss JIRA] (WFLY-6497) Unable to build wildfly master against wildfly-core master
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/WFLY-6497?page=com.atlassian.jira.plugin.... ]
Darran Lofthouse resolved WFLY-6497.
------------------------------------
Resolution: Rejected
The correct command to build WildFly with an updated core is: -
{noformat}
mvn clean install -Dversion.org.wildfly.core=3.0.0.Alpha1-SNAPSHOT -Djboss.test.xml.validation.future.schemas=3.0.0.Alpha1-SNAPSHOT/wildfly-config_5_0.xsd
{noformat}
> Unable to build wildfly master against wildfly-core master
> ----------------------------------------------------------
>
> Key: WFLY-6497
> URL: https://issues.jboss.org/browse/WFLY-6497
> Project: WildFly
> Issue Type: Bug
> Components: Test Suite
> Reporter: Darran Lofthouse
> Assignee: Darran Lofthouse
> Labels: affects_elytron
> Fix For: 10.1.0.Final
>
>
> WildFly Core contains a later version of the management schema, WildFly now is failing to build with: -
> {noformat}
> mvn clean install -Dversion.org.wildfly.core=3.0.0.Alpha1-SNAPSHOT
> {noformat}
> The general failure is: -
> {noformat}
> testStandalone(org.jboss.as.test.smoke.subsystem.xml.StandardConfigsXMLValidationUnitTestCase) Time elapsed: 0.238 sec <<< FAILURE!
> java.lang.AssertionError: 1:60 systemId='file:/home/darranl/src/wildfly10/wildfly/testsuite/integration/smoke/target/StandardConfigsXMLValidationUnitTestCase5907684017774151739xml' cvc-elt.1.a: Cannot find the declaration of element 'server'. a possible cause may be that a subsystem is not using the most up to date schema.
> at org.junit.Assert.fail(Assert.java:88)
> at org.jboss.as.test.smoke.subsystem.xml.AbstractValidationUnitTest$ErrorHandlerImpl.error(AbstractValidationUnitTest.java:323)
> at org.apache.xerces.util.ErrorHandlerWrapper.error(ErrorHandlerWrapper.java:135)
> at org.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:394)
> at org.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:325)
> at org.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:282)
> at org.apache.xerces.impl.xs.XMLSchemaValidator.handleStartElement(XMLSchemaValidator.java:2143)
> at org.apache.xerces.impl.xs.XMLSchemaValidator.startElement(XMLSchemaValidator.java:788)
> at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:283)
> at org.apache.xerces.impl.XMLNSDocumentScannerImpl$NSContentDispatcher.scanRootElementHook(XMLNSDocumentScannerImpl.java:733)
> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1754)
> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:324)
> at org.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:875)
> at org.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:798)
> at org.apache.xerces.jaxp.validation.StreamValidatorHelper.validate(StreamValidatorHelper.java:186)
> at org.apache.xerces.jaxp.validation.ValidatorImpl.validate(ValidatorImpl.java:129)
> at javax.xml.validation.Validator.validate(Validator.java:124)
> at org.jboss.as.test.smoke.subsystem.xml.StandardConfigsXMLValidationUnitTestCase.parseXml(StandardConfigsXMLValidationUnitTestCase.java:175)
> at org.jboss.as.test.smoke.subsystem.xml.StandardConfigsXMLValidationUnitTestCase.testStandalone(StandardConfigsXMLValidationUnitTestCase.java:97)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 1 month
[JBoss JIRA] (WFCORE-1466) Wildfly SSL Setup Fails on HSM-Backed Keystore
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1466?page=com.atlassian.jira.plugi... ]
Darran Lofthouse commented on WFCORE-1466:
------------------------------------------
Ok, I will approve the PR - we do have a better solution coming in the future that will also be compatible with PKCS#11 but for now this will cover the two file based types plus Luna
> Wildfly SSL Setup Fails on HSM-Backed Keystore
> ----------------------------------------------
>
> Key: WFCORE-1466
> URL: https://issues.jboss.org/browse/WFCORE-1466
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Security
> Environment: Any host environment with an HSM for key management
> Reporter: Gregory Ramsperger
> Assignee: Darran Lofthouse
>
> Using a keystore type that does not allow or returns empty from getEncoded() on private keys causes a KeyStoreException at startup. This is common in HSM-backed key operations.
> Storing SSL keys and certs in an HSM is a common method of securing keys and offloading SSL overhead.
> FileKeyStore.java copies a KeyStore.Entry value into a JKS KeyStore but JKS and PKCS12 KeyStore implementations maintain a copy of the encoded PKCS#8 data for private keys. When applying a KeyStore.Entry from a source that does not return the data for security reasons, the import fails.
> While it's still not guaranteed to work with all KeyStore providers, switching {{KeyStore.getInstance("JKS")}} to {{KeyStore.getInstance(provider)}} fixes the issue for SafeNet "Luna" and SunPKCS11 "PKCS11" KeyStore implementations while not breaking the "PKCS12" and "JKS" cases.
> See [https://github.com/wildfly/wildfly-core/blob/master/domain-management/src...]
> Log output:
> {noformat}
> 2016-04-04 18:53:51,100 i-4b6f79d1 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000001: Failed to start service jboss.server.controller.management.security_realm.test.key-manager: org.jboss.msc.service.StartException in service jboss.server.controller.management.security_realm.test.key-manager: JBAS015229: Unable to start service
> at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:148)
> at org.jboss.as.domain.management.security.FileKeyManagerService.start(FileKeyManagerService.java:119)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_60]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_60]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_60]
> Caused by: java.security.KeyStoreException: Cannot get key bytes, not PKCS#8 encoded
> at sun.security.provider.KeyProtector.protect(KeyProtector.java:174) [rt.jar:1.8.0_60]
> at sun.security.provider.JavaKeyStore.engineSetKeyEntry(JavaKeyStore.java:267) [rt.jar:1.8.0_60]
> at sun.security.provider.JavaKeyStore$JKS.engineSetKeyEntry(JavaKeyStore.java:56) [rt.jar:1.8.0_60]
> at java.security.KeyStoreSpi.engineSetEntry(KeyStoreSpi.java:537) [rt.jar:1.8.0_60]
> at sun.security.provider.KeyStoreDelegator.engineSetEntry(KeyStoreDelegator.java:179) [rt.jar:1.8.0_60]
> at sun.security.provider.JavaKeyStore$DualFormatJKS.engineSetEntry(JavaKeyStore.java:70) [rt.jar:1.8.0_60]
> at java.security.KeyStore.setEntry(KeyStore.java:1557) [rt.jar:1.8.0_60]
> at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:136)
> ... 6 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 1 month
[JBoss JIRA] (DROOLS-711) Kie Server unable to build KieBase which builds fine with kie-maven-plugin and unit test
by Kaan Bingol (JIRA)
[ https://issues.jboss.org/browse/DROOLS-711?page=com.atlassian.jira.plugin... ]
Kaan Bingol commented on DROOLS-711:
------------------------------------
My issue has been resolved by changing packing value kjar to jar in the pom file of the rule project.
> Kie Server unable to build KieBase which builds fine with kie-maven-plugin and unit test
> ----------------------------------------------------------------------------------------
>
> Key: DROOLS-711
> URL: https://issues.jboss.org/browse/DROOLS-711
> Project: Drools
> Issue Type: Bug
> Affects Versions: 6.2.0.CR4, 6.2.0.Final
> Environment: kie-server on WildFly 8.2.0.Final running with JDK 8
> Reporter: Matteo Mortari
> Assignee: Mario Fusco
> Attachments: 20150213.DROOLS-711.zip
>
>
> I have deployed Kie Server kie-server-distribution-wars-6.2.0.CR4-ee7.war on WildFly 8.2.0.Final.
> I have a simple rule base in the project {{client-test-rules}} as per below, in a kie project structured as kie-drools-archetype, which passes all test and install in m2 repo with {{mvn clean install}}.
> {code:java}
> package com.acme.client_test_rules;
> declare Measurement
> @role(event)
> end
> rule "keep latest Measurement ID"
> salience 1000
> no-loop
> when
> $old : Measurement( $id : id )
> $new : Measurement( id == $id , this after $old )
> then
> System.out.println("keep latest old : "+$old+" new (kept): "+$new);
> retract($old);
> end
> {code}
> Then I issue Kie Server commands, using the client library, using project {{client-test}} to instantiate a container for the same artifact, but in the wildfly logs I get the following error below - which doesn't happen for kie-maven-plugin nor during unit test. Please note I'm assuming user {{kieserver}} with password {{kieserver}} is a valid user for the Kie Server Realm, or configure wildfly or the code in the project {{client-test}} accordingly to your environment =).
> {code}
> 09:41:41,700 ERROR [org.drools.compiler.kie.builder.impl.AbstractKieModule] (default task-4) Unable to build KieBaseModel:kbase1
> Unable to create Field Extractor for 'id'Field/method 'id' not found for class 'com.acme.client_test_rules.Measurement'
> : [Rule name='keep latest Measurement ID']
> java.lang.RuntimeException: Field/method 'id' not found for class 'com.acme.client_test_rules.Measurement'
> Unable to create Field Extractor for 'id'Field/method 'id' not found for class 'com.acme.client_test_rules.Measurement'
> : [Rule name='keep latest Measurement ID']
> java.lang.RuntimeException: Field/method 'id' not found for class 'com.acme.client_test_rules.Measurement'
> Field Reader does not exist for declaration '$id' in 'predicate 'id == $id'' in the rule 'keep latest Measurement ID' : [Rule name='keep latest Measurement ID']
> Unable to Analyse Expression id == $id:
> [Error: unable to resolve method using strict-mode: com.acme.client_test_rules.Measurement.id()]
> [Near : {... id == $id ....}]
> ^
> [Line: 12, Column: 4] : [Rule name='keep latest Measurement ID']
> {code}
> Can you kindly verify if some bug is hiding behind this odd behavior, please?
> Thank you
> MM
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 1 month
[JBoss JIRA] (DROOLS-711) Kie Server unable to build KieBase which builds fine with kie-maven-plugin and unit test
by Mario Fusco (JIRA)
[ https://issues.jboss.org/browse/DROOLS-711?page=com.atlassian.jira.plugin... ]
Mario Fusco reassigned DROOLS-711:
----------------------------------
Assignee: Mario Fusco (was: Edson Tirelli)
> Kie Server unable to build KieBase which builds fine with kie-maven-plugin and unit test
> ----------------------------------------------------------------------------------------
>
> Key: DROOLS-711
> URL: https://issues.jboss.org/browse/DROOLS-711
> Project: Drools
> Issue Type: Bug
> Affects Versions: 6.2.0.CR4, 6.2.0.Final
> Environment: kie-server on WildFly 8.2.0.Final running with JDK 8
> Reporter: Matteo Mortari
> Assignee: Mario Fusco
> Attachments: 20150213.DROOLS-711.zip
>
>
> I have deployed Kie Server kie-server-distribution-wars-6.2.0.CR4-ee7.war on WildFly 8.2.0.Final.
> I have a simple rule base in the project {{client-test-rules}} as per below, in a kie project structured as kie-drools-archetype, which passes all test and install in m2 repo with {{mvn clean install}}.
> {code:java}
> package com.acme.client_test_rules;
> declare Measurement
> @role(event)
> end
> rule "keep latest Measurement ID"
> salience 1000
> no-loop
> when
> $old : Measurement( $id : id )
> $new : Measurement( id == $id , this after $old )
> then
> System.out.println("keep latest old : "+$old+" new (kept): "+$new);
> retract($old);
> end
> {code}
> Then I issue Kie Server commands, using the client library, using project {{client-test}} to instantiate a container for the same artifact, but in the wildfly logs I get the following error below - which doesn't happen for kie-maven-plugin nor during unit test. Please note I'm assuming user {{kieserver}} with password {{kieserver}} is a valid user for the Kie Server Realm, or configure wildfly or the code in the project {{client-test}} accordingly to your environment =).
> {code}
> 09:41:41,700 ERROR [org.drools.compiler.kie.builder.impl.AbstractKieModule] (default task-4) Unable to build KieBaseModel:kbase1
> Unable to create Field Extractor for 'id'Field/method 'id' not found for class 'com.acme.client_test_rules.Measurement'
> : [Rule name='keep latest Measurement ID']
> java.lang.RuntimeException: Field/method 'id' not found for class 'com.acme.client_test_rules.Measurement'
> Unable to create Field Extractor for 'id'Field/method 'id' not found for class 'com.acme.client_test_rules.Measurement'
> : [Rule name='keep latest Measurement ID']
> java.lang.RuntimeException: Field/method 'id' not found for class 'com.acme.client_test_rules.Measurement'
> Field Reader does not exist for declaration '$id' in 'predicate 'id == $id'' in the rule 'keep latest Measurement ID' : [Rule name='keep latest Measurement ID']
> Unable to Analyse Expression id == $id:
> [Error: unable to resolve method using strict-mode: com.acme.client_test_rules.Measurement.id()]
> [Near : {... id == $id ....}]
> ^
> [Line: 12, Column: 4] : [Rule name='keep latest Measurement ID']
> {code}
> Can you kindly verify if some bug is hiding behind this odd behavior, please?
> Thank you
> MM
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 1 month
[JBoss JIRA] (WFLY-6498) EJBClient UserTransactions with multiple method invocations generate lock conflicts
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-6498?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz commented on WFLY-6498:
-------------------------------------------
I seem to remember that because the first invocation creates a new batch and that batch is not committed/rolled back (as the outer txn has not finished and does not call release), the second invocation (on a different thread) creates a new batch, each batch has different lock owners, and they are trying to access the same session. So the lock conflicts arise.
So the problem here seems to be that after the first invocation, the Batch is closed but the count has not reached zero and so the txn is not comitted/rolledback, leaving the locks still held.
> EJBClient UserTransactions with multiple method invocations generate lock conflicts
> ------------------------------------------------------------------------------------
>
> Key: WFLY-6498
> URL: https://issues.jboss.org/browse/WFLY-6498
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.Final
> Reporter: Richard Achmatowicz
> Assignee: Paul Ferraro
> Fix For: 11.0.0.Alpha1
>
>
> Using the EJBClient library, we should be able to do the following from a standalone EJBClient application:
> // create a UserTransaction associated with a clustered server node X
> // make an invocation on an EJB on X
> // make an invocation on an EJB on X
> // commit the UserTransaction
> However, doing so results in this exception which occurs during processing of the second invocation:
> {noformat}
> [0m[31m11:16:22,580 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-57) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166] and requestor GlobalTransaction:<node-0>:120:local. Lock is held by GlobalTransaction:<node-0>:118:local
> at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:190)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:192)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:113)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:70)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:77)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)
> at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitReadCommand(StateTransferInterceptor.java:176)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitGetKeyValueCommand(StateTransferInterceptor.java:153)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:411)
> at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:443)
> at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
> at org.wildfly.clustering.ejb.infinispan.bean.InfinispanBeanFactory.findValue(InfinispanBeanFactory.java:87)
> at org.wildfly.clustering.ejb.infinispan.bean.InfinispanBeanFactory.findValue(InfinispanBeanFactory.java:49)
> at org.wildfly.clustering.ejb.infinispan.InfinispanBeanManager.findBean(InfinispanBeanManager.java:244)
> at org.jboss.as.ejb3.cache.distributable.DistributableCache.get(DistributableCache.java:124)
> at org.jboss.as.ejb3.component.stateful.StatefulComponentInstanceInterceptor.processInvocation(StatefulComponentInstanceInterceptor.java:59)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:254)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:333)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:327)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:200)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:262)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> {noformat}
> The exception does not arise if we perform only one invocation within the UserTransaction.
> This exception is repeatable.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 1 month
[JBoss JIRA] (WFLY-6498) EJBClient UserTransactions with multiple method invocations generate lock conflicts
by Richard Achmatowicz (JIRA)
[ https://issues.jboss.org/browse/WFLY-6498?page=com.atlassian.jira.plugin.... ]
Richard Achmatowicz commented on WFLY-6498:
-------------------------------------------
{noformat}
First invocation in userTransaction1 on target node-0
11:16:07,558 INFO [stdout] (default task-56) org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor: Invocation has transaction id: UserTransactionID [484948545469547054525453506851486850526669535566]
//
// UserTransaction defined, create a txn on the server
//
11:16:07,560 INFO [stdout] (default task-56) org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor: creating transaction
11:16:07,562 INFO [stdout] (default task-56) org.jboss.as.ejb3.tx.CMTTxInterceptor: got transaction from txn manager: status = 0
11:16:07,562 INFO [stdout] (default task-56) org.jboss.as.ejb3.component.stateful.StatefulComponentInstanceInterceptor: looking for stateful component instance wih session id: UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166]
11:16:07,562 INFO [stdout] (default task-56) org.jboss.as.ejb3.cache.distributable.DistributableCache: get(UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166]) - start
11:16:07,562 INFO [stdout] (default task-56) org.wildfly.clustering.ee.infinispan.InfinispanBatcher: entering createBatch()
11:16:07,562 INFO [stdout] (default task-56) org.wildfly.clustering.ee.infinispan.InfinispanBatcher: no current batch exists: creating new batch
11:16:07,563 INFO [stdout] (default task-56) org.wildfly.clustering.ejb.infinispan.InfinispanBeanManager: locating bean UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166]
11:16:07,564 INFO [stdout] (default task-56) org.jboss.as.ejb3.cache.distributable.DistributableCache: get() - end
11:16:07,564 INFO [stdout] (default task-56) org.jboss.as.ejb3.component.stateful.StatefulComponentInstanceInterceptor: invoking instance instance wih session id: UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166]
11:16:07,564 INFO [stdout] (default task-56) org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor: lock owner (based on active txn or thread) = 0:ffffc0a80067:-55fecb13:570527e9:d9
11:16:07,564 INFO [stdout] (default task-56) org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor: trying to acquire lock: org.jboss.as.ejb3.tx.OwnableReentrantLock@2345f95e[Unlocked] for stateful component instance: Instance of StatefulIncrementorBean {UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166]} during invocation: org.jboss.invocation.InterceptorContext@6e12e314
11:16:07,565 INFO [stdout] (default task-56) org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor: acquired lock: org.jboss.as.ejb3.tx.OwnableReentrantLock@2345f95e[Locked by 0:ffffc0a80067:-55fecb13:570527e9:d9] for stateful component instance: Instance of StatefulIncrementorBean {UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166]} during invocation: org.jboss.invocation.InterceptorContext@6e12e314
11:16:07,565 INFO [stdout] (default task-56) org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor: using container managed txns - check if we need to register sync
11:16:07,565 INFO [stdout] (default task-56) org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor: registered tx synchronization: org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor$StatefulSessionSynchronization@1a7a265f for tx: 0:ffffc0a80067:-55fecb13:570527e9:d9 associated with stateful component instance: Instance of StatefulIncrementorBean {UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166]}
//
// register a synchronization for the UserTransaction (to call release)
//
11:16:07,565 INFO [stdout] (default task-56) org.jboss.as.ejb3.cache.distributable.DistributableCache: getWeakAffinity() - start
11:16:07,565 INFO [stdout] (default task-56) org.wildfly.clustering.ee.infinispan.InfinispanBatcher: entering createBatch()
11:16:07,565 INFO [stdout] (default task-56) org.wildfly.clustering.ee.infinispan.InfinispanBatcher: current batch exists: interposing
11:16:07,565 INFO [stdout] (default task-56) org.wildfly.clustering.ee.infinispan.InfinispanBatch: interposing batch: count = 1
11:16:07,566 INFO [stdout] (default task-56) org.jboss.as.ejb3.cache.distributable.DistributableCache: getWeakAffinity() - end
11:16:07,566 INFO [stdout] (default task-56) org.wildfly.clustering.ee.infinispan.InfinispanBatch: closing batch - count = 1
--------------------------------------------------------------
Second invocation in userTransaction1 on target node-0
11:16:07,573 INFO [stdout] (default task-57) org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor: Invocation has transaction id: UserTransactionID [484948545469547054525453506851486850526669535566]
//
// UserTransaction defined, resume the txn on the server
//
11:16:07,573 INFO [stdout] (default task-57) org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor: resuming transaction
11:16:07,577 INFO [stdout] (default task-57) org.jboss.as.ejb3.tx.CMTTxInterceptor: got transaction from txn manager: status = 0
11:16:07,578 INFO [stdout] (default task-57) org.jboss.as.ejb3.component.stateful.StatefulComponentInstanceInterceptor: looking for stateful component instance wih session id: UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166]
11:16:07,578 INFO [stdout] (default task-57) org.jboss.as.ejb3.cache.distributable.DistributableCache: get(UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166]) - start
11:16:07,578 INFO [stdout] (default task-57) org.wildfly.clustering.ee.infinispan.InfinispanBatcher: entering createBatch()
11:16:07,578 INFO [stdout] (default task-57) org.wildfly.clustering.ee.infinispan.InfinispanBatcher: no current batch exists: creating new batch
11:16:07,578 INFO [stdout] (default task-57) org.wildfly.clustering.ejb.infinispan.InfinispanBeanManager: locating bean UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166]
11:16:22,580 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-57) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166] and requestor GlobalTransaction:<node-0>:120:local. Lock is held by GlobalTransaction:<node-0>:118:local
at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:190)
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:192)
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:113)
at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:70)
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:77)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)
at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.statetransfer.StateTransferInterceptor.visitReadCommand(StateTransferInterceptor.java:176)
at org.infinispan.statetransfer.StateTransferInterceptor.visitGetKeyValueCommand(StateTransferInterceptor.java:153)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:411)
at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:443)
at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
at org.wildfly.clustering.ejb.infinispan.bean.InfinispanBeanFactory.findValue(InfinispanBeanFactory.java:87)
at org.wildfly.clustering.ejb.infinispan.bean.InfinispanBeanFactory.findValue(InfinispanBeanFactory.java:49)
at org.wildfly.clustering.ejb.infinispan.InfinispanBeanManager.findBean(InfinispanBeanManager.java:244)
at org.jboss.as.ejb3.cache.distributable.DistributableCache.get(DistributableCache.java:124)
at org.jboss.as.ejb3.component.stateful.StatefulComponentInstanceInterceptor.processInvocation(StatefulComponentInstanceInterceptor.java:59)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:254)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:333)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:80)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:327)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:200)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:262)
at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
{noformat}
> EJBClient UserTransactions with multiple method invocations generate lock conflicts
> ------------------------------------------------------------------------------------
>
> Key: WFLY-6498
> URL: https://issues.jboss.org/browse/WFLY-6498
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.Final
> Reporter: Richard Achmatowicz
> Assignee: Paul Ferraro
> Fix For: 11.0.0.Alpha1
>
>
> Using the EJBClient library, we should be able to do the following from a standalone EJBClient application:
> // create a UserTransaction associated with a clustered server node X
> // make an invocation on an EJB on X
> // make an invocation on an EJB on X
> // commit the UserTransaction
> However, doing so results in this exception which occurs during processing of the second invocation:
> {noformat}
> [0m[31m11:16:22,580 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-57) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key UnknownSessionID [4967684957516649565452525270575756545166695455535750486549485166] and requestor GlobalTransaction:<node-0>:120:local. Lock is held by GlobalTransaction:<node-0>:118:local
> at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:236)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:190)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:192)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:113)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:70)
> at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:77)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)
> at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitReadCommand(StateTransferInterceptor.java:176)
> at org.infinispan.statetransfer.StateTransferInterceptor.visitGetKeyValueCommand(StateTransferInterceptor.java:153)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
> at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:411)
> at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:443)
> at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
> at org.wildfly.clustering.ejb.infinispan.bean.InfinispanBeanFactory.findValue(InfinispanBeanFactory.java:87)
> at org.wildfly.clustering.ejb.infinispan.bean.InfinispanBeanFactory.findValue(InfinispanBeanFactory.java:49)
> at org.wildfly.clustering.ejb.infinispan.InfinispanBeanManager.findBean(InfinispanBeanManager.java:244)
> at org.jboss.as.ejb3.cache.distributable.DistributableCache.get(DistributableCache.java:124)
> at org.jboss.as.ejb3.component.stateful.StatefulComponentInstanceInterceptor.processInvocation(StatefulComponentInstanceInterceptor.java:59)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:254)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:333)
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
> at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:327)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:200)
> at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:262)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.processMessage(VersionOneProtocolChannelReceiver.java:213)
> at org.jboss.as.ejb3.remote.protocol.versiontwo.VersionTwoProtocolChannelReceiver.processMessage(VersionTwoProtocolChannelReceiver.java:76)
> at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:159)
> {noformat}
> The exception does not arise if we perform only one invocation within the UserTransaction.
> This exception is repeatable.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 1 month
[JBoss JIRA] (WFLY-4616) Problem with distributable sessions on long IO
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/WFLY-4616?page=com.atlassian.jira.plugin.... ]
Paul Ferraro commented on WFLY-4616:
------------------------------------
It is like you need to modify the default infinispan configuration for web session caching. Long requests are not the norm, so the default configuration may not be suitable for your use case. I'm going to need many more details of your configuration/use case than just the stacktrace above.
> Problem with distributable sessions on long IO
> ----------------------------------------------
>
> Key: WFLY-4616
> URL: https://issues.jboss.org/browse/WFLY-4616
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 8.1.0.Final, 8.2.0.Final, 10.0.0.Final
> Reporter: Paweł Goździkowski
> Assignee: Paul Ferraro
> Priority: Critical
>
> Hi,
> I faced problem with wildfly <distributable/> what I want to ask is if it will be fixed/or is fixed in new versions on wildfly.
> The problem is when server starts performing long operation, ex. moving file to ntfs file system wildfly hangs over and starts to throw exceptions.. I tested it on wildfly 8.1 and 8.2 and wildfly and problem still occures.
> Disscusion about it you can find:
> https://developer.jboss.org/message/908454?et=watches.email.thread#908454
> Exception:
> 13:19:00,670 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-19) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [15 seconds] on key [j4-c64fI_msgIgF60Eqw_q72] for requestor [GlobalTransaction:<pgozdzik/web>:6832:local]! Lock held by [GlobalTransaction:<pgozdzik/web>:6830:local]
> at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:198)
> at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:171)
> at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:169)
> at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitGetKeyValueCommand(PessimisticLockingInterceptor.java:70)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:226)
> at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:221)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:263)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:247)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:92)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:79)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.CacheImpl.get(CacheImpl.java:377)
> at org.infinispan.DecoratedCache.get(DecoratedCache.java:396)
> at org.infinispan.AbstractDelegatingCache.get(AbstractDelegatingCache.java:271)
> at org.jboss.as.clustering.infinispan.invoker.Locator$FindOperation.invoke(Locator.java:54)
> at org.jboss.as.clustering.infinispan.invoker.Locator$LockingFindOperation.invoke(Locator.java:71)
> at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
> at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:82)
> at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:109)
> at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:55)
> at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:149)
> at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:115)
> at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:677) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:353) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]
> at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:168) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]
> at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:153) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]
> at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:216) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:239) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177) [undertow-core-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727) [undertow-core-1.0.15.Final.jar:1.0.15.Final]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_05]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_05]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_05]
>
> 13:19:00,727 ERROR [io.undertow.request] (default task-19) UT005023: Exception handling request to /test: java.lang.IllegalStateException: Transaction DummyTransaction{xid=DummyXid{id=6832}, status=1} is not in a valid state to be invoking cache operations on.
> at org.infinispan.interceptors.TxInterceptor.enlist(TxInterceptor.java:275)
> at org.infinispan.interceptors.TxInterceptor.enlistIfNeeded(TxInterceptor.java:231)
> at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:225)
> at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:221)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:263)
> at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:247)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:92)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)
> at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:79)
> at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74)
> at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)
> at org.infinispan.CacheImpl.get(CacheImpl.java:377)
> at org.infinispan.DecoratedCache.get(DecoratedCache.java:396)
> at org.infinispan.AbstractDelegatingCache.get(AbstractDelegatingCache.java:271)
> at org.jboss.as.clustering.infinispan.invoker.Locator$FindOperation.invoke(Locator.java:54)
> at org.jboss.as.clustering.infinispan.invoker.Locator$LockingFindOperation.invoke(Locator.java:71)
> at org.jboss.as.clustering.infinispan.invoker.SimpleCacheInvoker.invoke(SimpleCacheInvoker.java:34)
> at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:82)
> at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:109)
> at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionFactory.findValue(CoarseSessionFactory.java:55)
> at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:149)
> at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:115)
> at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:677) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:353) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]
> at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:168) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]
> at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:153) [weld-core-impl-2.1.2.Final.jar:2014-01-09 09:23]
> at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:216) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:239) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177) [undertow-core-1.0.15.Final.jar:1.0.15.Final]
> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727) [undertow-core-1.0.15.Final.jar:1.0.15.Final]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_05]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_05]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_05]
> - See more at: https://developer.jboss.org/message/908454?et=watches.email.thread#908454
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 1 month