[JBoss JIRA] (WFCORE-3796) Incorrect Elytron permission class-name or module should throw exception
by Ondrej Lukas (JIRA)
Ondrej Lukas created WFCORE-3796:
------------------------------------
Summary: Incorrect Elytron permission class-name or module should throw exception
Key: WFCORE-3796
URL: https://issues.jboss.org/browse/WFCORE-3796
Project: WildFly Core
Issue Type: Bug
Components: Security
Affects Versions: 5.0.0.Alpha4
Reporter: Ondrej Lukas
When non-existent class-name or module (e.g. when there is a typo) is added to any Elytron permission mapper (constant-permission-mapper or simple-permission-mapper) then exception should be thrown. Otherwise it can result to situation when due to a typo some permission is granted to any identity instead of denying it - when permission in used on 'deny' side.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
- 1
- 0
[JBoss JIRA] (WFLY-9494) Remove jboss-ejb-client.properties and wildfly-config.xml from default locations in the test suite
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/WFLY-9494?page=com.atlassian.jira.plugin.... ]
Darran Lofthouse commented on WFLY-9494:
----------------------------------------
Do we still have the wildfly-config.xml in place? I thought we had eliminated them.
> Remove jboss-ejb-client.properties and wildfly-config.xml from default locations in the test suite
> --------------------------------------------------------------------------------------------------
>
> Key: WFLY-9494
> URL: https://issues.jboss.org/browse/WFLY-9494
> Project: WildFly
> Issue Type: Task
> Components: EJB, Test Suite
> Reporter: Yeray Borges
> Assignee: Yeray Borges
>
> Currently, the test suite contains
> {noformat}
> testsuite/integration/basic/src/test/resources/jboss-ejb-client.properties
> testsuite/integration/basic/src/test/resources/wildfly-config.xml
> testsuite/integration/legacy-ejb-client/src/test/resources/jboss-ejb-client.properties
> testsuite/integration/manualmode/src/test/resources/jboss-ejb-client.properties
> {noformat}
> This means that all {{@RunAsClient}} tests pick them up by default and it's very difficult to add new tests which need to use the same configuration files but with different contents than the default ones, without negatively affecting other tests.
> For example, it's not easily possible to make a test completely ignore the {{jboss-ejb-client.properties}} because setting the {{jboss.ejb.client.properties.file.path}} property to an empty string still makes the EJB client use the file from the default location (it is possible to switch to a different valid {{jboss-ejb-client.properties}} but not to just tell the test to not use any at all).
> No such file should be present in the location from which it is picked up by default, tests which need such file should set up paths to them (for example, using {{wildfly.config.url}} and {{jboss.ejb.client.properties.file.path}} system properties).
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
- 1
- 0
[JBoss JIRA] (DROOLS-2462) kie-docs 6.5.x doesn't build offline
by Michael Biarnes Kiefer (JIRA)
[ https://issues.jboss.org/browse/DROOLS-2462?page=com.atlassian.jira.plugi... ]
Michael Biarnes Kiefer updated DROOLS-2462:
-------------------------------------------
Steps to Reproduce:
We saw that <xsl:import href="http://docbook.sourceforge.net/release/xsl/1.76.1/fo/docbook.xsl"/> is in the file: 23:33:38 Caused by: org.jboss.jdocbook.xslt.XSLTException: error rendering [Exception in extension function java.lang.NullPointerException; SystemID: jar:filehome/jenkins/.m2/repository/org/jboss/pressgang/pressgang-xslt-ns/3.1.3/pressgang-xslt-ns-3.1.3.jar!/xslt/org/jboss/pressgang/pdf.xsl; Line#: 866; Column#: -1] on master.xml
when decompressing the jar and looking into the pdf you can see that it tried to download this file.
was:
We saw that <xsl:import href="http://docbook.sourceforge.net/release/xsl/1.76.1/fo/docbook.xsl"/> is in the file: 23:33:38 Caused by: org.jboss.jdocbook.xslt.XSLTException: error rendering [Exception in extension function java.lang.NullPointerException; SystemID: jar:filehome/jenkins/.m2/repository/org/jboss/pressgang/pressgang-xslt-ns/3.1.3/pressgang-xslt-ns-3.1.3.jar!/xslt/org/jboss/pressgang/pdf.xsl; Line#: 866; Column#: -1] on master.xml
when decompressing the jar and looking into the pdf you can see thta itis tried to donload this file.
> kie-docs 6.5.x doesn't build offline
> ------------------------------------
>
> Key: DROOLS-2462
> URL: https://issues.jboss.org/browse/DROOLS-2462
> Project: Drools
> Issue Type: Bug
> Components: docs
> Affects Versions: 6.5.0.Final
> Reporter: Michael Biarnes Kiefer
> Assignee: Marek Novotny
>
> when the repository kie-docs 6.5.x is build offline
> _man clean install -Dfull -o_ (and without internet connection) this error is thrown: [error|https://gist.github.com/mbiarnes/bb3a46bebf788225f663f249fc9dc361]
> The general suspicion is that <xsl:import href="http://docbook.sourceforge.net/release/xsl/1.76.1/fo/docbook.xsl"/> can't be downloaded and this causes the error.
> We have on Jenkins each "some days" period that kie-docs fails, but when building again it builds successful. So when it fails it has no connection problems and this file can't be downloaded.
> A build without internet connection is not possible.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
- 1
- 0
[JBoss JIRA] (WFCORE-3795) Read Feature Description doesn't return ObjectAttributeDefinition capabilities
by ehsavoie Hugonnet (JIRA)
ehsavoie Hugonnet created WFCORE-3795:
-----------------------------------------
Summary: Read Feature Description doesn't return ObjectAttributeDefinition capabilities
Key: WFCORE-3795
URL: https://issues.jboss.org/browse/WFCORE-3795
Project: WildFly Core
Issue Type: Bug
Components: Management
Affects Versions: 5.0.0.Alpha4
Reporter: ehsavoie Hugonnet
Assignee: ehsavoie Hugonnet
ObjectAttributeDefinition are described as features but their required capabilities are not displayed.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
- 1
- 0
[JBoss JIRA] (WFLY-9494) Remove jboss-ejb-client.properties and wildfly-config.xml from default locations in the test suite
by Yeray Borges (JIRA)
[ https://issues.jboss.org/browse/WFLY-9494?page=com.atlassian.jira.plugin.... ]
Yeray Borges updated WFLY-9494:
-------------------------------
Priority: Major (was: Critical)
> Remove jboss-ejb-client.properties and wildfly-config.xml from default locations in the test suite
> --------------------------------------------------------------------------------------------------
>
> Key: WFLY-9494
> URL: https://issues.jboss.org/browse/WFLY-9494
> Project: WildFly
> Issue Type: Task
> Components: EJB, Test Suite
> Reporter: Yeray Borges
> Assignee: Yeray Borges
>
> Currently, the test suite contains
> {noformat}
> testsuite/integration/basic/src/test/resources/jboss-ejb-client.properties
> testsuite/integration/basic/src/test/resources/wildfly-config.xml
> testsuite/integration/legacy-ejb-client/src/test/resources/jboss-ejb-client.properties
> testsuite/integration/manualmode/src/test/resources/jboss-ejb-client.properties
> {noformat}
> This means that all {{@RunAsClient}} tests pick them up by default and it's very difficult to add new tests which need to use the same configuration files but with different contents than the default ones, without negatively affecting other tests.
> For example, it's not easily possible to make a test completely ignore the {{jboss-ejb-client.properties}} because setting the {{jboss.ejb.client.properties.file.path}} property to an empty string still makes the EJB client use the file from the default location (it is possible to switch to a different valid {{jboss-ejb-client.properties}} but not to just tell the test to not use any at all).
> No such file should be present in the location from which it is picked up by default, tests which need such file should set up paths to them (for example, using {{wildfly.config.url}} and {{jboss.ejb.client.properties.file.path}} system properties).
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
- 1
- 0
[JBoss JIRA] (ELY-1572) japicmp force source compatibility, but only binary compatibility is required
by Jan Kalina (JIRA)
[ https://issues.jboss.org/browse/ELY-1572?page=com.atlassian.jira.plugin.s... ]
Jan Kalina edited comment on ELY-1572 at 5/3/18 4:47 AM:
---------------------------------------------------------
In future, we can re-enable source compatibility check and ignore adding default methods into interfaces only:
{code}
<plugin>
<groupId>com.github.siom79.japicmp</groupId>
<artifactId>japicmp-maven-plugin</artifactId>
- <version>0.11.0</version>
+ <version>0.12.1-SNAPSHOT</version>
<configuration>
@@ -568,7 +568,21 @@
<parameter>
<breakBuildOnBinaryIncompatibleModifications>true</breakBuildOnBinaryIncompatibleModifications>
- <breakBuildOnSourceIncompatibleModifications>false</breakBuildOnSourceIncompatibleModifications>
+ <breakBuildOnSourceIncompatibleModifications>true</breakBuildOnSourceIncompatibleModifications>
+ <overrideCompatibilityChangeParameters>
+ <overrideCompatibilityChangeParameter>
+ <compatibilityChange>METHOD_NEW_DEFAULT</compatibilityChange>
+ <binaryCompatible>true</binaryCompatible>
+ <sourceCompatible>true</sourceCompatible>
+ </overrideCompatibilityChangeParameter>
+ </overrideCompatibilityChangeParameters>
{code}
But currently there is still bug (https://github.com/siom79/japicmp/issues/201#issuecomment-386222564), so METHOD_ADDED_TO_INTERFACE would have to be ignored too -> keeping source compatibility check disabled.
was (Author: honza889):
In future, we can re-enable source compatibility check and ignore adding default methods into interfaces only:
{code}
<plugin>
<groupId>com.github.siom79.japicmp</groupId>
<artifactId>japicmp-maven-plugin</artifactId>
- <version>0.11.0</version>
+ <version>0.12.1-SNAPSHOT</version>
<configuration>
@@ -568,7 +568,21 @@
<parameter>
<breakBuildOnBinaryIncompatibleModifications>true</breakBuildOnBinaryIncompatibleModifications>
- <breakBuildOnSourceIncompatibleModifications>false</breakBuildOnSourceIncompatibleModifications>
+ <breakBuildOnSourceIncompatibleModifications>true</breakBuildOnSourceIncompatibleModifications>
+ <overrideCompatibilityChangeParameters>
+ <overrideCompatibilityChangeParameter>
+ <compatibilityChange>METHOD_NEW_DEFAULT</compatibilityChange>
+ <binaryCompatible>true</binaryCompatible>
+ <sourceCompatible>true</sourceCompatible>
+ </overrideCompatibilityChangeParameter>
+ </overrideCompatibilityChangeParameters>
{code}
But currently there is still bug, so METHOD_ADDED_TO_INTERFACE would have to be ignored too -> keeping source compatibility check disabled.
> japicmp force source compatibility, but only binary compatibility is required
> -----------------------------------------------------------------------------
>
> Key: ELY-1572
> URL: https://issues.jboss.org/browse/ELY-1572
> Project: WildFly Elytron
> Issue Type: Bug
> Components: Build
> Affects Versions: 1.3.0.Final
> Reporter: Jan Kalina
> Assignee: Jan Kalina
>
> As part of ELY-1523 japicmp was configured to force binary and source compatibility.
> By last discussion, only binary compatibility is required.
> {panel}
> *Honza Kalina* jaspicmp reports it as source incompatibility, not as binary incompatibility - so it maybe it is false positive, as I would say it should be source-compatible...
> *Darran Lofthouse* Binary is the only one we are really interested in, i.e. will an update to a later version of Elytron break something. Most of our backwards compatibility strategy is more about breaking things and upsetting people
> {panel}
> Japicmp currently reports adding default method into interface as source incompatible change (see https://github.com/siom79/japicmp/issues/201) -> blocks adding IV into MaskedPassword in ELY-867
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
- 1
- 0
[JBoss JIRA] (ELY-1572) japicmp force source compatibility, but only binary compatibility is required
by Jan Kalina (JIRA)
[ https://issues.jboss.org/browse/ELY-1572?page=com.atlassian.jira.plugin.s... ]
Jan Kalina commented on ELY-1572:
---------------------------------
In future, we can re-enable source compatibility check and ignore adding default methods into interfaces only:
{code}
<plugin>
<groupId>com.github.siom79.japicmp</groupId>
<artifactId>japicmp-maven-plugin</artifactId>
- <version>0.11.0</version>
+ <version>0.12.1-SNAPSHOT</version>
<configuration>
@@ -568,7 +568,21 @@
<parameter>
<breakBuildOnBinaryIncompatibleModifications>true</breakBuildOnBinaryIncompatibleModifications>
- <breakBuildOnSourceIncompatibleModifications>false</breakBuildOnSourceIncompatibleModifications>
+ <breakBuildOnSourceIncompatibleModifications>true</breakBuildOnSourceIncompatibleModifications>
+ <overrideCompatibilityChangeParameters>
+ <overrideCompatibilityChangeParameter>
+ <compatibilityChange>METHOD_NEW_DEFAULT</compatibilityChange>
+ <binaryCompatible>true</binaryCompatible>
+ <sourceCompatible>true</sourceCompatible>
+ </overrideCompatibilityChangeParameter>
+ </overrideCompatibilityChangeParameters>
{code}
But currently there is still bug, so METHOD_ADDED_TO_INTERFACE would have to be ignored too -> keeping source compatibility check disabled.
> japicmp force source compatibility, but only binary compatibility is required
> -----------------------------------------------------------------------------
>
> Key: ELY-1572
> URL: https://issues.jboss.org/browse/ELY-1572
> Project: WildFly Elytron
> Issue Type: Bug
> Components: Build
> Affects Versions: 1.3.0.Final
> Reporter: Jan Kalina
> Assignee: Jan Kalina
>
> As part of ELY-1523 japicmp was configured to force binary and source compatibility.
> By last discussion, only binary compatibility is required.
> {panel}
> *Honza Kalina* jaspicmp reports it as source incompatibility, not as binary incompatibility - so it maybe it is false positive, as I would say it should be source-compatible...
> *Darran Lofthouse* Binary is the only one we are really interested in, i.e. will an update to a later version of Elytron break something. Most of our backwards compatibility strategy is more about breaking things and upsetting people
> {panel}
> Japicmp currently reports adding default method into interface as source incompatible change (see https://github.com/siom79/japicmp/issues/201) -> blocks adding IV into MaskedPassword in ELY-867
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
- 1
- 0
[JBoss JIRA] (WFLY-10317) CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException in a 4 node cluster during fail-over
by tommaso borgato (JIRA)
[ https://issues.jboss.org/browse/WFLY-10317?page=com.atlassian.jira.plugin... ]
tommaso borgato updated WFLY-10317:
-----------------------------------
Summary: CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException in a 4 node cluster during fail-over (was: TimeoutException CIRCULAR REFERENCE in a 4 node cluster during fail-over)
> CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException in a 4 node cluster during fail-over
> -------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10317
> URL: https://issues.jboss.org/browse/WFLY-10317
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 13.0.0.Beta1
> Reporter: tommaso borgato
> Assignee: Paul Ferraro
> Attachments: clusterbench-ee7.ear
>
>
> During one of the regular fail-over tests, we observed the following error when the cache is configured as replicated and synchronous.
> We did not have the error when the cache is configured as distributed and synchronous.
> We used master branch from [13.0.0.Alpha1-SNAPSHOT|https://github.com/wildfly/wildfly/tree/master].
> We observe lot of errors (more than a hundred on each node) like the following, note they occur right after one node has been stopped and restarted and then another node is stopped and restarted:
> {noformat}
> 2018/05/02 09:03:33:843 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started!
> [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-5) New session created: TXIN2deAqLp1QHmZzzQXXrUyTslOYCAx9pazOIo5
> [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-6) New session created: px4vAOKxK8pJ74234T406YgN3jG4e0Z72MDNEkZS
> [JBossINF] [0m[0m09:03:34,009 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-20) New session created: LzmtOORpjJqz7ZqRhmIg6rro3Top9GlWBixfcS7D
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-1) New session created: FYJS0bQVNA4bxHusRV2gI6stJCQxO0rJ2IuzA7b4
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-9) New session created: Jh5jc4oR_KeSt29lfasKTdszo4dWrr2_A8_nxm3x
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-2) New session created: 5AVElPWyS2H2WvqhfdXuZDOltSvQLyodrR1ckK5S
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-3) New session created: _selI14cpR26Q_zCUFCj5Vq_-B2YkPIsi6BB_kMv
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-7) New session created: DU8zan7woAW5gNRX_JraITeBnnXOTHla2q_5DOpZ
> [JBossINF] [0m[0m09:03:34,011 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-8) New session created: uU7mbyQQNWsdhKR091t66AZAQ089IFlYuCqZR0ml
> [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-4) New session created: Tm9d19lL4oFFotC_2PETa1WRKR3kypz3VEhrYF1f
> [JBossINF] [0m[0m09:04:35,431 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18]
> [JBossINF] [0m[0m09:04:35,437 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18]
> [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18]
> [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18]
> [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m09:05:43,742 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19]
> [JBossINF] [0m[0m09:05:43,744 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19]
> [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19]
> [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19]
> [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[31m09:05:53,824 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAttributesKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAccessMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21248 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> [JBossINF] [0m[31m09:05:53,840 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAccessMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAttributesKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> [JBossINF] [0m[31m09:05:53,843 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-12) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155)
> [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122)
> [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] ... 1 more
> [JBossINF] Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19
> [JBossINF] at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> [JBossINF] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> [JBossINF] at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37)
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155)
> [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122)
> [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] ... 1 more
> [JBossINF] [CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19]
> [JBossINF]
> {noformat}
> We tried also with tag [12.0.0.Final|https://github.com/wildfly/wildfly/tree/12.0.0.Final] and we did not have CIRCULAR REFERENCE; anyway we observed a high number (over a hundred for each node) of the following exceptions, note they occur (just like before) right after one node has been stopped and restarted and then another node is stopped and restarted:
> {noformat}
> 2018/05/02 11:17:02:743 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started!
> [JBossINF] [0m[0m11:18:06,946 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18]
> [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18]
> [JBossINF] [0m[0m11:18:06,957 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m11:18:06,960 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18]
> [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18]
> [JBossINF] [0m[0m11:18:06,962 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m11:19:18,519 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19]
> [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19]
> [JBossINF] [0m[0m11:19:18,522 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m11:19:18,523 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19]
> [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19]
> [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[31m11:19:28,697 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionAccessMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionAttributesKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionCreationMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAttributesKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAccessMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21977 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-14) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:166)
> [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:126)
> [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:446)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:493)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:577)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] ... 1 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
- 1
- 0
[JBoss JIRA] (WFLY-10317) TimeoutException CIRCULAR REFERENCE in a 4 node cluster during fail-over
by tommaso borgato (JIRA)
[ https://issues.jboss.org/browse/WFLY-10317?page=com.atlassian.jira.plugin... ]
tommaso borgato updated WFLY-10317:
-----------------------------------
Summary: TimeoutException CIRCULAR REFERENCE in a 4 node cluster during fail-over (was: TimeoutException caused by CIRCULAR REFERENCE in a 4 node cluster during fail-over)
> TimeoutException CIRCULAR REFERENCE in a 4 node cluster during fail-over
> ------------------------------------------------------------------------
>
> Key: WFLY-10317
> URL: https://issues.jboss.org/browse/WFLY-10317
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 13.0.0.Beta1
> Reporter: tommaso borgato
> Assignee: Paul Ferraro
> Attachments: clusterbench-ee7.ear
>
>
> During one of the regular fail-over tests, we observed the following error when the cache is configured as replicated and synchronous.
> We did not have the error when the cache is configured as distributed and synchronous.
> We used master branch from [13.0.0.Alpha1-SNAPSHOT|https://github.com/wildfly/wildfly/tree/master].
> We observe lot of errors (more than a hundred on each node) like the following, note they occur right after one node has been stopped and restarted and then another node is stopped and restarted:
> {noformat}
> 2018/05/02 09:03:33:843 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started!
> [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-5) New session created: TXIN2deAqLp1QHmZzzQXXrUyTslOYCAx9pazOIo5
> [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-6) New session created: px4vAOKxK8pJ74234T406YgN3jG4e0Z72MDNEkZS
> [JBossINF] [0m[0m09:03:34,009 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-20) New session created: LzmtOORpjJqz7ZqRhmIg6rro3Top9GlWBixfcS7D
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-1) New session created: FYJS0bQVNA4bxHusRV2gI6stJCQxO0rJ2IuzA7b4
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-9) New session created: Jh5jc4oR_KeSt29lfasKTdszo4dWrr2_A8_nxm3x
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-2) New session created: 5AVElPWyS2H2WvqhfdXuZDOltSvQLyodrR1ckK5S
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-3) New session created: _selI14cpR26Q_zCUFCj5Vq_-B2YkPIsi6BB_kMv
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-7) New session created: DU8zan7woAW5gNRX_JraITeBnnXOTHla2q_5DOpZ
> [JBossINF] [0m[0m09:03:34,011 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-8) New session created: uU7mbyQQNWsdhKR091t66AZAQ089IFlYuCqZR0ml
> [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-4) New session created: Tm9d19lL4oFFotC_2PETa1WRKR3kypz3VEhrYF1f
> [JBossINF] [0m[0m09:04:35,431 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18]
> [JBossINF] [0m[0m09:04:35,437 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18]
> [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18]
> [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18]
> [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m09:05:43,742 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19]
> [JBossINF] [0m[0m09:05:43,744 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19]
> [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19]
> [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19]
> [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[31m09:05:53,824 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAttributesKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAccessMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21248 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> [JBossINF] [0m[31m09:05:53,840 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAccessMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAttributesKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> [JBossINF] [0m[31m09:05:53,843 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-12) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155)
> [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122)
> [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] ... 1 more
> [JBossINF] Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19
> [JBossINF] at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> [JBossINF] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> [JBossINF] at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37)
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155)
> [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122)
> [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] ... 1 more
> [JBossINF] [CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19]
> [JBossINF]
> {noformat}
> We tried also with tag [12.0.0.Final|https://github.com/wildfly/wildfly/tree/12.0.0.Final] and we did not have CIRCULAR REFERENCE; anyway we observed a high number (over a hundred for each node) of the following exceptions, note they occur (just like before) right after one node has been stopped and restarted and then another node is stopped and restarted:
> {noformat}
> 2018/05/02 11:17:02:743 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started!
> [JBossINF] [0m[0m11:18:06,946 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18]
> [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18]
> [JBossINF] [0m[0m11:18:06,957 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m11:18:06,960 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18]
> [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18]
> [JBossINF] [0m[0m11:18:06,962 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m11:19:18,519 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19]
> [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19]
> [JBossINF] [0m[0m11:19:18,522 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m11:19:18,523 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19]
> [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19]
> [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[31m11:19:28,697 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionAccessMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionAttributesKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionCreationMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAttributesKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAccessMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21977 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-14) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:166)
> [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:126)
> [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:446)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:493)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:577)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] ... 1 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
- 1
- 0
[JBoss JIRA] (WFLY-10317) TimeoutException caused by CIRCULAR REFERENCE in a 4 node cluster during fail-over
by tommaso borgato (JIRA)
[ https://issues.jboss.org/browse/WFLY-10317?page=com.atlassian.jira.plugin... ]
tommaso borgato commented on WFLY-10317:
----------------------------------------
Thank you [~yersan]: I updated the description, should I add some more information?
> TimeoutException caused by CIRCULAR REFERENCE in a 4 node cluster during fail-over
> ----------------------------------------------------------------------------------
>
> Key: WFLY-10317
> URL: https://issues.jboss.org/browse/WFLY-10317
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 13.0.0.Beta1
> Reporter: tommaso borgato
> Assignee: Paul Ferraro
> Attachments: clusterbench-ee7.ear
>
>
> During one of the regular fail-over tests, we observed the following error when the cache is configured as replicated and synchronous.
> We did not have the error when the cache is configured as distributed and synchronous.
> We used master branch from [13.0.0.Alpha1-SNAPSHOT|https://github.com/wildfly/wildfly/tree/master].
> We observe lot of errors (more than a hundred on each node) like the following, note they occur right after one node has been stopped and restarted and then another node is stopped and restarted:
> {noformat}
> 2018/05/02 09:03:33:843 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started!
> [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-5) New session created: TXIN2deAqLp1QHmZzzQXXrUyTslOYCAx9pazOIo5
> [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-6) New session created: px4vAOKxK8pJ74234T406YgN3jG4e0Z72MDNEkZS
> [JBossINF] [0m[0m09:03:34,009 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-20) New session created: LzmtOORpjJqz7ZqRhmIg6rro3Top9GlWBixfcS7D
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-1) New session created: FYJS0bQVNA4bxHusRV2gI6stJCQxO0rJ2IuzA7b4
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-9) New session created: Jh5jc4oR_KeSt29lfasKTdszo4dWrr2_A8_nxm3x
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-2) New session created: 5AVElPWyS2H2WvqhfdXuZDOltSvQLyodrR1ckK5S
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-3) New session created: _selI14cpR26Q_zCUFCj5Vq_-B2YkPIsi6BB_kMv
> [JBossINF] [0m[0m09:03:34,010 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-7) New session created: DU8zan7woAW5gNRX_JraITeBnnXOTHla2q_5DOpZ
> [JBossINF] [0m[0m09:03:34,011 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-8) New session created: uU7mbyQQNWsdhKR091t66AZAQ089IFlYuCqZR0ml
> [JBossINF] [0m[0m09:03:34,007 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-4) New session created: Tm9d19lL4oFFotC_2PETa1WRKR3kypz3VEhrYF1f
> [JBossINF] [0m[0m09:04:35,431 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18]
> [JBossINF] [0m[0m09:04:35,437 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18]
> [JBossINF] [0m[0m09:04:35,438 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18]
> [JBossINF] [0m[0m09:04:35,440 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|6] (3) [perf21, perf20, perf18]
> [JBossINF] [0m[0m09:04:35,441 INFO [org.infinispan.CLUSTER] (thread-28,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m09:05:43,742 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19]
> [JBossINF] [0m[0m09:05:43,744 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19]
> [JBossINF] [0m[0m09:05:43,745 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19]
> [JBossINF] [0m[0m09:05:43,748 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf21|7] (4) [perf21, perf20, perf18, perf19]
> [JBossINF] [0m[0m09:05:43,749 INFO [org.infinispan.CLUSTER] (thread-32,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[31m09:05:53,824 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAttributesKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba), SessionAccessMetaDataKey(Rx1s7LTnrMx33fFVg13vLwN5GLYK049ZIbwqD-Ba)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21248 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> [JBossINF] [0m[31m09:05:53,840 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAccessMetaDataKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h), SessionAttributesKey(MXA09ctVC96ZOxYRU38Hg6A7xYgXFsizn31SR-1h)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> [JBossINF] [0m[31m09:05:53,843 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-12) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155)
> [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122)
> [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:164)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] ... 1 more
> [JBossINF] Suppressed: java.util.concurrent.ExecutionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19
> [JBossINF] at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> [JBossINF] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> [JBossINF] at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:82)
> [JBossINF] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:37)
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155)
> [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:122)
> [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:579)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1514)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] ... 1 more
> [JBossINF] [CIRCULAR REFERENCE:org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21249 from perf19]
> [JBossINF]
> {noformat}
> We tried also with tag [12.0.0.Final|https://github.com/wildfly/wildfly/tree/12.0.0.Final] and we did not have CIRCULAR REFERENCE; anyway we observed a high number (over a hundred for each node) of the following exceptions, note they occur (just like before) right after one node has been stopped and restarted and then another node is stopped and restarted:
> {noformat}
> 2018/05/02 11:17:02:743 EDT [DEBUG][Thread-82] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started!
> [JBossINF] [0m[0m11:18:06,946 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18]
> [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m11:18:06,956 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18]
> [JBossINF] [0m[0m11:18:06,957 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m11:18:06,960 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18]
> [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m11:18:06,961 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|6] (3) [perf20, perf21, perf18]
> [JBossINF] [0m[0m11:18:06,962 INFO [org.infinispan.CLUSTER] (thread-12,ejb,perf18) ISPN100001: Node perf19 left the cluster
> [JBossINF] [0m[0m11:19:18,519 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19]
> [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m11:19:18,521 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19]
> [JBossINF] [0m[0m11:19:18,522 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m11:19:18,523 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19]
> [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN000094: Received new cluster view for channel ejb: [perf20|7] (4) [perf20, perf21, perf18, perf19]
> [JBossINF] [0m[0m11:19:18,524 INFO [org.infinispan.CLUSTER] (thread-42,ejb,perf18) ISPN100000: Node perf19 joined the cluster
> [JBossINF] [0m[31m11:19:28,697 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionAccessMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionAttributesKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI), SessionCreationMetaDataKey(6t_HBz794DetKUT7Do1B8P1TZ2W8pynlk6NGYiBI)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (timeout-thread--p10-t1) ISPN000136: Error executing command PrepareCommand, writing keys [SessionCreationMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAttributesKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5), SessionAccessMetaDataKey(Uu_wR07Vr_DkG5twQ0wk7Cqo2RD6radIQk0Uj_s5)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21977 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF]
> [JBossINF] [0m[31m11:19:28,702 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (default task-14) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19
> [JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
> [JBossINF] at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
> [JBossINF] at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:166)
> [JBossINF] at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:126)
> [JBossINF] at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:446)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:493)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
> [JBossINF] at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
> [JBossINF] at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.$closeResource(DistributableSession.java:92)
> [JBossINF] at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:91)
> [JBossINF] at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:945)
> [JBossINF] at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:577)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:339)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
> [JBossINF] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
> [JBossINF] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
> [JBossINF] at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1526)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
> [JBossINF] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
> [JBossINF] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
> [JBossINF] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
> [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
> [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
> [JBossINF] at java.lang.Thread.run(Thread.java:748)
> [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 21976 from perf19
> [JBossINF] at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:163)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:86)
> [JBossINF] at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:21)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [JBossINF] ... 1 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 2 months
- 1
- 0