[JBoss JIRA] (WFCORE-1002) TransactionalProtocolOperationHandler.AbortOperationHandler must initialize the ExecuteRequestContext before responding
by Kabir Khan (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1002?page=com.atlassian.jira.plugi... ]
Kabir Khan updated WFCORE-1002:
-------------------------------
Fix Version/s: 2.0.0.CR7
(was: 2.0.0.CR6)
> TransactionalProtocolOperationHandler.AbortOperationHandler must initialize the ExecuteRequestContext before responding
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: WFCORE-1002
> URL: https://issues.jboss.org/browse/WFCORE-1002
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 2.0.0.CR3
> Reporter: Brian Stansberry
> Assignee: Brian Stansberry
> Fix For: 2.0.0.CR7
>
>
> If in a race a COMPLETE_TX_REQUEST cancelling an op is processed by a server/slave HC before the original EXECUTE_TX_REQUEST is processed, when the EXECUTE_TX_REQUEST comes in the AbortOperationHandler attempts to respond with a "outcome"=>"cancelled" response. This will fail because the ExecutionRequestContext will not be initialized.
> In this situation the AbortOperationHandler should initialize the ERC.
> With assertions enabled, this appears in the log:
> {code}
> 20:58:47,947 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) Exception in thread "Remoting "slave:main-three:MANAGEMENT" task-6" java.lang.AssertionError
> 20:58:47,947 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestContext.failed(TransactionalProtocolOperationHandler.java:458)
> 20:58:47,948 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$AbortOperationHandler.handleRequest(TransactionalProtocolOperationHandler.java:275)
> 20:58:47,948 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:270)
> 20:58:47,948 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:252)
> 20:58:47,948 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.protocol.mgmt.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:123)
> 20:58:47,949 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.protocol.mgmt.ManagementChannelReceiver$1.handleMessage(ManagementChannelReceiver.java:56)
> 20:58:47,949 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.as.protocol.mgmt.ManagementChannelReceiver.handleMessage(ManagementChannelReceiver.java:85)
> 20:58:47,949 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:451)
> 20:58:47,949 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at org.jboss.remoting3.EndpointImpl$TrackingExecutor$1.run(EndpointImpl.java:705)
> 20:58:47,950 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 20:58:47,950 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 20:58:47,950 ERROR [stderr] (Remoting "slave:main-three:MANAGEMENT" task-6) at java.lang.Thread.run(Thread.java:745)
> {code}
> Without assertions, a few lines later a call to sendResponse will result in an NPE.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 9 months
[JBoss JIRA] (WFCORE-994) Inconsistent :rrd(access-control) response when querying host resources
by Kabir Khan (JIRA)
[ https://issues.jboss.org/browse/WFCORE-994?page=com.atlassian.jira.plugin... ]
Kabir Khan updated WFCORE-994:
------------------------------
Fix Version/s: 2.0.0.CR7
(was: 2.0.0.CR6)
> Inconsistent :rrd(access-control) response when querying host resources
> -----------------------------------------------------------------------
>
> Key: WFCORE-994
> URL: https://issues.jboss.org/browse/WFCORE-994
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Reporter: Heiko Braun
> Assignee: Kabir Khan
> Priority: Critical
> Fix For: 2.0.0.CR7
>
>
> {noformat}
> [domain@localhost:9990 /] /host=*:read-resource-description(access-control=trim-descriptions){roles=Administrator}
> {
> "outcome" => "success",
> "result" => [
> {
> "address" => [("host" => "macbook-pro.fritz.box")],
> "outcome" => "success",
> "result" => {
> "access-control" => {
> "default" => {
> "read" => true,
> "write" => true,
> "attributes" => {}
> },
> "exceptions" => {}
> },
> "description" => undefined
> }
> },
> {
> "address" => [("host" => "master")],
> "outcome" => "success",
> "result" => {
> "description" => undefined,
> "attributes" => undefined,
> "operations" => undefined,
> "notifications" => undefined,
> "children" => {
> "core-service" => {"model-description" => undefined},
> "jvm" => {"model-description" => undefined},
> "socket-binding-group" => {"model-description" => undefined},
> "path" => {"model-description" => undefined},
> "server" => {"model-description" => undefined},
> "subsystem" => {"model-description" => undefined},
> "server-config" => {"model-description" => undefined},
> "interface" => {"model-description" => undefined},
> "system-property" => {"model-description" => undefined},
> "extension" => {"model-description" => undefined}
> },
> "access-control" => {
> "default" => {
> "read" => true,
> "write" => true,
> "attributes" => {
> "directory-grouping" => {
> "read" => true,
> "write" => true
> },
> "domain-controller" => {
> "read" => true,
> "write" => true
> },
> "domain-organization" => {
> "read" => true,
> "write" => true
> },
> "host-state" => {
> "read" => true,
> "write" => true
> },
> "management-major-version" => {
> "read" => true,
> "write" => true
> },
> "management-micro-version" => {
> "read" => true,
> "write" => true
> },
> "management-minor-version" => {
> "read" => true,
> "write" => true
> },
> "master" => {
> "read" => true,
> "write" => true
> },
> "name" => {
> "read" => true,
> "write" => true
> },
> "namespaces" => {
> "read" => true,
> "write" => true
> },
> "organization" => {
> "read" => true,
> "write" => true
> },
> "product-name" => {
> "read" => true,
> "write" => true
> },
> "product-version" => {
> "read" => true,
> "write" => true
> },
> "release-codename" => {
> "read" => true,
> "write" => true
> },
> "release-version" => {
> "read" => true,
> "write" => true
> },
> "running-mode" => {
> "read" => true,
> "write" => true
> },
> "schema-locations" => {
> "read" => true,
> "write" => true
> },
> "server-state" => {
> "read" => true,
> "write" => true
> },
> "suspend-state" => {
> "read" => true,
> "write" => true
> },
> "uuid" => {
> "read" => true,
> "write" => true
> }
> }
> },
> "exceptions" => {}
> }
> }
> }
> ]
> }
> [domain@localhost:9990 /]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 9 months
[JBoss JIRA] (WFCORE-989) Premature registration of reconnecting server
by Kabir Khan (JIRA)
[ https://issues.jboss.org/browse/WFCORE-989?page=com.atlassian.jira.plugin... ]
Kabir Khan updated WFCORE-989:
------------------------------
Fix Version/s: 2.0.0.CR7
(was: 2.0.0.CR6)
> Premature registration of reconnecting server
> ---------------------------------------------
>
> Key: WFCORE-989
> URL: https://issues.jboss.org/browse/WFCORE-989
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 2.0.0.CR1
> Reporter: Brian Stansberry
> Assignee: Brian Stansberry
> Fix For: 2.0.0.CR7
>
>
> ServerInventoryImpl.reconnectServer is registering the ProxyController for the reconnecting server with the DomainModelControllerService. The problem is that ProxyController is not yet in a state where it can forward requests to the server. That won't happen until the server calls back with a DomainServerProtocol.SERVER_RECONNECT_REQUEST and the ServerToHostProtocolHandler.ServerReconnectRequestHandler handles it. The effect is if a request for the server comes in during this window, it will fail. If that request is part of a domain operation rollout (because during the window the HC regarded the server as 'live' and instructed the DC to send rollout ops to it), then the domain rollout will be affected. This happened in the testsuite failure I just saw. Most significant logging is as follows:
> {code}
> 2015-09-18 17:59:32,229 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Core 2.0.0.CR2-SNAPSHOT "Kenny" (Host Controller) started in 93ms - Started 48 of 51 services (15 services are lazy, passive or on-demand)
> 2015-09-18 17:59:32,247 INFO [org.jboss.as.controller.management-operation] (Remoting "slave:MANAGEMENT" task-16) Initialized for 1849055188
> 2015-09-18 17:59:32,247 INFO [org.jboss.as.host.controller] (Remoting "slave:MANAGEMENT" task-14) WFLYHC0021: Server [Server:main-three] connected using connection [Channel ID 08aa200b (inbound) of Remoting connection 27e3d90d to /127.0.0.1:49272]
> 2015-09-18 17:59:32,247 INFO [org.jboss.as.host.controller] (Remoting "slave:MANAGEMENT" task-15) WFLYHC0021: Server [Server:other-two] connected using connection [Channel ID 02bd2646 (inbound) of Remoting connection 64a8b7f2 to /127.0.0.1:49273]
> 2015-09-18 17:59:32,247 INFO [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 34) Initialized for 1849055188
> 2015-09-18 17:59:32,252 INFO [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 34) sending prepared response for 1849055188 --- interrupted: false
> 2015-09-18 17:59:32,254 INFO [org.jboss.as.controller.management-operation] (Remoting "slave:MANAGEMENT" task-1) Initialized for 1933145457
> 2015-09-18 17:59:32,254 INFO [org.jboss.as.controller.management-operation] (Remoting "slave:MANAGEMENT" task-1) Initialized for 481103151
> 2015-09-18 17:59:32,254 INFO [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 36) Initialized for 1933145457
> 2015-09-18 17:59:32,254 INFO [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 37) Initialized for 481103151
> 2015-09-18 17:59:32,255 INFO [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 36) sending pre-prepare failed response for 1933145457 --- interrupted: false
> 2015-09-18 17:59:32,256 INFO [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 37) sending pre-prepare failed response for 481103151 --- interrupted: false
> {code}
> The HC completes boot. Then servers main-three and other-two register. The HC sends a prepared response to a request. This is the host rollout request to the slave from the DC with the response being the ops to invoke on the servers. The HC reports sending a "pre-prepare failed response" to two requests. These are the requests the DC has asked it to proxy to the servers.
> The result of all this for the client is the following failure of a management op:
> {code}
> Failed operation:
> {
> "operation" => "add",
> "address" => [("extension" => "org.jboss.as.test.hc.extension")]
> }
> Response:
> {
> "outcome" => "failed",
> "result" => undefined,
> "failure-description" => {"WFLYDC0074: Operation failed or was rolled back on all servers. Server failures:" => {"server-group" => {
> "main-server-group" => {"host" => {"slave" => {"main-three" => "WFLYHC0153: Channel closed"}}},
> "other-server-group" => {"host" => {"slave" => {"other-two" => "WFLYHC0153: Channel closed"}}}
> }}},
> "rolled-back" => true,
> "server-groups" => {
> "main-server-group" => {"host" => {
> "master" => {"main-one" => {"response" => {
> "outcome" => "failed",
> "result" => [("HC" => "1.1.1")],
> "rolled-back" => true
> }}},
> "slave" => {"main-three" => {"response" => {
> "outcome" => "failed",
> "result" => undefined,
> "failure-description" => "WFLYHC0153: Channel closed",
> "rolled-back" => true
> }}}
> }},
> "other-server-group" => {"host" => {"slave" => {"other-two" => {"response" => {
> "outcome" => "failed",
> "result" => undefined,
> "failure-description" => "WFLYHC0153: Channel closed",
> "rolled-back" => true
> }}}}}
> }
> }
> {code}
> The "WFLYHC0153: Channel closed" failure is what is produced when an attempt is made to invoke on a disconnected proxy controller.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 9 months
[JBoss JIRA] (WFCORE-988) Host controller fails to roll out management operations if assertions are enabled
by Kabir Khan (JIRA)
[ https://issues.jboss.org/browse/WFCORE-988?page=com.atlassian.jira.plugin... ]
Kabir Khan updated WFCORE-988:
------------------------------
Fix Version/s: 2.0.0.CR7
(was: 2.0.0.CR6)
> Host controller fails to roll out management operations if assertions are enabled
> ---------------------------------------------------------------------------------
>
> Key: WFCORE-988
> URL: https://issues.jboss.org/browse/WFCORE-988
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Reporter: James Perkins
> Assignee: Brian Stansberry
> Fix For: 2.0.0.CR7
>
>
> Enabling assertions, {{-ea}}, for the host controller then trying to execute an operation that needs to be rolled out to servers fails. The issue seems to be in the [{{OperationContextImpl}}|https://github.com/wildfly/wildfly-core/blob/2.0.0.CR1/controller/src/main/java/org/jboss/as/controller/OperationContextImpl.java#L826].
> Replacing the assertion with an exception I see the stack trace below.
> {code:title=Stack Trace}
> [Host Controller] 14:08:33,124 DEBUG [org.jboss.as.host.controller] (Host Controller Service Threads - 8) failed to process task {
> [Host Controller] "address" => [
> [Host Controller] ("host" => "master"),
> [Host Controller] ("server" => "server-one"),
> [Host Controller] ("subsystem" => "logging")
> [Host Controller] ],
> [Host Controller] "operation" => "write-attribute",
> [Host Controller] "name" => "use-deployment-logging-config",
> [Host Controller] "value" => true,
> [Host Controller] "operation-headers" => {
> [Host Controller] "access-mechanism" => "NATIVE",
> [Host Controller] "domain-uuid" => "1763a478-a1ad-42b5-a206-03db6e8bbc69",
> [Host Controller] "execute-for-coordinator" => true
> [Host Controller] }
> [Host Controller] }: java.lang.RuntimeException: Thread.currentThread=Thread[Host Controller Service Threads - 8,5,Host Controller Service Threads], controllingThread=null, initiatingThreadThread[management-handler-thread - 2,5,management-handler-thread]
> [Host Controller] at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:832)
> [Host Controller] at org.jboss.as.controller.transform.ResourceTransformationContextImpl.create(ResourceTransformationContextImpl.java:64)
> [Host Controller] at org.jboss.as.controller.transform.TransformersImpl.transformOperation(TransformersImpl.java:88)
> [Host Controller] at org.jboss.as.controller.TransformingProxyController$Factory$1.transformOperation(TransformingProxyController.java:102)
> [Host Controller] at org.jboss.as.controller.TransformingProxyController$TransformingProxyControllerImpl.transformOperation(TransformingProxyController.java:153)
> [Host Controller] at org.jboss.as.domain.controller.operations.coordination.MultiphaseOverallContext.transformServerOperation(MultiphaseOverallContext.java:176)
> [Host Controller] at org.jboss.as.domain.controller.operations.coordination.DomainRolloutStepHandler$2.execute(DomainRolloutStepHandler.java:315)
> [Host Controller] at org.jboss.as.domain.controller.plan.ServerTaskExecutor.executeTask(ServerTaskExecutor.java:83)
> [Host Controller] at org.jboss.as.domain.controller.plan.ConcurrentServerGroupUpdateTask.execute(ConcurrentServerGroupUpdateTask.java:53)
> [Host Controller] at org.jboss.as.domain.controller.plan.AbstractServerGroupRolloutTask$1.run(AbstractServerGroupRolloutTask.java:63)
> [Host Controller] at org.jboss.as.domain.controller.plan.AbstractServerGroupRolloutTask$1.run(AbstractServerGroupRolloutTask.java:59)
> [Host Controller] at java.security.AccessController.doPrivileged(Native Method)
> [Host Controller] at javax.security.auth.Subject.doAs(Subject.java:360)
> [Host Controller] at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:81)
> [Host Controller] at org.jboss.as.domain.controller.plan.AbstractServerGroupRolloutTask.run(AbstractServerGroupRolloutTask.java:59)
> [Host Controller] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [Host Controller] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [Host Controller] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [Host Controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [Host Controller] at java.lang.Thread.run(Thread.java:745)
> [Host Controller] at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 9 months
[JBoss JIRA] (WFCORE-745) Standalone powershell script does't support --debug and -secmgr option
by Kabir Khan (JIRA)
[ https://issues.jboss.org/browse/WFCORE-745?page=com.atlassian.jira.plugin... ]
Kabir Khan updated WFCORE-745:
------------------------------
Fix Version/s: 2.0.0.CR7
(was: 2.0.0.CR6)
> Standalone powershell script does't support --debug and -secmgr option
> ----------------------------------------------------------------------
>
> Key: WFCORE-745
> URL: https://issues.jboss.org/browse/WFCORE-745
> Project: WildFly Core
> Issue Type: Bug
> Components: Scripts
> Affects Versions: 2.0.0.Alpha3
> Reporter: Marek Kopecký
> Assignee: Tomaz Cerar
> Fix For: 2.0.0.CR7
>
>
> Standalone powershell script does't support --debug option. Also -secmgr option and SECMGR=true environment variable settings is missing in posershell script.
> *Steps to Reproduce:*
> # .\standalone.ps1 --debug
> *Actual results:*
> Debug mode is not active
> *Expected results:*
> {noformat}
> ...
> Listening for transport dt_socket at address: 8787
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 9 months
[JBoss JIRA] (WFCORE-743) Powershell script for domain is not finised
by Tomaz Cerar (JIRA)
[ https://issues.jboss.org/browse/WFCORE-743?page=com.atlassian.jira.plugin... ]
Tomaz Cerar updated WFCORE-743:
-------------------------------
Fix Version/s: 2.0.0.CR7
> Powershell script for domain is not finised
> -------------------------------------------
>
> Key: WFCORE-743
> URL: https://issues.jboss.org/browse/WFCORE-743
> Project: WildFly Core
> Issue Type: Bug
> Components: Scripts
> Affects Versions: 2.0.0.Alpha3
> Reporter: Marek Kopecký
> Assignee: Tomaz Cerar
> Fix For: 2.0.0.CR7
>
>
> domain.ps1 for now just calls domain.bat. Many arguments of domain.ps1 is not working, example:
> * .\domain.ps1 --Foo
> ** Actual results: domain starts
> ** Expected results: Invalid option '--foo'. Use domain --help for information on valid command line arguments and their syntax.
> * .\domain.ps1 --help
> ** Actual results: domain starts
> ** Expected results: domain prints help message
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 9 months
[JBoss JIRA] (DROOLS-935) tomcat7 unable to deploy Execution server
by Maciej Swiderski (JIRA)
[ https://issues.jboss.org/browse/DROOLS-935?page=com.atlassian.jira.plugin... ]
Maciej Swiderski commented on DROOLS-935:
-----------------------------------------
see the last comment in JBPM-4789 for the properties that need to be set to override default JNDI name and dialect for persistence. besides that workbench configuration is sufficient to setup bitronix as data source provider and transaction manager.
> tomcat7 unable to deploy Execution server
> -----------------------------------------
>
> Key: DROOLS-935
> URL: https://issues.jboss.org/browse/DROOLS-935
> Project: Drools
> Issue Type: Bug
> Components: kie server
> Affects Versions: 6.3.0.Final
> Environment: Windows 7 64bit sp1
> java 7u60 64bit
> tomcat 7.0.54
> Reporter: Siarhei Yarkavy
> Assignee: Maciej Swiderski
> Attachments: catalina.2015-10-01.zip
>
>
> Unable to properly deploy Execution server into tomcat7 container.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 9 months
[JBoss JIRA] (WFLY-5307) CacheException: Initial state transfer timed out for cache
by Michal Karm Babacek (JIRA)
[ https://issues.jboss.org/browse/WFLY-5307?page=com.atlassian.jira.plugin.... ]
Michal Karm Babacek updated WFLY-5307:
--------------------------------------
Affects Version/s: 10.0.0.CR2
> CacheException: Initial state transfer timed out for cache
> ----------------------------------------------------------
>
> Key: WFLY-5307
> URL: https://issues.jboss.org/browse/WFLY-5307
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.0.0.Beta1, 10.0.0.CR2
> Reporter: Michal Vinkler
> Assignee: Paul Ferraro
>
> EAP 7.0.0.DR9
> Scenario: failover-ejb-ejbremote-undeploy-dist-async
> During application deployment (after failover), perf18 logged this error (besides the Replication timeout WARN message):
> {code}
> [JBossINF] [0m[0m15:29:55,511 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 85) WFLYCLINF0002: Started dist cache from ejb container
> [JBossINF] [0m[0m15:29:55,513 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 86) WFLYCLINF0002: Started clusterbench-ee7.ear.clusterbench-ee7-web-default.war cache from web container
> [JBossINF] [0m[0m15:29:55,516 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 87) WFLYCLINF0002: Started clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war cache from web container
> [JBossINF] [0m[0m15:29:55,538 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 87) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
> [JBossINF] [0m[0m15:29:55,539 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 87) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
> [JBossINF] [0m[0m15:29:56,499 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 87) WFLYCLINF0002: Started clusterbench-ee7.ear/clusterbench-ee7-ejb.jar cache from ejb container
> [JBossINF] [0m[0m15:30:00,507 INFO [org.jboss.test.clusterbench.common.ejb.CommonStatefulSBImpl] (EJB default - 1) New SFSB created: org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl@5293c097.
> [JBossINF] [0m[0m15:30:01,323 INFO [org.jboss.test.clusterbench.common.ejb.CommonStatefulSBImpl] (EJB default - 2) New SFSB created: org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl@4dfb0a03.
> [JBossINF] [0m[33m15:30:55,461 WARN [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) ISPN000286: Issue when retrieving cluster listeners from perf19: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf19
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$79(JGroupsTransport.java:589)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$22/1199733093.apply(Unknown Source)
> [JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> [JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> [JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> [JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
> [JBossINF] at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
> [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF]
> [JBossINF] [0m[31m15:31:55,467 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 84) MSC000001: Failed to start service jboss.infinispan.web.dist: org.jboss.msc.service.StartException in service jboss.infinispan.web.dist: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [JBossINF] at java.lang.Thread.run(Thread.java:745)
> [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:320)
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
> [JBossINF] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
> [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
> [JBossINF] at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
> [JBossINF] at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:839)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:629)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:580)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:445)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:464)
> [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:455)
> [JBossINF] at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:120)
> [JBossINF] at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:111)
> [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(CacheBuilder.java:80)
> [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102)
> [JBossINF] ... 4 more
> [JBossINF] Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache dist on perf18
> [JBossINF] at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:225)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [JBossINF] at java.lang.reflect.Method.invoke(Method.java:497)
> [JBossINF] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
> [JBossINF] ... 19 more
> [JBossINF]
> [JBossINF] [0m[31m15:31:55,476 ERROR [org.jboss.as.server] (management-handler-thread - 2) WFLYSRV0021: Deploy of deployment "clusterbench-ee7.ear" was rolled back with the following failure message: undefined
> {code}
> Which ends up with WFLYSRV0021: *Deploy of deployment "clusterbench-ee7.ear" was rolled back*.
> In this very same testrun, there was another issue with application deployment on perf20, may be related: JBEAP-1043
> Server log:
> http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-e...
> This issue is similar to this EAP6 bug: [BZ927948|https://bugzilla.redhat.com/show_bug.cgi?id=927948]
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 9 months
[JBoss JIRA] (WFLY-5465) Occasional session loss after 500/503 status
by Richard Janík (JIRA)
Richard Janík created WFLY-5465:
-----------------------------------
Summary: Occasional session loss after 500/503 status
Key: WFLY-5465
URL: https://issues.jboss.org/browse/WFLY-5465
Project: WildFly
Issue Type: Bug
Components: Clustering
Reporter: Richard Janík
Assignee: Paul Ferraro
Hi,
we're occasionally seeing session being lost after an HTTP error status code (500/503). In other words, our test output looks like this:
{code}
...
2015/08/29 00:06:26:966 EDT [WARN ][Runner - 1549] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>>
2015/08/29 00:06:26:966 EDT [WARN ][Runner - 1549] SFCORE_LOG - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>>
...
... ~4 seconds later (meanwhile, other runners send their requests) ...
...
2015/08/29 00:06:31:050 EDT [WARN ][Runner - 1549] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 78, received 0, Runner: 1549>
2015/08/29 00:06:31:050 EDT [WARN ][Runner - 1549] SFCORE_LOG - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Stale session data received. Expected 78, received 0, Runner: 1549>
...
{code}
The session loss is not consistent - the runner from the example has caught a 500 about ~10 times before and has had many successful requests between those error status codes.
How it usually works (good scenario):
- GET --> 55, GET --> 56, GET --> 57, GET --> ERROR 500, GET --> 58, ...
How it occasionally works (bad scenario):
- GET --> 55, GET --> 56, GET --> 57, GET --> ERROR 500, GET --> 0, ...
When runner 1549 received a new session, the following output occurs on server side (perf19 - the server to which runner 1549 is then being directed):
{code}
[JBossINF] [0m[31m00:06:26,962 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-55) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf18
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$80(JGroupsTransport.java:602)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$34/1387701636.apply(Unknown Source)
[JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
[JBossINF]
[JBossINF] [0m[31m00:06:26,962 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-86) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf18
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$80(JGroupsTransport.java:602)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$34/1387701636.apply(Unknown Source)
[JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
[JBossINF]
[JBossINF] [0m[31m00:06:26,963 ERROR [io.undertow.request] (default task-55) UT005023: Exception handling request to /clusterbench/session: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf18
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$80(JGroupsTransport.java:602)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$34/1387701636.apply(Unknown Source)
[JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
[JBossINF]
[JBossINF] [0m[31m00:06:26,963 ERROR [io.undertow.request] (default task-86) UT005023: Exception handling request to /clusterbench/session: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf18
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$80(JGroupsTransport.java:602)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$34/1387701636.apply(Unknown Source)
[JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
[JBossINF]
[JBossINF] [0m[31m00:06:26,964 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-39) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf18
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$80(JGroupsTransport.java:602)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$34/1387701636.apply(Unknown Source)
[JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
[JBossINF]
[JBossINF] [0m[31m00:06:26,965 ERROR [io.undertow.request] (default task-39) UT005023: Exception handling request to /clusterbench/session: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf18
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$80(JGroupsTransport.java:602)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$34/1387701636.apply(Unknown Source)
[JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
[JBossINF]
[JBossINF] [0m[31m00:06:26,965 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-27) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf21
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$80(JGroupsTransport.java:602)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$34/1387701636.apply(Unknown Source)
[JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
[JBossINF]
[JBossINF] [0m[31m00:06:26,965 ERROR [io.undertow.request] (default task-27) UT005023: Exception handling request to /clusterbench/session: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf21
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$80(JGroupsTransport.java:602)
[JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$34/1387701636.apply(Unknown Source)
[JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
[JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
[JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] at java.lang.Thread.run(Thread.java:745)
[JBossINF]
[JBossINF] [0m[0m00:06:31,045 INFO [org.jboss.test.clusterbench.common.session.CommonHttpSessionServlet] (default task-90) New session created: iBowQB5GzE3tckL7V-5kzxBW4uvB6L9Ht0izC0PK
{code}
This is similar to https://bugzilla.redhat.com/show_bug.cgi?id=901164 except we're seeing it with more scenarios (see the names of jobs below).
Some jobs where this problem can be seen:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-...
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-...
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-...
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-http... <-- examples are taken from here
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-http...
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years, 9 months