[jboss-jira] [JBoss JIRA] (WFLY-12754) Clustering: cache inconsistencies in stress tests with replicated cache

Clustering: cache inconsistencies in stress tests with replicated cache ----------------------------------------------------------------------- Key: WFLY-12754 URL: https://issues.jboss.org/browse/WFLY-12754 Project: WildFly Issue Type: Bug Components: Clustering, Web (Undertow) Affects Versions: 18.0.0.Final Reporter: Tommasso Borgato Assignee: Paul Ferraro Priority: Critical The issue is about replicated-cache in stress tests. WildFly is started in clustered mode using a replicated cache for replicating HTTP session data across cluster nodes; all 4 nodes in the cluster are initialized with the following cli script (MOD_CLUSTER is used as load balancer): {noformat} embed-server --server-config=standalone-ha.xml /subsystem=modcluster/mod-cluster-config=configuration:write-attribute(name=advertise,value=false) /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=mod_cluster:add(host=10.0.145.184,port=7000) /subsystem=modcluster/mod-cluster-config=configuration:write-attribute(name=proxies,value=[mod_cluster]) /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp) /subsystem=infinispan/cache-container=web/replicated-cache=testRepl:add() /subsystem=infinispan/cache-container=web/replicated-cache=testRepl/component=locking:write-attribute(name=isolation, value=REPEATABLE_READ) /subsystem=infinispan/cache-container=web/replicated-cache=testRepl/component=transaction:write-attribute(name=mode, value=BATCH) /subsystem=infinispan/cache-container=web/replicated-cache=testRepl/store=file:add() /subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=testRepl) {noformat} Comparing with the previous version: - [wildfly-17.0.1.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redh...]: 0.86% Fail rate and all errors are "HTTP response code 503" errors - [wildfly-18.0.0.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redh...]: 0.9% Fail rate and all errors are sampling errors (inconsistencies in the cache returned values) One example of such a sampling error (note there is no jvmRoute change): {noformat} 2019-11-05 09:15:27,976 ERROR o.j.e.c.j.ClusteringHTTPRequestSampler: Invalid serial: Expected 200, received 2 (previous: JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} current: JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]}) samples sequence: 09:02:07.228 serial 0 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:02:11.231 serial 1 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:02:15.233 serial 2 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:02:19.236 serial 3 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:02:23.239 serial 4 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:02:27.241 serial 5 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:02:31.244 serial 6 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:02:35.247 serial 7 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:02:39.250 serial 8 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:02:43.253 serial 9 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:02:47.256 serial 10 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:02:51.259 serial 11 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:02:55.263 serial 12 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:02:59.266 serial 13 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:03.269 serial 14 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:07.274 serial 15 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:11.283 serial 16 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:15.286 serial 17 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:19.288 serial 18 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:23.290 serial 19 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:27.293 serial 20 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:31.295 serial 21 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:35.298 serial 22 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:39.308 serial 23 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:43.311 serial 24 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:47.313 serial 25 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:51.315 serial 26 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:55.318 serial 27 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:03:59.320 serial 28 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:03.323 serial 29 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:07.325 serial 30 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:11.329 serial 31 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:15.331 serial 32 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:19.335 serial 33 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:23.337 serial 34 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:27.341 serial 35 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:31.344 serial 36 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:35.349 serial 37 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:39.355 serial 38 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:43.357 serial 39 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:47.358 serial 40 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:51.360 serial 41 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:55.363 serial 42 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:04:59.365 serial 43 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:03.367 serial 44 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:07.369 serial 45 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:11.376 serial 46 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:15.378 serial 47 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:19.381 serial 48 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:23.385 serial 49 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:27.387 serial 50 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:31.394 serial 51 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:35.397 serial 52 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:39.400 serial 53 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:43.402 serial 54 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:47.404 serial 55 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:51.406 serial 56 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:55.408 serial 57 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:05:59.410 serial 58 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:03.413 serial 59 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:07.415 serial 60 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:11.417 serial 61 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:15.419 serial 62 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:19.422 serial 63 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:23.424 serial 64 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:27.426 serial 65 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:31.429 serial 66 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:35.432 serial 67 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:39.433 serial 68 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:43.448 serial 69 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:47.450 serial 70 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:51.452 serial 71 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:55.454 serial 72 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:06:59.457 serial 73 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:03.459 serial 74 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:07.461 serial 75 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:11.463 serial 76 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:15.465 serial 77 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:19.467 serial 78 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:23.469 serial 79 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:27.471 serial 80 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:31.474 serial 81 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:35.476 serial 82 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:39.478 serial 83 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:43.480 serial 84 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:47.482 serial 85 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:51.484 serial 86 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:55.487 serial 87 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:07:59.489 serial 88 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:03.491 serial 89 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:07.493 serial 90 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:11.495 serial 91 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:15.497 serial 92 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:19.499 serial 93 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:23.501 serial 94 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:27.506 serial 95 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:31.508 serial 96 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:35.510 serial 97 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:39.512 serial 98 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:43.514 serial 99 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:47.525 serial 100 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:51.527 serial 101 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:55.529 serial 102 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:08:59.531 serial 103 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:03.533 serial 104 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:07.536 serial 105 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:11.537 serial 106 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:15.540 serial 107 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:19.542 serial 108 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:23.544 serial 109 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:27.546 serial 110 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:31.551 serial 111 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:35.553 serial 112 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:39.555 serial 113 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:43.557 serial 114 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:47.585 serial 115 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:51.596 serial 116 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:55.598 serial 117 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:09:59.600 serial 118 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:03.602 serial 119 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:07.604 serial 120 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:11.606 serial 121 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:15.609 serial 122 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:19.611 serial 123 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:23.613 serial 124 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:27.615 serial 125 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:31.617 serial 126 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:35.620 serial 127 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:39.622 serial 128 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:43.624 serial 129 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:47.625 serial 130 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:51.628 serial 131 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:55.631 serial 132 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:10:59.633 serial 133 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:03.635 serial 134 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:07.637 serial 135 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:11.643 serial 136 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:15.645 serial 137 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:19.653 serial 138 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:23.655 serial 139 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:27.657 serial 140 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:31.659 serial 141 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:35.661 serial 142 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:39.663 serial 143 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:43.665 serial 144 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:47.668 serial 145 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:51.669 serial 146 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:55.671 serial 147 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:11:59.674 serial 148 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:03.677 serial 149 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:07.680 serial 150 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:11.685 serial 151 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:15.687 serial 152 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:19.704 serial 153 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:23.706 serial 154 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:27.709 serial 155 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:31.711 serial 156 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:35.713 serial 157 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:39.715 serial 158 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:43.717 serial 159 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:47.719 serial 160 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:51.721 serial 161 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:55.723 serial 162 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:12:59.725 serial 163 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:03.727 serial 164 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:07.730 serial 165 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:11.733 serial 166 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:15.736 serial 167 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:19.738 serial 168 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:23.740 serial 169 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:27.743 serial 170 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:31.745 serial 171 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:35.748 serial 172 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:39.752 serial 173 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:43.761 serial 174 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:47.777 serial 175 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:51.780 serial 176 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:55.786 serial 177 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:13:59.789 serial 178 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:03.805 serial 179 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:07.807 serial 180 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:11.809 serial 181 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:15.811 serial 182 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:19.812 serial 183 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:23.814 serial 184 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:27.816 serial 185 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:31.819 serial 186 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:35.821 serial 187 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:39.824 serial 188 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:43.827 serial 189 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:47.829 serial 190 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:51.831 serial 191 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:55.834 serial 192 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:14:59.835 serial 193 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:15:03.837 serial 194 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:15:07.839 serial 195 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:15:11.842 serial 196 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:15:15.844 serial 197 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:15:19.847 serial 198 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} 09:15:23.849 serial 199 JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC', jvmRoutes=[wildfly4]} {noformat}