]
Paul Ferraro commented on WFLY-12754:
-------------------------------------
[~tommaso-borgato] This is completely consistent with the behavior change I described in
In WF17, the cache put happens during the request, whereas in WF18, the cache put happens
after the response is committed.
Thus a given failure of this cache put will result in a 503 on WF17, but a sampling error
on WF18.
Clustering: cache inconsistencies in stress tests with replicated
cache
-----------------------------------------------------------------------
Key: WFLY-12754
URL:
https://issues.jboss.org/browse/WFLY-12754
Project: WildFly
Issue Type: Bug
Components: Clustering, Web (Undertow)
Affects Versions: 18.0.0.Final
Reporter: Tommasso Borgato
Assignee: Paul Ferraro
Priority: Critical
The issue is about replicated-cache in stress tests.
WildFly is started in clustered mode using a replicated cache for replicating HTTP
session data across cluster nodes; all 4 nodes in the cluster are initialized with the
following cli script (MOD_CLUSTER is used as load balancer):
{noformat}
embed-server --server-config=standalone-ha.xml
/subsystem=modcluster/mod-cluster-config=configuration:write-attribute(name=advertise,value=false)
/socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=mod_cluster:add(host=10.0.145.184,port=7000)
/subsystem=modcluster/mod-cluster-config=configuration:write-attribute(name=proxies,value=[mod_cluster])
/subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
/subsystem=infinispan/cache-container=web/replicated-cache=testRepl:add()
/subsystem=infinispan/cache-container=web/replicated-cache=testRepl/component=locking:write-attribute(name=isolation,
value=REPEATABLE_READ)
/subsystem=infinispan/cache-container=web/replicated-cache=testRepl/component=transaction:write-attribute(name=mode,
value=BATCH)
/subsystem=infinispan/cache-container=web/replicated-cache=testRepl/store=file:add()
/subsystem=infinispan/cache-container=web:write-attribute(name=default-cache,
value=testRepl)
{noformat}
Comparing with the previous version:
-
[
wildfly-17.0.1.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redh...]:
0.86% Fail rate and all errors are "HTTP response code 503" errors
-
[
wildfly-18.0.0.Final.zip|https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redh...]:
0.9% Fail rate and all errors are sampling errors (inconsistencies in the cache returned
values)
One example of such a sampling error (note there is no jvmRoute change):
{noformat}
2019-11-05 09:15:27,976 ERROR o.j.e.c.j.ClusteringHTTPRequestSampler: Invalid serial:
Expected 200, received 2 (previous:
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]} current:
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]})
samples sequence:
09:02:07.228 serial 0
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:02:11.231 serial 1
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:02:15.233 serial 2
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:02:19.236 serial 3
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:02:23.239 serial 4
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:02:27.241 serial 5
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:02:31.244 serial 6
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:02:35.247 serial 7
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:02:39.250 serial 8
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:02:43.253 serial 9
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:02:47.256 serial 10
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:02:51.259 serial 11
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:02:55.263 serial 12
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:02:59.266 serial 13
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:03.269 serial 14
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:07.274 serial 15
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:11.283 serial 16
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:15.286 serial 17
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:19.288 serial 18
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:23.290 serial 19
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:27.293 serial 20
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:31.295 serial 21
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:35.298 serial 22
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:39.308 serial 23
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:43.311 serial 24
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:47.313 serial 25
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:51.315 serial 26
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:55.318 serial 27
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:03:59.320 serial 28
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:03.323 serial 29
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:07.325 serial 30
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:11.329 serial 31
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:15.331 serial 32
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:19.335 serial 33
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:23.337 serial 34
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:27.341 serial 35
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:31.344 serial 36
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:35.349 serial 37
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:39.355 serial 38
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:43.357 serial 39
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:47.358 serial 40
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:51.360 serial 41
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:55.363 serial 42
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:04:59.365 serial 43
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:03.367 serial 44
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:07.369 serial 45
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:11.376 serial 46
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:15.378 serial 47
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:19.381 serial 48
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:23.385 serial 49
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:27.387 serial 50
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:31.394 serial 51
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:35.397 serial 52
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:39.400 serial 53
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:43.402 serial 54
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:47.404 serial 55
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:51.406 serial 56
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:55.408 serial 57
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:05:59.410 serial 58
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:03.413 serial 59
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:07.415 serial 60
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:11.417 serial 61
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:15.419 serial 62
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:19.422 serial 63
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:23.424 serial 64
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:27.426 serial 65
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:31.429 serial 66
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:35.432 serial 67
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:39.433 serial 68
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:43.448 serial 69
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:47.450 serial 70
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:51.452 serial 71
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:55.454 serial 72
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:06:59.457 serial 73
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:03.459 serial 74
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:07.461 serial 75
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:11.463 serial 76
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:15.465 serial 77
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:19.467 serial 78
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:23.469 serial 79
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:27.471 serial 80
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:31.474 serial 81
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:35.476 serial 82
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:39.478 serial 83
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:43.480 serial 84
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:47.482 serial 85
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:51.484 serial 86
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:55.487 serial 87
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:07:59.489 serial 88
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:03.491 serial 89
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:07.493 serial 90
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:11.495 serial 91
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:15.497 serial 92
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:19.499 serial 93
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:23.501 serial 94
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:27.506 serial 95
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:31.508 serial 96
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:35.510 serial 97
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:39.512 serial 98
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:43.514 serial 99
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:47.525 serial 100
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:51.527 serial 101
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:55.529 serial 102
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:08:59.531 serial 103
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:03.533 serial 104
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:07.536 serial 105
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:11.537 serial 106
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:15.540 serial 107
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:19.542 serial 108
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:23.544 serial 109
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:27.546 serial 110
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:31.551 serial 111
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:35.553 serial 112
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:39.555 serial 113
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:43.557 serial 114
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:47.585 serial 115
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:51.596 serial 116
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:55.598 serial 117
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:09:59.600 serial 118
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:03.602 serial 119
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:07.604 serial 120
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:11.606 serial 121
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:15.609 serial 122
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:19.611 serial 123
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:23.613 serial 124
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:27.615 serial 125
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:31.617 serial 126
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:35.620 serial 127
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:39.622 serial 128
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:43.624 serial 129
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:47.625 serial 130
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:51.628 serial 131
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:55.631 serial 132
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:10:59.633 serial 133
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:03.635 serial 134
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:07.637 serial 135
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:11.643 serial 136
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:15.645 serial 137
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:19.653 serial 138
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:23.655 serial 139
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:27.657 serial 140
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:31.659 serial 141
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:35.661 serial 142
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:39.663 serial 143
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:43.665 serial 144
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:47.668 serial 145
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:51.669 serial 146
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:55.671 serial 147
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:11:59.674 serial 148
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:03.677 serial 149
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:07.680 serial 150
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:11.685 serial 151
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:15.687 serial 152
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:19.704 serial 153
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:23.706 serial 154
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:27.709 serial 155
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:31.711 serial 156
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:35.713 serial 157
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:39.715 serial 158
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:43.717 serial 159
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:47.719 serial 160
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:51.721 serial 161
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:55.723 serial 162
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:12:59.725 serial 163
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:03.727 serial 164
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:07.730 serial 165
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:11.733 serial 166
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:15.736 serial 167
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:19.738 serial 168
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:23.740 serial 169
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:27.743 serial 170
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:31.745 serial 171
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:35.748 serial 172
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:39.752 serial 173
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:43.761 serial 174
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:47.777 serial 175
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:51.780 serial 176
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:55.786 serial 177
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:13:59.789 serial 178
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:03.805 serial 179
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:07.807 serial 180
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:11.809 serial 181
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:15.811 serial 182
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:19.812 serial 183
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:23.814 serial 184
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:27.816 serial 185
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:31.819 serial 186
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:35.821 serial 187
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:39.824 serial 188
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:43.827 serial 189
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:47.829 serial 190
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:51.831 serial 191
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:55.834 serial 192
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:14:59.835 serial 193
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:15:03.837 serial 194
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:15:07.839 serial 195
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:15:11.842 serial 196
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:15:15.844 serial 197
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:15:19.847 serial 198
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
09:15:23.849 serial 199
JSessionId{sessionId='yyBiQJw7wKDgQpBkJYa7RLRDyTJcciVeAlmDfMgC',
jvmRoutes=[wildfly4]}
{noformat}