[JBoss JIRA] (WFWIP-248) Avoid Route unavailability caused by load balancer recreation on CRD change
2:15 a.m.
[
https://issues.jboss.org/browse/WFWIP-248?page=com.atlassian.jira.plugin....
]
Jeff Mesnil resolved WFWIP-248.
-------------------------------
Resolution: Done
Avoid Route unavailability caused by load balancer recreation on CRD
change
---------------------------------------------------------------------------
Key: WFWIP-248
URL: https://issues.jboss.org/browse/WFWIP-248
Project: WildFly WIP
Issue Type: Bug
Components: OpenShift
Reporter: Petr Kremensky
Assignee: Jeff Mesnil
Priority: Blocker
Labels: operator
Tracked by https://github.com/wildfly/wildfly-operator/issues/117
Reproduce:
{code:sh}
### REPRODUCER TERMINAL
# deploy quickstart project
oc new-project wildfly-test
wildfly-operator]$ ./build/run-openshift.sh
oc apply -f deploy/crds/quickstart-cr.yaml
wildfly-operator]$ oc get pods
NAME READY STATUS RESTARTS AGE
quickstart-0 1/1 Running 0 34s
quickstart-1 1/1 Running 0 34s
wildfly-operator-7f555b86d5-jkctc 1/1 Running 0 99s
# verify the app is responsive
ROUTE=quickstart-route-wildfly-test.apps-crc.testing
curl $ROUTE
{"ip":"10.128.1.43"}
# watch pods
oc get pods -w &
# verify route readiness
ROUTE=quickstart-route-wildfly-test.apps-crc.testing
# I've seen some weird bash optimization using `while true` in past which I'd
like to avoid here...
for i in {0..100000} ; do
echo '----------------------------------------'
date
curl -sI $ROUTE | grep HTTP
curl -s $ROUTE | head -n 1
echo ''
oc describe service/quickstart-loadbalancer | grep Endpoints
sleep 0.5
done
### ADMIN TERMINAL - scale the application
oc patch wildflyserver quickstart -p '[{"op":"replace",
"path":"/spec/replicas", "value":3}]' --type json
{code}
output:
{noformat}
----------------------------------------
Mon 14 Oct 2019 01:12:09 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
quickstart-2 0/1 Pending 0 0s
quickstart-2 0/1 Pending 0 0s
quickstart-2 0/1 Pending 0 0s
quickstart-2 0/1 Pending 0 0s
quickstart-2 0/1 ContainerCreating 0 0s
quickstart-2 0/1 ContainerCreating 0 0s
Error from server (NotFound): services "quickstart-loadbalancer" not found
----------------------------------------
Mon 14 Oct 2019 01:12:09 PM CEST
HTTP/1.0 503 Service Unavailable
<html>
Endpoints: 10.128.1.43:8080,10.128.1.44:8080
----------------------------------------
Mon 14 Oct 2019 01:12:10 PM CEST
HTTP/1.0 503 Service Unavailable
<html>
Endpoints: 10.128.1.43:8080,10.128.1.44:8080
----------------------------------------
Mon 14 Oct 2019 01:12:11 PM CEST
HTTP/1.0 503 Service Unavailable
<html>
Endpoints: 10.128.1.43:8080,10.128.1.44:8080
----------------------------------------
Mon 14 Oct 2019 01:12:12 PM CEST
HTTP/1.0 503 Service Unavailable
<html>
Endpoints: 10.128.1.43:8080,10.128.1.44:8080
----------------------------------------
Mon 14 Oct 2019 01:12:13 PM CEST
HTTP/1.0 503 Service Unavailable
<html>
Endpoints: 10.128.1.43:8080,10.128.1.44:8080
----------------------------------------
Mon 14 Oct 2019 01:12:13 PM CEST
HTTP/1.0 503 Service Unavailable
<html>
Endpoints: 10.128.1.43:8080,10.128.1.44:8080
----------------------------------------
Mon 14 Oct 2019 01:12:14 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080
----------------------------------------
Mon 14 Oct 2019 01:12:15 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080
----------------------------------------
Mon 14 Oct 2019 01:12:16 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080
quickstart-2quickstart-2 0/10/1
ContainerCreatingContainerCreating 00 8s8s
----------------------------------------
Mon 14 Oct 2019 01:12:17 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080
quickstart-2 1/1 Running 0 8s
quickstart-2 1/1 Running 0 8s
----------------------------------------
Mon 14 Oct 2019 01:12:17 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:18 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:19 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:20 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:24 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:24 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:25 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:26 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:27 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:28 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:28 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:29 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:30 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:31 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:32 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:32 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:33 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:34 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.44"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:35 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.43"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
----------------------------------------
Mon 14 Oct 2019 01:12:36 PM CEST
HTTP/1.1 200 OK
{"ip":"10.128.1.47"}
Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
{noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
2475
days inactive
2475
days old
0 comments
1 participants
participants (1)
-
Jeff Mesnil (Jira)