[jboss-jira] [JBoss JIRA] (WFWIP-248) Avoid Route unavailability caused by load balancer recreation on CRD change
Jeff Mesnil (Jira)
issues at jboss.org
Mon Oct 14 11:50:00 EDT 2019
[ https://issues.jboss.org/browse/WFWIP-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798883#comment-13798883 ]
Jeff Mesnil commented on WFWIP-248:
-----------------------------------
[~pkremens] Can be tested with the image registry-proxy.engineering.redhat.com/rh-osbs/jboss-eap-7-tech-preview-eap-operator:jb-eap-7.3-operator-rhel8-containers-candidate-41808-20191014154416 built from https://code.engineering.redhat.com/gerrit/gitweb?p=wildfly/wildfly-operator.git;a=shortlog;h=refs/heads/build-b0d026bd8f1c6d496b9a2df7eb99ce9d6e923a71
> Avoid Route unavailability caused by load balancer recreation on CRD change
> ---------------------------------------------------------------------------
>
> Key: WFWIP-248
> URL: https://issues.jboss.org/browse/WFWIP-248
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Petr Kremensky
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: operator
>
> Tracked by https://github.com/wildfly/wildfly-operator/issues/117
> Reproduce:
> {code:sh}
> ### REPRODUCER TERMINAL
> # deploy quickstart project
> oc new-project wildfly-test
> wildfly-operator]$ ./build/run-openshift.sh
> oc apply -f deploy/crds/quickstart-cr.yaml
> wildfly-operator]$ oc get pods
> NAME READY STATUS RESTARTS AGE
> quickstart-0 1/1 Running 0 34s
> quickstart-1 1/1 Running 0 34s
> wildfly-operator-7f555b86d5-jkctc 1/1 Running 0 99s
> # verify the app is responsive
> ROUTE=quickstart-route-wildfly-test.apps-crc.testing
> curl $ROUTE
> {"ip":"10.128.1.43"}
> # watch pods
> oc get pods -w &
> # verify route readiness
> ROUTE=quickstart-route-wildfly-test.apps-crc.testing
> # I've seen some weird bash optimization using `while true` in past which I'd like to avoid here...
> for i in {0..100000} ; do
> echo '----------------------------------------'
> date
> curl -sI $ROUTE | grep HTTP
> curl -s $ROUTE | head -n 1
> echo ''
> oc describe service/quickstart-loadbalancer | grep Endpoints
> sleep 0.5
> done
> ### ADMIN TERMINAL - scale the application
> oc patch wildflyserver quickstart -p '[{"op":"replace", "path":"/spec/replicas", "value":3}]' --type json
> {code}
> output:
> {noformat}
> ----------------------------------------
> Mon 14 Oct 2019 01:12:09 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> quickstart-2 0/1 Pending 0 0s
> quickstart-2 0/1 Pending 0 0s
> quickstart-2 0/1 Pending 0 0s
> quickstart-2 0/1 Pending 0 0s
> quickstart-2 0/1 ContainerCreating 0 0s
> quickstart-2 0/1 ContainerCreating 0 0s
> Error from server (NotFound): services "quickstart-loadbalancer" not found
> ----------------------------------------
> Mon 14 Oct 2019 01:12:09 PM CEST
> HTTP/1.0 503 Service Unavailable
> <html>
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:10 PM CEST
> HTTP/1.0 503 Service Unavailable
> <html>
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:11 PM CEST
> HTTP/1.0 503 Service Unavailable
> <html>
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:12 PM CEST
> HTTP/1.0 503 Service Unavailable
> <html>
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:13 PM CEST
> HTTP/1.0 503 Service Unavailable
> <html>
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:13 PM CEST
> HTTP/1.0 503 Service Unavailable
> <html>
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:14 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:15 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:16 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080
> quickstart-2quickstart-2 0/10/1 ContainerCreatingContainerCreating 00 8s8s
> ----------------------------------------
> Mon 14 Oct 2019 01:12:17 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080
> quickstart-2 1/1 Running 0 8s
> quickstart-2 1/1 Running 0 8s
> ----------------------------------------
> Mon 14 Oct 2019 01:12:17 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:18 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:19 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:20 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:24 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:24 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:25 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:26 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:27 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:28 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:28 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:29 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:30 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:31 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:32 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:32 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:33 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:34 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.44"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:35 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.43"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> ----------------------------------------
> Mon 14 Oct 2019 01:12:36 PM CEST
> HTTP/1.1 200 OK
> {"ip":"10.128.1.47"}
> Endpoints: 10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
More information about the jboss-jira
mailing list