[jboss-jira] [JBoss JIRA] (WFWIP-206) scale down isn't successful when there are in-doubt transaction on pod

[ https://issues.jboss.org/browse/WFWIP-206?page=com.atlassian.jira.plugin.... ] Martin Simka edited comment on WFWIP-206 at 9/25/19 9:54 AM: ------------------------------------------------------------- with latest operator ([915fbc2db39df10b65cbe164f34affb695edb830|https://github.com/wildfly/wildf...]) scale down again doesn't work at all, pods are never scaled down. {noformat} {"level":"info","ts":1569419086.5349486,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server"} {"level":"info","ts":1569419086.5350897,"logger":"wildflyserver_controller","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.11","Pod State":"SCALING_DOWN_RECOVERY_INVESTIGATION","Pod Phase":"Running"} {"level":"info","ts":1569419086.6266701,"logger":"wildflyserver_controller","msg":"Verification the recovery listener is setup to run transaction recovery at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"} {"level":"info","ts":1569419089.6047068,"logger":"wildflyserver_controller","msg":"Query to find the transaction recovery port to force scan at pod tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"} {"level":"info","ts":1569419095.348274,"logger":"wildlfyserver_resources","msg":"Updating Resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server-0","Resource.Type":"*v1.Pod"} {"level":"error","ts":1569419095.3573227,"logger":"wildlfyserver_resources","msg":"Failed to update resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server-0","Resource.Type":"*v1.Pod","error":"Operation cannot be fulfilled on pods \"tx-server-0\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLog... {"level":"error","ts":1569419095.3575552,"logger":"wildflyserver_controller","msg":"Failures during scaling down recovery processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Desired replica size":0,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to update pod annotations, pod name tx-server-0, annotations to be set map[openshift.io/scc:restricted recovery-port:4712 wildfly.org/wildfly-server-generation:3], error: Operation cannot be fulfilled on pods \"tx-server-0\": the object has been modified; please apply your changes to the latest version and try again]],","stacktrace":"github.com/go-logr/zapr.(*zap... {"level":"info","ts":1569419095.3576174,"logger":"wildflyserver_controller","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"} {"level":"info","ts":1569419095.3576312,"logger":"wildflyserver_controller","msg":"Statefulset was not fully scaled to the desired replica size 0 (current StatefulSet size: 1). Some pods were not cleaned by recovery. Verify status of the WildflyServer tx-server","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"} {"level":"info","ts":1569419095.3576558,"logger":"wildlfyserver_resources","msg":"Resource generations do not match","WildFlyServer.Name":"tx-server","WildFlyServer.Namespace":"msimka-namespace","Resource.Name":"tx-server","Resource.Type":"*v1.StatefulSet","WildFlyServer.Generation":3,"Resource.Generation":"2"} {"level":"info","ts":1569419095.3577547,"logger":"wildlfyserver_resources","msg":"Updating Resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server","Resource.Type":"*v1.StatefulSet"} {"level":"info","ts":1569419095.4202075,"logger":"wildlfyserver_resources","msg":"Updated resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server","Resource.Type":"*v1.StatefulSet"} {"level":"info","ts":1569419095.4202409,"logger":"wildflyserver_controller","msg":"Updating StatefulSet to be up to date with the WildFlyServer Spec","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"} {"level":"info","ts":1569419095.4203625,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"} {"level":"info","ts":1569419095.4204185,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server"} {"level":"info","ts":1569419095.4204547,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server"} {"level":"info","ts":1569419095.4205728,"logger":"wildflyserver_controller","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.11","Pod State":"SCALING_DOWN_RECOVERY_INVESTIGATION","Pod Phase":"Running"} {"level":"info","ts":1569419095.4491572,"logger":"wildflyserver_controller","msg":"Verification the recovery listener is setup to run transaction recovery at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"} {"level":"info","ts":1569419098.3074143,"logger":"wildflyserver_controller","msg":"Query to find the transaction recovery port to force scan at pod tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"} {"level":"info","ts":1569419104.2142792,"logger":"wildlfyserver_resources","msg":"Updating Resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server-0","Resource.Type":"*v1.Pod"} {"level":"info","ts":1569419104.2362192,"logger":"wildlfyserver_resources","msg":"Updated resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server-0","Resource.Type":"*v1.Pod"} {"level":"info","ts":1569419104.236247,"logger":"wildflyserver_controller","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.11","Recovery port":4712} {"level":"info","ts":1569419120.079055,"logger":"wildlfyserver_resources","msg":"Updating status of WildFlyServer","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server"} {"level":"info","ts":1569419120.0828912,"logger":"wildlfyserver_resources","msg":"Updated status of WildFlyServer","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server"} {"level":"info","ts":1569419120.0829396,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"} {"level":"info","ts":1569419120.0829656,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server"} {"level":"info","ts":1569419120.082995,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server"} {"level":"info","ts":1569419120.0830805,"logger":"wildflyserver_controller","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"} {"level":"info","ts":1569419120.083089,"logger":"wildflyserver_controller","msg":"Scaling down and updating replica size to 0","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"} {"level":"info","ts":1569419120.0830944,"logger":"wildflyserver_controller","msg":"Updating statefulset","StatefulSet.Replicas":0} {"level":"info","ts":1569419120.1033726,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"} {"level":"info","ts":1569419120.1035216,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server"} {"level":"info","ts":1569419120.1035802,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server"} {"level":"info","ts":1569419120.1036997,"logger":"wildflyserver_controller","msg":"Number of pods does not match the WildFlyServer specification. Waiting to get numbers in sync.","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","WildflyServer specification":"tx-server","Expected number of pods":0,"Number of deployed pods":1,"StatefulSet spec size":0} {"level":"info","ts":1569419120.1235578,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"} {"level":"info","ts":1569419120.1236944,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server"} {"level":"info","ts":1569419120.123772,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server"} {"level":"info","ts":1569419120.1239598,"logger":"wildflyserver_controller","msg":"Number of pods does not match the WildFlyServer specification. Waiting to get numbers in sync.","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","WildflyServer specification":"tx-server","Expected number of pods":0,"Number of deployed pods":1,"StatefulSet spec size":0} {"level":"info","ts":1569419120.1254933,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"} {"level":"info","ts":1569419120.1255758,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server"} {"level":"info","ts":1569419120.1256313,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server"} {"level":"info","ts":1569419120.1258724,"logger":"wildflyserver_controller","msg":"Number of pods does not match the WildFlyServer specification. Waiting to get numbers in sync.","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","WildflyServer specification":"tx-server","Expected number of pods":0,"Number of deployed pods":1,"StatefulSet spec size":0} W0925 13:50:32.893831 1 reflector.go:270] pkg/mod/k8s.io/client-go@v0.0.0-20190228174230-b40b2a5939e4/tools/cache/reflector.go:95: watch of *v1.Route ended with: The resourceVersion for the provided watch is too old. {"level":"info","ts":1569419433.9129002,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-client"} {"level":"info","ts":1569419433.912962,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-client","Resource.Name":"tx-client"} {"level":"info","ts":1569419433.9130015,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-client","Resource.Name":"tx-client"} {"level":"info","ts":1569419433.9131017,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-client","Resource.Name":"tx-client-loadbalancer"} {"level":"info","ts":1569419433.9131217,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-client","Resource.Name":"tx-client-loadbalancer"} {"level":"info","ts":1569419433.9131458,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-client","Resource.Name":"tx-client-headless"} {"level":"info","ts":1569419433.9131584,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-client","Resource.Name":"tx-client-headless"} {"level":"info","ts":1569419433.913179,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-client","Resource.Name":"tx-client-route"} {"level":"info","ts":1569419433.913192,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-client","Resource.Name":"tx-client-route"} {"level":"info","ts":1569419433.9132183,"logger":"wildflyserver_controller","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"} {"level":"info","ts":1569419433.9132376,"logger":"wildlfyserver_resources","msg":"Getting resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server"} {"level":"info","ts":1569419433.9132624,"logger":"wildlfyserver_resources","msg":"Got resource","WildFlyServer.Namespace":"msimka-namespace","WildFlyServer.Name":"tx-server","Resource.Name":"tx-server"} {"level":"info","ts":1569419433.9133332,"logger":"wildflyserver_controller","msg":"Number of pods does not match the WildFlyServer specification. Waiting to get numbers in sync.","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","WildflyServer specification":"tx-server","Expected number of pods":0,"Number of deployed pods":1,"StatefulSet spec size":0} {noformat} was (Author: simkam): with latest operator ([915fbc2db39df10b65cbe164f34affb695edb830|https://github.com/wildfly/wildf...]) scale down again doesn't work at all, pods are never scaled down. -- This message was sent by Atlassian Jira (v7.13.8#713008)