[jboss-jira] [JBoss JIRA] (WFWIP-252) Error log messages in operator logs on CRD edit after scale down
Petr Kremensky (Jira)
issues at jboss.org
Wed Oct 16 05:09:00 EDT 2019
[ https://issues.jboss.org/browse/WFWIP-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800211#comment-13800211 ]
Petr Kremensky commented on WFWIP-252:
--------------------------------------
{noformat}
{"level":"error","ts":1571208568.7673173,"logger":"wildlfyserver_resources","msg":"Failed to update resource","WildFlyServer.Namespace":"pkremens-namespace","WildFlyServer.Name":"simple-jaxrs-health-eap-cd","Resource.Name":"simple-jaxrs-health-eap-cd","Resource.Type":"*v1.StatefulSet","error":"Operation cannot be fulfilled on statefulsets.apps \"simple-jaxrs-health-eap-cd\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/wildfly/wildfly-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/resources.Update\n\t/root/go/src/github.com/wildfly/wildfly-operator/pkg/resources/actions.go:72\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).checkStatefulSet\n\t/root/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:399\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/root/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:187\ngithub.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1571208568.7674172,"logger":"wildflyserver_controller","msg":"Failed to update StatefulSet.","StatefulSet.Namespace":"pkremens-namespace","StatefulSet.Name":"simple-jaxrs-health-eap-cd","error":"Operation cannot be fulfilled on statefulsets.apps \"simple-jaxrs-health-eap-cd\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/wildfly/wildfly-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).checkStatefulSet\n\t/root/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:400\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/root/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:187\ngithub.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1571208568.7675304,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"wildflyserver-controller","request":"pkremens-namespace/simple-jaxrs-health-eap-cd","error":"Operation cannot be fulfilled on statefulsets.apps \"simple-jaxrs-health-eap-cd\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/wildfly/wildfly-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/root/go/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{noformat}
This is the one I'm trying to hit now ^
> Error log messages in operator logs on CRD edit after scale down
> ----------------------------------------------------------------
>
> Key: WFWIP-252
> URL: https://issues.jboss.org/browse/WFWIP-252
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Petr Kremensky
> Assignee: Jeff Mesnil
> Priority: Major
> Labels: operator
> Attachments: operator-logs.txt
>
>
> Error messages can be found in operator logs after user performs a scale down task (WFWIP-251) and than edit the CRD.
> *reproduce*
> * deploy operator {noformat}./build/run-openshift.sh{noformat}
> * change the default number of initial replicas from 2 to 3 {noformat}sed -i 's/2/3/' deploy/crds/quickstart-cr.yaml{noformat}
> * deploy quickstart project from operator repository {noformat}oc apply -f deploy/crds/quickstart-cr.yaml{noformat}
> * wait for both replicas to be ready {noformat}oc get pods
> NAME READY STATUS RESTARTS AGE
> quickstart-0 1/1 Running 0 31s
> quickstart-1 1/1 Running 0 31s
> quickstart-2 1/1 Running 0 31s
> wildfly-operator-5bcb545b46-j788h 1/1 Running 0 41s{noformat}
> * scale down to 2 {noformat}oc patch wildflyserver quickstart -p '[{"op":"replace", "path":"/spec/replicas", "value":2}]' --type json{noformat}
> * wait for scale down to perform {noformat} Pods:
> Name: quickstart-0
> Pod IP: 10.128.1.193
> State: ACTIVE
> Name: quickstart-1
> Pod IP: 10.128.1.192
> State: ACTIVE
> Replicas: 2
> Scalingdown Pods: 0
> {noformat}
> * edit the CRD (e.g. add an env variable) {noformat}oc edit wildflyserver quickstart
> ...
> replicas: 2
> env:
> - name: FOO
> value: BAR{noformat}
> * wait for both pods to be reloaded
> * see the [^operator-logs.txt]
> ** apart WFWIP-251, there are 2 extra error messages
> There might be other ways to reproduce the issue, but these should make it 100% appearance (at least for me).
> *actual*
> Error messages in operator log
> {noformat}
> {"level":"error","ts":1571215869.7030337,"logger":"wildlfyserver_resources","msg":"Failed to update resource","WildFlyServer.Namespace":"wildfly-test","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart","Resource.Type":"*v1.StatefulSet","error":"Operation cannot be fulfilled on statefulsets.apps \"quickstart\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr at v0.1.1/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/resources.Update\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/resources/actions.go:72\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).checkStatefulSet\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:399\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:88"}
> {"level":"error","ts":1571215869.7032435,"logger":"wildflyserver_controller","msg":"Failed to update StatefulSet.","StatefulSet.Namespace":"wildfly-test","StatefulSet.Name":"quickstart","error":"Operation cannot be fulfilled on statefulsets.apps \"quickstart\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr at v0.1.1/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).checkStatefulSet\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:400\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:88"}
> {"level":"error","ts":1571215869.7034042,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"wildflyserver-controller","request":"wildfly-test/quickstart","error":"Operation cannot be fulfilled on statefulsets.apps \"quickstart\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr at v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:88"}
> {noformat}
> *expected*
> No error messages in operator log.
> Operator image build from [0a221ea|https://github.com/wildfly/wildfly-operator/commit/0a221eac5eb4628bd1cba27448e389bdbda8e813]
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
More information about the jboss-jira
mailing list