[jboss-jira] [JBoss JIRA] (WFWIP-251) Operator error logs on scale down
Petr Kremensky (Jira)
issues at jboss.org
Wed Oct 16 04:12:00 EDT 2019
[ https://issues.jboss.org/browse/WFWIP-251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Petr Kremensky updated WFWIP-251:
---------------------------------
Description:
Error messages can be found in operator logs after user performs a scale down task.
*reproduce*
* deploy operator {noformat}./build/run-openshift.sh{noformat}
* deploy quickstart project from operator repository {noformat}oc apply -f deploy/crds/quickstart-cr.yaml{noformat}
* wait for both replicas to be ready {noformat}oc get pods -w
NAME READY STATUS RESTARTS AGE
quickstart-0 1/1 Running 0 32s
quickstart-1 1/1 Running 0 32s
wildfly-operator-5bcb545b46-2bjs8 1/1 Running 0 102s{noformat}
* scale down to 1 {noformat}oc patch wildflyserver quickstart -p '[{"op":"replace", "path":"/spec/replicas", "value":1}]' --type json{noformat}
* see the operator logs [^operator-logs.txt]
*actual*
Error messages in operator log
{noformat}
{"level":"error","ts":1571213256.675551,"logger":"wildlfyserver_resources","msg":"Failed to update resource","WildFlyServer.Namespace":"wildfly-test","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart","Resource.Type":"*v1.StatefulSet","error":"Operation cannot be fulfilled on statefulsets.apps \"quickstart\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr at v0.1.1/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/resources.Update\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/resources/actions.go:72\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).checkStatefulSet\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:399\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1571213256.6756332,"logger":"wildflyserver_controller","msg":"Failed to update StatefulSet.","StatefulSet.Namespace":"wildfly-test","StatefulSet.Name":"quickstart","error":"Operation cannot be fulfilled on statefulsets.apps \"quickstart\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr at v0.1.1/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).checkStatefulSet\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:400\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1571213256.675672,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"wildflyserver-controller","request":"wildfly-test/quickstart","error":"Operation cannot be fulfilled on statefulsets.apps \"quickstart\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr at v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:88"}
{noformat}
*expected*
No error messages in operator log.
Operator image build from [0a221ea|https://github.com/wildfly/wildfly-operator/commit/0a221eac5eb4628bd1cba27448e389bdbda8e813]
was:
Error messages can be found in operator logs after user performs a scale down task.
*reproduce*
* deploy operator {noformat}./build/run-openshift.sh{noformat}
* deploy quickstart project from operator repository {noformat}oc apply -f deploy/crds/quickstart-cr.yaml{noformat}
* wait for both replicas to be ready {noformat}oc get pods -w
NAME READY STATUS RESTARTS AGE
quickstart-0 1/1 Running 0 32s
quickstart-1 1/1 Running 0 32s
wildfly-operator-5bcb545b46-2bjs8 1/1 Running 0 102s{noformat}
* scale down to 1 {noformat}oc patch wildflyserver quickstart -p '[{"op":"replace", "path":"/spec/replicas", "value":1}]' --type json{noformat}
* see the operator logs
*actual*
Error messages in operator log
{noformat}
{"level":"error","ts":1571213256.675551,"logger":"wildlfyserver_resources","msg":"Failed to update resource","WildFlyServer.Namespace":"wildfly-test","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart","Resource.Type":"*v1.StatefulSet","error":"Operation cannot be fulfilled on statefulsets.apps \"quickstart\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr at v0.1.1/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/resources.Update\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/resources/actions.go:72\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).checkStatefulSet\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:399\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1571213256.6756332,"logger":"wildflyserver_controller","msg":"Failed to update StatefulSet.","StatefulSet.Namespace":"wildfly-test","StatefulSet.Name":"quickstart","error":"Operation cannot be fulfilled on statefulsets.apps \"quickstart\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr at v0.1.1/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).checkStatefulSet\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:400\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1571213256.675672,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"wildflyserver-controller","request":"wildfly-test/quickstart","error":"Operation cannot be fulfilled on statefulsets.apps \"quickstart\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr at v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:88"}
{noformat}
*expected*
No error messages in operator log.
Operator image build from [0a221ea|https://github.com/wildfly/wildfly-operator/commit/0a221eac5eb4628bd1cba27448e389bdbda8e813]
> Operator error logs on scale down
> ---------------------------------
>
> Key: WFWIP-251
> URL: https://issues.jboss.org/browse/WFWIP-251
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Petr Kremensky
> Assignee: Jeff Mesnil
> Priority: Critical
> Labels: operator
> Attachments: operator-logs.txt
>
>
> Error messages can be found in operator logs after user performs a scale down task.
> *reproduce*
> * deploy operator {noformat}./build/run-openshift.sh{noformat}
> * deploy quickstart project from operator repository {noformat}oc apply -f deploy/crds/quickstart-cr.yaml{noformat}
> * wait for both replicas to be ready {noformat}oc get pods -w
> NAME READY STATUS RESTARTS AGE
> quickstart-0 1/1 Running 0 32s
> quickstart-1 1/1 Running 0 32s
> wildfly-operator-5bcb545b46-2bjs8 1/1 Running 0 102s{noformat}
> * scale down to 1 {noformat}oc patch wildflyserver quickstart -p '[{"op":"replace", "path":"/spec/replicas", "value":1}]' --type json{noformat}
> * see the operator logs [^operator-logs.txt]
> *actual*
> Error messages in operator log
> {noformat}
> {"level":"error","ts":1571213256.675551,"logger":"wildlfyserver_resources","msg":"Failed to update resource","WildFlyServer.Namespace":"wildfly-test","WildFlyServer.Name":"quickstart","Resource.Name":"quickstart","Resource.Type":"*v1.StatefulSet","error":"Operation cannot be fulfilled on statefulsets.apps \"quickstart\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr at v0.1.1/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/resources.Update\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/resources/actions.go:72\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).checkStatefulSet\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:399\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:88"}
> {"level":"error","ts":1571213256.6756332,"logger":"wildflyserver_controller","msg":"Failed to update StatefulSet.","StatefulSet.Namespace":"wildfly-test","StatefulSet.Name":"quickstart","error":"Operation cannot be fulfilled on statefulsets.apps \"quickstart\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr at v0.1.1/zapr.go:128\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).checkStatefulSet\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:400\ngithub.com/wildfly/wildfly-operator/pkg/controller/wildflyserver.(*ReconcileWildFlyServer).Reconcile\n\t/go/src/github.com/wildfly/wildfly-operator/pkg/controller/wildflyserver/wildflyserver_controller.go:187\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:215\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:88"}
> {"level":"error","ts":1571213256.675672,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"wildflyserver-controller","request":"wildfly-test/quickstart","error":"Operation cannot be fulfilled on statefulsets.apps \"quickstart\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr at v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime at v0.1.12/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery at v0.0.0-20190221213512-86fb29eff628/pkg/util/wait/wait.go:88"}
> {noformat}
> *expected*
> No error messages in operator log.
> Operator image build from [0a221ea|https://github.com/wildfly/wildfly-operator/commit/0a221eac5eb4628bd1cba27448e389bdbda8e813]
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
More information about the jboss-jira
mailing list