[JBoss JIRA] (WFWIP-206) scale down isn't successful when there are in-doubt transaction on pod
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-206?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil updated WFWIP-206:
------------------------------
Labels: operator (was: )
> scale down isn't successful when there are in-doubt transaction on pod
> ----------------------------------------------------------------------
>
> Key: WFWIP-206
> URL: https://issues.jboss.org/browse/WFWIP-206
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Environment: operator, built from [1984d98154f11a7473be065e4ae7f54b1812c9b0|https://github.com/wildfly/wildf...] :
> {noformat}
> docker-registry.engineering.redhat.com/jbossqe-eap/wildfly-operator:latest
> {noformat}
> EAP image:
> {noformat}
> docker-registry.engineering.redhat.com/ochaloup/wildfly18-snapshot:190909...
> {noformat}
> Reporter: Martin Simka
> Assignee: Ondrej Chaloupka
> Priority: Blocker
> Labels: operator
>
> While testing tx recovery in OpenShift I see that scale down of pod that has transaction in-doubt on it isn't successful
> Scenario:
> *ejb client* (app tx-client, pod tx-client-0):
> * EJB business method
> ** lookup remote EJB
> ** enlist XA resource 1 to transaction
> ** enlist XA resource 2 to transaction
> ** call remote EJB
> *ejb server* (app tx-server, pod tx-server-0):
> * EJB business method
> ** enlist XA resource 1 to transaction
> ** enlist XA resource 2 to transaction
> ejb server XA resource 2 fails with {{XAException(XAException.XAER_RMFAIL)}}
> Then the test calls scale down (size from 1 to 0) on tx-server pod. But scale down never completes.
> Log from operator:
> {noformat}
> {"level":"info","ts":1568905676.6303623,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905676.7313502,"logger":"controller_wildflyserver","msg":"Enabling recovery listener for processing scaledown at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905679.4325309,"logger":"controller_wildflyserver","msg":"Query to find the transaction recovery port to force scan at pod tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905686.7914035,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"info","ts":1568905702.0583296,"logger":"controller_wildflyserver","msg":"In-doubt transactions in object store","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","Message":"Recovery scan to be invoked as the transaction log storage is not empty for pod scaling down pod tx-server-0, transaction list: map[0:ffffac11000a:991c183:5d8399a1:13:map[age-in-seconds:23 id:0:ffffac11000a:991c183:5d8399a1:13 jmx-name:<nil> participants:map[java:/MockXAResource:map[eis-product-name:MockXAResource Test eis-product-version:0.1.Mock jmx-name:<nil> jndi-name:java:/MockXAResource status:PREPARED type:/StateManager/AbstractRecord/XAResourceRecord]] type:StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction/SubordinateAtomicAction/JCA]]"}
> {"level":"info","ts":1568905711.1034026,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905711.103548,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905711.109706,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"error","ts":1568905711.2608829,"logger":"controller_wildflyserver","msg":"Failures during scaling down recovery processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Desired replica size":0,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod tx-server-0. Please, verify the pod log file. Error: Error to get response for command SCAN sending to 172.17.0.10:4712, error: EOF]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905711.2609518,"logger":"controller_wildflyserver","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.2609615,"logger":"controller_wildflyserver","msg":"Statefulset was not fully scaled to the desired replica size 0 while StatefulSet is to be at size 1. Some pods were not cleaned by recovery. Verify status of the WildflyServer tx-server","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.2795022,"logger":"controller_wildflyserver","msg":"Updating StatefulSet to be up to date with the WildFlyServer Spec","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.2795491,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905711.2796504,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905711.2937052,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"error","ts":1568905711.294249,"logger":"controller_wildflyserver","msg":"Failures during scaling down recovery processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Desired replica size":0,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod tx-server-0. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.10:4712, error: dial tcp 172.17.0.10:4712: connect: connection refused]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905711.294342,"logger":"controller_wildflyserver","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.294417,"logger":"controller_wildflyserver","msg":"Statefulset was not fully scaled to the desired replica size 0 while StatefulSet is to be at size 1. Some pods were not cleaned by recovery. Verify status of the WildflyServer tx-server","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"error","ts":1568905711.311673,"logger":"controller_wildflyserver","msg":"Failed to Update StatefulSet.","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server","error":"Operation cannot be fulfilled on statefulsets.apps \"tx-server\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"error","ts":1568905711.311745,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"wildflyserver-controller","request":"msimka-namespace/tx-server","error":"Operation cannot be fulfilled on statefulsets.apps \"tx-server\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905712.3137681,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905712.3139439,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905712.3253288,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"error","ts":1568905712.3255754,"logger":"controller_wildflyserver","msg":"Failures during scaling down recovery processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Desired replica size":0,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod tx-server-0. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.10:4712, error: dial tcp 172.17.0.10:4712: connect: connection refused]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905712.3256311,"logger":"controller_wildflyserver","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905712.3256419,"logger":"controller_wildflyserver","msg":"Statefulset was not fully scaled to the desired replica size 0 while StatefulSet is to be at size 1. Some pods were not cleaned by recovery. Verify status of the WildflyServer tx-server","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (WFWIP-201) incomplete tx recovery on openshift
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-201?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil updated WFWIP-201:
------------------------------
Labels: operator (was: )
> incomplete tx recovery on openshift
> -----------------------------------
>
> Key: WFWIP-201
> URL: https://issues.jboss.org/browse/WFWIP-201
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Martin Simka
> Assignee: Ondrej Chaloupka
> Priority: Blocker
> Labels: operator
> Attachments: auth_fail.txt, tx-client-0.txt, tx-server-0.txt
>
>
> While testing tx recovery in OpenShift I see that recovery doesn't completely finish in some tests.
> Scenario:
> *ejb client* (app tx-client, pod tx-client-0):
> * EJB business method
> ** lookup remote EJB
> ** enlist XA resource 1 to transaction
> ** enlist XA resource 2 to transaction
> ** call remote EJB
> *ejb server* (app tx-server, pod tx-server-0):
> * EJB business method
> ** enlist XA resource 1 to transaction
> ** enlist XA resource 2 to transaction
> ejb server XA resource 2 crashes JVM in commit method phase. (the same outcome client crashes at commit phase)
> Test waits until crashed pod is restarted, then forces periodic recovery twice and then checks that transaction log store is empty. But it is not empty, however it looks like all XA resources were commited.
> Attached are logs from client and server pods.
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (WFWIP-196) Operator missing volumes from secrets and config maps
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-196?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil updated WFWIP-196:
------------------------------
Labels: operator (was: )
> Operator missing volumes from secrets and config maps
> ------------------------------------------------------
>
> Key: WFWIP-196
> URL: https://issues.jboss.org/browse/WFWIP-196
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Martin Choma
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: operator
>
> Operator is missing way of configuring volumes from secets and config maps
> - secrets
> -- configuring keystore in ssl
> -- configuring keystore for jgroups ASYNC encryption in jgroups (DNS_PING, KUBE_PING))
> - config map
> -- Microprofile Config and reading properties from ConfigMap
> -- running custom CLI script from config map (EAP7-892)
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (WFWIP-200) Runtime changes to sessionAffinity are not reflected in underlying objects
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-200?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil updated WFWIP-200:
------------------------------
Labels: operator (was: )
> Runtime changes to sessionAffinity are not reflected in underlying objects
> --------------------------------------------------------------------------
>
> Key: WFWIP-200
> URL: https://issues.jboss.org/browse/WFWIP-200
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Martin Choma
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: operator
>
> Any changes (adding, removing or updating) made to sessionAffinity attribute after WildFlyServer CR was created are not reflected in underlying Service kubernetes object.
> Reproducer:
> # create CR
> {code}
> apiVersion: wildfly.org/v1alpha1
> kind: WildFlyServer
> metadata:
> generation: 1
> name: eap-cd
> namespace: default
> spec:
> applicationImage: >-
> brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-eap-7-tech-preview/eap-cd-openshift-rhel8:17.0-4
> size: 1
> {code}
> # Edit CR with sessionAffinity: true
> {code}
> apiVersion: wildfly.org/v1alpha1
> kind: WildFlyServer
> metadata:
> generation: 1
> name: eap-cd
> namespace: default
> spec:
> applicationImage: >-
> brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-eap-7-tech-preview/eap-cd-openshift-rhel8:17.0-4
> size: 1
> sessionAffinity: true
> {code}
> # Service object eap-cd-loadbalancer is not updated
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (WFWIP-202) Runtime changes to disableHTTPRoute are not reflected in Route object
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-202?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil updated WFWIP-202:
------------------------------
Labels: operator (was: )
> Runtime changes to disableHTTPRoute are not reflected in Route object
> ---------------------------------------------------------------------
>
> Key: WFWIP-202
> URL: https://issues.jboss.org/browse/WFWIP-202
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Martin Choma
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: operator
>
> Any changes (adding, removing or updating) made to disableHTTPRoute attribute after WildFlyServer CR was created are not reflected in underlying Route object.
> Reproducer:
> # create CR
> {code}
> apiVersion: wildfly.org/v1alpha1
> kind: WildFlyServer
> metadata:
> generation: 1
> name: eap-cd
> namespace: default
> spec:
> applicationImage: >-
> brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-eap-7-tech-preview/eap-cd-openshift-rhel8:17.0-4
> size: 1
> disableHTTPRoute: false
> {code}
> # Edit CR with disableHTTPRoute: true. I would expect Route object will be deleted.
> {code}
> apiVersion: wildfly.org/v1alpha1
> kind: WildFlyServer
> metadata:
> generation: 1
> name: eap-cd
> namespace: default
> spec:
> applicationImage: >-
> brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-eap-7-tech-preview/eap-cd-openshift-rhel8:17.0-4
> size: 1
> disableHTTPRoute: true
> {code}
> # Route object eap-cd is not disabled/deleted
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (WFWIP-189) Operatoris not aware of Secret/ConfigMap updates (WildFlyServerSpec#envFrom) - this could lead to inconsistencies
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-189?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil updated WFWIP-189:
------------------------------
Labels: operator (was: )
> Operatoris not aware of Secret/ConfigMap updates (WildFlyServerSpec#envFrom) - this could lead to inconsistencies
> -----------------------------------------------------------------------------------------------------------------
>
> Key: WFWIP-189
> URL: https://issues.jboss.org/browse/WFWIP-189
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Petr Kremensky
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: operator
>
> User is able to share environment variables in a ConfigMap or Secret and pass it to Operator via [WildFlyServerSpec#envFrom|https://github.com/wildfly/wildfly-operator/blo...] field. Problem is, that Operator is not aware of changes in Secret/ConfigMap (only reference change is recognized - add or remove ConfigMap/Secret reference). This could lead to inconsistency of environment between pods in a single project (moreover this could lead also to inconsistency between projects in case that ConfigMap/Secret is shared by them).
> The reaction on ConfigMap/Secret content should be doable, see https://blog.questionable.services/article/kubernetes-deployments-configm...
> *reproduce*
> * create a config map with ("foo1", "bar1") entry
> * create an operator (size = 1) with a reference to the config map
> * update the config map - add ("foo2", "bar2") entry
> * resize the operator
> *actual*
> {code}
> $ oc get pods
> NAME READY STATUS RESTARTS AGE
> eap-cd-0 1/1 Running 0 112s
> eap-cd-1 1/1 Running 0 94s
> wildfly-operator-55b544c4bb-wts8l 1/1 Running 0 2m1s
> $ oc rsh pod/eap-cd-0
> sh-4.4$ env | grep foo
> foo1=bar1
> $ oc rsh pod/eap-cd-1
> sh-4.4$ env | grep foo
> foo1=bar1
> foo2=bar2
> {code}
> *expected*
> {code}
> $ oc get pods
> NAME READY STATUS RESTARTS AGE
> eap-cd-0 1/1 Running 0 112s
> eap-cd-1 1/1 Running 0 94s
> wildfly-operator-55b544c4bb-wts8l 1/1 Running 0 2m1s
> $ oc rsh pod/eap-cd-0
> sh-4.4$ env | grep foo
> foo1=bar1
> foo2=bar2
> $ oc rsh pod/eap-cd-1
> sh-4.4$ env | grep foo
> foo1=bar1
> foo2=bar2
> {code}
> *Environment:*
> *operator version:* [467407a|https://github.com/wildfly/wildfly-operator/commit/467407a6c21102...]
> *openshift version:*
> {noformat}
> OpenShift version: 4.1.11
> Kubernetes Master Version: v1.13.4+df9cebc
> {noformat}
> Out of curiosity, is there is way to rollout the pods manually as {{oc rollout}} was designed for DeploymentConfigs and cannot be used here?
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months