[JBoss JIRA] (WFWIP-206) scale down isn't successful when there are in-doubt transaction on pod
by Martin Simka (Jira)
[ https://issues.jboss.org/browse/WFWIP-206?page=com.atlassian.jira.plugin.... ]
Martin Simka edited comment on WFWIP-206 at 10/1/19 11:37 AM:
--------------------------------------------------------------
update:
* failure in {{testTxStatelessServerSecondCommitThrowRmFail}} - fails, I created WFWIP-218
* {{testTxStatelessClientSecondCommitThrowRmFail}} passes
* {{testTxStatelessServerSecondPrepareJvmHalt}} passes
* {{testTxStatelessClientSecondPrepareJvmHalt}} fails, but I'm not sure if it's expectations are correct, anyway I created WFWIP-222
was (Author: simkam):
update:
* failure in {{testTxStatelessServerSecondCommitThrowRmFail}} - fails, I created WFWIP-218
* {{testTxStatelessClientSecondCommitThrowRmFail}} passes
* {{testTxStatelessServerSecondPrepareJvmHalt}} passes
* {{testTxStatelessClientSecondPrepareJvmHalt}} pending retest
> scale down isn't successful when there are in-doubt transaction on pod
> ----------------------------------------------------------------------
>
> Key: WFWIP-206
> URL: https://issues.jboss.org/browse/WFWIP-206
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Environment: operator, built from [1984d98154f11a7473be065e4ae7f54b1812c9b0|https://github.com/wildfly/wildf...] :
> {noformat}
> docker-registry.engineering.redhat.com/jbossqe-eap/wildfly-operator:latest
> {noformat}
> EAP image:
> {noformat}
> docker-registry.engineering.redhat.com/ochaloup/wildfly18-snapshot:190909...
> {noformat}
> Reporter: Martin Simka
> Assignee: Ondrej Chaloupka
> Priority: Blocker
> Labels: operator
>
> While testing tx recovery in OpenShift I see that scale down of pod that has transaction in-doubt on it isn't successful
> Scenario:
> *ejb client* (app tx-client, pod tx-client-0):
> * EJB business method
> ** lookup remote EJB
> ** enlist XA resource 1 to transaction
> ** enlist XA resource 2 to transaction
> ** call remote EJB
> *ejb server* (app tx-server, pod tx-server-0):
> * EJB business method
> ** enlist XA resource 1 to transaction
> ** enlist XA resource 2 to transaction
> ejb server XA resource 2 fails with {{XAException(XAException.XAER_RMFAIL)}}
> Then the test calls scale down (size from 1 to 0) on tx-server pod. But scale down never completes.
> Log from operator:
> {noformat}
> {"level":"info","ts":1568905676.6303623,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905676.7313502,"logger":"controller_wildflyserver","msg":"Enabling recovery listener for processing scaledown at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905679.4325309,"logger":"controller_wildflyserver","msg":"Query to find the transaction recovery port to force scan at pod tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905686.7914035,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"info","ts":1568905702.0583296,"logger":"controller_wildflyserver","msg":"In-doubt transactions in object store","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","Message":"Recovery scan to be invoked as the transaction log storage is not empty for pod scaling down pod tx-server-0, transaction list: map[0:ffffac11000a:991c183:5d8399a1:13:map[age-in-seconds:23 id:0:ffffac11000a:991c183:5d8399a1:13 jmx-name:<nil> participants:map[java:/MockXAResource:map[eis-product-name:MockXAResource Test eis-product-version:0.1.Mock jmx-name:<nil> jndi-name:java:/MockXAResource status:PREPARED type:/StateManager/AbstractRecord/XAResourceRecord]] type:StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction/SubordinateAtomicAction/JCA]]"}
> {"level":"info","ts":1568905711.1034026,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905711.103548,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905711.109706,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"error","ts":1568905711.2608829,"logger":"controller_wildflyserver","msg":"Failures during scaling down recovery processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Desired replica size":0,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod tx-server-0. Please, verify the pod log file. Error: Error to get response for command SCAN sending to 172.17.0.10:4712, error: EOF]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905711.2609518,"logger":"controller_wildflyserver","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.2609615,"logger":"controller_wildflyserver","msg":"Statefulset was not fully scaled to the desired replica size 0 while StatefulSet is to be at size 1. Some pods were not cleaned by recovery. Verify status of the WildflyServer tx-server","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.2795022,"logger":"controller_wildflyserver","msg":"Updating StatefulSet to be up to date with the WildFlyServer Spec","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.2795491,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905711.2796504,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905711.2937052,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"error","ts":1568905711.294249,"logger":"controller_wildflyserver","msg":"Failures during scaling down recovery processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Desired replica size":0,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod tx-server-0. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.10:4712, error: dial tcp 172.17.0.10:4712: connect: connection refused]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905711.294342,"logger":"controller_wildflyserver","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.294417,"logger":"controller_wildflyserver","msg":"Statefulset was not fully scaled to the desired replica size 0 while StatefulSet is to be at size 1. Some pods were not cleaned by recovery. Verify status of the WildflyServer tx-server","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"error","ts":1568905711.311673,"logger":"controller_wildflyserver","msg":"Failed to Update StatefulSet.","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server","error":"Operation cannot be fulfilled on statefulsets.apps \"tx-server\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"error","ts":1568905711.311745,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"wildflyserver-controller","request":"msimka-namespace/tx-server","error":"Operation cannot be fulfilled on statefulsets.apps \"tx-server\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905712.3137681,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905712.3139439,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905712.3253288,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"error","ts":1568905712.3255754,"logger":"controller_wildflyserver","msg":"Failures during scaling down recovery processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Desired replica size":0,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod tx-server-0. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.10:4712, error: dial tcp 172.17.0.10:4712: connect: connection refused]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905712.3256311,"logger":"controller_wildflyserver","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905712.3256419,"logger":"controller_wildflyserver","msg":"Statefulset was not fully scaled to the desired replica size 0 while StatefulSet is to be at size 1. Some pods were not cleaned by recovery. Verify status of the WildflyServer tx-server","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 9 months
[JBoss JIRA] (WFWIP-206) scale down isn't successful when there are in-doubt transaction on pod
by Martin Simka (Jira)
[ https://issues.jboss.org/browse/WFWIP-206?page=com.atlassian.jira.plugin.... ]
Martin Simka edited comment on WFWIP-206 at 10/1/19 11:37 AM:
--------------------------------------------------------------
update:
* failure in {{testTxStatelessServerSecondCommitThrowRmFail}} - fails, I created WFWIP-218
* {{testTxStatelessClientSecondCommitThrowRmFail}} passes
* {{testTxStatelessServerSecondPrepareJvmHalt}} passes
* {{testTxStatelessClientSecondPrepareJvmHalt}} fails, but I'm not sure if its expectations are correct, anyway I created WFWIP-222
was (Author: simkam):
update:
* failure in {{testTxStatelessServerSecondCommitThrowRmFail}} - fails, I created WFWIP-218
* {{testTxStatelessClientSecondCommitThrowRmFail}} passes
* {{testTxStatelessServerSecondPrepareJvmHalt}} passes
* {{testTxStatelessClientSecondPrepareJvmHalt}} fails, but I'm not sure if it's expectations are correct, anyway I created WFWIP-222
> scale down isn't successful when there are in-doubt transaction on pod
> ----------------------------------------------------------------------
>
> Key: WFWIP-206
> URL: https://issues.jboss.org/browse/WFWIP-206
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Environment: operator, built from [1984d98154f11a7473be065e4ae7f54b1812c9b0|https://github.com/wildfly/wildf...] :
> {noformat}
> docker-registry.engineering.redhat.com/jbossqe-eap/wildfly-operator:latest
> {noformat}
> EAP image:
> {noformat}
> docker-registry.engineering.redhat.com/ochaloup/wildfly18-snapshot:190909...
> {noformat}
> Reporter: Martin Simka
> Assignee: Ondrej Chaloupka
> Priority: Blocker
> Labels: operator
>
> While testing tx recovery in OpenShift I see that scale down of pod that has transaction in-doubt on it isn't successful
> Scenario:
> *ejb client* (app tx-client, pod tx-client-0):
> * EJB business method
> ** lookup remote EJB
> ** enlist XA resource 1 to transaction
> ** enlist XA resource 2 to transaction
> ** call remote EJB
> *ejb server* (app tx-server, pod tx-server-0):
> * EJB business method
> ** enlist XA resource 1 to transaction
> ** enlist XA resource 2 to transaction
> ejb server XA resource 2 fails with {{XAException(XAException.XAER_RMFAIL)}}
> Then the test calls scale down (size from 1 to 0) on tx-server pod. But scale down never completes.
> Log from operator:
> {noformat}
> {"level":"info","ts":1568905676.6303623,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905676.7313502,"logger":"controller_wildflyserver","msg":"Enabling recovery listener for processing scaledown at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905679.4325309,"logger":"controller_wildflyserver","msg":"Query to find the transaction recovery port to force scan at pod tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905686.7914035,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"info","ts":1568905702.0583296,"logger":"controller_wildflyserver","msg":"In-doubt transactions in object store","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","Message":"Recovery scan to be invoked as the transaction log storage is not empty for pod scaling down pod tx-server-0, transaction list: map[0:ffffac11000a:991c183:5d8399a1:13:map[age-in-seconds:23 id:0:ffffac11000a:991c183:5d8399a1:13 jmx-name:<nil> participants:map[java:/MockXAResource:map[eis-product-name:MockXAResource Test eis-product-version:0.1.Mock jmx-name:<nil> jndi-name:java:/MockXAResource status:PREPARED type:/StateManager/AbstractRecord/XAResourceRecord]] type:StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction/SubordinateAtomicAction/JCA]]"}
> {"level":"info","ts":1568905711.1034026,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905711.103548,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905711.109706,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"error","ts":1568905711.2608829,"logger":"controller_wildflyserver","msg":"Failures during scaling down recovery processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Desired replica size":0,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod tx-server-0. Please, verify the pod log file. Error: Error to get response for command SCAN sending to 172.17.0.10:4712, error: EOF]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905711.2609518,"logger":"controller_wildflyserver","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.2609615,"logger":"controller_wildflyserver","msg":"Statefulset was not fully scaled to the desired replica size 0 while StatefulSet is to be at size 1. Some pods were not cleaned by recovery. Verify status of the WildflyServer tx-server","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.2795022,"logger":"controller_wildflyserver","msg":"Updating StatefulSet to be up to date with the WildFlyServer Spec","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.2795491,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905711.2796504,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905711.2937052,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"error","ts":1568905711.294249,"logger":"controller_wildflyserver","msg":"Failures during scaling down recovery processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Desired replica size":0,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod tx-server-0. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.10:4712, error: dial tcp 172.17.0.10:4712: connect: connection refused]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905711.294342,"logger":"controller_wildflyserver","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.294417,"logger":"controller_wildflyserver","msg":"Statefulset was not fully scaled to the desired replica size 0 while StatefulSet is to be at size 1. Some pods were not cleaned by recovery. Verify status of the WildflyServer tx-server","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"error","ts":1568905711.311673,"logger":"controller_wildflyserver","msg":"Failed to Update StatefulSet.","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server","error":"Operation cannot be fulfilled on statefulsets.apps \"tx-server\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"error","ts":1568905711.311745,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"wildflyserver-controller","request":"msimka-namespace/tx-server","error":"Operation cannot be fulfilled on statefulsets.apps \"tx-server\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905712.3137681,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905712.3139439,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905712.3253288,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"error","ts":1568905712.3255754,"logger":"controller_wildflyserver","msg":"Failures during scaling down recovery processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Desired replica size":0,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod tx-server-0. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.10:4712, error: dial tcp 172.17.0.10:4712: connect: connection refused]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905712.3256311,"logger":"controller_wildflyserver","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905712.3256419,"logger":"controller_wildflyserver","msg":"Statefulset was not fully scaled to the desired replica size 0 while StatefulSet is to be at size 1. Some pods were not cleaned by recovery. Verify status of the WildflyServer tx-server","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 9 months
[JBoss JIRA] (WFWIP-209) WildFlyServerStatus is not present in operator description
by Martin Choma (Jira)
[ https://issues.jboss.org/browse/WFWIP-209?page=com.atlassian.jira.plugin.... ]
Martin Choma closed WFWIP-209.
------------------------------
Resolution: Cannot Reproduce
Could not reproduce on latest image wildfly-operator:0.2.0-fe4dece.
> WildFlyServerStatus is not present in operator description
> -----------------------------------------------------------
>
> Key: WFWIP-209
> URL: https://issues.jboss.org/browse/WFWIP-209
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Petr Kremensky
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: operator
>
> WFWIP-163 is back.
> Operator image build from https://github.com/wildfly/wildfly-operator/commit/73722306343d8650b7d3f9...
> Install the operator using the run-openshift.sh script (yaml files from ^ are used)
> reproduce:
> {noformat}
> $ oc apply -f quickstart-cr.yaml
> $ oc get pods
> NAME READY STATUS RESTARTS AGE
> quickstart-0 1/1 Running 0 3m39s
> quickstart-1 1/1 Running 0 3m39s
> wildfly-operator-777b5ccd87-cf2vf 1/1 Running 0 4m22s
> $ oc describe wildflyserver quickstart
> Name: quickstart
> Namespace: pkremens-namespace
> Labels: <none>
> Annotations: kubectl.kubernetes.io/last-applied-configuration:
> {"apiVersion":"wildfly.org/v1alpha1","kind":"WildFlyServer","metadata":{"annotations":{},"name":"quickstart","namespace":"pkremens-namespa...
> API Version: wildfly.org/v1alpha1
> Kind: WildFlyServer
> Metadata:
> Creation Timestamp: 2019-09-26T10:10:43Z
> Generation: 1
> Resource Version: 337905
> Self Link: /apis/wildfly.org/v1alpha1/namespaces/pkremens-namespace/wildflyservers/quickstart
> UID: e63168ce-e045-11e9-9045-52fdfc072182
> Spec:
> Application Image: quay.io/wildfly-quickstarts/wildfly-operator-quickstart:17.0
> Size: 2
> Storage:
> Volume Claim Template:
> Spec:
> Resources:
> Requests:
> Storage: 3Gi
> Events: <none>
> {noformat}
> Status is missing.
> Operator log error:
> {noformat}
> {"level":"error","ts":1569492856.288481,"logger":"wildlfyserver_resources","msg":"Failed to update status of WildFlyServer","WildFlyServer.Namespace":"pkremens-namespace","WildFlyServer.Name":"quickstart","error":"the server could not find the requested resource (put wildflyservers.wildfly.org quickstart)","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"error","ts":1569492856.288641,"logger":"wildflyserver_controller","msg":"Failed to update WildFlyServer status.","Request.Namespace":"pkremens-namespace","Request.Name":"quickstart","error":"the server could not find the requested resource (put wildflyservers.wildfly.org quickstart)","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"error","ts":1569492856.288895,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"wildflyserver-controller","request":"pkremens-namespace/quickstart","error":"the server could not find the requested resource (put wildflyservers.wildfly.org quickstart)","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 9 months
[JBoss JIRA] (WFWIP-222) when client is scaled down with in-doubt transactions, tx participants on server are resolved with delay
by Martin Simka (Jira)
Martin Simka created WFWIP-222:
----------------------------------
Summary: when client is scaled down with in-doubt transactions, tx participants on server are resolved with delay
Key: WFWIP-222
URL: https://issues.jboss.org/browse/WFWIP-222
Project: WildFly WIP
Issue Type: Bug
Reporter: Martin Simka
Assignee: Ondrej Chaloupka
Attachments: tx-server.log
this follows up on WFWIP-206
While testing tx recovery in OpenShift I see that when client is scaled down with in-doubt transactions, tx participants on server are resolved with delay.
Scenario:
*ejb client* (app tx-client, pod tx-client-0):
* EJB business method
** lookup remote EJB
** enlist XA resource 1 to transaction
** enlist XA resource 2 to transaction
** call remote EJB
*ejb server* (app tx-server, pod tx-server-0):
* EJB business method
** enlist XA resource 1 to transaction
** enlist XA resource 2 to transaction
*testTxStatelessClientSecondPrepareJvmHalt*
JVM on client crashes in PREPARE phase of second XA resource on client. Then openshift restarts pod, but pod is immediately scaled down. Transactions on client pod are rollbacked during scale down, but on server they are rollbacked some time later. I'm not sure if it is periodic recovery or tx timeout.
server log with {{com.arjuna}} trace attached.
Feel free to reject if it is expected behavior.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 9 months
[JBoss JIRA] (WFLY-12624) Return hostname instead of IP address when generating default client mapping
by Richard Achmatowicz (Jira)
[ https://issues.jboss.org/browse/WFLY-12624?page=com.atlassian.jira.plugin... ]
Richard Achmatowicz commented on WFLY-12624:
--------------------------------------------
At present, the default client mapping returns 127.0.0.1/::1 rather than the hostname "localhost". The proposal here is to change this to return the hostname instead, avoiding the issue for the default case. This doesn't prevent a user from configuring IP addresses in user-specified client mappings.
> Return hostname instead of IP address when generating default client mapping
> -----------------------------------------------------------------------------
>
> Key: WFLY-12624
> URL: https://issues.jboss.org/browse/WFLY-12624
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 18.0.0.Beta1
> Environment: An EJB client application interacting with a cluster of Wildfly server nodes
> Reporter: Richard Achmatowicz
> Assignee: Richard Achmatowicz
> Priority: Major
>
> When an EJB client application interacts with a clustered Wildfly deployment, it receives topology updates from cluster nodes describing the membership of the cluster.
> For each node in the cluster, a set of one or more client mappings is provided to indicate how the client may connect to the node, if it hasn't already. If the node is connected to a single network, there will be one client mapping; if the node is multi-homed and connected to two networks, there will be two client mappings, etc. Client mappings specify three things: a CIDR representation of the network the client may be on, a destination hostname or IP address and a destination port.
> Client mappings may be generated by default (if none are provided in the server profile) or may be specified by the user via client mappings defined in the socket binding of the Remoting connector. For example:
> {noformat}
> <socket-binding name="remoting" port="1099">
> <client-mapping source-network="135.121.1.29/16" destination-address="135.121.1.29" destination-port="1099"/>
> </socketbinding>
> {noformat}
> When the client mapping information is received by the EJB client application, it is added to the discovered node registry (DNR) in the Discovery component of the EJB client. The DNR represents all known information about nodes with which the client can interact which was received from nodes in one or more clusters.
> When an invocation is attempted, the Discovery component uses this information to generate a set of ServiceURLs which represent candidate targets (i.e. servers containing the deployment and module the client is invoking on) for the invocation. The Discovery component uses "an algorithm" to take the information in the DNR (and other places) and convert that information to a corresponding set of ServiceURLs representing available targets. The Discovery component will then select one such ServiceURL and return this as the target for the invocation.
> Discovery obtains node information used in the algorithm from various sources: client mappings associated with cluster nodes, as described above, as well as Remoting endpoints associated with established connections to nodes. These pieces of information describe at a minimum a host and a port.
> The problem is that "the algorithm" used in Discovery to compute the set of ServiceURLs treats hostnames and IP addresses as simple strings. So "localhost" and "127.0.0.1" are treated as different hosts, even though they refer to the same host. If a mix of hostnames and IP addresses for the same node is received, this results in an incomplete/incorrect set of ServiceURLs being generated which in turn leads to incorrect Discovery failures.
>
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 9 months
[JBoss JIRA] (WFWIP-187) Changes to PVC are not reflected in Operator
by Martin Choma (Jira)
[ https://issues.jboss.org/browse/WFWIP-187?page=com.atlassian.jira.plugin.... ]
Martin Choma commented on WFWIP-187:
------------------------------------
It is really ignored. I am trying on OpenShift with wildfly-operator. If I add `sizeLimit: 2Mi` directly to StatefulSet it is throw away on Save. This behaves the same if I add any unknow parameter e.g. `a: b`
I will create OpenShift issue for that to be explained.
> Changes to PVC are not reflected in Operator
> --------------------------------------------
>
> Key: WFWIP-187
> URL: https://issues.jboss.org/browse/WFWIP-187
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Martin Choma
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: operator
>
> Any chnages (adding, removing or updating) made to PVC after WildFlyServer CR was created are not reflected in underlying PVC kubernetes object.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 9 months
[JBoss JIRA] (WFLY-12624) Return hostname instead of IP address when generating default client mapping
by Richard Achmatowicz (Jira)
[ https://issues.jboss.org/browse/WFLY-12624?page=com.atlassian.jira.plugin... ]
Richard Achmatowicz commented on WFLY-12624:
--------------------------------------------
There are two parts to this problem: the data input to the algorithm, and the algorithm itself.
The algorithm assumes that different strings (e..g "127.0.0.1" and "localhost") represent different hosts.
However, this assumption is not being met because:
* data from Remoting connections is always in hostname form
* data from client mappings can be either in hostname form or IP address form
So, we have this problem.
Fixing the algorithm to work under the assumption that two distinct strings may represent the same host, is possible. However, doing DNS reverse lookups to convert between IP addresses and hostnames is unwanted. It may be possible to work around this, based on the way information is stored in the DNR.
> Return hostname instead of IP address when generating default client mapping
> -----------------------------------------------------------------------------
>
> Key: WFLY-12624
> URL: https://issues.jboss.org/browse/WFLY-12624
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 18.0.0.Beta1
> Environment: An EJB client application interacting with a cluster of Wildfly server nodes
> Reporter: Richard Achmatowicz
> Assignee: Richard Achmatowicz
> Priority: Major
>
> When an EJB client application interacts with a clustered Wildfly deployment, it receives topology updates from cluster nodes describing the membership of the cluster.
> For each node in the cluster, a set of one or more client mappings is provided to indicate how the client may connect to the node, if it hasn't already. If the node is connected to a single network, there will be one client mapping; if the node is multi-homed and connected to two networks, there will be two client mappings, etc. Client mappings specify three things: a CIDR representation of the network the client may be on, a destination hostname or IP address and a destination port.
> Client mappings may be generated by default (if none are provided in the server profile) or may be specified by the user via client mappings defined in the socket binding of the Remoting connector. For example:
> {noformat}
> <socket-binding name="remoting" port="1099">
> <client-mapping source-network="135.121.1.29/16" destination-address="135.121.1.29" destination-port="1099"/>
> </socketbinding>
> {noformat}
> When the client mapping information is received by the EJB client application, it is added to the discovered node registry (DNR) in the Discovery component of the EJB client. The DNR represents all known information about nodes with which the client can interact which was received from nodes in one or more clusters.
> When an invocation is attempted, the Discovery component uses this information to generate a set of ServiceURLs which represent candidate targets (i.e. servers containing the deployment and module the client is invoking on) for the invocation. The Discovery component uses "an algorithm" to take the information in the DNR (and other places) and convert that information to a corresponding set of ServiceURLs representing available targets. The Discovery component will then select one such ServiceURL and return this as the target for the invocation.
> Discovery obtains node information used in the algorithm from various sources: client mappings associated with cluster nodes, as described above, as well as Remoting endpoints associated with established connections to nodes. These pieces of information describe at a minimum a host and a port.
> The problem is that "the algorithm" used in Discovery to compute the set of ServiceURLs treats hostnames and IP addresses as simple strings. So "localhost" and "127.0.0.1" are treated as different hosts, even though they refer to the same host. If a mix of hostnames and IP addresses for the same node is received, this results in an incomplete/incorrect set of ServiceURLs being generated which in turn leads to incorrect Discovery failures.
>
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 9 months
[JBoss JIRA] (WFWIP-221) jps can't see server process in Openshift
by Jan Blizňák (Jira)
[ https://issues.jboss.org/browse/WFWIP-221?page=com.atlassian.jira.plugin.... ]
Jan Blizňák updated WFWIP-221:
------------------------------
Description:
This is somewhat curious issue.
When pod of app image is deployed (server is started) and switch to terminal in Openshift GUI, doing `jps -l` does nothing although I can see in logs that server is up and I can find java process in /proc/*/cmdline (image has no `ps` utility).
This happens only in Openshift environment and it is a regression to what was working with latest published CD image.
Funnily, when I try to run the image in docker and start the server by /opt/eap/bin/openshift-launch.sh, then jps can see the java process.
{code:java}
# this works fine
docker run -it docker-registry.upshift.redhat.com/kwills/eap-cd-openshift-rhel8:18.0-EAP... sh
sh-4.4$ /opt/eap/bin/openshift-launch.sh &
sh-4.4$ jps -l
{code}
was:
This is somewhat curious issue.
When pod of app image is deployed (server is started) and switch to terminal in Openshift GUI, doing `jps -l` does nothing although I can see in logs that server is up and can find java process in /proc/*/cmdline (image has no `ps` utility).
This happens only in Openshift environment and is a regression to what was working with latest published CD image.
Funnily, when I try to run the image in docker and start the server by /opt/eap/bin/openshift-launch.sh, then jps can see the java process.
> jps can't see server process in Openshift
> -----------------------------------------
>
> Key: WFWIP-221
> URL: https://issues.jboss.org/browse/WFWIP-221
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Jan Blizňák
> Assignee: Jean Francois Denise
> Priority: Critical
>
> This is somewhat curious issue.
> When pod of app image is deployed (server is started) and switch to terminal in Openshift GUI, doing `jps -l` does nothing although I can see in logs that server is up and I can find java process in /proc/*/cmdline (image has no `ps` utility).
> This happens only in Openshift environment and it is a regression to what was working with latest published CD image.
> Funnily, when I try to run the image in docker and start the server by /opt/eap/bin/openshift-launch.sh, then jps can see the java process.
> {code:java}
> # this works fine
> docker run -it docker-registry.upshift.redhat.com/kwills/eap-cd-openshift-rhel8:18.0-EAP... sh
> sh-4.4$ /opt/eap/bin/openshift-launch.sh &
> sh-4.4$ jps -l
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 9 months
[JBoss JIRA] (WFWIP-221) jps can't see server process in Openshift
by Jan Blizňák (Jira)
[ https://issues.jboss.org/browse/WFWIP-221?page=com.atlassian.jira.plugin.... ]
Jan Blizňák updated WFWIP-221:
------------------------------
Steps to Reproduce:
Either in gui switch to Terminal tab and type `jps` or use remote shell
{code:java}
oc login
oc rsh <pod name> jps -l
{code}
> jps can't see server process in Openshift
> -----------------------------------------
>
> Key: WFWIP-221
> URL: https://issues.jboss.org/browse/WFWIP-221
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Jan Blizňák
> Assignee: Jean Francois Denise
> Priority: Critical
>
> This is somewhat curious issue.
> When pod of app image is deployed (server is started) and switch to terminal in Openshift GUI, doing `jps -l` does nothing although I can see in logs that server is up and can find java process in /proc/*/cmdline (image has no `ps` utility).
> This happens only in Openshift environment and is a regression to what was working with latest published CD image.
> Funnily, when I try to run the image in docker and start the server by /opt/eap/bin/openshift-launch.sh, then jps can see the java process.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 9 months