[JBoss JIRA] (WFWIP-206) scale down isn't successful when there are in-doubt transaction on pod
by Martin Simka (Jira)
[ https://issues.jboss.org/browse/WFWIP-206?page=com.atlassian.jira.plugin.... ]
Martin Simka updated WFWIP-206:
-------------------------------
Steps to Reproduce:
it can be reproduced using test:
{code}
git clone git@gitlab.mw.lab.eng.bos.redhat.com:msimka/openshift-eap-tests.git
cd openshift-eap-tests.git
git checkout EAP7-1192_scaledown
# create file test.properties
mvn clean test -P72 -Dtest=EjbTxnRemotingScaleDownTest -Dcheckstyle.skip -Dconsole-log-level=DEBUG
{code}
test.properties
{code}
xtf.openshift.url=https://master.all-in-one-msimka-002.dynamic.xpaas:8443
xtf.openshift.namespace=wip-namespace
xtf.bm.namespace=wip-builds-namespace
xtf.eap.72.image=docker-registry.engineering.redhat.com/ochaloup/wildfly1...
xtf.eap.72.properties.eap.imagestream.name=jboss-eap73-openshift
xtf.eap.72.version=7.2.0.GA
xtf.eap.properties.location=/opt/eap
xtf.eap.72.templates.repo=git://github.com/jboss-container-images/jboss-e...
xtf.eap.72.templates.branch=eap72,v7.3.0.GA
xtf.maven.proxy.url=http://maven.all-in-one-043.dynamic.xpaas/nexus/content/groups/public/
# this might be needed if oc login command fails, see test suite logs
#xtf.openshift.binary.path=/home/msimka/.minishift/cache/oc/v3.11.0/linux/oc
xtf.operator.image=docker-registry.engineering.redhat.com/jbossqe-eap/wil...
xtf.operator.resources.context=
# specify correct path oc binary
xtf.openshift.binary.path=<oc_binary_path>
{code}
was:
it can be reproduced using test:
{code}
git clone git@gitlab.mw.lab.eng.bos.redhat.com:msimka/openshift-eap-tests.git
cd openshift-eap-tests.git
git checkout EAP7-1192_scaledown
# create file test.properties
cd test-eap
mvn clean test -P72 -Dtest=EjbTxnRemotingScaleDownTest -Dcheckstyle.skip -Dconsole-log-level=DEBUG
{code}
test.properties
{code}
xtf.openshift.url=https://master.all-in-one-msimka-002.dynamic.xpaas:8443
xtf.openshift.namespace=wip-namespace
xtf.bm.namespace=wip-builds-namespace
xtf.eap.72.image=docker-registry.engineering.redhat.com/ochaloup/wildfly1...
xtf.eap.72.properties.eap.imagestream.name=jboss-eap73-openshift
xtf.eap.72.version=7.2.0.GA
xtf.eap.properties.location=/opt/eap
xtf.eap.72.templates.repo=git://github.com/jboss-container-images/jboss-e...
xtf.eap.72.templates.branch=eap72,v7.3.0.GA
xtf.maven.proxy.url=http://maven.all-in-one-043.dynamic.xpaas/nexus/content/groups/public/
# this might be needed if oc login command fails, see test suite logs
#xtf.openshift.binary.path=/home/msimka/.minishift/cache/oc/v3.11.0/linux/oc
xtf.operator.image=docker-registry.engineering.redhat.com/jbossqe-eap/wil...
xtf.operator.resources.context=
# specify correct path oc binary
xtf.openshift.binary.path=<oc_binary_path>
{code}
> scale down isn't successful when there are in-doubt transaction on pod
> ----------------------------------------------------------------------
>
> Key: WFWIP-206
> URL: https://issues.jboss.org/browse/WFWIP-206
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Environment: operator, built from [1984d98154f11a7473be065e4ae7f54b1812c9b0|https://github.com/wildfly/wildf...] :
> {noformat}
> docker-registry.engineering.redhat.com/jbossqe-eap/wildfly-operator:latest
> {noformat}
> EAP image:
> {noformat}
> docker-registry.engineering.redhat.com/ochaloup/wildfly18-snapshot:190909...
> {noformat}
> Reporter: Martin Simka
> Assignee: Ondrej Chaloupka
> Priority: Blocker
> Labels: operator
>
> While testing tx recovery in OpenShift I see that scale down of pod that has transaction in-doubt on it isn't successful
> Scenario:
> *ejb client* (app tx-client, pod tx-client-0):
> * EJB business method
> ** lookup remote EJB
> ** enlist XA resource 1 to transaction
> ** enlist XA resource 2 to transaction
> ** call remote EJB
> *ejb server* (app tx-server, pod tx-server-0):
> * EJB business method
> ** enlist XA resource 1 to transaction
> ** enlist XA resource 2 to transaction
> ejb server XA resource 2 fails with {{XAException(XAException.XAER_RMFAIL)}}
> Then the test calls scale down (size from 1 to 0) on tx-server pod. But scale down never completes.
> Log from operator:
> {noformat}
> {"level":"info","ts":1568905676.6303623,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905676.7313502,"logger":"controller_wildflyserver","msg":"Enabling recovery listener for processing scaledown at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905679.4325309,"logger":"controller_wildflyserver","msg":"Query to find the transaction recovery port to force scan at pod tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905686.7914035,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"info","ts":1568905702.0583296,"logger":"controller_wildflyserver","msg":"In-doubt transactions in object store","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","Message":"Recovery scan to be invoked as the transaction log storage is not empty for pod scaling down pod tx-server-0, transaction list: map[0:ffffac11000a:991c183:5d8399a1:13:map[age-in-seconds:23 id:0:ffffac11000a:991c183:5d8399a1:13 jmx-name:<nil> participants:map[java:/MockXAResource:map[eis-product-name:MockXAResource Test eis-product-version:0.1.Mock jmx-name:<nil> jndi-name:java:/MockXAResource status:PREPARED type:/StateManager/AbstractRecord/XAResourceRecord]] type:StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction/SubordinateAtomicAction/JCA]]"}
> {"level":"info","ts":1568905711.1034026,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905711.103548,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905711.109706,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"error","ts":1568905711.2608829,"logger":"controller_wildflyserver","msg":"Failures during scaling down recovery processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Desired replica size":0,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod tx-server-0. Please, verify the pod log file. Error: Error to get response for command SCAN sending to 172.17.0.10:4712, error: EOF]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905711.2609518,"logger":"controller_wildflyserver","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.2609615,"logger":"controller_wildflyserver","msg":"Statefulset was not fully scaled to the desired replica size 0 while StatefulSet is to be at size 1. Some pods were not cleaned by recovery. Verify status of the WildflyServer tx-server","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.2795022,"logger":"controller_wildflyserver","msg":"Updating StatefulSet to be up to date with the WildFlyServer Spec","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.2795491,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905711.2796504,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905711.2937052,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"error","ts":1568905711.294249,"logger":"controller_wildflyserver","msg":"Failures during scaling down recovery processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Desired replica size":0,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod tx-server-0. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.10:4712, error: dial tcp 172.17.0.10:4712: connect: connection refused]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905711.294342,"logger":"controller_wildflyserver","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905711.294417,"logger":"controller_wildflyserver","msg":"Statefulset was not fully scaled to the desired replica size 0 while StatefulSet is to be at size 1. Some pods were not cleaned by recovery. Verify status of the WildflyServer tx-server","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"error","ts":1568905711.311673,"logger":"controller_wildflyserver","msg":"Failed to Update StatefulSet.","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server","error":"Operation cannot be fulfilled on statefulsets.apps \"tx-server\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"error","ts":1568905711.311745,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"wildflyserver-controller","request":"msimka-namespace/tx-server","error":"Operation cannot be fulfilled on statefulsets.apps \"tx-server\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905712.3137681,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"msimka-namespace","Request.Name":"tx-server"}
> {"level":"info","ts":1568905712.3139439,"logger":"controller_wildflyserver","msg":"Transaction recovery scaledown processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod Name":"tx-server-0","IP Address":"172.17.0.10"}
> {"level":"info","ts":1568905712.3253288,"logger":"controller_wildflyserver","msg":"Executing recovery scan at tx-server-0","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Pod IP":"172.17.0.10","Recovery port":4712}
> {"level":"error","ts":1568905712.3255754,"logger":"controller_wildflyserver","msg":"Failures during scaling down recovery processing","Request.Namespace":"msimka-namespace","Request.Name":"tx-server","Desired replica size":0,"Number of pods to be removed":1,"error":"Found 1 errors:\n [[Failed to run transaction recovery scan for scaling down pod tx-server-0. Please, verify the pod log file. Error: Cannot process TCP connection to 172.17.0.10:4712, error: dial tcp 172.17.0.10:4712: connect: connection refused]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-l..."}
> {"level":"info","ts":1568905712.3256311,"logger":"controller_wildflyserver","msg":"Scaling down statefulset by verification if pods are clean by recovery","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {"level":"info","ts":1568905712.3256419,"logger":"controller_wildflyserver","msg":"Statefulset was not fully scaled to the desired replica size 0 while StatefulSet is to be at size 1. Some pods were not cleaned by recovery. Verify status of the WildflyServer tx-server","StatefulSet.Namespace":"msimka-namespace","StatefulSet.Name":"tx-server"}
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (WFLY-12580) Explicitly disable the inheritance of Galleon configurations from transitive feature pack
by Yeray Borges (Jira)
Yeray Borges created WFLY-12580:
-----------------------------------
Summary: Explicitly disable the inheritance of Galleon configurations from transitive feature pack
Key: WFLY-12580
URL: https://issues.jboss.org/browse/WFLY-12580
Project: WildFly
Issue Type: Task
Components: Build System
Affects Versions: 18.0.0.Beta1
Reporter: Yeray Borges
Assignee: Yeray Borges
Since galleon 4.0.1.Final, configurations from transitive feature pack dependencies are inherited by default. This is a bug in Galleon. As a temporal workaround in WildFly, we have to explicitly stop such inheritance until having the proper fix with a Galleon upgrade.
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (JGRP-2382) JGroups version 4.0.13.Final.jar is causing memory leaks
by Bela Ban (Jira)
[ https://issues.jboss.org/browse/JGRP-2382?page=com.atlassian.jira.plugin.... ]
Bela Ban commented on JGRP-2382:
--------------------------------
Not on this ticket. Use my email instead.
> JGroups version 4.0.13.Final.jar is causing memory leaks
> --------------------------------------------------------
>
> Key: JGRP-2382
> URL: https://issues.jboss.org/browse/JGRP-2382
> Project: JGroups
> Issue Type: Feature Request
> Affects Versions: 4.0.13
> Environment: AIX machine 7.1 with JDK 1.8
> Reporter: Rashmi Acharya
> Assignee: Bela Ban
> Priority: Major
> Attachments: dumps_TEST_node1_20190918_after_3_hours.zip, dumps_TEST_node1_20190918_right_after_restart.zip, dumps_TEST_node2_20190918_after_3_hours.zip, dumps_TEST_node2_20190918_right_after_restart.zip
>
>
> We are observing a constant memory growth and leak with JGroup version 4.0.13 ..
> One of our customer is having two node cluster environment and in one node we are observing org.Group.Messages which contain org.groups.Header and org.groups.Stack.ipAddress objects.. these are not getting cleared from memory..
> We dont see any exception related to Jgroups from logs and but it is causing a gradual emory growth and OOM.
> Here is the Jgroups cluster configuration we have:
> dynamic.cluster.property_string
> <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
> <TCP bind_addr="&HOST_ADDR;" bind_port="&MULTICAST_NODE_PORT2;"/>
> <TCPPING async_discovery="true" initial_hosts="&CLUSTER_INITIAL_HOSTS;" port_range="0" send_cache_on_join="true"/>
> <MERGE3 min_interval="3000" max_interval="5000" />
> <FD_ALL timeout="20000" interval="15000"/>
> <FD_SOCK/>
> <FD timeout="5000" max_tries="48" />
> <VERIFY_SUSPECT timeout="1500"/>
> <BARRIER/>
> <pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true"/>
> <UNICAST3/>
> <pbcast.STABLE desired_avg_gossip="20000" max_bytes="0" stability_delay="1000"/>
> <pbcast.GMS print_local_addr="true" join_timeout="15000" />
> </config>
> =================================
> dynamic.cluster.distribution_property_string
> <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
> <TCP bind_port="&MULTICAST_NODE_PORT1;" />
> <TCPPING async_discovery="true" initial_hosts="&CLUSTER_INITIAL_HOSTS;" port_range="0" send_cache_on_join="true"/>
> <MERGE3 min_interval="3000" max_interval="5000"/>
> <FD_SOCK/>
> <FD timeout="5000" max_tries="48"/>
> <VERIFY_SUSPECT timeout="1500"/>
> <BARRIER/>
> <pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true"/>
> <UNICAST3/>
> <pbcast.STABLE desired_avg_gossip="20000" max_bytes="0" stability_delay="1000" />
> <pbcast.GMS print_local_addr="true" join_timeout="5000"/>
> </config>
> ================================
> dynamic.cluster.lock.protocolStack
> <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
> <TCP bind_addr="&HOST_ADDR;" bind_port="&MULTICAST_NODE_PORT3;"/>
> <TCPPING async_discovery="true" initial_hosts="&CLUSTER_INITIAL_HOSTS;" port_range="0" send_cache_on_join="true"/>
> <MERGE3 min_interval="3000" max_interval="5000"/>
> <FD_ALL timeout="20000" interval="5000"/>
> <FD timeout="5000" max_tries="48"/>
> <VERIFY_SUSPECT timeout="1500"/>
> <BARRIER/>
> <pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true"/>
> <UNICAST3 /> <pbcast.STABLE desired_avg_gossip="20000" />
> <pbcast.GMS print_local_addr="true" join_timeout="5000"/>
> <FRAG2 frag_size="8096"/>
> <CENTRAL_LOCK2/>
> </config>
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months