[JBoss JIRA] (JGRP-2382) JGroups version 4.0.13.Final.jar is causing memory leaks
by Rashmi Acharya (Jira)
[ https://issues.jboss.org/browse/JGRP-2382?page=com.atlassian.jira.plugin.... ]
Rashmi Acharya commented on JGRP-2382:
--------------------------------------
Thanks for the confirmation..Let me know the feasible duration , so I can schedule a call. I work from India .
-Regards
Rashmi
> JGroups version 4.0.13.Final.jar is causing memory leaks
> --------------------------------------------------------
>
> Key: JGRP-2382
> URL: https://issues.jboss.org/browse/JGRP-2382
> Project: JGroups
> Issue Type: Feature Request
> Affects Versions: 4.0.13
> Environment: AIX machine 7.1 with JDK 1.8
> Reporter: Rashmi Acharya
> Assignee: Bela Ban
> Priority: Major
> Attachments: dumps_TEST_node1_20190918_after_3_hours.zip, dumps_TEST_node1_20190918_right_after_restart.zip, dumps_TEST_node2_20190918_after_3_hours.zip, dumps_TEST_node2_20190918_right_after_restart.zip
>
>
> We are observing a constant memory growth and leak with JGroup version 4.0.13 ..
> One of our customer is having two node cluster environment and in one node we are observing org.Group.Messages which contain org.groups.Header and org.groups.Stack.ipAddress objects.. these are not getting cleared from memory..
> We dont see any exception related to Jgroups from logs and but it is causing a gradual emory growth and OOM.
> Here is the Jgroups cluster configuration we have:
> dynamic.cluster.property_string
> <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
> <TCP bind_addr="&HOST_ADDR;" bind_port="&MULTICAST_NODE_PORT2;"/>
> <TCPPING async_discovery="true" initial_hosts="&CLUSTER_INITIAL_HOSTS;" port_range="0" send_cache_on_join="true"/>
> <MERGE3 min_interval="3000" max_interval="5000" />
> <FD_ALL timeout="20000" interval="15000"/>
> <FD_SOCK/>
> <FD timeout="5000" max_tries="48" />
> <VERIFY_SUSPECT timeout="1500"/>
> <BARRIER/>
> <pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true"/>
> <UNICAST3/>
> <pbcast.STABLE desired_avg_gossip="20000" max_bytes="0" stability_delay="1000"/>
> <pbcast.GMS print_local_addr="true" join_timeout="15000" />
> </config>
> =================================
> dynamic.cluster.distribution_property_string
> <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
> <TCP bind_port="&MULTICAST_NODE_PORT1;" />
> <TCPPING async_discovery="true" initial_hosts="&CLUSTER_INITIAL_HOSTS;" port_range="0" send_cache_on_join="true"/>
> <MERGE3 min_interval="3000" max_interval="5000"/>
> <FD_SOCK/>
> <FD timeout="5000" max_tries="48"/>
> <VERIFY_SUSPECT timeout="1500"/>
> <BARRIER/>
> <pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true"/>
> <UNICAST3/>
> <pbcast.STABLE desired_avg_gossip="20000" max_bytes="0" stability_delay="1000" />
> <pbcast.GMS print_local_addr="true" join_timeout="5000"/>
> </config>
> ================================
> dynamic.cluster.lock.protocolStack
> <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
> <TCP bind_addr="&HOST_ADDR;" bind_port="&MULTICAST_NODE_PORT3;"/>
> <TCPPING async_discovery="true" initial_hosts="&CLUSTER_INITIAL_HOSTS;" port_range="0" send_cache_on_join="true"/>
> <MERGE3 min_interval="3000" max_interval="5000"/>
> <FD_ALL timeout="20000" interval="5000"/>
> <FD timeout="5000" max_tries="48"/>
> <VERIFY_SUSPECT timeout="1500"/>
> <BARRIER/>
> <pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true"/>
> <UNICAST3 /> <pbcast.STABLE desired_avg_gossip="20000" />
> <pbcast.GMS print_local_addr="true" join_timeout="5000"/>
> <FRAG2 frag_size="8096"/>
> <CENTRAL_LOCK2/>
> </config>
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (JGRP-2382) JGroups version 4.0.13.Final.jar is causing memory leaks
by Rashmi Acharya (Jira)
[ https://issues.jboss.org/browse/JGRP-2382?page=com.atlassian.jira.plugin.... ]
Rashmi Acharya edited comment on JGRP-2382 at 9/20/19 4:29 AM:
---------------------------------------------------------------
Is it possible to get into a call with us to help us set/tune these parameters ?
We are part of IBM B2B Sterling Integrator team and we have lot of usecases around Jgroups and It is better to guide what all we need to cover when we upgrade,. Also help us on solving customer Usecase.
was (Author: rashmia5):
Is it possible to get into a call with US to help us set/tune these parameters ?
We are part of IBM B2B Sterling Integrator team and we have lot of usecases around Jgroups and It is better to guide what all we need to cover when we upgrade,. Also help us on solving customer Usecase.
> JGroups version 4.0.13.Final.jar is causing memory leaks
> --------------------------------------------------------
>
> Key: JGRP-2382
> URL: https://issues.jboss.org/browse/JGRP-2382
> Project: JGroups
> Issue Type: Feature Request
> Affects Versions: 4.0.13
> Environment: AIX machine 7.1 with JDK 1.8
> Reporter: Rashmi Acharya
> Assignee: Bela Ban
> Priority: Major
> Attachments: dumps_TEST_node1_20190918_after_3_hours.zip, dumps_TEST_node1_20190918_right_after_restart.zip, dumps_TEST_node2_20190918_after_3_hours.zip, dumps_TEST_node2_20190918_right_after_restart.zip
>
>
> We are observing a constant memory growth and leak with JGroup version 4.0.13 ..
> One of our customer is having two node cluster environment and in one node we are observing org.Group.Messages which contain org.groups.Header and org.groups.Stack.ipAddress objects.. these are not getting cleared from memory..
> We dont see any exception related to Jgroups from logs and but it is causing a gradual emory growth and OOM.
> Here is the Jgroups cluster configuration we have:
> dynamic.cluster.property_string
> <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
> <TCP bind_addr="&HOST_ADDR;" bind_port="&MULTICAST_NODE_PORT2;"/>
> <TCPPING async_discovery="true" initial_hosts="&CLUSTER_INITIAL_HOSTS;" port_range="0" send_cache_on_join="true"/>
> <MERGE3 min_interval="3000" max_interval="5000" />
> <FD_ALL timeout="20000" interval="15000"/>
> <FD_SOCK/>
> <FD timeout="5000" max_tries="48" />
> <VERIFY_SUSPECT timeout="1500"/>
> <BARRIER/>
> <pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true"/>
> <UNICAST3/>
> <pbcast.STABLE desired_avg_gossip="20000" max_bytes="0" stability_delay="1000"/>
> <pbcast.GMS print_local_addr="true" join_timeout="15000" />
> </config>
> =================================
> dynamic.cluster.distribution_property_string
> <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
> <TCP bind_port="&MULTICAST_NODE_PORT1;" />
> <TCPPING async_discovery="true" initial_hosts="&CLUSTER_INITIAL_HOSTS;" port_range="0" send_cache_on_join="true"/>
> <MERGE3 min_interval="3000" max_interval="5000"/>
> <FD_SOCK/>
> <FD timeout="5000" max_tries="48"/>
> <VERIFY_SUSPECT timeout="1500"/>
> <BARRIER/>
> <pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true"/>
> <UNICAST3/>
> <pbcast.STABLE desired_avg_gossip="20000" max_bytes="0" stability_delay="1000" />
> <pbcast.GMS print_local_addr="true" join_timeout="5000"/>
> </config>
> ================================
> dynamic.cluster.lock.protocolStack
> <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
> <TCP bind_addr="&HOST_ADDR;" bind_port="&MULTICAST_NODE_PORT3;"/>
> <TCPPING async_discovery="true" initial_hosts="&CLUSTER_INITIAL_HOSTS;" port_range="0" send_cache_on_join="true"/>
> <MERGE3 min_interval="3000" max_interval="5000"/>
> <FD_ALL timeout="20000" interval="5000"/>
> <FD timeout="5000" max_tries="48"/>
> <VERIFY_SUSPECT timeout="1500"/>
> <BARRIER/>
> <pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true"/>
> <UNICAST3 /> <pbcast.STABLE desired_avg_gossip="20000" />
> <pbcast.GMS print_local_addr="true" join_timeout="5000"/>
> <FRAG2 frag_size="8096"/>
> <CENTRAL_LOCK2/>
> </config>
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (WFWIP-165) You dont need to be cluster-admin to be able to work with operator on OpenShift
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-165?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil updated WFWIP-165:
------------------------------
Labels: operator (was: )
> You dont need to be cluster-admin to be able to work with operator on OpenShift
> -------------------------------------------------------------------------------
>
> Key: WFWIP-165
> URL: https://issues.jboss.org/browse/WFWIP-165
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Martin Choma
> Assignee: Jeff Mesnil
> Priority: Major
> Labels: operator
>
> Instructions how to install operator on OpenShift [1] are shown under cluster-admin role.
> As role, role-binding and crd are cluster scoped resources I would expect that only cluster-admin can specify that.
> However I wouldt expect to have cluster-admin role to be able to deploy application using operator with
> oc apply -f deploy/crds/quickstart-cr.yaml
> To be able to run this as developer user have to be granted (by cluster-admin) these permission:
> {code}
> oc create role wildfly-server --verb=* --resource=wildflyservers.wildfly.org -n <namespace>
> oc adm policy add-role-to-user wildfly-server developer --role-namespace=<namespace> -n <namespace>
> {code}
> Please make this clear in install procedure.
> [1] https://github.com/wildfly/wildfly-operator#OpenShift
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (WFWIP-190) cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-190?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil updated WFWIP-190:
------------------------------
Labels: operator (was: )
> cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on
> ---------------------------------------------------------------------------------------------------
>
> Key: WFWIP-190
> URL: https://issues.jboss.org/browse/WFWIP-190
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Martin Choma
> Assignee: Jeff Mesnil
> Priority: Major
> Labels: operator
>
> This error message occurs always with operator start.
> {code}{"level":"info","ts":1568012414.9460623,"logger":"cmd","msg":"failed to create or get service for metrics: services \"wildfly-operator\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>"}{code}
> Is this error message safe?
> How can I set debug level log for operator?
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (WFWIP-204) Status is not set when envFrom: [] is configured
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-204?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil updated WFWIP-204:
------------------------------
Labels: operator (was: )
> Status is not set when envFrom: [] is configured
> ------------------------------------------------
>
> Key: WFWIP-204
> URL: https://issues.jboss.org/browse/WFWIP-204
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Martin Choma
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: operator
>
> # Create {{WildFlyServer}} resource with {{envFrom: []}}
> {code:yaml|title=empty-envfrom-reproducer.yaml}
> apiVersion: wildfly.org/v1alpha1
> kind: WildFlyServer
> metadata:
> name: empty-envfrom-reproducer
> spec:
> applicationImage: 'brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-eap-7-tech-preview/eap-cd-openshift-rhel8:17.0-4'
> size: 1
> envFrom: []
> {code}
> {{oc create -f empty-envfrom-reproducer.yaml}}
> # Pods are started properly in {{Ready}} state. But {{empty-envfrom-reproducer}} is not updated with status stanza.
> # After manually deleting {{envFrom: []}} from resource, status is created and filled properly
> {code:yaml}
> status:
> hosts:
> - >-
> empty-envfrom-reproducer-mchoma.apps.multinode-nfs-mchoma-001.dynamic.xpaas
> pods:
> - name: empty-envfrom-reproducer-0
> podIP: 10.130.0.208
> {code}
> # Operator pod log
> {code:json|operator-pod.log}
> {"level":"info","ts":1568705927.510753,"logger":"cmd","msg":"Go Version: go1.12.7"}
> {"level":"info","ts":1568705927.5108342,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
> {"level":"info","ts":1568705927.5108497,"logger":"cmd","msg":"Version of operator-sdk: v0.8.1"}
> {"level":"info","ts":1568705927.5111423,"logger":"leader","msg":"Trying to become the leader."}
> {"level":"info","ts":1568705927.594317,"logger":"leader","msg":"No pre-existing lock was found."}
> {"level":"info","ts":1568705927.5975504,"logger":"leader","msg":"Became the leader."}
> {"level":"info","ts":1568705927.6472175,"logger":"cmd","msg":"Registering Components."}
> {"level":"info","ts":1568705927.6474228,"logger":"utils","msg":"detecting platform version..."}
> {"level":"info","ts":1568705927.6507683,"logger":"utils","msg":"route.openshift.io found in apis, platform is OpenShift"}
> {"level":"info","ts":1568705927.6508126,"logger":"utils","msg":"PlatformInfo [Name: OpenShift, K8SVersion: 1.11+, OS: linux/amd64]"}
> {"level":"info","ts":1568705927.6509123,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"wildflyserver-controller","source":"kind source: /, Kind="}
> {"level":"info","ts":1568705927.651141,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"wildflyserver-controller","source":"kind source: /, Kind="}
> {"level":"info","ts":1568705927.6512737,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"wildflyserver-controller","source":"kind source: /, Kind="}
> {"level":"info","ts":1568705927.6513739,"logger":"utils","msg":"detecting platform version..."}
> {"level":"info","ts":1568705927.653909,"logger":"utils","msg":"route.openshift.io found in apis, platform is OpenShift"}
> {"level":"info","ts":1568705927.653929,"logger":"utils","msg":"PlatformInfo [Name: OpenShift, K8SVersion: 1.11+, OS: linux/amd64]"}
> {"level":"info","ts":1568705927.65394,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"wildflyserver-controller","source":"kind source: /, Kind="}
> {"level":"info","ts":1568705927.7098546,"logger":"cmd","msg":"failed to create or get service for metrics: services \"wildfly-operator\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: no RBAC policy matched, <nil>"}
> {"level":"info","ts":1568705927.7099428,"logger":"cmd","msg":"Starting the Cmd."}
> {"level":"info","ts":1568705927.8103025,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"wildflyserver-controller"}
> {"level":"info","ts":1568705927.9105484,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"wildflyserver-controller","worker count":1}
> {"level":"info","ts":1568705999.590361,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568705999.590594,"logger":"controller_wildflyserver","msg":"Creating a new StatefulSet.","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568705999.6067576,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568705999.6069276,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568705999.6118605,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568705999.6120145,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568705999.6286259,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568705999.6287735,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568705999.636309,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568705999.6363795,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568705999.7196531,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568705999.7197673,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568705999.8841014,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568705999.88426,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706000.2099004,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706000.210038,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706000.8545172,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706000.8546531,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706002.1387296,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706002.138901,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706002.885167,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706002.8852742,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706004.7032814,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706004.7033875,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706014.9472132,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706014.947392,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706035.435317,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706035.4354827,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706076.402556,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706076.4027927,"logger":"controller_wildflyserver","msg":"Updating envFrom","StatefulSet.Namespace":"mchoma","StatefulSet.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706088.455402,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706088.4555542,"logger":"controller_wildflyserver","msg":"Creating a new LoadBalancer.","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer","LoadBalancer.Namespace":"mchoma","LoadBalancer.Name":"empty-envfrom-reproducer-loadbalancer"}
> {"level":"info","ts":1568706088.4717329,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706088.4718812,"logger":"controller_wildflyserver","msg":"Creating a new Route.","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer","Route.Namespace":"mchoma","Route.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706088.4799702,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706088.5806746,"logger":"controller_wildflyserver","msg":"Updating hosts","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer","WildFlyServer":{"kind":"WildFlyServer","apiVersion":"wildfly.org/v1alpha1","metadata":{"name":"empty-envfrom-reproducer","namespace":"mchoma","selfLink":"/apis/wildfly.org/v1alpha1/namespaces/mchoma/wildflyservers/empty-envfrom-reproducer","uid":"46bcc345-d91e-11e9-b23d-fa163e9a0fb6","resourceVersion":"7076790","generation":2,"creationTimestamp":"2019-09-17T07:39:27Z"},"spec":{"applicationImage":"brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-eap-7-tech-preview/eap-cd-openshift-rhel8:17.0-4","size":1},"status":{}}}
> {"level":"info","ts":1568706088.5878873,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {"level":"info","ts":1568706088.5880494,"logger":"controller_wildflyserver","msg":"Updating hosts","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer","WildFlyServer":{"kind":"WildFlyServer","apiVersion":"wildfly.org/v1alpha1","metadata":{"name":"empty-envfrom-reproducer","namespace":"mchoma","selfLink":"/apis/wildfly.org/v1alpha1/namespaces/mchoma/wildflyservers/empty-envfrom-reproducer","uid":"46bcc345-d91e-11e9-b23d-fa163e9a0fb6","resourceVersion":"7076797","generation":2,"creationTimestamp":"2019-09-17T07:39:27Z"},"spec":{"applicationImage":"brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-eap-7-tech-preview/eap-cd-openshift-rhel8:17.0-4","size":1},"status":{"pods":[{"name":"empty-envfrom-reproducer-0","podIP":"10.130.0.208"}],"hosts":["empty-envfrom-reproducer-mchoma.apps.multinode-nfs-mchoma-001.dynamic.xpaas"]}}}
> {"level":"info","ts":1568706088.593367,"logger":"controller_wildflyserver","msg":"Reconciling WildFlyServer","Request.Namespace":"mchoma","Request.Name":"empty-envfrom-reproducer"}
> {code}
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months
[JBoss JIRA] (WFWIP-203) Transaction recovery may hit a wrong server when remote side works with multiple pods
by Jeff Mesnil (Jira)
[ https://issues.jboss.org/browse/WFWIP-203?page=com.atlassian.jira.plugin.... ]
Jeff Mesnil updated WFWIP-203:
------------------------------
Labels: operator (was: )
> Transaction recovery may hit a wrong server when remote side works with multiple pods
> -------------------------------------------------------------------------------------
>
> Key: WFWIP-203
> URL: https://issues.jboss.org/browse/WFWIP-203
> Project: WildFly WIP
> Issue Type: Bug
> Components: OpenShift
> Reporter: Ondrej Chaloupka
> Assignee: Ondrej Chaloupka
> Priority: Blocker
> Labels: operator
> Attachments: tx-client-0.log, tx-server-0.log, tx-server-1.log
>
>
> When server to server calls ejb remote calls where transaction context is propagated then ejb call can be routed to a one pod where the recovery call may directed to a different pod.
> Such situation causes a consistency issue.
> Let's say the scenario: the first server (let's call it `tx-client`) makes remote ejb call to remote server which is on of the servers joint in cluster named `tx-server-0` and `tx-server-1`. The `tx-client` calls the `tx-server-1`. The processing continues up to the start of the 2PC and the `tx-server-1` crashes (or host goes down, network issue happens...).
> `tx-client` understands that the process was not succesful and ask recovery manager to retry and finish.
> The recovery manager starts to call the remote server based on data saved in the object store of `tx-client`.
> But unfortunately the recovery remote call goes *not* to the `tx-server-1` but to `tx-server-0`. The `tx-client` gets error code `XAException.XAER_NOTA` (`-4`) and removes data from its object store (`/opt/eap/standalone/data/tx-object-store/`, `/opt/eap/standalone/data/ejb-xa-recovery`) and then never finishes in-doubt transactions at `tx-server-1`.
> It's in doubt if it's issue of OpenShift configuration or if it's a trouble of WFTC/ejb/remoting layer in WildFly.
> This is tested with WFLY Operator from 2019-09-26 `@90a2b3b`.
--
This message was sent by Atlassian Jira
(v7.13.5#713005)
6 years, 10 months