[JBoss JIRA] (WFLY-10047) OOM caused by jgroups objects UNICAST3$SenderEntry#1
by Erich Duda (JIRA)
[ https://issues.jboss.org/browse/WFLY-10047?page=com.atlassian.jira.plugin... ]
Erich Duda commented on WFLY-10047:
-----------------------------------
Heap dump, logs from server, test, configuration, etc. you can find on this link - http://download.eng.brq.redhat.com/scratch/eduda/jira/WFLY-10047/WFLY-100...
> OOM caused by jgroups objects UNICAST3$SenderEntry#1
> ----------------------------------------------------
>
> Key: WFLY-10047
> URL: https://issues.jboss.org/browse/WFLY-10047
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 13.0.0.Beta1
> Reporter: Erich Duda
> Assignee: Paul Ferraro
> Priority: Blocker
> Attachments: heapdump.png
>
>
> JGroups objects UNICAST3$SenderEntry#1 caused OOM on Wildfly server during the boot. See attached picture. !heapdump.png|thumbnail!
> *User impact:* If users use JGroups for clustering, the server may get OOM what can cause undefined behavior.
> The *blocker* priority was set, because this is regression against previous versions of Wildfly and the OOM is serious error which prevents server to work properly.
> The issue was hit in following scenario.
> # start two servers (nodes) in cluster with one queue
> # producer starts to send messages to queue to node-1
> # node-2 is killed and restarted during sending messages <---- *Here the test failed, when the node-2 was started after that it had been killed.*
> # start consumer on node-2 which reads messages from queue
> # servers are stopped
> The Wildfly was built from following source code:
> repo: https://github.com/jmesnil/wildfly
> branch: WFLY-9407_upgrade_artemis_2.5.0
> commit SHA: 06c878a313d3cad323889d017e60fd5533204d1a
> JGroups version: 4.0.10.Final
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (WFLY-10047) OOM caused by jgroups objects UNICAST3$SenderEntry#1
by Erich Duda (JIRA)
Erich Duda created WFLY-10047:
---------------------------------
Summary: OOM caused by jgroups objects UNICAST3$SenderEntry#1
Key: WFLY-10047
URL: https://issues.jboss.org/browse/WFLY-10047
Project: WildFly
Issue Type: Bug
Components: Clustering
Affects Versions: 13.0.0.Beta1
Reporter: Erich Duda
Assignee: Paul Ferraro
Priority: Blocker
Attachments: heapdump.png
JGroups objects UNICAST3$SenderEntry#1 caused OOM on Wildfly server during the boot. See attached picture. !heapdump.png|thumbnail!
*User impact:* If users use JGroups for clustering, the server may get OOM what can cause undefined behavior.
The *blocker* priority was set, because this is regression against previous versions of Wildfly and the OOM is serious error which prevents server to work properly.
The issue was hit in following scenario.
# start two servers (nodes) in cluster with one queue
# producer starts to send messages to queue to node-1
# node-2 is killed and restarted during sending messages <---- *Here the test failed, when the node-2 was started after that it had been killed.*
# start consumer on node-2 which reads messages from queue
# servers are stopped
The Wildfly was built from following source code:
repo: https://github.com/jmesnil/wildfly
branch: WFLY-9407_upgrade_artemis_2.5.0
commit SHA: 06c878a313d3cad323889d017e60fd5533204d1a
JGroups version: 4.0.10.Final
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (WFLY-10046) Cannot create cluster with JGroups Gossip router
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-10046?page=com.atlassian.jira.plugin... ]
Radoslav Husar commented on WFLY-10046:
---------------------------------------
I would be surprised if this ever worked. Definitely not a valid configuration: "gossip_router_hosts" requires hosts to connect to and address 0.0.0.0 is a non-routable target.
> Cannot create cluster with JGroups Gossip router
> ------------------------------------------------
>
> Key: WFLY-10046
> URL: https://issues.jboss.org/browse/WFLY-10046
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 12.0.0.Final
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Attachments: standalone-full-ha-gosship1.xml, standalone-full-ha-gosship2.xml
>
>
> I've used config from EAP 7.1.0/WF11 where gossip router was configured in udp stack like:
> {code}
> <stack name="udp">
> <transport type="UDP" shared="false" socket-binding="jgroups-udp"/>
> <protocol type="PING"/>
> <protocol type="MERGE3"/>
> <protocol type="FD_SOCK"/>
> <protocol type="FD_ALL"/>
> <protocol type="VERIFY_SUSPECT"/>
> <protocol type="pbcast.NAKACK2"/>
> <protocol type="UNICAST3"/>
> <protocol type="pbcast.STABLE"/>
> <protocol type="pbcast.GMS"/>
> <protocol type="UFC"/>
> <protocol type="MFC"/>
> <protocol type="FRAG2"/>
> <protocol type="TUNNEL">
> <property name="gossip_router_hosts">
> 0.0.0.0[12001]
> </property>
> </protocol>
> </stack>
> {code}
> Gossip router was started on localhost:
> {{java -cp $JBOSS_HOME//bin/client/jboss-client.jar org.jgroups.stack.GossipRouter -port 12001 -bindaddress 0.0.0.0}}
> but cluster does not form up. The same configuration works in EAP 7.1.
> Attaching xml config files for both of the servers.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (WFLY-10046) Cannot create cluster with JGroups Gossip router
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-10046?page=com.atlassian.jira.plugin... ]
Radoslav Husar updated WFLY-10046:
----------------------------------
Description:
I've used config from EAP 7.1.0/WF11 where gossip router was configured in udp stack like:
{code}
<stack name="udp">
<transport type="UDP" shared="false" socket-binding="jgroups-udp"/>
<protocol type="PING"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
<protocol type="TUNNEL">
<property name="gossip_router_hosts">
0.0.0.0[12001]
</property>
</protocol>
</stack>
{code}
Gossip router was started on localhost:
{{java -cp $JBOSS_HOME//bin/client/jboss-client.jar org.jgroups.stack.GossipRouter -port 12001 -bindaddress 0.0.0.0}}
but cluster does not form up. The same configuration works in EAP 7.1.
Attaching xml config files for both of the servers.
was:
I've used config from EAP 7.1.0/WF11 where gosship router was configured in udp stack like:
{code}
<stack name="udp">
<transport type="UDP" shared="false" socket-binding="jgroups-udp"/>
<protocol type="PING"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
<protocol type="TUNNEL">
<property name="gossip_router_hosts">
0.0.0.0[12001]
</property>
</protocol>
</stack>
{code}
Gosshop router was started on localhost:
{{java -cp $JBOSS_HOME//bin/client/jboss-client.jar org.jgroups.stack.GossipRouter -port 12001 -bindaddress 0.0.0.0}}
but cluster does not form up. The same configuration works in EAP 7.1.
Attaching xml config files for both of the servers.
> Cannot create cluster with JGroups Gossip router
> ------------------------------------------------
>
> Key: WFLY-10046
> URL: https://issues.jboss.org/browse/WFLY-10046
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 12.0.0.Final
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Attachments: standalone-full-ha-gosship1.xml, standalone-full-ha-gosship2.xml
>
>
> I've used config from EAP 7.1.0/WF11 where gossip router was configured in udp stack like:
> {code}
> <stack name="udp">
> <transport type="UDP" shared="false" socket-binding="jgroups-udp"/>
> <protocol type="PING"/>
> <protocol type="MERGE3"/>
> <protocol type="FD_SOCK"/>
> <protocol type="FD_ALL"/>
> <protocol type="VERIFY_SUSPECT"/>
> <protocol type="pbcast.NAKACK2"/>
> <protocol type="UNICAST3"/>
> <protocol type="pbcast.STABLE"/>
> <protocol type="pbcast.GMS"/>
> <protocol type="UFC"/>
> <protocol type="MFC"/>
> <protocol type="FRAG2"/>
> <protocol type="TUNNEL">
> <property name="gossip_router_hosts">
> 0.0.0.0[12001]
> </property>
> </protocol>
> </stack>
> {code}
> Gossip router was started on localhost:
> {{java -cp $JBOSS_HOME//bin/client/jboss-client.jar org.jgroups.stack.GossipRouter -port 12001 -bindaddress 0.0.0.0}}
> but cluster does not form up. The same configuration works in EAP 7.1.
> Attaching xml config files for both of the servers.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (DROOLS-2395) [DMN Designer] Invocation parameter and table output are not paired correctly
by Michael Anstis (JIRA)
[ https://issues.jboss.org/browse/DROOLS-2395?page=com.atlassian.jira.plugi... ]
Michael Anstis commented on DROOLS-2395:
----------------------------------------
[~jomarko] Please note {{Context}} was equally affected.
A {{DecisionTable}} {{OutputClause}} column title is the name of the outer {{InformationItem}} containing the table. Likewise when the {{DecisionTable}} has more than one {{OutputClause}} columns the "top level" title is also the name of the outer {{InformationItem}} containing the table. The secondary column titles are not bound to anything in the outer container.
> [DMN Designer] Invocation parameter and table output are not paired correctly
> -----------------------------------------------------------------------------
>
> Key: DROOLS-2395
> URL: https://issues.jboss.org/browse/DROOLS-2395
> Project: Drools
> Issue Type: Bug
> Components: DMN Editor
> Affects Versions: 7.7.0.Final
> Reporter: Jozef Marko
> Assignee: Michael Anstis
> Attachments: Screenshot from 2018-03-16 12-51-03.png, Screenshot from 2018-03-16 12-51-14.png, Screenshot from 2018-03-16 12-51-22.png
>
>
> Invocation parameter is not updated if user updates output of the decision table output.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (WFLY-10046) Cannot create cluster with JGroups Gossip router
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-10046?page=com.atlassian.jira.plugin... ]
Radoslav Husar updated WFLY-10046:
----------------------------------
Summary: Cannot create cluster with JGroups Gossip router (was: Cannot create cluster with JGroups Gosship router)
> Cannot create cluster with JGroups Gossip router
> ------------------------------------------------
>
> Key: WFLY-10046
> URL: https://issues.jboss.org/browse/WFLY-10046
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 12.0.0.Final
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Attachments: standalone-full-ha-gosship1.xml, standalone-full-ha-gosship2.xml
>
>
> I've used config from EAP 7.1.0/WF11 where gosship router was configured in udp stack like:
> {code}
> <stack name="udp">
> <transport type="UDP" shared="false" socket-binding="jgroups-udp"/>
> <protocol type="PING"/>
> <protocol type="MERGE3"/>
> <protocol type="FD_SOCK"/>
> <protocol type="FD_ALL"/>
> <protocol type="VERIFY_SUSPECT"/>
> <protocol type="pbcast.NAKACK2"/>
> <protocol type="UNICAST3"/>
> <protocol type="pbcast.STABLE"/>
> <protocol type="pbcast.GMS"/>
> <protocol type="UFC"/>
> <protocol type="MFC"/>
> <protocol type="FRAG2"/>
> <protocol type="TUNNEL">
> <property name="gossip_router_hosts">
> 0.0.0.0[12001]
> </property>
> </protocol>
> </stack>
> {code}
> Gosshop router was started on localhost:
> {{java -cp $JBOSS_HOME//bin/client/jboss-client.jar org.jgroups.stack.GossipRouter -port 12001 -bindaddress 0.0.0.0}}
> but cluster does not form up. The same configuration works in EAP 7.1.
> Attaching xml config files for both of the servers.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (WFCORE-1427) Add a timeout param to reload op and use it for "graceful reload"
by Yeray Borges (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1427?page=com.atlassian.jira.plugi... ]
Yeray Borges updated WFCORE-1427:
---------------------------------
Issue Type: Feature Request (was: Enhancement)
> Add a timeout param to reload op and use it for "graceful reload"
> -----------------------------------------------------------------
>
> Key: WFCORE-1427
> URL: https://issues.jboss.org/browse/WFCORE-1427
> Project: WildFly Core
> Issue Type: Feature Request
> Components: CLI, Domain Management
> Reporter: Brian Stansberry
> Assignee: Yeray Borges
>
> So instead of
> {code}
> :suspend(20)
> :reload
> {code}
> It's just
> {code}
> :reload(20)
> {code}
> The high level 'reload' command in the CLI should take a --timeout param as well.
> If doing the graceful suspend as part of server side ":reload" handling proves problematic (I haven't looked into it at all before filing this) then a simpler alternative is to only go with the --timeout param on the CLI reload command, and have the CLI implement the graceful behavior internally by first calling :suspend and then :reload. Web console could do the same thing.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (WFLY-10046) Cannot create cluster with JGroups Gosship router
by Miroslav Novak (JIRA)
[ https://issues.jboss.org/browse/WFLY-10046?page=com.atlassian.jira.plugin... ]
Miroslav Novak updated WFLY-10046:
----------------------------------
Steps to Reproduce:
* Unzip WF12 to two different directories
* Download attached standalone xml files and copy instead of standalone-full-ha.xml
* Start gosship router by: {{java -cp $JBOSS_HOME//bin/client/jboss-client.jar org.jgroups.stack.GossipRouter -port 12001 -bindaddress 0.0.0.0}}
* Start first by: {{sh standalone.sh -c standalone-full-ha.xml}}
* Start second by: {{sh standalone.sh -c standalone-full-ha.xml -Djboss.socket.binding.port-offset=1000}}
There should appear message like: {code}
11:56:17,607 INFO [org.apache.activemq.artemis.core.server] (Thread-9 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@49be2674)) AMQ221027: Bridge ClusterConnectionBridge@6a0e45dc [name=sf.my-cluster.3c2415d7-2b61-11e8-81e5-f496342f6705, queue=QueueImpl[name=sf.my-cluster.3c2415d7-2b61-11e8-81e5-f496342f6705, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=3edb46eb-2b61-11e8-86e1-54e1ad305985]]@87f9619 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@6a0e45dc [name=sf.my-cluster.3c2415d7-2b61-11e8-81e5-f496342f6705, queue=QueueImpl[name=sf.my-cluster.3c2415d7-2b61-11e8-81e5-f496342f6705, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=3edb46eb-2b61-11e8-86e1-54e1ad305985]]@87f9619 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=9080&host=localhost], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1691717715[nodeUUID=3edb46eb-2b61-11e8-86e1-54e1ad305985, connector=TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=localhost, address=jms, server=ActiveMQServerImpl::serverUUID=3edb46eb-2b61-11e8-86e1-54e1ad305985])) [initialConnectors=[TransportConfiguration(name=connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=9080&host=localhost], discoveryGroupConfiguration=null]] is connected
{code}
which informs that Artemis cluster was formed but it does not.
was:
* Unzip WF12 to two different directories
* Download attached standalone xml files and copy instead of standalone-full-ha.xml
* Start gosship router by: {{java -cp $JBOSS_HOME//bin/client/jboss-client.jar org.jgroups.stack.GossipRouter -port 12001 -bindaddress 0.0.0.0}}
* Start first by: {{sh standalone.sh -c standalone-full-ha.xml}}
* Start second by: {{sh standalone.sh -c standalone-full-ha.xml -Djboss.socket.binding.port-offset=1000}}
> Cannot create cluster with JGroups Gosship router
> -------------------------------------------------
>
> Key: WFLY-10046
> URL: https://issues.jboss.org/browse/WFLY-10046
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 12.0.0.Final
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Attachments: standalone-full-ha-gosship1.xml, standalone-full-ha-gosship2.xml
>
>
> I've used config from EAP 7.1.0/WF11 where gosship router was configured in udp stack like:
> {code}
> <stack name="udp">
> <transport type="UDP" shared="false" socket-binding="jgroups-udp"/>
> <protocol type="PING"/>
> <protocol type="MERGE3"/>
> <protocol type="FD_SOCK"/>
> <protocol type="FD_ALL"/>
> <protocol type="VERIFY_SUSPECT"/>
> <protocol type="pbcast.NAKACK2"/>
> <protocol type="UNICAST3"/>
> <protocol type="pbcast.STABLE"/>
> <protocol type="pbcast.GMS"/>
> <protocol type="UFC"/>
> <protocol type="MFC"/>
> <protocol type="FRAG2"/>
> <protocol type="TUNNEL">
> <property name="gossip_router_hosts">
> 0.0.0.0[12001]
> </property>
> </protocol>
> </stack>
> {code}
> Gosshop router was started on localhost:
> {{java -cp $JBOSS_HOME//bin/client/jboss-client.jar org.jgroups.stack.GossipRouter -port 12001 -bindaddress 0.0.0.0}}
> but cluster does not form up. The same configuration works in EAP 7.1.
> Attaching xml config files for both of the servers.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (WFLY-10046) Cannot create cluster with JGroups Gosship router
by Miroslav Novak (JIRA)
[ https://issues.jboss.org/browse/WFLY-10046?page=com.atlassian.jira.plugin... ]
Miroslav Novak updated WFLY-10046:
----------------------------------
Attachment: standalone-full-ha-gosship1.xml
standalone-full-ha-gosship2.xml
> Cannot create cluster with JGroups Gosship router
> -------------------------------------------------
>
> Key: WFLY-10046
> URL: https://issues.jboss.org/browse/WFLY-10046
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 12.0.0.Final
> Reporter: Miroslav Novak
> Assignee: Paul Ferraro
> Priority: Blocker
> Attachments: standalone-full-ha-gosship1.xml, standalone-full-ha-gosship2.xml
>
>
> I've used config from EAP 7.1.0/WF11 where gosship router was configured in udp stack like:
> {code}
> <stack name="udp">
> <transport type="UDP" shared="false" socket-binding="jgroups-udp"/>
> <protocol type="PING"/>
> <protocol type="MERGE3"/>
> <protocol type="FD_SOCK"/>
> <protocol type="FD_ALL"/>
> <protocol type="VERIFY_SUSPECT"/>
> <protocol type="pbcast.NAKACK2"/>
> <protocol type="UNICAST3"/>
> <protocol type="pbcast.STABLE"/>
> <protocol type="pbcast.GMS"/>
> <protocol type="UFC"/>
> <protocol type="MFC"/>
> <protocol type="FRAG2"/>
> <protocol type="TUNNEL">
> <property name="gossip_router_hosts">
> 0.0.0.0[12001]
> </property>
> </protocol>
> </stack>
> {code}
> Gosshop router was started on localhost:
> {{java -cp $JBOSS_HOME//bin/client/jboss-client.jar org.jgroups.stack.GossipRouter -port 12001 -bindaddress 0.0.0.0}}
> but cluster does not form up. The same configuration works in EAP 7.1.
> Attaching xml config files for both of the servers.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month
[JBoss JIRA] (WFLY-10046) Cannot create cluster with JGroups Gosship router
by Miroslav Novak (JIRA)
Miroslav Novak created WFLY-10046:
-------------------------------------
Summary: Cannot create cluster with JGroups Gosship router
Key: WFLY-10046
URL: https://issues.jboss.org/browse/WFLY-10046
Project: WildFly
Issue Type: Bug
Components: Clustering
Affects Versions: 12.0.0.Final
Reporter: Miroslav Novak
Assignee: Paul Ferraro
Priority: Blocker
Attachments: standalone-full-ha-gosship1.xml, standalone-full-ha-gosship2.xml
I've used config from EAP 7.1.0/WF11 where gosship router was configured in udp stack like:
{code}
<stack name="udp">
<transport type="UDP" shared="false" socket-binding="jgroups-udp"/>
<protocol type="PING"/>
<protocol type="MERGE3"/>
<protocol type="FD_SOCK"/>
<protocol type="FD_ALL"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="pbcast.NAKACK2"/>
<protocol type="UNICAST3"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
<protocol type="TUNNEL">
<property name="gossip_router_hosts">
0.0.0.0[12001]
</property>
</protocol>
</stack>
{code}
Gosshop router was started on localhost:
{{java -cp $JBOSS_HOME//bin/client/jboss-client.jar org.jgroups.stack.GossipRouter -port 12001 -bindaddress 0.0.0.0}}
but cluster does not form up. The same configuration works in EAP 7.1.
Attaching xml config files for both of the servers.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 1 month