[JBoss JIRA] (WFLY-11682) Clustered SLSB membership anomalies when all cluster members removed
by Richard Achmatowicz (Jira)
[ https://issues.jboss.org/browse/WFLY-11682?page=com.atlassian.jira.plugin... ]
Richard Achmatowicz commented on WFLY-11682:
--------------------------------------------
On the server side, the old EJB client implementation classes EJBRemoteConnectorService and VersionOneProtocolChannelReceiver have been replaced by a number of classes which work together to do the same thing: EJBRemoteConnectorService, RemoteEJBService, AssociationService, AssociationImpl and EJBServerChannel.
The VersionOneProtocolChannelReceiver held an instance of RegistryCollector (which holds client mappings entries for each cluster the server is a member of), which allowed supporting membership of multiple clusters at the same time well as notifying the server when a cluster was being removed (registryAdded()/registryRemoved()). A call to registryRemoved() by the last node in the cluster would cause the server to send a message to the client indicating the cluster was removed entirely, and was a way for the last node to clear that cluster's information from the client.
The new arrangement only supports server membership of one cluster and AssociationImpl, rather than holding a reference to RegistryCollector, now only holds a single Registry to store the client mappings for the single cluster this server is a member of. Consequently, unlike before, we have no notification of the last node leaving via the RegistryCollector. We need to recreate this somehow.
We want to send out a CLUSTER_REMOVED message to all connected clients from the last node in the cluster when we are sure that the last node is in fact shutting down. Looking at the dependencies between the various services, we have EJBRemoteConnector -> AssociationService, so that the EJBRemoteConnector will shut down before the AssociationService. Also the EJBRemoteConnector, in its stop() method, shuts down the Endpoint for the Remoting connector, making all channel connections to clients inaccessible thereafter. So it seems the only reasonable place to send out this message is at the beginning of the stop() method of EJBRemoteConnector while the channel connections are still open.
I have implemented this and it works based on my limited testing. I'll push a PR for now so that I can get some feedback.
As Paul noted, this does not solve the problem in the case where the last node crashes (does not get a chance to cleanly shut down) and the only reasonable place to deal with this special case is on the client (somehow).. For now though, i'm making this change available to handle the case of clean shutdown.
Also, i'll implement the missing suspend/resume behavior in another issue. It's unrelated to this one. We should also probably revisit the limitation of single cluster membership at some time soon.
> Clustered SLSB membership anomalies when all cluster members removed
> --------------------------------------------------------------------
>
> Key: WFLY-11682
> URL: https://issues.jboss.org/browse/WFLY-11682
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, EJB
> Affects Versions: 15.0.1.Final
> Environment: WildFly running in an n-node cluster with an EJB client sending requests even during the time the cluster is down.
> Reporter: Jörg Bäsner
> Assignee: Richard Achmatowicz
> Priority: Major
> Attachments: node1.txt, node12.txt, node2.txt, node3.txt, playground.zip
>
>
> This description will be based on a 3 node cluster. Cluster node 1 and 2 are configured in the {{PROVIDER_URL}}, node 3 is not.
> The client has a custom ClusterNodeSelector implementation that is printing the {{connectedNodes}} and the {{availableNodes}} and doing a random balancing.
> As long as all nodes are up and running the client is calling EJBs in a balanced way.
> When node1 is shut down, the client get the notification below:
> {code}...
> DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
> DEBUG (XNIO-1 task-4) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
> DEBUG (XNIO-1 task-4) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
> DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
> DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received CLUSTER_TOPOLOGY_NODE_REMOVAL(18) message for (cluster, node) = (ejb, node1)
> DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received CLUSTER_TOPOLOGY_NODE_REMOVAL(18) message for (cluster, node) = (ejb, node1)
> ...
> {code}
> Then node2 is shut down. Again the client get the information, see:
> {code}
> ...
> DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
> DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
> DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received CLUSTER_TOPOLOGY_NODE_REMOVAL(18) message for (cluster, node) = (ejb, node2)
> ...
> {code}
> Finally node3 is being shut down. Now the client only get the following information:
> {code}
> ...
> DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
> DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
> ...
> {code}
> This mean the _node3_ is not being informed about the fact that the last node of the cluster has been stopped.
> From this point on the client is always getting {{Caused by: java.net.ConnectException: Connection refused}}
> Now node1 is started again, resulting in the following output for {{connectedNodes}} and the {{availableNodes}}:
> {code}
> ...
> INFO (ThreadPoolTaskExecutor-1) [com.jboss.examples.ejb.CustomClusterNodeSelector] connectedNodes(1) '[node1]', availableNodes(2) '[node3, node1]'
> ...
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months
[JBoss JIRA] (WFLY-11793) JPA EntityManager merge infinite loop - java.lang.StackOverflowError
by Bruno Maioli Martins (Jira)
[ https://issues.jboss.org/browse/WFLY-11793?page=com.atlassian.jira.plugin... ]
Bruno Maioli Martins edited comment on WFLY-11793 at 3/2/19 11:20 AM:
----------------------------------------------------------------------
Scott Marlow, thanks for the response.
I did the steps you suggested, and I was also suspecting Hibernate ORM Core, but the interesting thing is that if I run the same application, but in desktop app, I get satisfactory result.
I added the server log and jstack generated after the failure.
I will investigate the failure better, and report back to the Hibernate ORM Core team.
Thank you
was (Author: brunomaioli):
Scott Marlow, thanks for the response.
I did the steps you suggested, and I was also suspecting Hibernate ORM Core, but the interesting thing is that if I run the same application, but in desktop app, I get satisfactory result.
I added the server log and jstack generated after the failure.
> JPA EntityManager merge infinite loop - java.lang.StackOverflowError
> --------------------------------------------------------------------
>
> Key: WFLY-11793
> URL: https://issues.jboss.org/browse/WFLY-11793
> Project: WildFly
> Issue Type: Bug
> Components: JPA / Hibernate
> Affects Versions: 15.0.1.Final
> Environment: MAC OS Mojave
> Java 11
> Wildfly 15.0.1-Final
> Data Base Postgresql
> Reporter: Bruno Maioli Martins
> Assignee: Scott Marlow
> Priority: Critical
> Attachments: jstack.txt, server+querys.log, server-1.log, teste.zip
>
>
> Dear,
> When executing the update of a record invoking getEntityManager().merge(obj) the application goes into infinite loop. This behavior only occurs when there is a OneToMany bidirectional mapping of the entity Papel -> PermissaoAtribuida.
> If the same code runs outside of the Wildfly server context its execution occurs normally.
> Apparently the call getEntityManager.().merge(obj) attempts infinite selects to sync the object with transaction context before doing the merge.
> Attach the source code of my test classes.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months
[JBoss JIRA] (WFLY-11793) JPA EntityManager merge infinite loop - java.lang.StackOverflowError
by Bruno Maioli Martins (Jira)
[ https://issues.jboss.org/browse/WFLY-11793?page=com.atlassian.jira.plugin... ]
Bruno Maioli Martins updated WFLY-11793:
----------------------------------------
Attachment: server-1.log
server+querys.log
jstack.txt
> JPA EntityManager merge infinite loop - java.lang.StackOverflowError
> --------------------------------------------------------------------
>
> Key: WFLY-11793
> URL: https://issues.jboss.org/browse/WFLY-11793
> Project: WildFly
> Issue Type: Bug
> Components: JPA / Hibernate
> Affects Versions: 15.0.1.Final
> Environment: MAC OS Mojave
> Java 11
> Wildfly 15.0.1-Final
> Data Base Postgresql
> Reporter: Bruno Maioli Martins
> Assignee: Scott Marlow
> Priority: Critical
> Attachments: jstack.txt, server+querys.log, server-1.log, teste.zip
>
>
> Dear,
> When executing the update of a record invoking getEntityManager().merge(obj) the application goes into infinite loop. This behavior only occurs when there is a OneToMany bidirectional mapping of the entity Papel -> PermissaoAtribuida.
> If the same code runs outside of the Wildfly server context its execution occurs normally.
> Apparently the call getEntityManager.().merge(obj) attempts infinite selects to sync the object with transaction context before doing the merge.
> Attach the source code of my test classes.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months
[JBoss JIRA] (WFLY-11793) JPA EntityManager merge infinite loop - java.lang.StackOverflowError
by Bruno Maioli Martins (Jira)
[ https://issues.jboss.org/browse/WFLY-11793?page=com.atlassian.jira.plugin... ]
Bruno Maioli Martins commented on WFLY-11793:
---------------------------------------------
Scott Marlow, thanks for the response.
I did the steps you suggested, and I was also suspecting Hibernate ORM Core, but the interesting thing is that if I run the same application, but in desktop app, I get satisfactory result.
I added the server log and jstack generated after the failure.
> JPA EntityManager merge infinite loop - java.lang.StackOverflowError
> --------------------------------------------------------------------
>
> Key: WFLY-11793
> URL: https://issues.jboss.org/browse/WFLY-11793
> Project: WildFly
> Issue Type: Bug
> Components: JPA / Hibernate
> Affects Versions: 15.0.1.Final
> Environment: MAC OS Mojave
> Java 11
> Wildfly 15.0.1-Final
> Data Base Postgresql
> Reporter: Bruno Maioli Martins
> Assignee: Scott Marlow
> Priority: Critical
> Attachments: teste.zip
>
>
> Dear,
> When executing the update of a record invoking getEntityManager().merge(obj) the application goes into infinite loop. This behavior only occurs when there is a OneToMany bidirectional mapping of the entity Papel -> PermissaoAtribuida.
> If the same code runs outside of the Wildfly server context its execution occurs normally.
> Apparently the call getEntityManager.().merge(obj) attempts infinite selects to sync the object with transaction context before doing the merge.
> Attach the source code of my test classes.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months
[JBoss JIRA] (WFCORE-4355) Add utility for event logging such as audit and access logging
by James Perkins (Jira)
James Perkins created WFCORE-4355:
-------------------------------------
Summary: Add utility for event logging such as audit and access logging
Key: WFCORE-4355
URL: https://issues.jboss.org/browse/WFCORE-4355
Project: WildFly Core
Issue Type: Feature Request
Components: Logging
Reporter: James Perkins
Assignee: James Perkins
There are open feature requests to write audit and access logging to {{stdout}} in JSON format to be consumed by log aggregators. It would be useful to have a minimal API to handle this so that output can be standard. This should go through standard logging.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months
[JBoss JIRA] (DROOLS-3724) Defining non default Kie Base and Kie Session results in a hang in the Test Scenario Runner
by Toni Rikkola (Jira)
[ https://issues.jboss.org/browse/DROOLS-3724?page=com.atlassian.jira.plugi... ]
Toni Rikkola updated DROOLS-3724:
---------------------------------
Sprint: 2019 Week 08-10
> Defining non default Kie Base and Kie Session results in a hang in the Test Scenario Runner
> -------------------------------------------------------------------------------------------
>
> Key: DROOLS-3724
> URL: https://issues.jboss.org/browse/DROOLS-3724
> Project: Drools
> Issue Type: Bug
> Components: Test Scenarios Editor
> Reporter: Jim Tyrrell
> Assignee: Toni Rikkola
> Priority: Blocker
> Labels: ScenarioSimulation
>
> If I use the Settings of a project to created a Kie Base and Kie Session I end up with the UI hanging, and I can't continue until I do a control-R to reload the web application. Web Applications should never hang.
> Given a DRL like this:
> package com.myspace.annual_rent_decision;
> import com.myspace.annual_rent_decision.Lease;
> rule "Is signoff required"
> when
> lease : Lease(leaseAgreementAmount >= 80000)
>
> // eval(true)
> then
> System.out.println("Setting signOffRequired to True");
> lease.setSignOffRequired(true);
> end
> A simple fact model to support it. You will find the below issue.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
5 years, 2 months