]
Mike Zhary commented on WFLY-14234:
-----------------------------------
Hi.
there are to be 2 servers and haproxy between them.
[wildfly-server1] <-> [haproxy] <-> [wildfly-server2]
there are to be defined remote connections to haproxy server
like this:
{code}
<http-connector name="http-remoting-connector"
connector-ref="default" security-realm="ApplicationRealm"/>
<outbound-connections>
<remote-outbound-connection name="remote-ejb-connection"
outbound-socket-binding-ref="remote-ejb" protocol="http-remoting"
security-realm="ejb-security-realm" username="ejb">
<properties>
<property name="SASL_POLICY_NOANONYMOUS"
value="false"/>
<property name="SSL_ENABLED"
value="false"/>
</properties>
</remote-outbound-connection>
</outbound-connections>
{code}
{code}
<outbound-socket-binding name="remote-ejb">
<remote-destination host="test-ejb-balance01"
port="30001"/>
</outbound-socket-binding>
{code}
haproxy.cfg
{code}
### start config for haproxy_host.name balance
frontend balance_test-wf01
bind 192.168.230.13:30000
mode tcp
default_backend test-wf01_servers
option tcplog
log global
capture request header X-Forwarded-For len 15
backend test-wf01_servers
balance roundrobin
server server-test-wf01 test-wf01:8080 check
### stop config for haproxy_host.name balance
### start config for haproxy_host.name balance
frontend balance_test-wf02
bind 192.168.230.13:30001
mode tcp
default_backend test-wf02_servers
option tcplog
log global
capture request header X-Forwarded-For len 15
backend test-wf02_servers
balance roundrobin
server server-test-wf02 test-wf02:8080 check
### stop config for haproxy_host.name balance
{code}
You can see tcp mode in *haproxy.cfg*, but I'd tried even http mode.
In order to reproduce this scheme I deployed application which calls remote-ejb on
wildfly-server2.
When wildfly-server2 is online there are no problems.
But if I'll shutdown wildfly there I'll have situation I described before.
EJB bean method direct memory buffer leaking
--------------------------------------------
Key: WFLY-14234
URL:
https://issues.redhat.com/browse/WFLY-14234
Project: WildFly
Issue Type: Bug
Components: EJB, IO
Affects Versions: 19.0.0.Final
Reporter: Mike Zhary
Assignee: Flavia Rainone
Priority: Major
Attachments: 2020-11-10-112753_563x276_scrot.png,
src_main_java_org_jboss_remoting3_remote_RemoteConnectionProvider.java
Wildfly.19.0.1.Final standalone
We're using haproxy to balance ejb calls between wildfly servers.
That's very useful model, but there are problems with it.
All connections are declared as remote-outbound-connection and it works.
{code}
<remote-outbound-connection name="{{ c.name }}"
outbound-socket-binding-ref="{{ c.socket_binding }}"
protocol="http-remoting" security-realm="{{ c.security_realm }}"
username="{{ c.user }}">
<properties>
<property name="SASL_POLICY_NOANONYMOUS"
value="false"/>
<property name="SSL_ENABLED" value="false"/>
</properties>
</remote-outbound-connection>
{code}
But there is the problem when some from servers is offline. haproxy send response by http
and sends RST by tcp, but WF can't make sense of this response.
Problem can be reprouced when servers listening port, but there is no repsonse for ejb
call(in our case it means haproxy doesn't have suitable backend online)
But everything is clear, when port closed or server don't response for a calls.
Haproxy doesn't have feature to don't listen port if backed is offline.
At OS problems shows as huge count(equal to thread count) of connections in state
CLOSE_WAIT. That means server had send packet TCP_FIN and client accepted it, but there
were no respond on it. In other words WF and it's subsystems didn't agree with
closing connection and it becomes to CLOSE_WAIT. At OS side closing connection in sometime
and took it as dead(it's normal case)
At WF memory it appears as growing count of objects
org.xnio.ByteBufferSlicePool$PooledByteBuffer, that have attribute buffer that holds about
2mb each.
Take a look:
!2020-11-10-112753_563x276_scrot.png|thumbnail!
Problem allocated in XNIO library. There were found a mistake and looks very similar to
our problem. XNIO-254
The same time we'd found XNIO-380 and we tried it with last version of this
library(snapshot 3.9), no results were there.
We tried get undrestanding ow DMB clean, but did nothing due to too complicated code
design.
As per investigations DMB used for an SSL connection, but we don't it.
So we were to patch jboss-remoting class RemoteConnectionProvider.
after that we'd solved issue with leaking of DMB.
Our patched library:
[^src_main_java_org_jboss_remoting3_remote_RemoteConnectionProvider.java]
Please take look how this bug can be fixed.