[jboss-jira] [JBoss JIRA] (WFLY-6583) Session leak on SmartOS hosts

[Michael Noack (JIRA)]

    [ https://issues.jboss.org/browse/WFLY-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200660#comment-13200660 ] 

Michael Noack commented on WFLY-6583:
-------------------------------------

[~pferraro] I used both approaches. I started out with the HttpSessionListener for unrelated reasons. When wildfly would start to crash every so often being OOM, I started to investigate and found unclosed sessions. I had help from my database here and only confirmed later that the session close was missing in the logs as well.

So when running a test "grep sessionCreated server.log | wc -l" returns a number higher than "grep sessionDestroyed server.log | wc -l" on both server nodes.
Running the management operation to count active sessions on both nodes returns a similar discrepancy.
Both approaches have been taken after all sessions should have been timed out long ago.

I can only reproduce this behavior when running on Solaris kernels. So far I have only tried it on SmartOS and BrandZ kernels running on SmartOS though. I would expect to see the same issue on Illumous and OpenSolaris as well.

> Session leak on SmartOS hosts
> -----------------------------
>
>                 Key: WFLY-6583
>                 URL: https://issues.jboss.org/browse/WFLY-6583
>             Project: WildFly
>          Issue Type: Bug
>          Components: Clustering
>    Affects Versions: 8.2.0.Final, 9.0.0.Final, 10.0.0.Final
>         Environment: CentOS 7 or SmartOS instance using Joyents Infrastructure/Bare metal container.
> [root at 979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [root at 979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# uname -a
> Linux 979638eb-b45c-45b3-9fdb-d7f48276e4ef 3.10.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
> [root at 979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# cat /etc/issue
> \S
> Kernel \r on an \m
>            Reporter: Michael Noack
>            Assignee: Paul Ferraro
>            Priority: Minor
>
> When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first. It took me a while to connect the dots here.
> When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
> The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)