[JBoss JIRA] (WFLY-6583) Session leak on SmartOS hosts
by Michael Noack (JIRA)
[ https://issues.jboss.org/browse/WFLY-6583?page=com.atlassian.jira.plugin.... ]
Michael Noack edited comment on WFLY-6583 at 5/4/16 12:46 PM:
--------------------------------------------------------------
[~pferraro] I certainly understand why this could lead to wrong assumptions on a live system. However on my test system, where I control every single request made to the system and thus every session ever created, I would expect this management operation to return zero, if the system has been sitting without requests for longer than the configured session timeout.
And this is indeed what I can see on KVM virtualized CentOS instances!
The very same configuration will never again return to zero sessions on said SmartOS instances or on virtual CentOS instances using a BrandZ kernel. And I can also see the live system getting OOM every few days, if not restarted on a daily basis. So those session do indeed still hang around and take memory.
I thought that we might hit a bug on some of the replication libraries at first, which is why I took up tests again using wildfly-10.0.0.Final. But since I can reproduce the issue with an application as simple as this one: https://github.com/liweinan/cluster-demo I no longer believe this. I've since been able to establish the link between the issue and infrastructure instances on Joyents plattform and can reproduce the issue reliably. KVM instances are free of said session leak. If you want, you can provide me with a public ssh-key and I can give you access to such a CentOS 7 with BrandZ kernel test instance to take a look for yourself.
I'm about to migrate our live system to said KVM instances because of this, despite those being twice as expensive, since this is the only work-around I've found to far. I'm not complaining here, and I'm well aware that this is very suspicous. The OS kernel shouldn't be able to affect the session handling of an application running on top of a JVM. I'm merely documenting this here for reference in case others might see similar issues.
was (Author: michael.noack):
[~pferraro] I certainly understand why this could lead to wrong assumptions on a live system. However on my test system, where I control every single request made to the system and thus every session ever created, I would expect this management operation to return zero, if the system has been sitting without requests for longer than the configured session timeout.
And this is indeed what I can see on KVM virtualized CentOS instances!
The very same configuration will never again return to zero sessions on said SmartOS instances or on virtual CentOS instances using a BrandZ kernel. And I can also see the live system getting OOM every few days, if not restarted on a daily basis. So those session do indeed still hang around and take memory.
I thought that we might hit a bug on some of the replication libraries at first, which is why I took up tests again using wildfly-10.0.0.Final. But since I can reproduce the issue with an application as simple as this one: https://github.com/liweinan/cluster-demo I no longer believe this. I've since been able to establish the link between the issue and infrastructure instances on Joyents plattform and can reproduce the issue reliably. KVM instances are free of said session leak. If you want, you can provide me with a public ssh-key and I can give you access to such a BrandZ test instance to take a look for yourself.
I'm about to migrate our live system to said KVM instances because of this, despite those being twice as expensive, since this is the only work-around I've found to far. I'm not complaining here, and I'm well aware that this is very suspicous. The OS kernel shouldn't be able to affect the session handling of an application running on top of a JVM. I'm merely documenting this here for reference in case others might see similar issues.
> Session leak on SmartOS hosts
> -----------------------------
>
> Key: WFLY-6583
> URL: https://issues.jboss.org/browse/WFLY-6583
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 8.2.0.Final, 9.0.0.Final, 10.0.0.Final
> Environment: CentOS 7 or SmartOS instance using Joyents Infrastructure/Bare metal container.
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# uname -a
> Linux 979638eb-b45c-45b3-9fdb-d7f48276e4ef 3.10.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# cat /etc/issue
> \S
> Kernel \r on an \m
> Reporter: Michael Noack
> Assignee: Paul Ferraro
> Priority: Minor
>
> When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first. It took me a while to connect the dots here.
> When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
> The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (WFLY-6583) Session leak on SmartOS hosts
by Michael Noack (JIRA)
[ https://issues.jboss.org/browse/WFLY-6583?page=com.atlassian.jira.plugin.... ]
Michael Noack edited comment on WFLY-6583 at 5/4/16 12:45 PM:
--------------------------------------------------------------
[~pferraro] I certainly understand why this could lead to wrong assumptions on a live system. However on my test system, where I control every single request made to the system and thus every session ever created, I would expect this management operation to return zero, if the system has been sitting without requests for longer than the configured session timeout.
And this is indeed what I can see on KVM virtualized CentOS instances!
The very same configuration will never again return to zero sessions on said SmartOS instances or on virtual CentOS instances using a BrandZ kernel. And I can also see the live system getting OOM every few days, if not restarted on a daily basis. So those session do indeed still hang around and take memory.
I thought that we might hit a bug on some of the replication libraries at first, which is why I took up tests again using wildfly-10.0.0.Final. But since I can reproduce the issue with an application as simple as this one: https://github.com/liweinan/cluster-demo I no longer believe this. I've since been able to establish the link between the issue and infrastructure instances on Joyents plattform and can reproduce the issue reliably. KVM instances are free of said session leak. If you want, you can provide me with a public ssh-key and I can give you access to such a BrandZ test instance to take a look for yourself.
I'm about to migrate our live system to said KVM instances because of this, despite those being twice as expensive, since this is the only work-around I've found to far. I'm not complaining here, and I'm well aware that this is very suspicous. The OS kernel shouldn't be able to affect the session handling of an application running on top of a JVM. I'm merely documenting this here for reference in case others might see similar issues.
was (Author: michael.noack):
[~pferraro] I certainly understand why this could lead to wrong assumptions on a live system. However on my test system, where I control every single request made to the system and thus every session ever created, I would expect this management operation to return zero, if the system has been sitting without requests for longer than the configured session timeout.
And this is indeed what I can see on KVM virtualized CentOS instances!
The very same configuration will never again return to zero sessions on said SmartOS instances or on virtual CentOS instances using a BrandZ kernel. And I can also see the live system getting OOM every few days, if not restarted on a daily basis. So those session do indeed still hang around and take memory.
I thought that we might hit a bug on some of the replication libraries at first, which is why I took up tests again using wildfly-10.0.0.Final. But since I can reproduce the issue with an application as simple as this one: https://github.com/liweinan/cluster-demo I no longer believe this. I've since been able to establish the link between KVM and infrastructure instances on Joyents plattform and can reproduce the issue reliably. If you want, you can provide me with a public ssh-key and I can give you access to such a BrandZ test instance.
I'm about to migrate our live system to said KVM instances because of this, despite those being twice as expensive, since this is the only work-around I've found to far. I'm not complaining here, and I'm well aware that this is very suspicous. The OS kernel shouldn't be able to affect the session handling of an application running on top of a JVM. I'm merely documenting this here for reference in case others might see similar issues.
> Session leak on SmartOS hosts
> -----------------------------
>
> Key: WFLY-6583
> URL: https://issues.jboss.org/browse/WFLY-6583
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 8.2.0.Final, 9.0.0.Final, 10.0.0.Final
> Environment: CentOS 7 or SmartOS instance using Joyents Infrastructure/Bare metal container.
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# uname -a
> Linux 979638eb-b45c-45b3-9fdb-d7f48276e4ef 3.10.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# cat /etc/issue
> \S
> Kernel \r on an \m
> Reporter: Michael Noack
> Assignee: Paul Ferraro
> Priority: Minor
>
> When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first. It took me a while to connect the dots here.
> When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
> The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (WFLY-6583) Session leak on SmartOS hosts
by Michael Noack (JIRA)
[ https://issues.jboss.org/browse/WFLY-6583?page=com.atlassian.jira.plugin.... ]
Michael Noack commented on WFLY-6583:
-------------------------------------
[~pferraro] I certainly understand why this could lead to wrong assumptions on a live system. However on my test system, where I control every single request made to the system and thus every session ever created, I would expect this management operation to return zero, if the system has been sitting without requests for longer than the configured session timeout.
And this is indeed what I can see on KVM virtualized CentOS instances!
The very same configuration will never again return to zero sessions on said SmartOS instances or on virtual CentOS instances using a BrandZ kernel. And I can also see the live system getting OOM every few days, if not restarted on a daily basis. So those session do indeed still hang around and take memory.
I thought that we might hit a bug on some of the replication libraries at first, which is why I took up tests again using wildfly-10.0.0.Final. But since I can reproduce the issue with an application as simple as this one: https://github.com/liweinan/cluster-demo I no longer believe this. I've since been able to establish the link between KVM and infrastructure instances on Joyents plattform and can reproduce the issue reliably. If you want, you can provide me with a public ssh-key and I can give you access to such a BrandZ test instance.
I'm about to migrate our live system to said KVM instances because of this, despite those being twice as expensive, since this is the only work-around I've found to far. I'm not complaining here, and I'm well aware that this is very suspicous. The OS kernel shouldn't be able to affect the session handling of an application running on top of a JVM. I'm merely documenting this here for reference in case others might see similar issues.
> Session leak on SmartOS hosts
> -----------------------------
>
> Key: WFLY-6583
> URL: https://issues.jboss.org/browse/WFLY-6583
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 8.2.0.Final, 9.0.0.Final, 10.0.0.Final
> Environment: CentOS 7 or SmartOS instance using Joyents Infrastructure/Bare metal container.
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# uname -a
> Linux 979638eb-b45c-45b3-9fdb-d7f48276e4ef 3.10.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# cat /etc/issue
> \S
> Kernel \r on an \m
> Reporter: Michael Noack
> Assignee: Paul Ferraro
> Priority: Minor
>
> When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first. It took me a while to connect the dots here.
> When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
> The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (ELY-524) Caching support in the LDAP realm
by David Lloyd (JIRA)
David Lloyd created ELY-524:
-------------------------------
Summary: Caching support in the LDAP realm
Key: ELY-524
URL: https://issues.jboss.org/browse/ELY-524
Project: WildFly Elytron
Issue Type: Feature Request
Components: Realms
Reporter: David Lloyd
Priority: Critical
Fix For: 1.1.0.Beta6
The LDAP realm should use a caching strategy to avoid excessive database load in the presence of per-request authentication traffic.
The realm implementation could maintain a synchronized LRU cache of one-time-initialize references to a cached DirContext or Attributes or binding or some combination of these. Because the cache is synchronized, the one-time-initialize object would be added under the lock and then the lock released before the object is populated and returned as a cached credential, allowing atomic action with a minimum of contention.
For each cached entity, a NamingListener could be established which would invalidate (or possibly update) the cached value as the database changes.
Alternatively, a NamingListener could be established for all identities, and each update would invalidate or update any cached values corresponding to the DN or resolved name.
This is a complex design topic so discussion is welcome.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (WFLY-6583) Session leak on SmartOS hosts
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/WFLY-6583?page=com.atlassian.jira.plugin.... ]
Paul Ferraro edited comment on WFLY-6583 at 5/4/16 12:26 PM:
-------------------------------------------------------------
The active-sessions metric uses the SessionManager.getActiveSessions() method - which, for a distributed session manager, will return the identifiers of the sessions residing in memory (excluding passivated sessions) of the target node. If the SessionManager is backed by a distributed cache (as is the case with the default configuration), this will include sessions owned by the target node, as well as sessions for which the target node is a backup. For a cluster size of 3, where each node has just created N sessions, the value for active-sessions would return, on average, 2*N (i.e. N local sessions plus N/2 sessions for each remote node). Likewise, if the SessionManager is backed by a replicated cache, the active-sessions metric would return 3*N (i.e. N local sessions, plus N sessions for each remote node). You can see why I am cautious about drawing premature conclusions from this mechanism of measuring the number of sessions.
was (Author: pferraro):
The active-sessions metric uses the SessionManager.getActiveSessions() method - which for a distributed session manager will return the identifiers of the sessions residing in memory (excluding passivated sessions) on the node. If the SessionManager is backed by a distributed cache (as is the case with the default configuration), this will include sessions owned by the target node, as well as sessions for which the target node is a backup. For a cluster size of 3, where each node has just created N sessions, the value for active-sessions would return, on average, 2*N (i.e. N local sessions plus N/2 sessions for each remote node). Likewise, if the SessionManager is backed by a replicated cache, the active-sessions metric would return 3*N (i.e. N local sessions, plus N sessions for each remote node). You can see why I am cautious about drawing premature conclusions from this mechanism of measuring the number of sessions.
> Session leak on SmartOS hosts
> -----------------------------
>
> Key: WFLY-6583
> URL: https://issues.jboss.org/browse/WFLY-6583
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 8.2.0.Final, 9.0.0.Final, 10.0.0.Final
> Environment: CentOS 7 or SmartOS instance using Joyents Infrastructure/Bare metal container.
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# uname -a
> Linux 979638eb-b45c-45b3-9fdb-d7f48276e4ef 3.10.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# cat /etc/issue
> \S
> Kernel \r on an \m
> Reporter: Michael Noack
> Assignee: Paul Ferraro
> Priority: Minor
>
> When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first. It took me a while to connect the dots here.
> When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
> The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (WFLY-6583) Session leak on SmartOS hosts
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/WFLY-6583?page=com.atlassian.jira.plugin.... ]
Paul Ferraro commented on WFLY-6583:
------------------------------------
The active-sessions metric uses the SessionManager.getActiveSessions() method - which for a distributed session manager will return the identifiers of the sessions residing in memory (excluding passivated sessions) on the node. If the SessionManager is backed by a distributed cache (as is the case with the default configuration), this will include sessions owned by the target node, as well as sessions for which the target node is a backup. For a cluster size of 3, where each node has just created N sessions, the value for active-sessions would return, on average, 2*N (i.e. N local sessions plus N/2 sessions for each remote node). Likewise, if the SessionManager is backed by a replicated cache, the active-sessions metric would return 3*N (i.e. N local sessions, plus N sessions for each remote node). You can see why I am cautious about drawing premature conclusions from this mechanism of measuring the number of sessions.
> Session leak on SmartOS hosts
> -----------------------------
>
> Key: WFLY-6583
> URL: https://issues.jboss.org/browse/WFLY-6583
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 8.2.0.Final, 9.0.0.Final, 10.0.0.Final
> Environment: CentOS 7 or SmartOS instance using Joyents Infrastructure/Bare metal container.
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# uname -a
> Linux 979638eb-b45c-45b3-9fdb-d7f48276e4ef 3.10.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# cat /etc/issue
> \S
> Kernel \r on an \m
> Reporter: Michael Noack
> Assignee: Paul Ferraro
> Priority: Minor
>
> When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first. It took me a while to connect the dots here.
> When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
> The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (WFCORE-1525) ageout-history does not report failures
by Bartosz Spyrko-Śmietanko (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1525?page=com.atlassian.jira.plugi... ]
Bartosz Spyrko-Śmietanko moved WFLY-6572 to WFCORE-1525:
--------------------------------------------------------
Project: WildFly Core (was: WildFly)
Key: WFCORE-1525 (was: WFLY-6572)
Component/s: Patching
(was: Patching)
Affects Version/s: 2.1.0.Final
(was: 10.0.0.Final)
> ageout-history does not report failures
> ----------------------------------------
>
> Key: WFCORE-1525
> URL: https://issues.jboss.org/browse/WFCORE-1525
> Project: WildFly Core
> Issue Type: Bug
> Components: Patching
> Affects Versions: 2.1.0.Final
> Reporter: Dennis Reed
> Assignee: Bartosz Spyrko-Śmietanko
>
> When ageout-history cannot delete the old CP files, it silently fails and returns "success".
> LocalAgeoutHistoryHandler#recursiveDelete does keep track of whether the
> delete failed or succeeded, but the return value from this method is ignored
> in execute(...).
> If recursiveDelete returns false, it should report a failure.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years
[JBoss JIRA] (WFLY-6572) ageout-history does not report failures
by Bartosz Spyrko-Śmietanko (JIRA)
[ https://issues.jboss.org/browse/WFLY-6572?page=com.atlassian.jira.plugin.... ]
Bartosz Spyrko-Śmietanko updated WFLY-6572:
-------------------------------------------
Component/s: Patching
> ageout-history does not report failures
> ----------------------------------------
>
> Key: WFLY-6572
> URL: https://issues.jboss.org/browse/WFLY-6572
> Project: WildFly
> Issue Type: Bug
> Components: Patching
> Affects Versions: 10.0.0.Final
> Reporter: Dennis Reed
> Assignee: Bartosz Spyrko-Śmietanko
>
> When ageout-history cannot delete the old CP files, it silently fails and returns "success".
> LocalAgeoutHistoryHandler#recursiveDelete does keep track of whether the
> delete failed or succeeded, but the return value from this method is ignored
> in execute(...).
> If recursiveDelete returns false, it should report a failure.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
10 years