[JBoss JIRA] (WFLY-6583) Session leak on SmartOS hosts
by Michael Noack (JIRA)
[ https://issues.jboss.org/browse/WFLY-6583?page=com.atlassian.jira.plugin.... ]
Michael Noack updated WFLY-6583:
--------------------------------
Steps to Reproduce:
Spin up any 3 SmartOS or Infrastructure Container instances on Joyent cloud service (or install SmartOS on 3 physical devices). For reference I will call those server 1, 2 and 3.
Install JDK 7 or 8 (8 required for 10.0.0) on all of them.
Deploy wildfly-10.0.0-Final to all three instances and follow the clustering guide.
I made server 1 the DC without any servers being specified in host.xml
I used the following type of servers on server 2+3:
<servers>
<server name="slave64-server-210" group="server-group-b" auto-start="true">
<socket-bindings port-offset="210"/>
</server>
</servers>
Deploy any clusterized webapp, I used our production app as well as https://github.com/liweinan/cluster-demo for tests, the app doesn't matter
Repeatedly call any webapp uri that creates a session, for example using:
wget http://$server:8290/cluster-demo/index.jsp -O status.$server -q -o /dev/null
Monitor the session count of the deployment on both servers, example:
/opt/wildfly/bin/./jboss-cli.sh --connect --user="password" --password="username" --command="/host=slave2/server=slave2-server-210/deployment=cluster-demo.war/subsystem=undertow :read-attribute(name=active-sessions)
was:
Spin up any 3 SmartOS or Infrastructure Container instances on Joyent cloud service (or install SmartOS on 3 physical devices). For reference I will call those server 1, 2 and 3.
Install JDK 7 or 8 (8 required for 10.0.0) on all of them.
Deploy wildfly-10.0.0-Final to all three instances and follow the clustering guide.
I made server 1 the DC without any servers being specified in host.xml
I used the following type of servers on server 2+3:
<servers>
<server name="slave64-server-210" group="server-group-b" auto-start="true">
<socket-bindings port-offset="210"/>
</server>
</servers>
Deploy any clusterized webapp, I used our production app as well as https://github.com/liweinan/cluster-demo for tests, the app doesn't matter
Repeatedly call any webapp uri that creates a session, for example using:
wget http://$server:8290/cluster-demo/index.jsp -O status.$server -q -o /dev/null
Monitor the session count of the deployment on both servers, example:
/opt/wildfly/bin/./jboss-cli.sh --connect --user="luckycat" --password="luckycat" --command="/host=slave2/server=slave2-server-210/deployment=cluster-demo.war/subsystem=undertow :read-attribute(name=active-sessions)
> Session leak on SmartOS hosts
> -----------------------------
>
> Key: WFLY-6583
> URL: https://issues.jboss.org/browse/WFLY-6583
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 8.2.0.Final, 9.0.0.Final, 10.0.0.Final
> Environment: CentOS 7 or SmartOS instance using Joyents Infrastructure/Bare metal container.
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# uname -a
> Linux 979638eb-b45c-45b3-9fdb-d7f48276e4ef 3.10.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# cat /etc/issue
> \S
> Kernel \r on an \m
> Reporter: Michael Noack
> Assignee: Paul Ferraro
> Priority: Minor
>
> When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first. It took me a while to connect the dots here.
> When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
> The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 1 month
[JBoss JIRA] (WFCORE-1512) DelegatingBasicLogger could delegate to a Logger provider
by James Perkins (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1512?page=com.atlassian.jira.plugi... ]
James Perkins closed WFCORE-1512.
---------------------------------
Resolution: Rejected
> DelegatingBasicLogger could delegate to a Logger provider
> ---------------------------------------------------------
>
> Key: WFCORE-1512
> URL: https://issues.jboss.org/browse/WFCORE-1512
> Project: WildFly Core
> Issue Type: Feature Request
> Components: Logging
> Reporter: Jean-Francois Denise
> Assignee: James Perkins
>
> Today the DelegatingBasicLogger subclass generated from @MessageLogger annotation is bound to a Logger at construction time. It could be of interest (e.g: I ran by accident on WFCORE-1187) to have the logger to be resolved dynamically.
> I didn't evaluate the impact on the logging project/tools/performances of a solution like the following but it captures the idea of a logger provider.
> public class DelegatingBasicLogger {
> public interface LoggerProvider
> { public Logger getLogger(); }
> DelegatingBasicLogger(Logger logger) {
> { this(()->{return logger;})}
> DelegatingBasicLogger(LoggerProvider provider)
> { this.provider = provider; }
> ....
> }
> A subclass could use LogContext to retrieve the right Logger.
> class MyLogger extends DelegatingBasicLogger {
> MyLogger(String category)
> { super(()->LogContext.getLogContext().getLogger()); }
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 1 month
[JBoss JIRA] (ELY-523) Token-based Security Realm
by Pedro Igor (JIRA)
Pedro Igor created ELY-523:
------------------------------
Summary: Token-based Security Realm
Key: ELY-523
URL: https://issues.jboss.org/browse/ELY-523
Project: WildFly Elytron
Issue Type: Feature Request
Components: Realms
Affects Versions: 1.1.0.Beta5
Reporter: Pedro Igor
Assignee: Pedro Igor
Provide a Token-based realm capable of managing identities based on different token formats.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 1 month
[JBoss JIRA] (WFLY-6583) Session leak on SmartOS hosts
by Paul Ferraro (JIRA)
[ https://issues.jboss.org/browse/WFLY-6583?page=com.atlassian.jira.plugin.... ]
Paul Ferraro commented on WFLY-6583:
------------------------------------
How exactly do you determine that a session was not destroyed?
For distributed web applications, there's no guarantee that the node that created a session will be the node that destroys the session.
> Session leak on SmartOS hosts
> -----------------------------
>
> Key: WFLY-6583
> URL: https://issues.jboss.org/browse/WFLY-6583
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 8.2.0.Final, 9.0.0.Final, 10.0.0.Final
> Environment: CentOS 7 or SmartOS instance using Joyents Infrastructure/Bare metal container.
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# uname -a
> Linux 979638eb-b45c-45b3-9fdb-d7f48276e4ef 3.10.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# cat /etc/issue
> \S
> Kernel \r on an \m
> Reporter: Michael Noack
> Assignee: Paul Ferraro
> Priority: Minor
>
> When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first. It took me a while to connect the dots here.
> When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
> The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 1 month
[JBoss JIRA] (WFLY-6583) Session leak on SmartOS hosts
by Tomaz Cerar (JIRA)
[ https://issues.jboss.org/browse/WFLY-6583?page=com.atlassian.jira.plugin.... ]
Tomaz Cerar commented on WFLY-6583:
-----------------------------------
What is/was the exact jvm used on systems that this occurs?
> Session leak on SmartOS hosts
> -----------------------------
>
> Key: WFLY-6583
> URL: https://issues.jboss.org/browse/WFLY-6583
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 8.2.0.Final, 9.0.0.Final, 10.0.0.Final
> Environment: CentOS 7 or SmartOS instance using Joyents Infrastructure/Bare metal container.
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# uname -a
> Linux 979638eb-b45c-45b3-9fdb-d7f48276e4ef 3.10.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# cat /etc/issue
> \S
> Kernel \r on an \m
> Reporter: Michael Noack
> Assignee: Paul Ferraro
> Priority: Minor
>
> When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first. It took me a while to connect the dots here.
> When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
> The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 1 month
[JBoss JIRA] (WFLY-6583) Session leak on SmartOS hosts
by Michael Noack (JIRA)
[ https://issues.jboss.org/browse/WFLY-6583?page=com.atlassian.jira.plugin.... ]
Michael Noack updated WFLY-6583:
--------------------------------
Description:
When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left onclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first.
When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
was:
When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first.
When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
> Session leak on SmartOS hosts
> -----------------------------
>
> Key: WFLY-6583
> URL: https://issues.jboss.org/browse/WFLY-6583
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 8.2.0.Final, 9.0.0.Final, 10.0.0.Final
> Environment: CentOS 7 or SmartOS instance using Joyents Infrastructure/Bare metal container.
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# uname -a
> Linux 979638eb-b45c-45b3-9fdb-d7f48276e4ef 3.10.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# cat /etc/issue
> \S
> Kernel \r on an \m
> Reporter: Michael Noack
> Assignee: Paul Ferraro
> Priority: Minor
>
> When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left onclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first.
> When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
> The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 1 month
[JBoss JIRA] (WFLY-6583) Session leak on SmartOS hosts
by Michael Noack (JIRA)
[ https://issues.jboss.org/browse/WFLY-6583?page=com.atlassian.jira.plugin.... ]
Michael Noack updated WFLY-6583:
--------------------------------
Description:
When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first.
When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
was:
When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left onclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first.
When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
> Session leak on SmartOS hosts
> -----------------------------
>
> Key: WFLY-6583
> URL: https://issues.jboss.org/browse/WFLY-6583
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 8.2.0.Final, 9.0.0.Final, 10.0.0.Final
> Environment: CentOS 7 or SmartOS instance using Joyents Infrastructure/Bare metal container.
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# uname -a
> Linux 979638eb-b45c-45b3-9fdb-d7f48276e4ef 3.10.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# cat /etc/issue
> \S
> Kernel \r on an \m
> Reporter: Michael Noack
> Assignee: Paul Ferraro
> Priority: Minor
>
> When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first.
> When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
> The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 1 month
[JBoss JIRA] (WFLY-6583) Session leak on SmartOS hosts
by Michael Noack (JIRA)
[ https://issues.jboss.org/browse/WFLY-6583?page=com.atlassian.jira.plugin.... ]
Michael Noack updated WFLY-6583:
--------------------------------
Description:
When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first. It took me a while to connect the dots here.
When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
was:
When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first.
When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
> Session leak on SmartOS hosts
> -----------------------------
>
> Key: WFLY-6583
> URL: https://issues.jboss.org/browse/WFLY-6583
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 8.2.0.Final, 9.0.0.Final, 10.0.0.Final
> Environment: CentOS 7 or SmartOS instance using Joyents Infrastructure/Bare metal container.
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# java -version
> java version "1.8.0_77"
> Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# uname -a
> Linux 979638eb-b45c-45b3-9fdb-d7f48276e4ef 3.10.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
> [root@979638eb-b45c-45b3-9fdb-d7f48276e4ef /]# cat /etc/issue
> \S
> Kernel \r on an \m
> Reporter: Michael Noack
> Assignee: Paul Ferraro
> Priority: Minor
>
> When running Wildfly 8.2.0-Final, 9.0.0-Final or 10.0.0-Final in domain mode using the full-ha profile some sessions never get closed when running on SmartOS or a BrandZ kernel on SmartOS. The amount of unclosed sessions rises slowly. With 1 session per second and server created, roughly 30-50 sessions are left unclosed on each server. I've been keeping track of this issue for almost a year now and handled it by restarting the entire cluster at first. It took me a while to connect the dots here.
> When registering a HttpSessionListener and logging any sessionCreated(HttpSessionEvent se) and sessionDestroyed(HttpSessionEvent se) one can cleary see some sessions never generate the sessionDestroyed event.
> The problem disappears when running the very same setup on a KVM instance of CentOS 6 or 7 (regardless whether the KVM host is SmartOS or Linux).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 1 month
[JBoss JIRA] (WFCORE-1524) Scanner deployment failing at boot passes readiness/liveness checks
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1524?page=com.atlassian.jira.plugi... ]
Brian Stansberry commented on WFCORE-1524:
------------------------------------------
[~ehugonnet] Quick speculation:
I think this all relates to how we handle boot ops. The scanner generated deployment ops are inserted as steps into the overall boot op (the 2nd op, that does subsystems.)
That overall op does not fail because we have rollback-on-runtime-failure set to false.
But I suspect we are losing track of what is happening to the particular steps that are added by the scanner and therefore are not reporting things correctly.
> Scanner deployment failing at boot passes readiness/liveness checks
> -------------------------------------------------------------------
>
> Key: WFCORE-1524
> URL: https://issues.jboss.org/browse/WFCORE-1524
> Project: WildFly Core
> Issue Type: Bug
> Components: Deployment Scanner, Domain Management
> Environment: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-eap-7/eap70-openshift:1.3-11
> Reporter: Marek Schmidt
> Assignee: ehsavoie Hugonnet
>
> During larger scale-ups (e.g. from 1 to 6 on nodes under load) deployments sometimes fail with "org.infinispan.util.concurrent.TimeoutException: Replication timeout for foo-1-xxxxx".
> That by itself is expected with the default timeouts, however, the problem is that such failed deployments pass the default EAP readiness and liveness checks, which results in HTTP 404 responses.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 1 month