[JBoss JIRA] (WFCORE-1529) New threads named ServerService Thread Pool -- xyz created with every deploy and undeploy and removed after ~20 seconds
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1529?page=com.atlassian.jira.plugi... ]
Brian Stansberry updated WFCORE-1529:
-------------------------------------
Description:
tl;dr version from Brian:
The ServerService Threads and Host Controller Service Threads pools have no core size and a thread timeout of 20 secs, so if a management request comes in periodically but less often then every 20 secs (says twice a minute) the pool will continually churn threads. That kind of periodic polling is not an unusual scenario, so, we'll put in a core size of one or two.
Original details from Rostislav:
I did quite simple thing in the loop :
* deploy attached application into jboss-eap-7.0/standalone/deployments/
* wait 1 minute
* remove anything from jboss-eap-7.0/standalone/deployments/
* wait 1 minute
I connected with jvisualvm and was watching monitor tab for cpu / classes / heap / threads overview statistics. After some time I spotted jumps (up and down) on threads.
After some time I see threads named "ServerService Thread Pool -- 346", I ended with number bigger than 1400. In my case 15 threads are created for 20 seconds when deploying and another 15 threads are created for 20 seconds when undeploying. Nothing seems to be reused
Setup for threads is done in https://github.com/wildfly/wildfly-core/blob/master/server/src/main/java/... method addService
Is it intentional to have threads alive only 20 seconds and then scratch them ? Keeping a lot of threads means usage of system resources, creating new threads has some overhead too. Maybe the question is what is the right balance ?
Note: It would be nice to have "Possible Bug" Issue Type for issues like this ;)
was:
I did quite simple thing in the loop :
* deploy attached application into jboss-eap-7.0/standalone/deployments/
* wait 1 minute
* remove anything from jboss-eap-7.0/standalone/deployments/
* wait 1 minute
I connected with jvisualvm and was watching monitor tab for cpu / classes / heap / threads overview statistics. After some time I spotted jumps (up and down) on threads.
After some time I see threads named "ServerService Thread Pool -- 346", I ended with number bigger than 1400. In my case 15 threads are created for 20 seconds when deploying and another 15 threads are created for 20 seconds when undeploying. Nothing seems to be reused
Setup for threads is done in https://github.com/wildfly/wildfly-core/blob/master/server/src/main/java/... method addService
Is it intentional to have threads alive only 20 seconds and then scratch them ? Keeping a lot of threads means usage of system resources, creating new threads has some overhead too. Maybe the question is what is the right balance ?
Note: It would be nice to have "Possible Bug" Issue Type for issues like this ;)
> New threads named ServerService Thread Pool -- xyz created with every deploy and undeploy and removed after ~20 seconds
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: WFCORE-1529
> URL: https://issues.jboss.org/browse/WFCORE-1529
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 2.1.0.Final
> Reporter: Rostislav Svoboda
> Assignee: Brian Stansberry
>
> tl;dr version from Brian:
> The ServerService Threads and Host Controller Service Threads pools have no core size and a thread timeout of 20 secs, so if a management request comes in periodically but less often then every 20 secs (says twice a minute) the pool will continually churn threads. That kind of periodic polling is not an unusual scenario, so, we'll put in a core size of one or two.
> Original details from Rostislav:
> I did quite simple thing in the loop :
> * deploy attached application into jboss-eap-7.0/standalone/deployments/
> * wait 1 minute
> * remove anything from jboss-eap-7.0/standalone/deployments/
> * wait 1 minute
> I connected with jvisualvm and was watching monitor tab for cpu / classes / heap / threads overview statistics. After some time I spotted jumps (up and down) on threads.
> After some time I see threads named "ServerService Thread Pool -- 346", I ended with number bigger than 1400. In my case 15 threads are created for 20 seconds when deploying and another 15 threads are created for 20 seconds when undeploying. Nothing seems to be reused
> Setup for threads is done in https://github.com/wildfly/wildfly-core/blob/master/server/src/main/java/... method addService
> Is it intentional to have threads alive only 20 seconds and then scratch them ? Keeping a lot of threads means usage of system resources, creating new threads has some overhead too. Maybe the question is what is the right balance ?
> Note: It would be nice to have "Possible Bug" Issue Type for issues like this ;)
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months
[JBoss JIRA] (WFCORE-1529) New threads named ServerService Thread Pool -- xyz created with every deploy and undeploy and removed after ~20 seconds
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1529?page=com.atlassian.jira.plugi... ]
Brian Stansberry updated WFCORE-1529:
-------------------------------------
Priority: Minor (was: Major)
> New threads named ServerService Thread Pool -- xyz created with every deploy and undeploy and removed after ~20 seconds
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: WFCORE-1529
> URL: https://issues.jboss.org/browse/WFCORE-1529
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 2.1.0.Final
> Reporter: Rostislav Svoboda
> Assignee: Brian Stansberry
> Priority: Minor
>
> tl;dr version from Brian:
> The ServerService Threads and Host Controller Service Threads pools have no core size and a thread timeout of 20 secs, so if a management request comes in periodically but less often then every 20 secs (says twice a minute) the pool will continually churn threads. That kind of periodic polling is not an unusual scenario, so, we'll put in a core size of one or two.
> Original details from Rostislav:
> I did quite simple thing in the loop :
> * deploy attached application into jboss-eap-7.0/standalone/deployments/
> * wait 1 minute
> * remove anything from jboss-eap-7.0/standalone/deployments/
> * wait 1 minute
> I connected with jvisualvm and was watching monitor tab for cpu / classes / heap / threads overview statistics. After some time I spotted jumps (up and down) on threads.
> After some time I see threads named "ServerService Thread Pool -- 346", I ended with number bigger than 1400. In my case 15 threads are created for 20 seconds when deploying and another 15 threads are created for 20 seconds when undeploying. Nothing seems to be reused
> Setup for threads is done in https://github.com/wildfly/wildfly-core/blob/master/server/src/main/java/... method addService
> Is it intentional to have threads alive only 20 seconds and then scratch them ? Keeping a lot of threads means usage of system resources, creating new threads has some overhead too. Maybe the question is what is the right balance ?
> Note: It would be nice to have "Possible Bug" Issue Type for issues like this ;)
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months
[JBoss JIRA] (WFCORE-1529) New threads named ServerService Thread Pool -- xyz created with every deploy and undeploy and removed after ~20 seconds
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1529?page=com.atlassian.jira.plugi... ]
Brian Stansberry moved JBEAP-4546 to WFCORE-1529:
-------------------------------------------------
Project: WildFly Core (was: JBoss Enterprise Application Platform)
Key: WFCORE-1529 (was: JBEAP-4546)
Workflow: GIT Pull Request workflow (was: CDW with loose statuses v1)
Component/s: Domain Management
(was: Domain Management)
Target Release: (was: 7.backlog.GA)
Affects Version/s: 2.1.0.Final
(was: 7.0.0.CR2)
> New threads named ServerService Thread Pool -- xyz created with every deploy and undeploy and removed after ~20 seconds
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: WFCORE-1529
> URL: https://issues.jboss.org/browse/WFCORE-1529
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management
> Affects Versions: 2.1.0.Final
> Reporter: Rostislav Svoboda
> Assignee: Brian Stansberry
>
> I did quite simple thing in the loop :
> * deploy attached application into jboss-eap-7.0/standalone/deployments/
> * wait 1 minute
> * remove anything from jboss-eap-7.0/standalone/deployments/
> * wait 1 minute
> I connected with jvisualvm and was watching monitor tab for cpu / classes / heap / threads overview statistics. After some time I spotted jumps (up and down) on threads.
> After some time I see threads named "ServerService Thread Pool -- 346", I ended with number bigger than 1400. In my case 15 threads are created for 20 seconds when deploying and another 15 threads are created for 20 seconds when undeploying. Nothing seems to be reused
> Setup for threads is done in https://github.com/wildfly/wildfly-core/blob/master/server/src/main/java/... method addService
> Is it intentional to have threads alive only 20 seconds and then scratch them ? Keeping a lot of threads means usage of system resources, creating new threads has some overhead too. Maybe the question is what is the right balance ?
> Note: It would be nice to have "Possible Bug" Issue Type for issues like this ;)
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months
[JBoss JIRA] (WFCORE-1528) EAP 7 with secured management become unavailable after ~8 days
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1528?page=com.atlassian.jira.plugi... ]
Brian Stansberry commented on WFCORE-1528:
------------------------------------------
I believe WFCORE-1428 is cause of WFCORE-1528, as it brings a non-fixed-size thread pool into Undertow's request handling, and Undertow's internal caching mechanisms are not designed for use with a non-fixed pool. As the pool churns threads, pooled ByteBuffers are not getting freed.
It's also better to use a fixed pool for the WFCORE-1428 work as this call path is for end-user requests and we want to limit concurrency of those a la what we do for native clients at https://github.com/wildfly/wildfly-core/blob/master/controller/src/main/j....
> EAP 7 with secured management become unavailable after ~8 days
> --------------------------------------------------------------
>
> Key: WFCORE-1528
> URL: https://issues.jboss.org/browse/WFCORE-1528
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Security
> Affects Versions: 2.1.0.Final
> Environment: EAP 7.0.0.CR01 2-way ssl over remote https communication
> Reporter: Simeon Pinder
> Assignee: Brian Stansberry
> Priority: Critical
> Fix For: 3.0.0.Alpha1
>
>
> Appears there may be an EAP 7.0.0.CR01 2-way ssl issue in domain mode.
> Issue discovered by JON QE after testing for 8 days with SSL.
> JON QE did not see the same issue when running identical test against EAP 6.
> Restarting the remote client(JON Agent) did not repair/fix the issue, and only was repaired after EAP instance restarted.
> See BZ https://bugzilla.redhat.com/show_bug.cgi?id=1330180#c0 for more details.
> JON team would like some evaluation done to determine:
> i)if this is EAP 7 issue or
> ii)somehow just a JON issue
> iii)suggestions on what to do when we try to reproduce properly or ways to shorten reproduce cycle.
> This issue could affect GA release schedules.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months
[JBoss JIRA] (WFCORE-1528) EAP 7 with secured management become unavailable after ~8 days
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1528?page=com.atlassian.jira.plugi... ]
Brian Stansberry updated WFCORE-1528:
-------------------------------------
Priority: Critical (was: Major)
> EAP 7 with secured management become unavailable after ~8 days
> --------------------------------------------------------------
>
> Key: WFCORE-1528
> URL: https://issues.jboss.org/browse/WFCORE-1528
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Security
> Affects Versions: 2.1.0.Final
> Environment: EAP 7.0.0.CR01 2-way ssl over remote https communication
> Reporter: Simeon Pinder
> Assignee: Brian Stansberry
> Priority: Critical
> Fix For: 3.0.0.Alpha1
>
>
> Appears there may be an EAP 7.0.0.CR01 2-way ssl issue in domain mode.
> Issue discovered by JON QE after testing for 8 days with SSL.
> JON QE did not see the same issue when running identical test against EAP 6.
> Restarting the remote client(JON Agent) did not repair/fix the issue, and only was repaired after EAP instance restarted.
> See BZ https://bugzilla.redhat.com/show_bug.cgi?id=1330180#c0 for more details.
> JON team would like some evaluation done to determine:
> i)if this is EAP 7 issue or
> ii)somehow just a JON issue
> iii)suggestions on what to do when we try to reproduce properly or ways to shorten reproduce cycle.
> This issue could affect GA release schedules.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months
[JBoss JIRA] (WFCORE-1528) EAP 7 with secured management become unavailable after ~8 days
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1528?page=com.atlassian.jira.plugi... ]
Brian Stansberry updated WFCORE-1528:
-------------------------------------
Fix Version/s: 3.0.0.Alpha1
> EAP 7 with secured management become unavailable after ~8 days
> --------------------------------------------------------------
>
> Key: WFCORE-1528
> URL: https://issues.jboss.org/browse/WFCORE-1528
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Security
> Affects Versions: 2.1.0.Final
> Environment: EAP 7.0.0.CR01 2-way ssl over remote https communication
> Reporter: Simeon Pinder
> Assignee: Brian Stansberry
> Priority: Critical
> Fix For: 3.0.0.Alpha1
>
>
> Appears there may be an EAP 7.0.0.CR01 2-way ssl issue in domain mode.
> Issue discovered by JON QE after testing for 8 days with SSL.
> JON QE did not see the same issue when running identical test against EAP 6.
> Restarting the remote client(JON Agent) did not repair/fix the issue, and only was repaired after EAP instance restarted.
> See BZ https://bugzilla.redhat.com/show_bug.cgi?id=1330180#c0 for more details.
> JON team would like some evaluation done to determine:
> i)if this is EAP 7 issue or
> ii)somehow just a JON issue
> iii)suggestions on what to do when we try to reproduce properly or ways to shorten reproduce cycle.
> This issue could affect GA release schedules.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months
[JBoss JIRA] (WFCORE-1528) EAP 7 with secured management become unavailable after ~8 days
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-1528?page=com.atlassian.jira.plugi... ]
Brian Stansberry moved JBEAP-4545 to WFCORE-1528:
-------------------------------------------------
Project: WildFly Core (was: JBoss Enterprise Application Platform)
Key: WFCORE-1528 (was: JBEAP-4545)
Workflow: GIT Pull Request workflow (was: CDW with loose statuses v1)
Component/s: Domain Management
Security
(was: Domain Management)
(was: Security)
Target Release: (was: 7.backlog.GA)
Affects Version/s: 2.1.0.Final
(was: 7.0.0.CR1)
> EAP 7 with secured management become unavailable after ~8 days
> --------------------------------------------------------------
>
> Key: WFCORE-1528
> URL: https://issues.jboss.org/browse/WFCORE-1528
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Security
> Affects Versions: 2.1.0.Final
> Environment: EAP 7.0.0.CR01 2-way ssl over remote https communication
> Reporter: Simeon Pinder
> Assignee: Brian Stansberry
>
> Appears there may be an EAP 7.0.0.CR01 2-way ssl issue in domain mode.
> Issue discovered by JON QE after testing for 8 days with SSL.
> JON QE did not see the same issue when running identical test against EAP 6.
> Restarting the remote client(JON Agent) did not repair/fix the issue, and only was repaired after EAP instance restarted.
> See BZ https://bugzilla.redhat.com/show_bug.cgi?id=1330180#c0 for more details.
> JON team would like some evaluation done to determine:
> i)if this is EAP 7 issue or
> ii)somehow just a JON issue
> iii)suggestions on what to do when we try to reproduce properly or ways to shorten reproduce cycle.
> This issue could affect GA release schedules.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months
[JBoss JIRA] (WFCORE-1527) Server group / JVM composite add op and minssing parent
by Ken Wills (JIRA)
Ken Wills created WFCORE-1527:
---------------------------------
Summary: Server group / JVM composite add op and minssing parent
Key: WFCORE-1527
URL: https://issues.jboss.org/browse/WFCORE-1527
Project: WildFly Core
Issue Type: Bug
Components: Domain Management
Reporter: Ken Wills
Assignee: Ken Wills
The test AutoIgnoreDomain tests when adding an ignored server group, like this:
https://github.com/luck3y/wildfly-core/blob/6cdd4af55b7723ff943a3743143a6...
Also calld the JvmAddHandler. During the JVMH add, the prior add of the server group is not visible in the model and the op fails.
Workaround in JVMAddHandler.populateModel, to attempt to RRFR, and fail silently if the parent is not available, but something prior in the chain is violating the readDomainFromRoot contract, as the server element should be a visible, but uncommitted change in the model.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
8 years, 2 months