[JBoss JIRA] (WFLY-9397) Once hitting the max-connections limit, new connections are not accepted any more and CLOSE_WAIT connections increase gradually
by Masafumi Miura (JIRA)
[ https://issues.jboss.org/browse/WFLY-9397?page=com.atlassian.jira.plugin.... ]
Masafumi Miura updated WFLY-9397:
---------------------------------
Steps to Reproduce:
# Set {{max-connections}} attribute in listener and Start WildFly:
{code}
<http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true" max-connections="10"/>
{code}
# Sent concurrent requests which can overflow the max-connections limit:
{code}
ab -c 100 -n 100 http://localhost:8080/
{code}
You will see most requests will be failed with "The timeout specified has expired". Once the above command hit the max-connections limit, WildFly does not accept new connection any more even after all request processing are completed.
- Try {{curl}} command to confirm you can not connect:
{code}
curl -v http://localhost:8080/
{code}
- Check an output of {{netstat}}. {{CLOSE_WAIT}} will be increased one by one when you execute curl command and cancel it:
{code}
netstat -tunapl | grep 8080
{code}
was:
# Set {{max-connections}} attribute in listener and Start JBoss:
{code}
<http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true" max-connections="10"/>
{code}
# Sent concurrent requests which can overflow the max-connections limit:
{code}
ab -c 100 -n 100 http://localhost:8080/
{code}
You will see most requests will be failed with "The timeout specified has expired". Once the above command hit the max-connections limit, JBoss does not accept new connection any more even after all request processing are completed.
- Try {{curl}} command to confirm you can not connect:
{code}
curl -v http://localhost:8080/
{code}
- Check an output of {{netstat}}. {{CLOSE_WAIT}} will be increased one by one when you execute curl command and cancel it:
{code}
netstat -tunapl | grep 8080
{code}
> Once hitting the max-connections limit, new connections are not accepted any more and CLOSE_WAIT connections increase gradually
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-9397
> URL: https://issues.jboss.org/browse/WFLY-9397
> Project: WildFly
> Issue Type: Bug
> Components: Web (Undertow)
> Affects Versions: 11.0.0.CR1
> Reporter: Masafumi Miura
> Assignee: David Lloyd
>
> Once hitting the {{max-connections}} limit of listener, JBoss EAP does not accept new connections any more. Even after all request processing are completed, JBoss does not accept. So, any client can not connect to JBoss EAP.
> In addition, when a client send a new request and close the connection from client-side due to timeout or cancellation in such situation, CLOSE_WAIT connections remain until JBoss Java process is stopped.
> I think this issue is related to XNIO-276 and XNIO-279, which both fixes are merged in XNIO 3.4 branch but it's not merged (or partially merged) in XNIO 3.5 branch:
> - XNIO-276 - If QueuedNioTcpServer hits the max connection limit it gets stuck in an infinite loop
> This issue is fixed by the two commits [6a5d04e|https://github.com/xnio/xnio/commit/6a5d04e3be98ee68a9fb1f9f98eb4...] and [8c0af87|https://github.com/xnio/xnio/commit/8c0af87f5dfdee96af2e3000e47e6...] which were merged in 3.4 branch. However, 8c0af87 is not included in 3.5 branch.
> - XNIO-279 - QueuedNioTcpServer can go into an infinite loop if accept fails
> This issue is fixed by the commit [86d0f6c|https://github.com/xnio/xnio/commit/86d0f6cc4bc41443b7e9442989e56...] in 3.4 branch. However, this is not included in 3.5 branch.
> Note: JBEAP-6397 and JBEAP-8080 were raised to track the issues for 7.1.0. I think both are marked as resolved because 7.1.0 DR came with XNIO 3.4.x at that time. However, the latest 7.1.0 CR now comes with XNIO 3.5.x, so it regresses in the latest release.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 6 months
[JBoss JIRA] (WFLY-9397) Once hitting the max-connections limit, new connections are not accepted any more and CLOSE_WAIT connections increase gradually
by Masafumi Miura (JIRA)
[ https://issues.jboss.org/browse/WFLY-9397?page=com.atlassian.jira.plugin.... ]
Masafumi Miura moved JBEAP-13284 to WFLY-9397:
----------------------------------------------
Project: WildFly (was: JBoss Enterprise Application Platform)
Key: WFLY-9397 (was: JBEAP-13284)
Workflow: GIT Pull Request workflow (was: CDW with loose statuses v1)
Component/s: Web (Undertow)
(was: Web (Undertow))
Affects Version/s: 11.0.0.CR1
(was: 7.1.0.CR1)
(was: 7.1.0.CR2)
> Once hitting the max-connections limit, new connections are not accepted any more and CLOSE_WAIT connections increase gradually
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-9397
> URL: https://issues.jboss.org/browse/WFLY-9397
> Project: WildFly
> Issue Type: Bug
> Components: Web (Undertow)
> Affects Versions: 11.0.0.CR1
> Reporter: Masafumi Miura
> Assignee: David Lloyd
>
> Once hitting the {{max-connections}} limit of listener, JBoss EAP does not accept new connections any more. Even after all request processing are completed, JBoss does not accept. So, any client can not connect to JBoss EAP.
> In addition, when a client send a new request and close the connection from client-side due to timeout or cancellation in such situation, CLOSE_WAIT connections remain until JBoss Java process is stopped.
> I think this issue is related to XNIO-276 and XNIO-279, which both fixes are merged in XNIO 3.4 branch but it's not merged (or partially merged) in XNIO 3.5 branch:
> - XNIO-276 - If QueuedNioTcpServer hits the max connection limit it gets stuck in an infinite loop
> This issue is fixed by the two commits [6a5d04e|https://github.com/xnio/xnio/commit/6a5d04e3be98ee68a9fb1f9f98eb4...] and [8c0af87|https://github.com/xnio/xnio/commit/8c0af87f5dfdee96af2e3000e47e6...] which were merged in 3.4 branch. However, 8c0af87 is not included in 3.5 branch.
> - XNIO-279 - QueuedNioTcpServer can go into an infinite loop if accept fails
> This issue is fixed by the commit [86d0f6c|https://github.com/xnio/xnio/commit/86d0f6cc4bc41443b7e9442989e56...] in 3.4 branch. However, this is not included in 3.5 branch.
> Note: JBEAP-6397 and JBEAP-8080 were raised to track the issues for 7.1.0. I think both are marked as resolved because 7.1.0 DR came with XNIO 3.4.x at that time. However, the latest 7.1.0 CR now comes with XNIO 3.5.x, so it regresses in the latest release.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 6 months
[JBoss JIRA] (WFLY-9262) adding undertow using the cli requires a batch
by Alexey Loubyansky (JIRA)
[ https://issues.jboss.org/browse/WFLY-9262?page=com.atlassian.jira.plugin.... ]
Alexey Loubyansky closed WFLY-9262.
-----------------------------------
Resolution: Won't Fix
This issue is not relevant to me any more. Unless there are others seeing a reason to do something about this, I'm closing it as "won't fix".
> adding undertow using the cli requires a batch
> ----------------------------------------------
>
> Key: WFLY-9262
> URL: https://issues.jboss.org/browse/WFLY-9262
> Project: WildFly
> Issue Type: Enhancement
> Components: Web (Undertow)
> Affects Versions: 11.0.0.Beta1
> Reporter: Alexey Loubyansky
> Assignee: Tomaz Cerar
> Fix For: 12.0.0.Alpha1
>
>
> Undertow appears to be the only subsystem now requiring a composite operation when it is added to the domain management model using management operations from a management client, e.g. cli.
> The exact error is
> {quote}
> [standalone@embedded /] /subsystem=undertow:add
> {
> "outcome" => "failed",
> "failure-description" => "WFLYCTL0369: Required capabilities are not available:
> org.wildfly.undertow.host.default-server.default-host; Possible registration points for this capability:
> /subsystem=undertow/server=*/host=*
> org.wildfly.undertow.server.default-server; Possible registration points for this capability:
> /subsystem=undertow/server=*
> org.wildfly.undertow.servlet-container.default; Possible registration points for this capability:
> /subsystem=undertow/servlet-container=*",
> "rolled-back" => true,
> "response-headers" => {"process-state" => "reload-required"}
> }
> {quote}
> The sequence that actually works is
> {quote}
> batch
> /subsystem=undertow:add
> /subsystem=undertow/buffer-cache=default:add
> /subsystem=undertow/configuration=filter:add
> /subsystem=undertow/configuration=handler:add
> /subsystem=undertow/server=default-server:add
> /subsystem=undertow/server=default-server/host=default-host:add(alias=["localhost"])
> /subsystem=undertow/server=default-server/http-listener=default:add(socket-binding="http",redirect-socket="https",enable-http2="true")
> /subsystem=undertow/servlet-container=default:add
> run-batch
> {quote}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 6 months
[JBoss JIRA] (WFLY-9370) Fix failing SingletonTunnelTestCase(ASYNC-tunnel)#testSingletonService
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-9370?page=com.atlassian.jira.plugin.... ]
Radoslav Husar updated WFLY-9370:
---------------------------------
Fix Version/s: 11.0.0.Final
> Fix failing SingletonTunnelTestCase(ASYNC-tunnel)#testSingletonService
> -----------------------------------------------------------------------
>
> Key: WFLY-9370
> URL: https://issues.jboss.org/browse/WFLY-9370
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 11.0.0.CR1
> Reporter: Radoslav Husar
> Assignee: Radoslav Husar
> Fix For: 11.0.0.Final
>
>
> The problem is that this test is not run as part of regular CI execution thus is destined to be broken. The test should be moved to the right location and renamed (i.e. drop tunnel since that has nothing to do with what it is testing).
> {noformat}
> &#27;[0m&#27;[31m04:44:19,235 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 1) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "singleton.war")]) - failure description: {
> "WFLYCTL0412: Required services that are not installed:" => ["jboss.clustering.group.default"],
> "WFLYCTL0180: Services with missing/unavailable dependencies" => [
> "jboss.test1.service.default is missing [jboss.clustering.group.default]",
> "jboss.test2.service.default is missing [jboss.clustering.group.default]"
> ],
> "WFLYCTL0288: One or more services were unable to start due to one or more indirect dependencies not being available." => {
> "Services that were unable to start:" => [
> "jboss.deployment.discovery.\"singleton.war\"",
> "jboss.deployment.unit.\"singleton.war\".CdiValidatorFactoryService",
> "jboss.deployment.unit.\"singleton.war\".WeldStartService",
> "jboss.deployment.unit.\"singleton.war\".component.TopologyChangeListenerBean.JndiBindingsService",
> "jboss.deployment.unit.\"singleton.war\".component.TopologyChangeListenerBean.START",
> "jboss.deployment.unit.\"singleton.war\".component.TopologyChangeListenerBean.VIEW.\"org.jboss.as.test.clustering.TopologyChangeListener\".REMOTE",
> "jboss.deployment.unit.\"singleton.war\".component.TopologyChangeListenerBean.WeldInstantiator",
> "jboss.deployment.unit.\"singleton.war\".component.TopologyChangeListenerBean.WeldInterceptorBindingsService",
> "jboss.deployment.unit.\"singleton.war\".component.\"com.sun.faces.config.ConfigureListener\".START",
> "jboss.deployment.unit.\"singleton.war\".component.\"com.sun.faces.config.ConfigureListener\".WeldInstantiator",
> "jboss.deployment.unit.\"singleton.war\".component.\"javax.faces.webapp.FacetTag\".START",
> "jboss.deployment.unit.\"singleton.war\".component.\"javax.faces.webapp.FacetTag\".WeldInstantiator",
> "jboss.deployment.unit.\"singleton.war\".component.\"javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV\".START",
> "jboss.deployment.unit.\"singleton.war\".component.\"javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV\".WeldInstantiator",
> "jboss.deployment.unit.\"singleton.war\".component.\"javax.servlet.jsp.jstl.tlv.ScriptFreeTLV\".START",
> "jboss.deployment.unit.\"singleton.war\".component.\"javax.servlet.jsp.jstl.tlv.ScriptFreeTLV\".WeldInstantiator",
> "jboss.deployment.unit.\"singleton.war\".component.\"org.jboss.as.test.clustering.TopologyChangeListenerServlet\".START",
> "jboss.deployment.unit.\"singleton.war\".component.\"org.jboss.as.test.clustering.TopologyChangeListenerServlet\".WeldInstantiator",
> "jboss.deployment.unit.\"singleton.war\".component.\"org.jboss.as.test.clustering.cluster.singleton.service.NodeServiceServlet\".START",
> "jboss.deployment.unit.\"singleton.war\".component.\"org.jboss.as.test.clustering.cluster.singleton.service.NodeServiceServlet\".WeldInstantiator",
> "jboss.deployment.unit.\"singleton.war\".component.\"org.jboss.as.test.clustering.cluster.singleton.service.ValueServiceServlet\".START",
> "jboss.deployment.unit.\"singleton.war\".component.\"org.jboss.as.test.clustering.cluster.singleton.service.ValueServiceServlet\".WeldInstantiator",
> "jboss.deployment.unit.\"singleton.war\".component.\"org.jboss.weld.servlet.WeldInitialListener\".START",
> "jboss.deployment.unit.\"singleton.war\".component.\"org.jboss.weld.servlet.WeldInitialListener\".WeldInstantiator",
> "jboss.deployment.unit.\"singleton.war\".component.\"org.jboss.weld.servlet.WeldTerminalListener\".START",
> "jboss.deployment.unit.\"singleton.war\".component.\"org.jboss.weld.servlet.WeldTerminalListener\".WeldInstantiator",
> "jboss.deployment.unit.\"singleton.war\".deploymentCompleteService",
> "jboss.deployment.unit.\"singleton.war\".ejb3.client-context.registration-service",
> "jboss.deployment.unit.\"singleton.war\".jndiDependencyService",
> "jboss.deployment.unit.\"singleton.war\".moduleDeploymentRuntimeInformation",
> "jboss.deployment.unit.\"singleton.war\".moduleDeploymentRuntimeInformationStart",
> "jboss.naming.context.java.app.singleton.singleton.TopologyChangeListenerBean",
> "jboss.naming.context.java.app.singleton.singleton.\"TopologyChangeListenerBean!org.jboss.as.test.clustering.TopologyChangeListener\"",
> "jboss.naming.context.java.global.singleton.TopologyChangeListenerBean",
> "jboss.naming.context.java.global.singleton.\"TopologyChangeListenerBean!org.jboss.as.test.clustering.TopologyChangeListener\"",
> "jboss.naming.context.java.module.singleton.singleton.TopologyChangeListenerBean",
> "jboss.naming.context.java.module.singleton.singleton.\"TopologyChangeListenerBean!org.jboss.as.test.clustering.TopologyChangeListener\"",
> "jboss.undertow.deployment.default-server.default-host./singleton",
> "jboss.undertow.deployment.default-server.default-host./singleton.UndertowDeploymentInfoService"
> ],
> "Services that may be the cause:" => [
> "jboss.clustering.group.default",
> "org.wildfly.network.socket-binding.undefined"
> ]
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 6 months
[JBoss JIRA] (WFLY-9371) Fix failing RemoteEJBTwoClusterTestCase
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-9371?page=com.atlassian.jira.plugin.... ]
Radoslav Husar updated WFLY-9371:
---------------------------------
Fix Version/s: 11.0.0.Final
> Fix failing RemoteEJBTwoClusterTestCase
> ---------------------------------------
>
> Key: WFLY-9371
> URL: https://issues.jboss.org/browse/WFLY-9371
> Project: WildFly
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Radoslav Husar
> Fix For: 11.0.0.Final
>
>
> The problem is that this test is not run as part of regular CI execution thus is destined to be broken. The test should be run as part of regular CI set of tests; if intermittently failing then it should be @Ignore-d in the way intermittently failing tests are.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 6 months
[JBoss JIRA] (WFLY-9371) Fix failing RemoteEJBTwoClusterTestCase
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-9371?page=com.atlassian.jira.plugin.... ]
Radoslav Husar updated WFLY-9371:
---------------------------------
Component/s: Test Suite
> Fix failing RemoteEJBTwoClusterTestCase
> ---------------------------------------
>
> Key: WFLY-9371
> URL: https://issues.jboss.org/browse/WFLY-9371
> Project: WildFly
> Issue Type: Bug
> Components: Clustering, Test Suite
> Affects Versions: 10.1.0.Final
> Reporter: Radoslav Husar
> Assignee: Radoslav Husar
> Fix For: 11.0.0.Final
>
>
> The problem is that this test is not run as part of regular CI execution thus is destined to be broken. The test should be run as part of regular CI set of tests; if intermittently failing then it should be @Ignore-d in the way intermittently failing tests are.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 6 months
[JBoss JIRA] (WFLY-9396) Improve EJB client clustering test coverage tracker
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-9396?page=com.atlassian.jira.plugin.... ]
Radoslav Husar updated WFLY-9396:
---------------------------------
Description:
A bunch of areas have been analyzed that need better test coverage.
* fix and move to proper execution EJB2 and two cluster test cases
* coverage for reported Jira issues (e.g. WFLY-9358 (PR already sent), JBEAP-13224/EJBCLIENT-273)
* coverage for different client configuration methods
* port testcases from https://github.com/wfink/jboss-eap7.1-playground
* etc
was:
A bunch of areas have been analyzed that need better test coverage.
* fix and move to proper execution EJB2 and two cluster test cases
* coverage for reported Jira issues (e.g. WFLY-9358 (PR sent), JBEAP-13224/EJBCLIENT-273)
* coverage for different client configuration methods
* port testcases from https://github.com/wfink/jboss-eap7.1-playground
* etc
> Improve EJB client clustering test coverage tracker
> ---------------------------------------------------
>
> Key: WFLY-9396
> URL: https://issues.jboss.org/browse/WFLY-9396
> Project: WildFly
> Issue Type: Task
> Components: Test Suite
> Affects Versions: 11.0.0.CR1
> Reporter: Radoslav Husar
> Assignee: Radoslav Husar
>
> A bunch of areas have been analyzed that need better test coverage.
> * fix and move to proper execution EJB2 and two cluster test cases
> * coverage for reported Jira issues (e.g. WFLY-9358 (PR already sent), JBEAP-13224/EJBCLIENT-273)
> * coverage for different client configuration methods
> * port testcases from https://github.com/wfink/jboss-eap7.1-playground
> * etc
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 6 months
[JBoss JIRA] (WFLY-9396) Improve EJB client clustering test coverage tracker
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-9396?page=com.atlassian.jira.plugin.... ]
Radoslav Husar updated WFLY-9396:
---------------------------------
Description:
A bunch of areas have been analyzed that need better test coverage.
* fix and move to proper execution EJB2 and two cluster test cases
* coverage for reported Jira issues (e.g. WFLY-9358 (PR sent), JBEAP-13224/EJBCLIENT-273)
* coverage for different client configuration methods
* port testcases from https://github.com/wfink/jboss-eap7.1-playground
* etc
was:
A bunch of areas have been analyzed that need better test coverage.
* fix and move to proper execution EJB2 and two cluster test cases
* coverage for reported Jira issues (e.g. JBEAP-13224/EJBCLIENT-273)
* coverage for different client configuration methods
* port testcases from https://github.com/wfink/jboss-eap7.1-playground
* etc
> Improve EJB client clustering test coverage tracker
> ---------------------------------------------------
>
> Key: WFLY-9396
> URL: https://issues.jboss.org/browse/WFLY-9396
> Project: WildFly
> Issue Type: Task
> Components: Test Suite
> Affects Versions: 11.0.0.CR1
> Reporter: Radoslav Husar
> Assignee: Radoslav Husar
>
> A bunch of areas have been analyzed that need better test coverage.
> * fix and move to proper execution EJB2 and two cluster test cases
> * coverage for reported Jira issues (e.g. WFLY-9358 (PR sent), JBEAP-13224/EJBCLIENT-273)
> * coverage for different client configuration methods
> * port testcases from https://github.com/wfink/jboss-eap7.1-playground
> * etc
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 6 months
[JBoss JIRA] (WFLY-9396) Improve EJB client clustering test coverage tracker
by Radoslav Husar (JIRA)
[ https://issues.jboss.org/browse/WFLY-9396?page=com.atlassian.jira.plugin.... ]
Radoslav Husar updated WFLY-9396:
---------------------------------
Description:
A bunch of areas have been analyzed that need better test coverage.
* fix and move to proper execution EJB2 and two cluster test cases
* coverage for reported Jira issues (e.g. JBEAP-13224/EJBCLIENT-273)
* coverage for different client configuration methods
* port testcases from https://github.com/wfink/jboss-eap7.1-playground
* etc
was:
A bunch of areas have been analyzed that need better test coverage.
* fix and move to proper execution EJB2 and two cluster test cases
* coverage for reported Jira issues (e.g. JBEAP-13224/EJBCLIENT-273)
* coverage for different client configuration methods
* etc
> Improve EJB client clustering test coverage tracker
> ---------------------------------------------------
>
> Key: WFLY-9396
> URL: https://issues.jboss.org/browse/WFLY-9396
> Project: WildFly
> Issue Type: Task
> Components: Test Suite
> Affects Versions: 11.0.0.CR1
> Reporter: Radoslav Husar
> Assignee: Radoslav Husar
>
> A bunch of areas have been analyzed that need better test coverage.
> * fix and move to proper execution EJB2 and two cluster test cases
> * coverage for reported Jira issues (e.g. JBEAP-13224/EJBCLIENT-273)
> * coverage for different client configuration methods
> * port testcases from https://github.com/wfink/jboss-eap7.1-playground
> * etc
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
6 years, 6 months