[JBoss JIRA] (WFLY-4397) WildFly.8.1.0.Final's Asynchronous servlets cause connection leak

Monday, 9 March 2015

    [
https://issues.jboss.org/browse/WFLY-4397?page=com.atlassian.jira.plugin....
] 

Wind Wild commented on WFLY-4397:
---------------------------------

Hi,Stuart Douglas
Thanks for your reply firstly.
My  jmeter script as following:

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="2.6" jmeter="2.11
r1554548">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan"
testname="Test Plan" enabled="true">
      <stringProp name="TestPlan.comments"></stringProp>
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <boolProp
name="TestPlan.serialize_threadgroups">false</boolProp>
      <elementProp name="TestPlan.user_defined_variables"
elementType="Arguments" guiclass="ArgumentsPanel"
testclass="Arguments" testname="User Defined Variables"
enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <stringProp
name="TestPlan.user_define_classpath"></stringProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui"
testclass="ThreadGroup" testname="Thread Group"
enabled="true">
        <stringProp
name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller"
elementType="LoopController" guiclass="LoopControlPanel"
testclass="LoopController" testname="Loop Controller"
enabled="true">
          <boolProp
name="LoopController.continue_forever">false</boolProp>
          <intProp name="LoopController.loops">-1</intProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">50</stringProp>
        <stringProp name="ThreadGroup.ramp_time"></stringProp>
        <longProp
name="ThreadGroup.start_time">1425010031000</longProp>
        <longProp
name="ThreadGroup.end_time">1425010031000</longProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
      </ThreadGroup>
      <hashTree>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui"
testclass="HTTPSamplerProxy" testname="HTTP Request"
enabled="true">
          <elementProp name="HTTPsampler.Arguments"
elementType="Arguments" guiclass="HTTPArgumentsPanel"
testclass="Arguments" testname="User Defined Variables"
enabled="true">
            <collectionProp name="Arguments.arguments"/>
          </elementProp>
          <stringProp
name="HTTPSampler.domain">10.19.220.172</stringProp>
          <stringProp name="HTTPSampler.port">8080</stringProp>
          <stringProp
name="HTTPSampler.connect_timeout"></stringProp>
          <stringProp
name="HTTPSampler.response_timeout"></stringProp>
          <stringProp name="HTTPSampler.protocol"></stringProp>
          <stringProp
name="HTTPSampler.contentEncoding"></stringProp>
          <stringProp
name="HTTPSampler.path">/payload/asyncDemoServlet</stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <boolProp
name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp
name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp
name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp
name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <boolProp name="HTTPSampler.monitor">false</boolProp>
          <stringProp
name="HTTPSampler.embedded_url_re"></stringProp>
        </HTTPSamplerProxy>
        <hashTree/>
        <ResultCollector guiclass="StatVisualizer"
testclass="ResultCollector" testname="Aggregate Report"
enabled="true">
          <boolProp
name="ResultCollector.error_logging">false</boolProp>
          <objProp>
            <name>saveConfig</name>
            <value class="SampleSaveConfiguration">
              <time>true</time>
              <latency>true</latency>
              <timestamp>true</timestamp>
              <success>true</success>
              <label>true</label>
              <code>true</code>
              <message>true</message>
              <threadName>true</threadName>
              <dataType>true</dataType>
              <encoding>false</encoding>
              <assertions>true</assertions>
              <subresults>true</subresults>
              <responseData>false</responseData>
              <samplerData>false</samplerData>
              <xml>false</xml>
              <fieldNames>false</fieldNames>
              <responseHeaders>false</responseHeaders>
              <requestHeaders>false</requestHeaders>
              <responseDataOnError>false</responseDataOnError>

<saveAssertionResultsFailureMessage>false</saveAssertionResultsFailureMessage>
              <assertionsResultsToSave>0</assertionsResultsToSave>
              <bytes>true</bytes>
            </value>
          </objProp>
          <stringProp name="filename"></stringProp>
        </ResultCollector>
        <hashTree/>
        <ResultCollector guiclass="SummaryReport"
testclass="ResultCollector" testname="Summary Report"
enabled="true">
          <boolProp
name="ResultCollector.error_logging">false</boolProp>
          <objProp>
            <name>saveConfig</name>
            <value class="SampleSaveConfiguration">
              <time>true</time>
              <latency>true</latency>
              <timestamp>true</timestamp>
              <success>true</success>
              <label>true</label>
              <code>true</code>
              <message>true</message>
              <threadName>true</threadName>
              <dataType>true</dataType>
              <encoding>false</encoding>
              <assertions>true</assertions>
              <subresults>true</subresults>
              <responseData>false</responseData>
              <samplerData>false</samplerData>
              <xml>false</xml>
              <fieldNames>false</fieldNames>
              <responseHeaders>false</responseHeaders>
              <requestHeaders>false</requestHeaders>
              <responseDataOnError>false</responseDataOnError>

<saveAssertionResultsFailureMessage>false</saveAssertionResultsFailureMessage>
              <assertionsResultsToSave>0</assertionsResultsToSave>
              <bytes>true</bytes>
            </value>
          </objProp>
          <stringProp name="filename"></stringProp>
        </ResultCollector>
        <hashTree/>
        <ResultCollector guiclass="ViewResultsFullVisualizer"
testclass="ResultCollector" testname="View Results Tree"
enabled="true">
          <boolProp
name="ResultCollector.error_logging">false</boolProp>
          <objProp>
            <name>saveConfig</name>
            <value class="SampleSaveConfiguration">
              <time>true</time>
              <latency>true</latency>
              <timestamp>true</timestamp>
              <success>true</success>
              <label>true</label>
              <code>true</code>
              <message>true</message>
              <threadName>true</threadName>
              <dataType>true</dataType>
              <encoding>false</encoding>
              <assertions>true</assertions>
              <subresults>true</subresults>
              <responseData>false</responseData>
              <samplerData>false</samplerData>
              <xml>false</xml>
              <fieldNames>false</fieldNames>
              <responseHeaders>false</responseHeaders>
              <requestHeaders>false</requestHeaders>
              <responseDataOnError>false</responseDataOnError>

<saveAssertionResultsFailureMessage>false</saveAssertionResultsFailureMessage>
              <assertionsResultsToSave>0</assertionsResultsToSave>
              <bytes>true</bytes>
            </value>
          </objProp>
          <stringProp name="filename"></stringProp>
        </ResultCollector>
        <hashTree/>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

you can test again,when you stop the request,you can watch the file-hander use command
"lsof -p pid | can't identify protocol | wc -l",maybe you can see there are
lots of "can't identify protocol".

I'm looking forward to your reply,thank you!

...
 WildFly.8.1.0.Final's Asynchronous servlets cause connection
leak
 -----------------------------------------------------------------

                 Key: WFLY-4397
                 URL: https://issues.jboss.org/browse/WFLY-4397
             Project: WildFly
          Issue Type: Bug
          Components: Web (Undertow)
    Affects Versions: 8.1.0.Final, 8.2.0.Final, 9.0.0.Alpha1
         Environment: Java(TM) SE Runtime Environment (build 1.7.0_25)
            Reporter: Wind Wild
            Assignee: Stuart Douglas

 Recently, I used Asynchronous servlets and AsyncListeners on my project.But when it runs
on WildFly.8.1.0.Final,a problem appeared as following:

 2015-01-10 10:08:40,936 ERROR [org.jboss.as.server.deployment.scanner]
(DeploymentScanner-threads - 2) JBAS015004: Caught exception writing deployment marker
file /opt/jboss/standalone/deployments/OpenESBHttp_full_prd.war.isundeploying:
java.io.FileNotFoundException:
/opt/jboss/standalone/deployments/OpenESBHttp_full_prd.war.isundeploying (Too many open
files)  
 at java.io.FileOutputStream.open(Native Method) [rt.jar:1.7.0_25]  
 at java.io.FileOutputStream.<init>(FileOutputStream.java:212) [rt.jar:1.7.0_25]  
 at java.io.FileOutputStream.<init>(FileOutputStream.java:165) [rt.jar:1.7.0_25]  
 at
org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.createMarkerFile(FileSystemDeploymentService.java:984)
[wildfly-deployment-scanner-8.1.0.Final.jar:8.1.0.Final]  
 at
org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.access$2800(FileSystemDeploymentService.java:83)
[wildfly-deployment-scanner-8.1.0.Final.jar:8.1.0.Final]  
 at
org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScannerTask.recordInProgress(FileSystemDeploymentService.java:1044)
[wildfly-deployment-scanner-8.1.0.Final.jar:8.1.0.Final]  
 at
org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:431)
[wildfly-deployment-scanner-8.1.0.Final.jar:8.1.0.Final]  
 at
org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:147)
[wildfly-deployment-scanner-8.1.0.Final.jar:8.1.0.Final]  
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[rt.jar:1.7.0_25]  
 at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
[rt.jar:1.7.0_25]  
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [rt.jar:1.7.0_25]  
 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[rt.jar:1.7.0_25]  
 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.7.0_25]  
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[rt.jar:1.7.0_25]  
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[rt.jar:1.7.0_25]  
 at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]  
 at org.jboss.threads.JBossThread.run(JBossThread.java:122)
[jboss-threads-2.1.1.Final.jar:2.1.1.Final]  

 2015-01-10 10:08:41,329 INFO [org.wildfly.extension.undertow] (MSC service thread 1-5)
JBAS017535: Unregistered web context: /ServiceWeb  
 2015-01-10 10:08:41,481 INFO [net.sf.json.xml.XMLSerializer] (HttpServiceHandler-50543)
Using default type string  
 2015-01-10 10:08:42,225 INFO [org.hibernate.validator.internal.util.Version] (MSC service
thread 1-3)   
 2015-01-10 10:08:43,504 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3)
JBAS015877: Stopped deployment OpenESBHttp_full_prd.war (runtime-name:
OpenESBHttp_full_prd.war) in 2373ms  

 From this server log,we can see that there is "Too open many files" problem.and
on the WildFly‘s manager console,we can see that my project was undeployed.When i used
command "lsof -p pid | wc -l" to find the count of open file-hander, 50000 open
file,because there were a lot of "can't identify protocol" as following:
 java 25625 jbossuser 460u sock 0,6 0t0 62184686 can't identify protocol  
 java 25625 jbossuser 461u sock 0,6 0t0 62170175 can't identify protocol  
 java 25625 jbossuser 462u sock 0,6 0t0 62159509 can't identify protocol  
 java 25625 jbossuser 463u sock 0,6 0t0 62193366 can't identify protocol  
 java 25625 jbossuser 464u sock 0,6 0t0 62181816 can't identify protocol  
 java 25625 jbossuser 465u sock 0,6 0t0 62159028 can't identify protocol  
 java 25625 jbossuser 466u sock 0,6 0t0 62181150 can't identify protocol  
 java 25625 jbossuser 467u sock 0,6 0t0 62165298 can't identify protocol  
 java 25625 jbossuser 468u sock 0,6 0t0 62181859 can't identify protocol  
 java 25625 jbossuser 469u sock 0,6 0t0 62155350 can't identify protocol  
 java 25625 jbossuser 470u sock 0,6 0t0 62184687 can't identify protocol  
 java 25625 jbossuser 471u sock 0,6 0t0 62150063 can't identify protocol  
 java 25625 jbossuser 472u IPv4 70228479 0t0 TCP
lesbprdapp16:webcache->192.168.53.177:23420 (ESTABLISHED)  

 So i know there maybe occur connection leak on WildFly, and i made a test as follow:
 I use apache's jmeter to test this project,i use 50 users to request it,then use
"lsof -p pid | can't identify protocol | wc -l" to find is there have
“can't identify protocol”,the number is 0.But when i stop the request,there appeared
lots of "can't identify protocol",the same phenomenon appeared on
wildfly-9.0.0.Alpha1 and wildfly-8.2.0.Final but not appeared on tomcat.So i'm doubt
that this is wildfly's bug,Is there anyony encounter this problem and tell me how to
solve it.

 Moreover, i found a issue is like above problem:
 [WFLY-3652] Network connection leak - JBoss Issue Tracker

 but don't think they are the same probem,because it say the problem alrendy was fixed
in wildfly-9.0.0.Alpha1,but above problem wasn't fixed when i test my project on
wildfly-9.0.0.Alpha1.

 Who can help me solve this problem? thanks! 

--
This message was sent by Atlassian JIRA
(v6.3.11#6341)

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006