[mod_cluster-issues] [JBoss JIRA] (MODCLUSTER-536) List of open files grows steadily during load test through mod_cluster

[Wayne Wang (JIRA)]

    [ https://issues.jboss.org/browse/MODCLUSTER-536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287623#comment-13287623 ] 

Wayne Wang commented on MODCLUSTER-536:
---------------------------------------

New information about using mod_cluster v1.2.6 for comparison.

I have executed 10 load tests now, and I found v1.2.6 is a lot more performing (processed a lot more requests), and I have NOT found the issue of "Too many open files". During load tests, I monitored the lsof every 3 seconds, and I found the outcome could increase up to a bit more than 300 (314 was the highest number I observed), but it fluctuated between 200 and 300 for most of the time.

Summary of the differences between v1.3.1 and v1.2.6
(1) with v1.3.1, the web server was out of service for a period of time during load tests, and this did not happen with v1.2.6 (/server-status page was always up)
(2) much more requests got processed by v1.2.6. Jmeter showed 70% requests got OK status with v1.2.6 vs less than 50% got OK status with v1.3.1
(3) The number of open files under httpd processes stabilized at around 300 during peak load for v1.2.6, while this number could shoot up to more than 3000 with v1.3.1 and the server will print out errors after even one load test.
(4) once the JMeter send button is back (becomes green), the lsof result from v1.2.6 will decrease to around 75 in all 10 load tests. I also did not see any performance degradation in v1.2.6 during all the 10 load tests.

It looks like v1.3.1 may have some issue managing "open file/socket" during the shutdown of wildfly instances. In addition, it is obviously less performing than v1.2.6

> List of open files grows steadily during load test through mod_cluster
> ----------------------------------------------------------------------
>
>                 Key: MODCLUSTER-536
>                 URL: https://issues.jboss.org/browse/MODCLUSTER-536
>             Project: mod_cluster
>          Issue Type: Bug
>          Components: Core & Container Integration (Java)
>    Affects Versions: 1.3.1.Final
>         Environment: Wildfly10.0.0.Final
> mod_cluster-1.3.1.Final-linux2-x64-ssl
> CentOS7 (virtualbox)
>            Reporter: Wayne Wang
>            Assignee: Michal Karm Babacek
>         Attachments: error_log, httpd-mpm.conf, httpd.conf, server.log, standalone-full-ha-snippet.xml
>
>
> I was able to configure wildfly 10 modcluster to work with Apache mod_cluster (1.3.1). However, when I was doing a load test, I found out that the test through web server eventually caused error in wildfly instance and I also saw error log in Apache web server
> The obvious error in wildfly instance is the so-called "java.net.SocketException: Too many files open". When I used the command lsop -u   | grep TCP | wc -l, I can see the number grew steadily until the wildfly instance reported the error. This was when I sent requests through web server.
> However, when I sent the requests through wildfly instance (app server) directly, the number did not grow, and the app server can take a lot heavier load without this issue.
> The issue did not happen until many rounds of load tests were executed through web server. If I restart the web server, everything is working fine until I execute many rounds of load tests again



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)