[mod_cluster-issues] [JBoss JIRA] (MODCLUSTER-536) List of open files grows steadily during load test through mod_cluster

[Wayne Wang (JIRA)]

    [ https://issues.jboss.org/browse/MODCLUSTER-536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13287587#comment-13287587 ] 

Wayne Wang edited comment on MODCLUSTER-536 at 9/1/16 11:51 AM:
----------------------------------------------------------------

The lsof command used for wildfly instance in one virtualbox vm was

lsof -u wayne | grep TCP | wc -l

The wildfly instance was started under user wayne. I can also find out the pid of the wildfly instance and change the command to lsof -p <pid> | grep TCP | wc -l. However, the result will be the same, so it was convenient for me to use the -u command

The lsof command used for apache web server in another virtualbox vm was
lsof -u daemon | grep TCP | wc -l

I also use the following to monitor the outcome of the command

{{watch -n 5 '${the above command}'}}

(1) if there was no shutdown of wildfly instance, the outcome of the command will keep low (about 200) and does not show the pattern of steady increase
(2) if I shut down the wildfly instance and let another standby instance to become active, and do this multiple times, I will see the outcome of the command show steady increase, and when it reach > 3000, server will print out error.  The Apache web server needs to be restarted in order to support another load test. The wildfly instance also needs to be restarted due to the error of "Too many open files" and the outcome of lsof command remains high and unchanged.

Note the soft limit was set at 4096 and hard limit was set at 8192


was (Author: wwang2016):
The lsof command used for wildfly instance in one virtualbox vm was

lsof -u wayne | grep TCP | wc -l

The wildfly instance was started under user wayne. I can also find out the pid of the wildfly instance and change the command to lsof -p <pid> | grep TCP | wc -l. However, the result will be the same, so it was convenient for me to use the -u command

The lsof command used for apache web server in another virtualbox vm was
lsof -u daemon | grep TCP | wc -l

I also use the following to monitor the outcome of the command

{quote}watch -n 5 '${the above command}'{quote}

(1) if there was no shutdown of wildfly instance, the outcome of the command will keep low (about 200) and does not show the pattern of steady increase
(2) if I shut down the wildfly instance and let another standby instance to become active, and do this multiple times, I will see the outcome of the command show steady increase, and when it reach > 3000, server will print out error.  The Apache web server needs to be restarted in order to support another load test. The wildfly instance also needs to be restarted due to the error of "Too many open files" and the outcome of lsof command remains high and unchanged.

Note the soft limit was set at 4096 and hard limit was set at 8192

> List of open files grows steadily during load test through mod_cluster
> ----------------------------------------------------------------------
>
>                 Key: MODCLUSTER-536
>                 URL: https://issues.jboss.org/browse/MODCLUSTER-536
>             Project: mod_cluster
>          Issue Type: Bug
>          Components: Core & Container Integration (Java)
>    Affects Versions: 1.3.1.Final
>         Environment: Wildfly10.0.0.Final
> mod_cluster-1.3.1.Final-linux2-x64-ssl
> CentOS7 (virtualbox)
>            Reporter: Wayne Wang
>            Assignee: Michal Karm Babacek
>         Attachments: error_log, httpd-mpm.conf, httpd.conf, server.log, standalone-full-ha-snippet.xml
>
>
> I was able to configure wildfly 10 modcluster to work with Apache mod_cluster (1.3.1). However, when I was doing a load test, I found out that the test through web server eventually caused error in wildfly instance and I also saw error log in Apache web server
> The obvious error in wildfly instance is the so-called "java.net.SocketException: Too many files open". When I used the command lsop -u   | grep TCP | wc -l, I can see the number grew steadily until the wildfly instance reported the error. This was when I sent requests through web server.
> However, when I sent the requests through wildfly instance (app server) directly, the number did not grow, and the app server can take a lot heavier load without this issue.
> The issue did not happen until many rounds of load tests were executed through web server. If I restart the web server, everything is working fine until I execute many rounds of load tests again



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)