[mod_cluster-issues] [JBoss JIRA] (MODCLUSTER-536) List of open files grows steadily during load test through mod_cluster

[Wayne Wang (JIRA)]

    [ https://issues.jboss.org/browse/MODCLUSTER-536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286898#comment-13286898 ] 

Wayne Wang edited comment on MODCLUSTER-536 at 8/31/16 11:35 AM:
-----------------------------------------------------------------

Additional information:

I could shut down the instance in two different ways during load tests
(1) After Jmeter send button is back (all requests were sent to a specific wildfly instance for processing)
(2) Before Jmeter send button is back, and the rest of requests need to be routed to the soon-to-be active wildfly instance that is currently inactive.

I found one round of load test with the second way of shutting down the wildfly instance will cause the "Too many open files" issue in the next round of load test. If I use the first way of shutting down the wildfly instance, it needs three times of load tests to replicate the issue.

So it looks like shutting down a wildfly instance may be one potential cause of the issue. In addition, the activation of the standby instance takes some time, and the rest of requests will go nowhere during that period of time until the standby instance becomes active. I am not sure if this can bring additional issue


was (Author: wwang2016):
Additional information:

I could shut down the instance in two different ways during load tests
(1) After Jmeter send button is back (all requests were sent to a specific wildfly instance for processing)
(2) Before Jmeter send button is back, and the rest of requests need to be routed to the soon-to-be active wildfly instance that is currently inactive.

I found one round of load test with the second way of shutting down the wildfly instance will cause the "Too many open files" issue in the next round of load test. If I use the first way of shutting down the wildfly instance, it needs three times of load tests to replicate the issue.

> List of open files grows steadily during load test through mod_cluster
> ----------------------------------------------------------------------
>
>                 Key: MODCLUSTER-536
>                 URL: https://issues.jboss.org/browse/MODCLUSTER-536
>             Project: mod_cluster
>          Issue Type: Bug
>          Components: Core & Container Integration (Java)
>    Affects Versions: 1.3.1.Final
>         Environment: Wildfly10.0.0.Final
> mod_cluster-1.3.1.Final-linux2-x64-ssl
> CentOS7 (virtualbox)
>            Reporter: Wayne Wang
>            Assignee: Michal Karm Babacek
>         Attachments: error_log, httpd-mpm.conf, httpd.conf, server.log, standalone-full-ha-snippet.xml
>
>
> I was able to configure wildfly 10 modcluster to work with Apache mod_cluster (1.3.1). However, when I was doing a load test, I found out that the test through web server eventually caused error in wildfly instance and I also saw error log in Apache web server
> The obvious error in wildfly instance is the so-called "java.net.SocketException: Too many files open". When I used the command lsop -u   | grep TCP | wc -l, I can see the number grew steadily until the wildfly instance reported the error. This was when I sent requests through web server.
> However, when I sent the requests through wildfly instance (app server) directly, the number did not grow, and the app server can take a lot heavier load without this issue.
> The issue did not happen until many rounds of load tests were executed through web server. If I restart the web server, everything is working fine until I execute many rounds of load tests again



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)