[jboss-jira] [JBoss JIRA] (WFLY-10022) [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
Miroslav Novak (JIRA)
issues at jboss.org
Wed Mar 14 10:46:01 EDT 2018
[ https://issues.jboss.org/browse/WFLY-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13545823#comment-13545823 ]
Miroslav Novak commented on WFLY-10022:
---------------------------------------
{{lsof -p <pid>}} shows that there is huge number of open files with large messages:
{code}
...
java 28326 mnovak 7419u REG 253,0 153600 575410126 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20182.msg
java 28326 mnovak 7420u REG 253,0 153600 575410127 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20195.msg
java 28326 mnovak 7421u REG 253,0 153600 575410135 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20502.msg
java 28326 mnovak 7423u REG 253,0 153600 576675698 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37277.msg
java 28326 mnovak 7425u REG 253,0 153600 576675699 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37335.msg
java 28326 mnovak 7426u REG 253,0 153600 573979023 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7226.msg
java 28326 mnovak 7427u REG 253,0 153600 576675700 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37357.msg
java 28326 mnovak 7428u REG 253,0 153600 573979024 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7240.msg
java 28326 mnovak 7429u REG 253,0 153600 576675701 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37390.msg
java 28326 mnovak 7430u REG 253,0 153600 573979025 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7253.msg
java 28326 mnovak 7431u REG 253,0 153600 576675702 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37403.msg
java 28326 mnovak 7432u REG 253,0 153600 576675706 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37563.msg
java 28326 mnovak 7433u REG 253,0 153600 576675703 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37416.msg
java 28326 mnovak 7434u REG 253,0 153600 576675704 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37429.msg
java 28326 mnovak 7435u REG 253,0 153600 576675705 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37484.msg
java 28326 mnovak 7436u REG 253,0 153600 576675707 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37627.msg
java 28326 mnovak 7437u REG 253,0 153600 576675708 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37685.msg
java 28326 mnovak 7438u REG 253,0 153600 576675709 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37727.msg
java 28326 mnovak 7439u REG 253,0 153600 575410130 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20279.msg
java 28326 mnovak 7440u REG 253,0 153600 575410131 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20362.msg
java 28326 mnovak 7441u REG 253,0 153600 575410132 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20375.msg
java 28326 mnovak 7442u REG 253,0 153600 576675710 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37740.msg
java 28326 mnovak 7443u REG 253,0 153600 573978975 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/6350.msg
java 28326 mnovak 7444u REG 253,0 153600 575410138 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20562.msg
java 28326 mnovak 7445u REG 253,0 153600 573979026 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7287.msg
java 28326 mnovak 7447u REG 253,0 153600 576786943 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50501.msg
java 28326 mnovak 7448u REG 253,0 153600 577760256 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50514.msg
java 28326 mnovak 7449u REG 253,0 153600 577760257 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50527.msg
java 28326 mnovak 7450u REG 253,0 153600 577760258 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50540.msg
java 28326 mnovak 7451u REG 253,0 153600 577760259 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50574.msg
java 28326 mnovak 7452u REG 253,0 153600 577760260 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50717.msg
java 28326 mnovak 7453u REG 253,0 153600 576677792 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37808.msg
java 28326 mnovak 7454u REG 253,0 153600 573979028 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7318.msg
java 28326 mnovak 7458u REG 253,0 153600 575410134 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20487.msg
java 28326 mnovak 7459u REG 253,0 153600 576675711 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37753.msg
java 28326 mnovak 7460u REG 253,0 153600 573979027 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7305.msg
java 28326 mnovak 7462u REG 253,0 153600 575410137 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20549.msg
java 28326 mnovak 7463u REG 253,0 153600 575410136 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20515.msg
java 28326 mnovak 7464u REG 253,0 153600 576677793 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37821.msg
java 28326 mnovak 7465u REG 253,0 153600 577760256 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50514.msg
java 28326 mnovak 7466u REG 253,0 153600 573979029 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7351.msg
java 28326 mnovak 7467u REG 253,0 153600 576677794 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37897.msg
java 28326 mnovak 7468u REG 253,0 153600 573979030 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7365.msg
...
{code}
> [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10022
> URL: https://issues.jboss.org/browse/WFLY-10022
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Jeff Mesnil
>
> There is file leak in Artemis master branch (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7) when large messages are consumed from topic. This is regression against Artemis 1.5.x.
> Test Scenario:
> * Start WF12 (Jeff's WF WFLY-9407_upgrade_artemis_2.5.0 branch - 06c878a313d3cad323889d017e60fd5533204d1a) with deployed "testTopic"
> * create durable subscriptions on testTopic
> * start sending large messages on testTopic
> * subscribers start to consume one by one so there is a huge difference in number of consumed messages between subscriptions
> Pass Criteria: Subscribers must receive correct number of messages.
> Actual Result:
> Artemis server shutdown/crashes itself on :
> {code}
> 15:21:54,717 WARN [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5 at 1787a3ac)) AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg, message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:397) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:376) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(JournalStorageManager.java:495) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendLarge(ServerSessionPacketHandler.java:937) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:302) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:281) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> Caused by: java.io.FileNotFoundException: /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method) [rt.jar:1.8.0_131]
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316) [rt.jar:1.8.0_131]
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) [rt.jar:1.8.0_131]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> ... 15 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
More information about the jboss-jira
mailing list