[JBoss JIRA] (WFLY-10022) [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
9:46 a.m.
[
https://issues.jboss.org/browse/WFLY-10022?page=com.atlassian.jira.plugin...
]
Miroslav Novak commented on WFLY-10022:
---------------------------------------
{{lsof -p <pid>}} shows that there is huge number of open files with large
messages:
{code}
...
java 28326 mnovak 7419u REG 253,0 153600 575410126
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20182.msg
java 28326 mnovak 7420u REG 253,0 153600 575410127
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20195.msg
java 28326 mnovak 7421u REG 253,0 153600 575410135
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20502.msg
java 28326 mnovak 7423u REG 253,0 153600 576675698
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37277.msg
java 28326 mnovak 7425u REG 253,0 153600 576675699
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37335.msg
java 28326 mnovak 7426u REG 253,0 153600 573979023
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7226.msg
java 28326 mnovak 7427u REG 253,0 153600 576675700
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37357.msg
java 28326 mnovak 7428u REG 253,0 153600 573979024
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7240.msg
java 28326 mnovak 7429u REG 253,0 153600 576675701
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37390.msg
java 28326 mnovak 7430u REG 253,0 153600 573979025
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7253.msg
java 28326 mnovak 7431u REG 253,0 153600 576675702
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37403.msg
java 28326 mnovak 7432u REG 253,0 153600 576675706
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37563.msg
java 28326 mnovak 7433u REG 253,0 153600 576675703
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37416.msg
java 28326 mnovak 7434u REG 253,0 153600 576675704
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37429.msg
java 28326 mnovak 7435u REG 253,0 153600 576675705
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37484.msg
java 28326 mnovak 7436u REG 253,0 153600 576675707
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37627.msg
java 28326 mnovak 7437u REG 253,0 153600 576675708
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37685.msg
java 28326 mnovak 7438u REG 253,0 153600 576675709
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37727.msg
java 28326 mnovak 7439u REG 253,0 153600 575410130
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20279.msg
java 28326 mnovak 7440u REG 253,0 153600 575410131
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20362.msg
java 28326 mnovak 7441u REG 253,0 153600 575410132
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20375.msg
java 28326 mnovak 7442u REG 253,0 153600 576675710
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37740.msg
java 28326 mnovak 7443u REG 253,0 153600 573978975
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/6350.msg
java 28326 mnovak 7444u REG 253,0 153600 575410138
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20562.msg
java 28326 mnovak 7445u REG 253,0 153600 573979026
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7287.msg
java 28326 mnovak 7447u REG 253,0 153600 576786943
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50501.msg
java 28326 mnovak 7448u REG 253,0 153600 577760256
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50514.msg
java 28326 mnovak 7449u REG 253,0 153600 577760257
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50527.msg
java 28326 mnovak 7450u REG 253,0 153600 577760258
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50540.msg
java 28326 mnovak 7451u REG 253,0 153600 577760259
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50574.msg
java 28326 mnovak 7452u REG 253,0 153600 577760260
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50717.msg
java 28326 mnovak 7453u REG 253,0 153600 576677792
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37808.msg
java 28326 mnovak 7454u REG 253,0 153600 573979028
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7318.msg
java 28326 mnovak 7458u REG 253,0 153600 575410134
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20487.msg
java 28326 mnovak 7459u REG 253,0 153600 576675711
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37753.msg
java 28326 mnovak 7460u REG 253,0 153600 573979027
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7305.msg
java 28326 mnovak 7462u REG 253,0 153600 575410137
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20549.msg
java 28326 mnovak 7463u REG 253,0 153600 575410136
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20515.msg
java 28326 mnovak 7464u REG 253,0 153600 576677793
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37821.msg
java 28326 mnovak 7465u REG 253,0 153600 577760256
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50514.msg
java 28326 mnovak 7466u REG 253,0 153600 573979029
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7351.msg
java 28326 mnovak 7467u REG 253,0 153600 576677794
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37897.msg
java 28326 mnovak 7468u REG 253,0 153600 573979030
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7365.msg
...
{code}
[Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO
Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg,
message=.../largemessages/55851.msg (Too many open files)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key: WFLY-10022
URL: https://issues.jboss.org/browse/WFLY-10022
Project: WildFly
Issue Type: Bug
Components: JMS
Reporter: Miroslav Novak
Assignee: Jeff Mesnil
There is file leak in Artemis master branch (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7)
when large messages are consumed from topic. This is regression against Artemis 1.5.x.
Test Scenario:
* Start WF12 (Jeff's WF WFLY-9407_upgrade_artemis_2.5.0 branch -
06c878a313d3cad323889d017e60fd5533204d1a) with deployed "testTopic"
* create durable subscriptions on testTopic
* start sending large messages on testTopic
* subscribers start to consume one by one so there is a huge difference in number of
consumed messages between subscriptions
Pass Criteria: Subscribers must receive correct number of messages.
Actual Result:
Artemis server shutdown/crashes itself on :
{code}
15:21:54,717 WARN [org.apache.activemq.artemis.core.server] (Thread-21
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1787a3ac))
AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg,
message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg
(Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR
message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg
(Too many open files)]
at
org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87)
[artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73)
[artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:397)
[artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:376)
[artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(JournalStorageManager.java:495)
[artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendLarge(ServerSessionPacketHandler.java:937)
[artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:302)
[artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:281)
[artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
[artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
[artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
[artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
[artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
[artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[rt.jar:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[rt.jar:1.8.0_131]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
Caused by: java.io.FileNotFoundException:
/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg
(Too many open files)
at java.io.RandomAccessFile.open0(Native Method) [rt.jar:1.8.0_131]
at java.io.RandomAccessFile.open(RandomAccessFile.java:316) [rt.jar:1.8.0_131]
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
[rt.jar:1.8.0_131]
at
org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79)
[artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
... 15 more
{code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
2996
days inactive
2996
days old
0 comments
1 participants
participants (1)
-
Miroslav Novak (JIRA)