[JBoss JIRA] (WFLY-10023) [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
by Jeff Mesnil (JIRA)
Jeff Mesnil created WFLY-10023:
----------------------------------
Summary: [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
Key: WFLY-10023
URL: https://issues.jboss.org/browse/WFLY-10023
Project: WildFly
Issue Type: Bug
Components: JMS
Reporter: Miroslav Novak
Assignee: Jeff Mesnil
Priority: Blocker
There is file leak in Artemis master branch (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7) when large messages are consumed from topic. This is regression against Artemis 1.5.x.
Test Scenario:
* Start WF12 (Jeff's WF WFLY-9407_upgrade_artemis_2.5.0 branch - 06c878a313d3cad323889d017e60fd5533204d1a) with deployed "testTopic"
* create durable subscriptions on testTopic
* start sending large messages on testTopic
* subscribers start to consume one by one so there is a huge difference in number of consumed messages between subscriptions
Pass Criteria: Subscribers must receive correct number of messages.
Actual Result:
Artemis server shutdown/crashes itself on (ulimit for max number of open files was set to 8196):
{code}
15:21:54,717 WARN [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1787a3ac)) AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg, message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:397) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:376) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(JournalStorageManager.java:495) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendLarge(ServerSessionPacketHandler.java:937) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:302) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:281) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
Caused by: java.io.FileNotFoundException: /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)
at java.io.RandomAccessFile.open0(Native Method) [rt.jar:1.8.0_131]
at java.io.RandomAccessFile.open(RandomAccessFile.java:316) [rt.jar:1.8.0_131]
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) [rt.jar:1.8.0_131]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
... 15 more
{code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 4 months
[JBoss JIRA] (WFLY-10022) [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
by Miroslav Novak (JIRA)
[ https://issues.jboss.org/browse/WFLY-10022?page=com.atlassian.jira.plugin... ]
Miroslav Novak updated WFLY-10022:
----------------------------------
Description:
There is file leak in Artemis master branch (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7) when large messages are consumed from topic. This is regression against Artemis 1.5.x.
Test Scenario:
* Start WF12 (Jeff's WF WFLY-9407_upgrade_artemis_2.5.0 branch - 06c878a313d3cad323889d017e60fd5533204d1a) with deployed "testTopic"
* create durable subscriptions on testTopic
* start sending large messages on testTopic
* subscribers start to consume one by one so there is a huge difference in number of consumed messages between subscriptions
Pass Criteria: Subscribers must receive correct number of messages.
Actual Result:
Artemis server shutdown/crashes itself on (ulimit for max number of open files was set to 8196):
{code}
15:21:54,717 WARN [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1787a3ac)) AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg, message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:397) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:376) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(JournalStorageManager.java:495) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendLarge(ServerSessionPacketHandler.java:937) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:302) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:281) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
Caused by: java.io.FileNotFoundException: /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)
at java.io.RandomAccessFile.open0(Native Method) [rt.jar:1.8.0_131]
at java.io.RandomAccessFile.open(RandomAccessFile.java:316) [rt.jar:1.8.0_131]
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) [rt.jar:1.8.0_131]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
... 15 more
{code}
was:
There is file leak in Artemis master branch (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7) when large messages are consumed from topic. This is regression against Artemis 1.5.x.
Test Scenario:
* Start WF12 (Jeff's WF WFLY-9407_upgrade_artemis_2.5.0 branch - 06c878a313d3cad323889d017e60fd5533204d1a) with deployed "testTopic"
* create durable subscriptions on testTopic
* start sending large messages on testTopic
* subscribers start to consume one by one so there is a huge difference in number of consumed messages between subscriptions
Pass Criteria: Subscribers must receive correct number of messages.
Actual Result:
Artemis server shutdown/crashes itself on :
{code}
15:21:54,717 WARN [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1787a3ac)) AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg, message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:397) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:376) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(JournalStorageManager.java:495) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendLarge(ServerSessionPacketHandler.java:937) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:302) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:281) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
Caused by: java.io.FileNotFoundException: /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)
at java.io.RandomAccessFile.open0(Native Method) [rt.jar:1.8.0_131]
at java.io.RandomAccessFile.open(RandomAccessFile.java:316) [rt.jar:1.8.0_131]
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) [rt.jar:1.8.0_131]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
... 15 more
{code}
> [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10022
> URL: https://issues.jboss.org/browse/WFLY-10022
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq
>
> There is file leak in Artemis master branch (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7) when large messages are consumed from topic. This is regression against Artemis 1.5.x.
> Test Scenario:
> * Start WF12 (Jeff's WF WFLY-9407_upgrade_artemis_2.5.0 branch - 06c878a313d3cad323889d017e60fd5533204d1a) with deployed "testTopic"
> * create durable subscriptions on testTopic
> * start sending large messages on testTopic
> * subscribers start to consume one by one so there is a huge difference in number of consumed messages between subscriptions
> Pass Criteria: Subscribers must receive correct number of messages.
> Actual Result:
> Artemis server shutdown/crashes itself on (ulimit for max number of open files was set to 8196):
> {code}
> 15:21:54,717 WARN [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1787a3ac)) AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg, message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:397) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:376) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(JournalStorageManager.java:495) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendLarge(ServerSessionPacketHandler.java:937) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:302) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:281) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> Caused by: java.io.FileNotFoundException: /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method) [rt.jar:1.8.0_131]
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316) [rt.jar:1.8.0_131]
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) [rt.jar:1.8.0_131]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> ... 15 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 4 months
[JBoss JIRA] (WFLY-10022) [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
by Miroslav Novak (JIRA)
[ https://issues.jboss.org/browse/WFLY-10022?page=com.atlassian.jira.plugin... ]
Miroslav Novak updated WFLY-10022:
----------------------------------
Steps to Reproduce:
git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout f3de9baf1f8b39b810bf3d55d45f4341e3b2aa87
groovy -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/217/artifact/jboss-eap.zip PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
cd ../jboss-hornetq-testsuite/
ulimit -n 8196
mvn clean test -Dtest=DurableSubscriptionsTestCase#largeByteMessagesTest -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1520950239-SNAPSHOT -DfailIfNoTests=false | tee log
> [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10022
> URL: https://issues.jboss.org/browse/WFLY-10022
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq
>
> There is file leak in Artemis master branch (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7) when large messages are consumed from topic. This is regression against Artemis 1.5.x.
> Test Scenario:
> * Start WF12 (Jeff's WF WFLY-9407_upgrade_artemis_2.5.0 branch - 06c878a313d3cad323889d017e60fd5533204d1a) with deployed "testTopic"
> * create durable subscriptions on testTopic
> * start sending large messages on testTopic
> * subscribers start to consume one by one so there is a huge difference in number of consumed messages between subscriptions
> Pass Criteria: Subscribers must receive correct number of messages.
> Actual Result:
> Artemis server shutdown/crashes itself on (ulimit for max number of open files was set to 8196):
> {code}
> 15:21:54,717 WARN [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1787a3ac)) AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg, message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:397) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:376) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(JournalStorageManager.java:495) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendLarge(ServerSessionPacketHandler.java:937) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:302) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:281) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> Caused by: java.io.FileNotFoundException: /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method) [rt.jar:1.8.0_131]
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316) [rt.jar:1.8.0_131]
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) [rt.jar:1.8.0_131]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> ... 15 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 4 months
[JBoss JIRA] (WFLY-10022) [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
by Jeff Mesnil (JIRA)
[ https://issues.jboss.org/browse/WFLY-10022?page=com.atlassian.jira.plugin... ]
Jeff Mesnil updated WFLY-10022:
-------------------------------
Labels: activemq (was: )
> [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10022
> URL: https://issues.jboss.org/browse/WFLY-10022
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Jeff Mesnil
> Priority: Blocker
> Labels: activemq
>
> There is file leak in Artemis master branch (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7) when large messages are consumed from topic. This is regression against Artemis 1.5.x.
> Test Scenario:
> * Start WF12 (Jeff's WF WFLY-9407_upgrade_artemis_2.5.0 branch - 06c878a313d3cad323889d017e60fd5533204d1a) with deployed "testTopic"
> * create durable subscriptions on testTopic
> * start sending large messages on testTopic
> * subscribers start to consume one by one so there is a huge difference in number of consumed messages between subscriptions
> Pass Criteria: Subscribers must receive correct number of messages.
> Actual Result:
> Artemis server shutdown/crashes itself on :
> {code}
> 15:21:54,717 WARN [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1787a3ac)) AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg, message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:397) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:376) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(JournalStorageManager.java:495) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendLarge(ServerSessionPacketHandler.java:937) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:302) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:281) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> Caused by: java.io.FileNotFoundException: /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method) [rt.jar:1.8.0_131]
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316) [rt.jar:1.8.0_131]
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) [rt.jar:1.8.0_131]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> ... 15 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 4 months
[JBoss JIRA] (WFLY-10022) [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
by Miroslav Novak (JIRA)
[ https://issues.jboss.org/browse/WFLY-10022?page=com.atlassian.jira.plugin... ]
Miroslav Novak updated WFLY-10022:
----------------------------------
Priority: Blocker (was: Major)
> [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10022
> URL: https://issues.jboss.org/browse/WFLY-10022
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Jeff Mesnil
> Priority: Blocker
>
> There is file leak in Artemis master branch (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7) when large messages are consumed from topic. This is regression against Artemis 1.5.x.
> Test Scenario:
> * Start WF12 (Jeff's WF WFLY-9407_upgrade_artemis_2.5.0 branch - 06c878a313d3cad323889d017e60fd5533204d1a) with deployed "testTopic"
> * create durable subscriptions on testTopic
> * start sending large messages on testTopic
> * subscribers start to consume one by one so there is a huge difference in number of consumed messages between subscriptions
> Pass Criteria: Subscribers must receive correct number of messages.
> Actual Result:
> Artemis server shutdown/crashes itself on :
> {code}
> 15:21:54,717 WARN [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1787a3ac)) AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg, message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:397) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:376) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(JournalStorageManager.java:495) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendLarge(ServerSessionPacketHandler.java:937) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:302) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:281) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> Caused by: java.io.FileNotFoundException: /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method) [rt.jar:1.8.0_131]
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316) [rt.jar:1.8.0_131]
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) [rt.jar:1.8.0_131]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> ... 15 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 4 months
[JBoss JIRA] (WFLY-10022) [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
by Miroslav Novak (JIRA)
[ https://issues.jboss.org/browse/WFLY-10022?page=com.atlassian.jira.plugin... ]
Miroslav Novak commented on WFLY-10022:
---------------------------------------
{{lsof -p <pid>}} shows that there is huge number of open files with large messages:
{code}
...
java 28326 mnovak 7419u REG 253,0 153600 575410126 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20182.msg
java 28326 mnovak 7420u REG 253,0 153600 575410127 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20195.msg
java 28326 mnovak 7421u REG 253,0 153600 575410135 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20502.msg
java 28326 mnovak 7423u REG 253,0 153600 576675698 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37277.msg
java 28326 mnovak 7425u REG 253,0 153600 576675699 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37335.msg
java 28326 mnovak 7426u REG 253,0 153600 573979023 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7226.msg
java 28326 mnovak 7427u REG 253,0 153600 576675700 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37357.msg
java 28326 mnovak 7428u REG 253,0 153600 573979024 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7240.msg
java 28326 mnovak 7429u REG 253,0 153600 576675701 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37390.msg
java 28326 mnovak 7430u REG 253,0 153600 573979025 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7253.msg
java 28326 mnovak 7431u REG 253,0 153600 576675702 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37403.msg
java 28326 mnovak 7432u REG 253,0 153600 576675706 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37563.msg
java 28326 mnovak 7433u REG 253,0 153600 576675703 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37416.msg
java 28326 mnovak 7434u REG 253,0 153600 576675704 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37429.msg
java 28326 mnovak 7435u REG 253,0 153600 576675705 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37484.msg
java 28326 mnovak 7436u REG 253,0 153600 576675707 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37627.msg
java 28326 mnovak 7437u REG 253,0 153600 576675708 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37685.msg
java 28326 mnovak 7438u REG 253,0 153600 576675709 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37727.msg
java 28326 mnovak 7439u REG 253,0 153600 575410130 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20279.msg
java 28326 mnovak 7440u REG 253,0 153600 575410131 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20362.msg
java 28326 mnovak 7441u REG 253,0 153600 575410132 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20375.msg
java 28326 mnovak 7442u REG 253,0 153600 576675710 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37740.msg
java 28326 mnovak 7443u REG 253,0 153600 573978975 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/6350.msg
java 28326 mnovak 7444u REG 253,0 153600 575410138 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20562.msg
java 28326 mnovak 7445u REG 253,0 153600 573979026 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7287.msg
java 28326 mnovak 7447u REG 253,0 153600 576786943 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50501.msg
java 28326 mnovak 7448u REG 253,0 153600 577760256 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50514.msg
java 28326 mnovak 7449u REG 253,0 153600 577760257 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50527.msg
java 28326 mnovak 7450u REG 253,0 153600 577760258 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50540.msg
java 28326 mnovak 7451u REG 253,0 153600 577760259 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50574.msg
java 28326 mnovak 7452u REG 253,0 153600 577760260 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50717.msg
java 28326 mnovak 7453u REG 253,0 153600 576677792 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37808.msg
java 28326 mnovak 7454u REG 253,0 153600 573979028 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7318.msg
java 28326 mnovak 7458u REG 253,0 153600 575410134 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20487.msg
java 28326 mnovak 7459u REG 253,0 153600 576675711 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37753.msg
java 28326 mnovak 7460u REG 253,0 153600 573979027 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7305.msg
java 28326 mnovak 7462u REG 253,0 153600 575410137 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20549.msg
java 28326 mnovak 7463u REG 253,0 153600 575410136 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/20515.msg
java 28326 mnovak 7464u REG 253,0 153600 576677793 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37821.msg
java 28326 mnovak 7465u REG 253,0 153600 577760256 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/50514.msg
java 28326 mnovak 7466u REG 253,0 153600 573979029 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7351.msg
java 28326 mnovak 7467u REG 253,0 153600 576677794 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/37897.msg
java 28326 mnovak 7468u REG 253,0 153600 573979030 /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/7365.msg
...
{code}
> [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: WFLY-10022
> URL: https://issues.jboss.org/browse/WFLY-10022
> Project: WildFly
> Issue Type: Bug
> Components: JMS
> Reporter: Miroslav Novak
> Assignee: Jeff Mesnil
>
> There is file leak in Artemis master branch (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7) when large messages are consumed from topic. This is regression against Artemis 1.5.x.
> Test Scenario:
> * Start WF12 (Jeff's WF WFLY-9407_upgrade_artemis_2.5.0 branch - 06c878a313d3cad323889d017e60fd5533204d1a) with deployed "testTopic"
> * create durable subscriptions on testTopic
> * start sending large messages on testTopic
> * subscribers start to consume one by one so there is a huge difference in number of consumed messages between subscriptions
> Pass Criteria: Subscribers must receive correct number of messages.
> Actual Result:
> Artemis server shutdown/crashes itself on :
> {code}
> 15:21:54,717 WARN [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1787a3ac)) AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg, message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:397) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:376) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(JournalStorageManager.java:495) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendLarge(ServerSessionPacketHandler.java:937) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:302) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:281) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> Caused by: java.io.FileNotFoundException: /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method) [rt.jar:1.8.0_131]
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316) [rt.jar:1.8.0_131]
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) [rt.jar:1.8.0_131]
> at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
> ... 15 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 4 months
[JBoss JIRA] (WFLY-10022) [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
by Miroslav Novak (JIRA)
Miroslav Novak created WFLY-10022:
-------------------------------------
Summary: [Artemis 2.x Upgrade] File leak - Artemis crashes on Critical IO Error, shutting down the server. file=NIOSequentialFile .../largemessages/55851.msg, message=.../largemessages/55851.msg (Too many open files)
Key: WFLY-10022
URL: https://issues.jboss.org/browse/WFLY-10022
Project: WildFly
Issue Type: Bug
Components: JMS
Reporter: Miroslav Novak
Assignee: Jeff Mesnil
There is file leak in Artemis master branch (577b62d5210cdcc0f86ab9bb1b24e944c877dfe7) when large messages are consumed from topic. This is regression against Artemis 1.5.x.
Test Scenario:
* Start WF12 (Jeff's WF WFLY-9407_upgrade_artemis_2.5.0 branch - 06c878a313d3cad323889d017e60fd5533204d1a) with deployed "testTopic"
* create durable subscriptions on testTopic
* start sending large messages on testTopic
* subscribers start to consume one by one so there is a huge difference in number of consumed messages between subscriptions
Pass Criteria: Subscribers must receive correct number of messages.
Actual Result:
Artemis server shutdown/crashes itself on :
{code}
15:21:54,717 WARN [org.apache.activemq.artemis.core.server] (Thread-21 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1787a3ac)) AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg, message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files): ActiveMQIOErrorException[errorType=IO_ERROR message=/home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:87) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:73) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile(LargeServerMessageImpl.java:397) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile(LargeServerMessageImpl.java:376) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(JournalStorageManager.java:495) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.sendLarge(ServerSessionPacketHandler.java:937) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:302) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:281) [artemis-server-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_131]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
Caused by: java.io.FileNotFoundException: /home/mnovak/hornetq_eap6_dev/internal/eap-tests-hornetq/scripts/journal-A/largemessages/55851.msg (Too many open files)
at java.io.RandomAccessFile.open0(Native Method) [rt.jar:1.8.0_131]
at java.io.RandomAccessFile.open(RandomAccessFile.java:316) [rt.jar:1.8.0_131]
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243) [rt.jar:1.8.0_131]
at org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:79) [artemis-journal-2.5.0-SNAPSHOT.jar:2.5.0-SNAPSHOT]
... 15 more
{code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 4 months
[JBoss JIRA] (WFLY-9574) Distribution files does not have POSIX permissions perfectly set
by Romain Pelisse (JIRA)
[ https://issues.jboss.org/browse/WFLY-9574?page=com.atlassian.jira.plugin.... ]
Romain Pelisse reopened WFLY-9574:
----------------------------------
> Distribution files does not have POSIX permissions perfectly set
> ----------------------------------------------------------------
>
> Key: WFLY-9574
> URL: https://issues.jboss.org/browse/WFLY-9574
> Project: WildFly
> Issue Type: Enhancement
> Components: Build System
> Affects Versions: 11.0.0.Final
> Reporter: Romain Pelisse
> Assignee: Romain Pelisse
> Priority: Minor
>
> The server provisioning copy (and extract) files in order to assemble the distribution based the information of the feature-pack.
> While doing so on a POSIX system, it keeps the permissions of the original files, which are not always optimum (see JBEAP-12374). Specifically, .properties and .jar files are associated with the mask rw-rw-r-- giving access to it to any other and allowing group member to modify the file ;
> On a "regular" Maven project, all of those changes could be specified in the assembly.xml, however in Wildfly cases, this is not really option because the provisioning-maven-plugin and feature-pack-build-maven-plugin are manipulating the content of the archive being built. Also, using assembly.xml would mean edit and update the 4 or 5 different assembly.xml in the project directory tree.
> I plan thus to propose a fix for wildfly-build-tools to address all those (small) problems.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 4 months
[JBoss JIRA] (WFLY-6634) Add remote-cache-container configuration to Infinispan subsystem
by Richard Janík (JIRA)
[ https://issues.jboss.org/browse/WFLY-6634?page=com.atlassian.jira.plugin.... ]
Richard Janík commented on WFLY-6634:
-------------------------------------
Agreed with [~pferraro], input from [~bilge] would be helpful on that.
I understand injecting RemoteCacheManagers is not supported, that is fine and I do not intend to test it. But the configuration for {{remote-cache-manager}} is still public and supported I assume. Since the PRs add configuration, there needs to be an RFE for that. Right now, EAP7-942 covers only the security configuration for the {{remote-cache-manager}}.
As [~mjurc] says: If we want to expose other configuration elements for {{remote-cache-manager}}, the configuration needs to be tested. If we do not want to support such configuration, those configuration elements must not be present in model and other, hidden mechanisms must be used.
We’re not asking for 1:1 coupling between WFLY and EAP7 issues, just that anything that is configurable is tracked in EAP7.
> Add remote-cache-container configuration to Infinispan subsystem
> ----------------------------------------------------------------
>
> Key: WFLY-6634
> URL: https://issues.jboss.org/browse/WFLY-6634
> Project: WildFly
> Issue Type: Feature Request
> Components: Clustering
> Affects Versions: 10.0.0.Final
> Reporter: Paul Ferraro
> Assignee: Radoslav Husar
>
> Need to establish schema/model.
> We probably want to create new capabilities for existing container/caches to expose generic BasicCache/BasicCacheContainer types.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years, 4 months