[jboss-jira] [JBoss JIRA] (AS7-3919) HornetQ causing deadlock when max-size-bytes reaches its limit

[Ove Ranheim (JIRA)]

    [ https://issues.jboss.org/browse/AS7-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671451#comment-12671451 ] 

Ove Ranheim commented on AS7-3919:
----------------------------------

We use Seam 3.1.0.Final together with JBoss AS 7.1.0.Final.

In Seam we bind the CDI Event to the JMS Queue by:

{noformat}
public interface JmsMailRoutingListener {

    final static String MAIL_JMS_DESTINATION = "queue/sendMail";

    @Routing(RouteType.EGRESS)
    public void mapAsyncMailToQueue(
            @Observes AsyncEmailMessage mailMessage,
            @JmsDestination(jndiName = MAIL_JMS_DESTINATION) Queue q
    );
}
{noformat}

The MDB is implemented as:

{noformat}
@MessageDriven(name = "MailProcessor", activationConfig = {
        @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
        @ActivationConfigProperty(propertyName = "destination", propertyValue = JmsMailRoutingListener.MAIL_JMS_DESTINATION)})
public class MailMessageDrivenBean implements MessageListener {
    @Override
    public void onMessage(Message message) {
        AsyncEmailMessage asyncEmailMessage = objectMessageExtractor.extractObject(message, AsyncEmailMessage.class);
        // handling code goes here
    }

}
{noformat}

To put an object on the queue, we do:

{noformat}
@ApplicationScoped
public class ContractMessagingBean implements Serializable {

    @Inject
    private Event<AsyncEmailMessage> notificationEvent;

    private String sendNotification(PerformanceContract contract, String templateName, List<EmailContact> recipients, ContractPartyName recipient, boolean id) {
        AsyncEmailMessage notification = notificationInstance.get();
        notificationEvent.fire(notification);
    }
}
{noformat}

So what's happening here are:

1. We bind with RoutType.EGRESS so all CDI Event objects of type AsyncEmailMessage will be sent as a persisted object onto the sendMailQueue through JmsMailRoutingListener (this is a great capability of Seam JMS)
2. We fire AsyncMailMessage onto the CDI Event observer from ContractMessagingBean.sendNotification(..)
3. The message will be picked up when notified by HornetQ on MailMessageDrivenBean.onMessage(..)
4. We extract the object and handle it. Finally we send a mail out through the smtp gateway.

After sending approx 30 messages onto the queue, we get the exception described in this issue.

                
> HornetQ causing deadlock when max-size-bytes reaches its limit
> --------------------------------------------------------------
>
>                 Key: AS7-3919
>                 URL: https://issues.jboss.org/browse/AS7-3919
>             Project: Application Server 7
>          Issue Type: Bug
>          Components: JMS
>    Affects Versions: 7.1.0.Final
>            Reporter: Ove Ranheim
>            Assignee: Andy Taylor
>            Priority: Critical
>             Fix For: 7.1.1.Final
>
>         Attachments: jstack-2012-02-27_08_49.txt, jstack-2012-02-27_09_58_crash.txt, standalone-musific.xml
>
>
> I have journal-file-size set to 10 Mb and max-size-bytes set 10 Mb, with a address policy of BLOCK. When my sendMailQueue reaches its limit a deadlock occurs. This is further emphasized in the JBossAS Forum reference.
> Here's a snippet from jstack after HornetQ crashes.
> {noformat}
> Found one Java-level deadlock:
> =============================
> "Thread-878 (HornetQ-client-global-threads-1151364488)":
>   waiting to lock monitor 0x0000000041df8640 (object 0x00000000ea17d6e0, a java.lang.Object),
>   which is held by "Thread-873 (HornetQ-client-global-threads-1151364488)"
> "Thread-873 (HornetQ-client-global-threads-1151364488)":
>   waiting to lock monitor 0x000000004107c8b8 (object 0x00000000ea17d6d0, a java.lang.Object),
>   which is held by "Thread-878 (HornetQ-client-global-threads-1151364488)"
> Java stack information for the threads listed above:
> ===================================================
> "Thread-878 (HornetQ-client-global-threads-1151364488)":
> 	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:461)
> 	- waiting to lock <0x00000000ea17d6e0> (a java.lang.Object)
> 	- locked <0x00000000ea17d6d0> (a java.lang.Object)
> 	at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1294)
> 	at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1238)
> 	at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:392)
> 	at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
> 	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
> 	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
> 	- locked <0x00000000f0222060> (a java.lang.Object)
> 	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
> 	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
> 	at org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
> 	at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> "Thread-873 (HornetQ-client-global-threads-1151364488)":
> 	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:458)
> 	- waiting to lock <0x00000000ea17d6d0> (a java.lang.Object)
> 	at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.close(HornetQXAResourceWrapper.java:391)
> 	at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connectionFailed(HornetQXAResourceWrapper.java:232)
> 	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:905)
> 	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:690)
> 	- locked <0x00000000ea17d6e0> (a java.lang.Object)
> 	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:556)
> 	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:79)
> 	at org.hornetq.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:386)
> 	at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> Found 1 deadlock.
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira