[jboss-dev-forums] [Design of Messaging on JBoss (Messaging/JBoss)] - Scaling UP tests

[clebert.suconic@jboss.com]

I have been trying to scale up few tests with nonTransactionSends/Persistent (that means.. every send requires transaction), and this scaled pretty nicely with AIO.

With NIO I've got about 350 sends / second with a single perfSender, because of the syncs on every send.

With AIO I've got 1200 sends /s econd with a single perfSender. 
I've also tested with 5 perfSender, and I've got the same rate on every perfSender. That means.. it scaled without any problem.


*However* when I added a listener, things were much different.


The rate on the send was terrible, and I got a bunch of threads waiting on this condition:



  |      [java] "pool-7-thread-4" prio=1 tid=0x00007fb8e96d56c0 nid=0x3d44 waiting for monitor entry [0x000000004396f000..0x000000004396fb20]
  |      [java]     at org.jboss.messaging.core.server.impl.QueueImpl.addLast(QueueImpl.java:159)
  |      [java]     - waiting to lock <0x00007fb8f0a62968> (a org.jboss.messaging.core.server.impl.QueueImpl)
  |      [java]     at org.jboss.messaging.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:321)
  |      [java]     at org.jboss.messaging.core.server.impl.ServerProducerImpl.send(ServerProducerImpl.java:103)
  |      [java]     at org.jboss.messaging.core.server.impl.ServerProducerPacketHandler.doHandle(ServerProducerPacketHandler.java:61)
  |      [java]     at org.jboss.messaging.core.server.impl.ServerPacketHandlerSupport.handle(ServerPacketHandlerSupport.java:48)
  |      [java]     at org.jboss.messaging.core.remoting.impl.PacketDispatcherImpl.dispatch(PacketDispatcherImpl.java:139)
  |      [java]     at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.messageReceivedInternal(MinaHandler.java:283)
  |      [java]     at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.access$000(MinaHandler.java:32)
  |      [java]     at org.jboss.messaging.core.remoting.impl.mina.MinaHandler$1.run(MinaHandler.java:165)
  |      [java]     at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:70)
  |      [java]     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
  |      [java]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
  |      [java]     at java.lang.Thread.run(Thread.java:595)
  |      [java]
  | 


While this thread was holding the lock:


     [java] "pool-7-thread-1" prio=1 tid=0x00007fb8e9870bc0 nid=0x3d33 in Object.wait() [0x00000000413a4000..0x00000000413a4da0]
  |      [java]     at java.lang.Object.wait(Native Method)
  |      [java]     - waiting on <0x00007fb8f09b0528> (a org.jboss.messaging.core.remoting.impl.mina.MinaHandler)
  |      [java]     at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.checkWrite(MinaHandler.java:210)
  |      [java]     - locked <0x00007fb8f09b0528> (a org.jboss.messaging.core.remoting.impl.mina.MinaHandler)
  |      [java]     at org.jboss.messaging.core.remoting.impl.mina.MinaHandler$2.send(MinaHandler.java:251)
  |      [java]     at org.jboss.messaging.core.server.impl.DeliveryImpl.deliver(DeliveryImpl.java:80)
  |      [java]     at org.jboss.messaging.core.server.impl.ServerSessionImpl.handleDelivery(ServerSessionImpl.java:222)
  |      [java]     at org.jboss.messaging.core.server.impl.ServerConsumerImpl.handle(ServerConsumerImpl.java:217)
  |      [java]     - locked <0x00007fb8f0a62a90> (a java.lang.Object)
  |      [java]     at org.jboss.messaging.core.server.impl.QueueImpl.deliver(QueueImpl.java:631)
  |      [java]     at org.jboss.messaging.core.server.impl.QueueImpl.add(QueueImpl.java:528)
  |      [java]     at org.jboss.messaging.core.server.impl.QueueImpl.addLast(QueueImpl.java:159)
  |      [java]     - locked <0x00007fb8f0a62968> (a org.jboss.messaging.core.server.impl.QueueImpl)
  |      [java]     at org.jboss.messaging.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:321)
  |      [java]     at org.jboss.messaging.core.server.impl.ServerProducerImpl.send(ServerProducerImpl.java:103)
  |      [java]     at org.jboss.messaging.core.server.impl.ServerProducerPacketHandler.doHandle(ServerProducerPacketHandler.java:61)
  |      [java]     at org.jboss.messaging.core.server.impl.ServerPacketHandlerSupport.handle(ServerPacketHandlerSupport.java:48)
  |      [java]     at org.jboss.messaging.core.remoting.impl.PacketDispatcherImpl.dispatch(PacketDispatcherImpl.java:139)
  |      [java]     at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.messageReceivedInternal(MinaHandler.java:283)
  |      [java]     at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.access$000(MinaHandler.java:32)
  |      [java]     at org.jboss.messaging.core.remoting.impl.mina.MinaHandler$1.run(MinaHandler.java:165)
  |      [java]     at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:70)
  |      [java]     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
  |      [java]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
  |      [java]     at java.lang.Thread.run(Thread.java:595)
  |      [java]
  | 


I haven't taken any conclusions about what's wrong as I haven't looked on the code yet. These are just figures from my test.

View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4153096#4153096

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4153096