[jboss-jira] [JBoss JIRA] Closed: (JBMESSAGING-771) One way invocations do not return the connection to the pool

One way invocations do not return the connection to the pool ------------------------------------------------------------ Key: JBMESSAGING-771 URL: http://jira.jboss.com/jira/browse/JBMESSAGING-771 Project: JBoss Messaging Issue Type: Bug Affects Versions: 1.2.0.Beta1 Reporter: Tim Fox Assigned To: Ovidiu Feodorov Priority: Blocker Fix For: 1.2.0.Beta2 When using one way invocations to delivery messages from server to client, then after sending about 50 messages, it fails, with: 16:36:34,140 TRACE @Thread-6 [ChannelSupport] Queue[0/testQueue] removing first ref in memory 16:36:34,140 TRACE @Thread-6 [ChannelSupport] Queue[0/testQueue] pushing Reference[305]:RELIABLE 16:36:34,140 TRACE @Thread-6 [ServerConsumerEndpoint] ConsumerEndpoint[8] receives Reference[305]:RELIABLE for delivery 16:36:34,140 TRACE @Thread-6 [ServerConsumerEndpoint] ConsumerEndpoint[8] has the main lock, preparing the message for delivery 16:36:34,140 TRACE @Thread-6 [ServerSessionEndpoint] SessionEndpoint[7] added delivery 50: Delivery[Reference[305]:RELIABLE](active) 16:36:34,140 TRACE @Thread-6 [ServerConsumerEndpoint] ConsumerEndpoint[8] submitting message JBossMessage[305]:PERSISTENT to the remoting layer to be sent asynchronously 16:36:34,140 DEBUG @Thread-6 [ServerInvokerCallbackHandler] ServerInvokerCallbackHandler[5c4o1b-9oatvb-ex94gcjm-1-ex94gea4-c+5c4o1b-9oatvb-ex94gcjm-1-ex94gebu-e] got PUSH callback InvocationRequest[952905] 16:36:34,140 DEBUG @Thread-6 [ServerInvokerCallbackHandler] ServerInvokerCallbackHandler[5c4o1b-9oatvb-ex94gcjm-1-ex94gea4-c+5c4o1b-9oatvb-ex94gcjm-1-ex94gebu-e] sending ASYNCHRONOUSLY the callback to the client 16:36:34,140 TRACE @Thread-6 [MicroRemoteClientInvoker] SocketClientInvoker[10d0eae, socket://192.168.1.11:2745](1) invoking InvocationRequest[b83be0] with parameter OnewayInvocation[InternalInvocation[1631573]] 16:36:34,140 TRACE @Thread-6 [MicroSocketClientInvoker] SocketClientInvoker[10d0eae, socket://192.168.1.11:2745] creating socket 49, attempt 1 16:36:34,140 TRACE @Thread-6 [SocketWrapper] creating SocketWrapper for Socket[addr=/192.168.1.11,port=2745,localport=1453], using timeout 0 16:36:34,140 TRACE @Thread-6 [MicroSocketClientInvoker] SocketClientInvoker[10d0eae, socket://192.168.1.11:2745] writing version 2 on output stream 16:36:34,140 TRACE @Thread-6 [MicroSocketClientInvoker] SocketClientInvoker[10d0eae, socket://192.168.1.11:2745] writing invocation on output stream 16:36:34,140 TRACE @Thread-6 [MicroSocketClientInvoker] SocketClientInvoker[10d0eae, socket://192.168.1.11:2745] done writing invocation on output stream 16:36:34,140 TRACE @Thread-6 [MicroSocketClientInvoker] SocketClientInvoker[10d0eae, socket://192.168.1.11:2745] sent oneway invocation, so not waiting for response, returning null 16:36:34,140 TRACE @Thread-6 [RoundRobinPointToPointRouter] receiver ConsumerEndpoint[8] handled Reference[305]:RELIABLE and returned Delivery[Reference[305]:RELIABLE](active) 16:36:34,140 TRACE @Thread-6 [ChannelSupport] Queue[0/testQueue]: Delivery[Reference[305]:RELIABLE](active) returned for message Reference[305]:RELIABLE 16:36:34,140 TRACE @Thread-6 [ChannelSupport] Queue[0/testQueue] removing first ref in memory 16:36:34,140 TRACE @Thread-6 [ChannelSupport] Queue[0/testQueue] pushing Reference[306]:RELIABLE 16:36:34,140 TRACE @Thread-6 [ServerConsumerEndpoint] ConsumerEndpoint[8] receives Reference[306]:RELIABLE for delivery 16:36:34,140 TRACE @Thread-6 [ServerConsumerEndpoint] ConsumerEndpoint[8] has the main lock, preparing the message for delivery 16:36:34,140 TRACE @Thread-6 [ServerSessionEndpoint] SessionEndpoint[7] added delivery 51: Delivery[Reference[306]:RELIABLE](active) 16:36:34,140 TRACE @Thread-6 [ServerConsumerEndpoint] ConsumerEndpoint[8] submitting message JBossMessage[306]:PERSISTENT to the remoting layer to be sent asynchronously 16:36:34,140 DEBUG @Thread-6 [ServerInvokerCallbackHandler] ServerInvokerCallbackHandler[5c4o1b-9oatvb-ex94gcjm-1-ex94gea4-c+5c4o1b-9oatvb-ex94gcjm-1-ex94gebu-e] got PUSH callback InvocationRequest[1533c8] 16:36:34,140 DEBUG @Thread-6 [ServerInvokerCallbackHandler] ServerInvokerCallbackHandler[5c4o1b-9oatvb-ex94gcjm-1-ex94gea4-c+5c4o1b-9oatvb-ex94gcjm-1-ex94gebu-e] sending ASYNCHRONOUSLY the callback to the client 16:36:34,140 TRACE @Thread-6 [MicroRemoteClientInvoker] SocketClientInvoker[10d0eae, socket://192.168.1.11:2745](1) invoking InvocationRequest[1faa614] with parameter OnewayInvocation[InternalInvocation[ad7d80]] 16:36:35,140 TRACE @Thread-6 [DefaultCallbackErrorHandler] Caught org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for locator - InvokerLocator [socket://192.168.1.11:2745/null?callbackServerHost=192.168.1.11&callbackServerPort=2745&callbackServerProtocol=socket&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&datatype=jms&serializationtype=jms&serverSocketClass=org.jboss.jms.server.remoting.ServerSocketWrapper] exception performing callback. Number of errors sending callbacks is 1 16:36:35,140 ERROR @Thread-6 [ServerInvokerCallbackHandler] Error handling callback. org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for locator - InvokerLocator [socket://192.168.1.11:2745/null?callbackServerHost=192.168.1.11&callbackServerPort=2745&callbackServerProtocol=socket&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&datatype=jms&serializationtype=jms&serverSocketClass=org.jboss.jms.server.remoting.ServerSocketWrapper] at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:328) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:118) at org.jboss.remoting.Client.invoke(Client.java:1414) at org.jboss.remoting.Client.invoke(Client.java:511) at org.jboss.remoting.Client.invokeOneway(Client.java:559) at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:686) at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:563) at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:248) at org.jboss.messaging.core.local.RoundRobinPointToPointRouter.handle(RoundRobinPointToPointRouter.java:108) at org.jboss.messaging.core.ChannelSupport.deliverInternal(ChannelSupport.java:635) at org.jboss.messaging.core.ChannelSupport$InMemoryCallback.doAfterCommit(ChannelSupport.java:1199) at org.jboss.messaging.core.ChannelSupport$InMemoryCallback.run(ChannelSupport.java:1058) at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:89) at java.lang.Thread.run(Thread.java:534) Caused by: java.net.SocketException: Can not obtain client socket connection from pool. Have waited 1 seconds for available connection (50 in use) at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:749) at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:324) ... 13 more Unfortunately I do not have the source code that corresponds to the version of remoting we are using, but, assuming things haven't changed too much, then it seems pretty clear why this bug is happening. MicroSocketClientInvoker::transport: There is a glaring and obvious bug here: if(metadata != null) { Object val = metadata.get(org.jboss.remoting.Client.ONEWAY_FLAG); if(val != null && val instanceof String && Boolean.valueOf((String)val).booleanValue()) { if(isTraceEnabled) { log.trace("Oneway invocation, so not waiting for response. Returning null."); } return null; } } I.e. if the invocation is one way then it just returns null immediately, WITHOUT returning the connection to the pool first!! The returning of the connection should be done in a finally block, since it needs to be done if an exception occurs as well. Doesn't remoting have stress tests that catch these kinds of things?