[JBoss Remoting] - Remote Client.addCallbackListener hang
by david malechek
david malechek [http://community.jboss.org/people/dmalechek] created the discussion
"Remote Client.addCallbackListener hang"
To view the discussion, visit: http://community.jboss.org/message/599414#599414
--------------------------------------------------------------
Hi,
I'm having an issue where a remote client call to createQueueConnection that hangs and never returns until a socket timeout.
We are using JBoss 4.2.3GA, Messaging 1.4.4, Remoting 2.2.3SP3. The system involves 500+ remote client machines sending 500K+ messages a day to a Queue with a message listener attached. There are a few other minor JMS queue activities as well.
It runs well until about once a day, usually during heavy load, all the clients hang, requiring a restart of the server. The CPU runs at 20% with plenty of heap and a strong Oracle box.
At this point I'm trying to figure out if I have a remoting configuration mistake, a client code mistake, a possible remoting bug, or just a limitation of the architecture during a high stress point. In the client, I don't re-use any of the JMS objects from one send to the next. I haven't been able to reproduce in a development environment yet.
I did try to back out the change made in JBREM-1081 to ServerInvoker in 2.2.3 to look like the 2.5.4 version of getCallbackHandler. It did removed the BLOCKED threads I show below, but it still fails somewhere in the ServerInvokerCallbackHandler.connect() call, probably in BisocketClientInvoker.handleConnect(). My client is reluctant to turn on trace mode in production so I'll continue to add custom logging until I track this down. My next configuration change will be trying a smaller pingFrequency.
The remote client side Thread dump looks like this.
3XMTHREADINFO "SEQMsgHandling" (TID:0x094A4800, sys_thread_t:0x094C9FB0, state:R, native ID:0x00007B14) prio=5
4XESTACKTRACE at java/net/SocketInputStream.socketRead0(Native Method)
4XESTACKTRACE at java/net/SocketInputStream.read(SocketInputStream.java:155(Compiled Code))
4XESTACKTRACE at java/io/BufferedInputStream.fill(BufferedInputStream.java:229(Compiled Code))
4XESTACKTRACE at java/io/BufferedInputStream.read(BufferedInputStream.java:246(Compiled Code))
4XESTACKTRACE at java/io/FilterInputStream.read(FilterInputStream.java:89(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/transport/socket/MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1077(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/transport/socket/MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:682(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/transport/bisocket/BisocketClientInvoker.transport(BisocketClientInvoker.java:458(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/Client.invoke(Client.java:1925(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/Client.addCallbackListener(Client.java:1994(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/Client.addListener(Client.java:1158(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/remoting/JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:259(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/remoting/JMSRemotingConnection.start(JMSRemotingConnection.java:388(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/container/StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80(Compiled Code))
4XESTACKTRACE at org/jboss/aop/advice/org/jboss/jms/client/container/StateCreationAspect0.invoke(StateCreationAspect0.java(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/delegate/ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101(Compiled Code))
4XESTACKTRACE at org/jboss/jms/client/JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95(Compiled Code))
...
The server side thread dump has all the ServerThread like on of these two:
Thread: WorkerThread#0[10.31.48.175:46194] : priority:5, demon:false, threadId:494, threadState:BLOCKED, lockName:java.util.HashMap@18acff0
org.jboss.remoting.ServerInvoker.removeCallbackHandler(ServerInvoker.java:1949)
org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1514)
org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:648)
org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:849)
org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:611)
org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:423)
org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:176)
Thread: WorkerThread#4[10.26.68.175:57044] : priority:5, demon:false, threadId:598, threadState:BLOCKED, lockName:java.util.HashMap@18acff0
org.jboss.remoting.ServerInvoker.getCallbackHandler(ServerInvoker.java:1927)
org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1501)
org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:648)
org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:849)
org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:611)
org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:423)
org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:176)
My Server side remoting-bisocket-service.xml contains:
<mbean code="org.jboss.remoting.transport.Connector"
name="jboss.messaging:service=Connector,transport=bisocket"
display-name="Bisocket Transport Connector">
<attribute name="Configuration">
<config>
<invoker transport="bisocket">
<!-- There should be no reason to change these parameters - warning!
Changing them may stop JBoss Messaging working correctly -->
<attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
<attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
<attribute name="dataType" isParam="true">jms</attribute>
<attribute name="socket.check_connection" isParam="true">false</attribute>
<attribute name="serverBindAddress">${jboss.bind.address}</attribute>
<attribute name="serverBindPort">14457</attribute>
<attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
<attribute name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>
<attribute name="numberOfCallRetries" isParam="true">1</attribute>
<attribute name="pingFrequency" isParam="true">214748364</attribute>
<attribute name="pingWindowFactor" isParam="true">10</attribute>
<attribute name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>
<!-- End immutable parameters -->
<attribute name="stopLeaseOnFailure" isParam="true">true</attribute>
<!-- Periodicity of client pings. Server window by default is twice this figure -->
<attribute name="clientLeasePeriod" isParam="true">10000</attribute>
<attribute name="validatorPingPeriod" isParam="true">10000</attribute>
<attribute name="validatorPingTimeout" isParam="true">5000</attribute>
<attribute name="failureDisconnectTimeout" isParam="true">0</attribute>
<attribute name="callbackErrorsAllowed">1</attribute>
<attribute name="registerCallbackListener">false</attribute>
<attribute name="useClientConnectionIdentity" isParam="true">true</attribute>
<attribute name="timeout" isParam="true">120000</attribute>
<!-- Number of seconds to wait for a connection in the client pool to become free -->
<attribute name="numberOfRetries" isParam="true">10</attribute>
<!-- Max Number of connections in client pool. This should be significantly higher than
the max number of sessions/consumers you expect -->
<attribute name="JBM_clientMaxPoolSize" isParam="true">200</attribute>
<!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->
<attribute name="callbackTimeout">10000</attribute>
<!-- Use these parameters to specify values for binding and connecting control connections to
work with your firewall/NAT configuration
<attribute name="secondaryBindPort">xyz</attribute>
<attribute name="secondaryConnectPort">abc</attribute>
-->
</invoker>
<handlers>
<handler subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>
</handlers>
</config>
</attribute>
</mbean>
Thanks in advance for any insights.
David Malechek
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/599414#599414]
Start a new discussion in JBoss Remoting at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&cont...]
15 years
[jBPM] - Re: Drools, Spring integration and JTA in container
by Fabio Wang
Fabio Wang [http://community.jboss.org/people/fabiowg] created the discussion
"Re: Drools, Spring integration and JTA in container"
To view the discussion, visit: http://community.jboss.org/message/598937#598937
--------------------------------------------------------------
Hi there,
After some time struggling to make this "Spring + JTA + Drools" integration, I could make it work, but I'm still unsure if it's the right way. As everyone pointed out, the SingleSessionComand#initTransactionManager ignores the case where a org.springframework.transaction.jta.JtaTransactionManager is used. So I had to manually create an Environment like this:
Environment environment = EnvironmentFactory.newEnvironment();
environment.set(EnvironmentName.ENTITY_MANAGER_FACTORY, emf);
DroolsSpringTransactionManager jtaTransactionManager = new DroolsSpringTransactionManager(ptm);
this.environment.set(EnvironmentName.TRANSACTION_MANAGER, jtaTransactionManager);
DroolsSpringJpaManager pcm = new DroolsSpringJpaManager(environment);
environment.set(EnvironmentName.PERSISTENCE_CONTEXT_MANAGER, pcm);
And, after that, I create my ksession (unfortunately, without the <drools:ksession/> shortcut):
ksession = JPAKnowledgeService.newStatefulKnowledgeSession(this.kbase, null, environment);
Hope this helps.
Fábio
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/598937#598937]
Start a new discussion in jBPM at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&cont...]
15 years
[JBoss Messaging] - Does QueueSession require transaction?
by Bernard Labno
Bernard Labno [http://community.jboss.org/people/blabno] created the discussion
"Does QueueSession require transaction?"
To view the discussion, visit: http://community.jboss.org/message/599318#599318
--------------------------------------------------------------
Hi, I'm trying to make my pojo seam component to act as MessageListener. I've experienced something that is strange for me.
If i create transacted queue session (XASession) (without creating transaction, it's done in @Create method of component in application context at application startup), then start transaction and try to read message and then rollback the transaction then message does not get back to the queue.
However if i start transaction, create transacted queue session, pick message from queue and rollback transaction then message is put back in the queue.
The difference is that i start transaction before creating the session.
This limits me to using one session for each message if I want to have the atomicity. This way i have to use readNoWait instead of providing MessageListener implementing class.
Am I missing something? How is it done form MDBs? Is session created for every single message?
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/599318#599318]
Start a new discussion in JBoss Messaging at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&cont...]
15 years
[jBPM] - Executing business logic as part of the user task
by Sujeevan Nagarajah
Sujeevan Nagarajah [http://community.jboss.org/people/sujee] created the discussion
"Executing business logic as part of the user task"
To view the discussion, visit: http://community.jboss.org/message/599305#599305
--------------------------------------------------------------
Hi All,
I'm doing a POC workflow engine using jBPM5. Nodes in the workflow represent the actions which will cause a change in the workflow status. for e.g. I have 2 nodes - create (status after execution = "Draft") and approve (status after execution = "Approved"). All workflow changes will be triggered by human actors and once triggered it will execute some business logic. In BPMN 2.0 and jBPM5, these are two different tasks. Human actor triggering the workflow is represented by user task type. Executing business logic is represented by service task type. Is it possible to run them together in the same transaction? what is the best way to design this in BPMN 2.0 and jBPM5? If I model a user task followed by a service task, how to rollback user task when service task is failed?
Thank you.
--------------------------------------------------------------
Reply to this message by going to Community
[http://community.jboss.org/message/599305#599305]
Start a new discussion in jBPM at Community
[http://community.jboss.org/choose-container!input.jspa?contentType=1&cont...]
15 years