[Remoting] - EJB3/Socket invoker - connection timeouts
by javajedi
We're running into a problem with a brand new production system that has a single remote EJB. We are using JBoss EJB3, which uses JBoss Remoting to do the remote communication. We are using the standard socket invoker to handle the communication. We are using JBoss Serialization instead of Java Serialization, which, in tests, provided significantly better overall performance. We've been testing this in a stage environment for several months, with no problems. Once we got it into production, we started to hit a wall. We reach a point where the socket connections between the client and the server just start timing out. A thread dump on the client shows a bunch of threads waiting for a socket connection to the server. A thread dump on the server shows several SeverThreads waiting to be used (via ServerThread.wakeup()). The problem appears to be that the thread in SocketServerInvoker that is responsible for the calls to serverSocket.accept() is getting stuck, or at least severely slowed. In the server side thread dump, here is the stack of that accept thread:
Thread: SocketServerInvoker#0-3873 : priority:5, demon:false, threadId:23, threadState:RUNNABLE, threadLockName:null
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
java.io.BufferedInputStream.read(BufferedInputStream.java:317)
java.io.FilterInputStream.read(FilterInputStream.java:90)
org.jboss.serial.io.JBossObjectInputStream.checkSignature(JBossObjectInputStream.java:110)
org.jboss.serial.io.JBossObjectInputStream.(JBossObjectInputStream.java:94)
org.jboss.serial.io.JBossObjectInputStream.(JBossObjectInputStream.java:83)
org.jboss.remoting.serialization.impl.jboss.JBossSerializationManager.createInput(JBossSerializationManager.java:57)
org.jboss.remoting.transport.socket.ServerSocketWrapper.createInputStream(ServerSocketWrapper.java:56)
org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:76)
org.jboss.remoting.transport.socket.ClientSocketWrapper.(ClientSocketWrapper.java:54)
org.jboss.remoting.transport.socket.ServerSocketWrapper.(ServerSocketWrapper.java:50)
sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
org.jboss.remoting.transport.socket.ServerThread.createServerSocket(ServerThread.java:185)
org.jboss.remoting.transport.socket.ServerThread.(ServerThread.java:87)
org.jboss.remoting.transport.socket.SocketServerInvoker.processInvocation(SocketServerInvoker.java:429)
org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:391)
java.lang.Thread.run(Thread.java:619)
It appears that this thread, which ought to be quickly finding a ServerThread to handle the incoming request, calling ServerThread.start(), and getting back to the call to serverSocket.accept() as quickly as possible, is instead doing some socket I/O. ServerSocketWrapper.java has a method called createInputStream() that creates a new ObjectInputStream. The stack trace shows that the JBossObjectInputStream constructor is reading some bytes from the input stream, but I don't think that this problem is specific to JBoss Serialization, as java.io.ObjectInputStream's constructor also reads bytes from the stream.
It seems to me that this could be the cause of the problem that we are seeing. Any time that you're doing a blocking socket read like this, you can potentially just sit for a while until the socket times out. When you've got a thread calling serverSocket.accept(), you can't normally afford to do something like this on that thread.
Does this analysis make sense? Any thoughts/suggestions would be greatly appreciated.
Thanks.
--Tim
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4057841#4057841
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4057841
18Â years, 10Â months
[Messaging, JMS & JBossMQ] - SpyJMSException !!!
by lethrech
hi,
i have configured jboss (with an ESB service) with oracle 10g in stead of hqldb, the server start without problems but when i execute my application which use ESB i have the following problem:
2007 06 26 - 11:47:33 INFO org.jboss.internal.soa.esb.couriers.JmsCourier.jmsConnectRetry [2007 06
| 26 - 11:47:33 ERROR org.jboss.internal.soa.esb.couriers.JmsCourier.jmsConnectRetry [JMS error. Atte
| mpting JMS reconnect.]
| org.jboss.mq.SpyJMSException: Could not store message: 1 msg=0 hard NOT_STORED PERSISTENT queue=QUEU
| E.A priority=4 lateClone=false hashCode=32771983; - nested throwable: (java.sql.SQLException: Type d
| e colonne non valide)
| at org.jboss.mq.pm.jdbc2.PersistenceManager.add(PersistenceManager.java:941)
| at org.jboss.mq.server.PersistentQueue.addMessage(PersistentQueue.java:58)
| at org.jboss.mq.server.JMSQueue.addMessage(JMSQueue.java:179)
| at org.jboss.mq.server.JMSDestinationManager.addMessage(JMSDestinationManager.java:406)
| at org.jboss.mq.server.JMSDestinationManager.addMessage(JMSDestinationManager.java:390)
| at org.jboss.mq.server.JMSServerInterceptorSupport.addMessage(JMSServerInterceptorSupport.java:150)
|
| at org.jboss.mq.security.ServerSecurityInterceptor.addMessage(ServerSecurityInterceptor.java:168)
| at org.jboss.mq.server.TracingInterceptor.addMessage(TracingInterceptor.java:266)
| at org.jboss.mq.server.JMSServerInvoker.addMessage(JMSServerInvoker.java:151)
| at org.jboss.mq.il.uil2.ServerSocketManagerHandler.handleMsg(ServerSocketManagerHandler.java:108)
| at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleMsg(SocketManager.java:396)
| at org.jboss.mq.il.uil2.msgs.BaseMsg.run(BaseMsg.java:392)
| at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)
| at java.lang.Thread.run(Thread.java:595)
| Caused by: java.sql.SQLException: Type de colonne non valide
folowing are the script of jboss schema:
CREATE TABLE TIMERS(TIMERID VARCHAR(80) NOT NULL,TARGETID VARCHAR(250) NOT NULL,INITIALDATE TIMESTAM
| P NOT NULL,TIMERINTERVAL SYS.ANYDATA,INSTANCEPK SYS.ANYDATA,INFO SYS.ANYDATA,CONSTRAINT TIMERS_PK PR
| IMARY KEY(TIMERID,TARGETID));
| CREATE TABLE HILOSEQUENCES(SEQUENCENAME VARCHAR(50) NOT NULL,HIGHVALUES INTEGER NOT NULL,CONSTRAINT
| HILO_PK PRIMARY KEY(SEQUENCENAME));
| CREATE TABLE JMS_USERS(USERID VARCHAR(32) NOT NULL PRIMARY KEY,PASSWD VARCHAR(32) NOT NULL,CLIENTID
| VARCHAR(128));
| CREATE TABLE JMS_ROLES(ROLEID VARCHAR(32) NOT NULL,USERID VARCHAR(32) NOT NULL,PRIMARY KEY(USERID,RO
| LEID));
| CREATE TABLE JMS_SUBSCRIPTIONS(CLIENTID VARCHAR(128) NOT NULL,SUBNAME VARCHAR(128) NOT NULL,TOPIC VA
| RCHAR(255) NOT NULL,SELECTOR VARCHAR(255),PRIMARY KEY(CLIENTID,SUBNAME));
| CREATE TABLE JMS_MESSAGES(MESSAGEID INTEGER NOT NULL,DESTINATION VARCHAR(255) NOT NULL,TXID INTEGER,
| TXOP CHAR(1),MESSAGEBLOB SYS.ANYDATA,PRIMARY KEY(MESSAGEID,DESTINATION));
| CREATE INDEX JMS_MESSAGES_TXOP_TXID ON JMS_MESSAGES(TXOP,TXID);
| CREATE INDEX JMS_MESSAGES_DESTINATION ON JMS_MESSAGES(DESTINATION);
| CREATE TABLE JMS_TRANSACTIONS(TXID INTEGER NOT NULL PRIMARY KEY);
i m using jboss4.0.4
DB: oracle10g 10.2.0.3
OS: XP professional 2002 SP2
can you help me
best regards
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4057838#4057838
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4057838
18Â years, 10Â months