[
https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji...
]
Howard Gao commented on JBMESSAGING-1456:
-----------------------------------------
Thanks Tim, I do need help. :)
Today I tried to setup the cluster on one machine and spring test clients on another. I
got outofmemory error on a server node about 10 more min after the clients have been
started, before seeing any stuck messages. I enlarged the memory on the server node and
reran the tests, I got the remoting exception I mentioned before (Error notifying
connection listeners of disconnected client connection). So I think maybe it's because
of I am using the Community JBoss. So I downloaded the JBoss EAP 4.3 cp02 and set up the
cluster again. For the moment I'm have problems with JNDI lookup issues. Please help
me to quickly reproduce the issue.
The jndi problem is that I have successfully started up the cluster, with jmx-console
access OK from remote machine. (Tim point out the doc for me where I can remove the
security constraints for EAP)
but the strange thing is the jndi look up all failed, even from localhost (client in the
same host as the server nodes). I have no idea. I changed the JNDI configuration using the
server's IP and the port is correct (this works with the community jboss before). But
there is still 127.0.0.1 appearing in the error, as
$ java -Xms256m -Xmx256m -jar ./TestClient.jar
conf/spring-contexts/testClientConnectivityContext.xml
log4j:WARN No appenders could be found for logger
(org.springframework.context.support.FileSystemXmlApplicationContext).
log4j:WARN Please initialize the log4j system properly.
Exception in thread "main"
org.springframework.beans.factory.BeanCreationException: Error creating bean with name
'authWrapperJbossLoginFactory' defined in file
[/home/howard/projects/issues/229464-1456/testclient/conf/spring-contexts/testClientConnectivityContext.xml]:
Cannot resolve reference to bean 'clusteredConnectionFactory' while setting bean
property 'targetConnectionFactory'; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating bean with name
'clusteredConnectionFactory' defined in file
[/home/howard/projects/issues/229464-1456/testclient/conf/spring-contexts/testClientConnectivityContext.xml]:
Invocation of init method failed; nested exception is javax.naming.CommunicationException
[Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1;
nested exception is:
java.net.ConnectException: Connection refused]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'clusteredConnectionFactory' defined in file
[/home/howard/projects/issues/229464-1456/testclient/conf/spring-contexts/testClientConnectivityContext.xml]:
Invocation of init method failed; nested exception is javax.naming.CommunicationException
[Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1;
nested exception is:
java.net.ConnectException: Connection refused]
Caused by: javax.naming.CommunicationException [Root exception is
java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused]
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:780)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
at javax.naming.InitialContext.lookup(InitialContext.java:351)
at org.springframework.jndi.JndiTemplate$1.doInContext(JndiTemplate.java:123)
at org.springframework.jndi.JndiTemplate.execute(JndiTemplate.java:85)
at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:121)
at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:146)
at org.springframework.jndi.JndiLocatorSupport.lookup(JndiLocatorSupport.java:93)
at org.springframework.jndi.JndiObjectLocator.lookup(JndiObjectLocator.java:105)
at
org.springframework.jndi.JndiObjectFactoryBean.lookupWithFallback(JndiObjectFactoryBean.java:197)
at
org.springframework.jndi.JndiObjectFactoryBean.afterPropertiesSet(JndiObjectFactoryBean.java:184)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1201)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1171)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:425)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:251)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:156)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:248)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160)
at
org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:261)
at
org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:109)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1099)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:861)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:421)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:251)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:156)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:248)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:287)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:352)
at
org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:93)
at
org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:77)
at
org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:67)
at
com.ssn.jms.integration.test.bootstrap.SimpleSpringContextBootStrap.run(SimpleSpringContextBootStrap.java:29)
at
com.ssn.jms.integration.test.bootstrap.SimpleSpringContextBootStrap.main(SimpleSpringContextBootStrap.java:20)
Caused by: java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested
exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:574)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:185)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:94)
at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:667)
... 33 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:520)
at java.net.Socket.connect(Socket.java:470)
at java.net.Socket.<init>(Socket.java:367)
at java.net.Socket.<init>(Socket.java:180)
at
sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at
sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:569)
... 38 more
Please do help me. Thanks.
Messages stuck in being-delivered state in cluster
--------------------------------------------------
Key: JBMESSAGING-1456
URL:
https://jira.jboss.org/jira/browse/JBMESSAGING-1456
Project: JBoss Messaging
Issue Type: Bug
Affects Versions: 1.4.0.SP3_CP03
Reporter: Justin Bertram
Assignee: Howard Gao
Priority: Critical
Attachments: BaseProducerThread.java, kill3_thread_dump.txt, thread_dump.txt
Messages become "stuck" in being-delivered state when clients use a clustered
XA connection factory in a cluster of at least 2 nodes.
JBoss setup:
-2 nodes of JBoss EAP 4.3 CP02
-commented out "ClusterPullConnectionFactory" in messaging-service.xml to
prevent message redistribution and eliminate the "message suckers" as the
potential culprit
-MySQL backend using the default mysql-persistence-service.xml (from
<JBOSS_HOME>/docs/examples/jms)
Client setup:
-both nodes have a client which is a separate process (i.e. not inside JBoss)
-clients are Spring based
-one client produces and consumes, the other client just consumes
-both clients use the ClusteredXAConnectionFactory from the default
connection-factories-service.xml
-both clients publish to and consume from "queue/testDistributedQueue"
-clients are configured to send persistent messages, use AUTO_ACKNOWLEDGE, and
transacted sessions
Symptoms of the issue:
-when running the clients I watch the JMX-Console for the
"queue/testDistributedQueue"
-as the consumers pull messages off the queue I can see the MessageCount and
DeliveringCount go to 0 every so often
-after a period of time (usually a few hours) the MessageCount and DeliveringCount
never go back to 0
-I "kill" the clients and wait for the DeliveringCount to go to 0, but it
never does
-after the clients are killed the ConsumerCount for the queue will drop, but never to 0
when messages are "stuck"
-a thread dump reveals at least one JBM server session that is apparently stuck (it
never goes away) - ostensibly this is the consumer that is showing in the JMX-Console for
"queue/testDistributedQueue"
-a "killall -3 java" doesn't produce anything from the clients so I know
their dead
-nothing is in any DLQ or expiry queue
-the database contains as many rows in the JBM_MSG and JBM_MSG_REF tables as the
DeliveringCount in the JMX-Console
-rebooting the node with the stuck messages frees the messages to be consumed (i.e.
un-sticks them)
Other notes:
-nothing else is happening on either node but running the client and running JBoss
-this only appears to happen when a clustered connection factory is used. I tested
using a normal connection factory and after 24 hours couldn't reproduce a stuck
message.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira