[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Justin Bertram updated JBMESSAGING-1456: ---------------------------------------- Affects Version/s: 1.4.0.SP3_CP03 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Justin Bertram updated JBMESSAGING-1456: ---------------------------------------- Attachment: thread_dump.txt This is a thread dump (from the JMX-Console, not from a kill -3) from JBoss when 72 messages were stuck in "queue/testDistributedQueue" and the ConsumerCount was "1". All the clients were, of course, dead. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Jay Howell commented on JBMESSAGING-1456: ----------------------------------------- In order to run the test client, you need to run... "java -Xms256m -Xmx256m -jar TestClient.jar conf/spring-contexts/testClientConnectivityContext.xml" -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Tim Fox commented on JBMESSAGING-1456: -------------------------------------- I'd also be interested i knowing if failover has occurred. What happens if supportsFailover is set to false in the connection factory? -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Tim Fox updated JBMESSAGING-1456: --------------------------------- Assignee: Howard Gao (was: Tim Fox) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Zach Kurey commented on JBMESSAGING-1456: ----------------------------------------- Hey Howard, I'm the customer associated with the issue. Sorry I didn't clean out these classes, they are unneeded to run the sample client I provided. You can enable or disable consuming/producing by modifying the file conf/configuration/testclient.configuration.properties. There are producer.enabled/consumer.enabled properties you can modify. You can ignore the 'heartbeat' related properties as they are not pertinent to the issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Got it running finally, thanks to the java decompiler. Thanks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Zach Kurey commented on JBMESSAGING-1456: ----------------------------------------- Howard, Sorry for the slow response, glad you got the client up and running. I just wanted to respond to some of your assumptions/how the test is being run. "This is because Spring will hold a pool of consumers for efficiency. When all messages are received, the number of consumers will drop down to 5 according to my observation. " The 'cacheLevel' property for the listener container in this test client is set to 0(CACHE_NONE). There should be no pooling of connections/sessions/nor consumers. This leads Spring to continually open a connection, create a session, create a consumer, then close the consumer/session/connection, with ever poll checking for new messages. This bug would probably not be observed so long as the underlying connection factory was a proper connection pool, but in the default case there is no pool. This leads to the undesirable case of opening and closing connections quite rapidly. "The mean time to failure for a stuck message may vary depending on the different test client configures and on different hardwares(machines). That may cause misunderstandings between us because your observation may be very different from mine." The way I've been running the test is to produce messages in bursts of 350 messages at a time. Then we sleep for 30-40 seconds between bursts letting the queue completely drain before sending more messages. You can control this with the message.delay property in the client. Also I've been running with two nodes so each node only gets messages published to it every 60 seconds as the producer is round robining on the two. This makes it much easier to witness consumer counts getting reduced on a particular instance below 5. 5 is just another client setting: consumer.min.count. Once messages are published in a burst to a particular node the clients threads should all be busy working on that one node; though they should still be creating and closing connections on the non-busy node quite quickly. The connections on the non-busy node may be being opened/closed quickly enough to not be visible in the console leading to an appearance of a consumer count of 0, at times. Regardless eventually a node, usually the one that is also running the MySQL instance and consumer locally on the same server, eventually will have its consumer count reach a non-zero floor. More importantly the delivering count will reach a non-zero floor which should never happen in this test scenario as each node should get drained entirely with each burst. Also I'm not sure what prefetch size is set on the connection factory your using, but leaving it at the default, 150, leads to the issue being reproduced much more quickly. Reducing the pre-fetch size to 1 the issue can still be reproduced but much less frequently. "However, you cannot shut down the consumer process by kill and then observe -- messages perhaps are being received and not acknowledged. The consumer processes (or threads) must be alive all the time when observing." The issue is always initially observed with the client consumers still active. Once it is clear that the message delivering count is never going to reach 0 again(wait any amount of time it won't go back to 0), then we've killed the clients and observed the delivering count is still non-zero. As far as whether or not killing the clients is a valid test, I disagree. If I kill the clients I would always expect the state of the broker to eventually determine the clients no longer exist and that message delivery was not successful returning the delivering and client counts both to 0, but maybe I'm misinterpreting your comment. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao updated JBMESSAGING-1456: ------------------------------------ Attachment: BaseProducerThread.java This file I changed to check message count before sending. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Thanks Tim, I do need help. :) Today I tried to setup the cluster on one machine and spring test clients on another. I got outofmemory error on a server node about 10 more min after the clients have been started, before seeing any stuck messages. I enlarged the memory on the server node and reran the tests, I got the remoting exception I mentioned before (Error notifying connection listeners of disconnected client connection). So I think maybe it's because of I am using the Community JBoss. So I downloaded the JBoss EAP 4.3 cp02 and set up the cluster again. For the moment I'm have problems with JNDI lookup issues. Please help me to quickly reproduce the issue. The jndi problem is that I have successfully started up the cluster, with jmx-console access OK from remote machine. (Tim point out the doc for me where I can remove the security constraints for EAP) but the strange thing is the jndi look up all failed, even from localhost (client in the same host as the server nodes). I have no idea. I changed the JNDI configuration using the server's IP and the port is correct (this works with the community jboss before). But there is still 127.0.0.1 appearing in the error, as $ java -Xms256m -Xmx256m -jar ./TestClient.jar conf/spring-contexts/testClientConnectivityContext.xml log4j:WARN No appenders could be found for logger (org.springframework.context.support.FileSystemXmlApplicationContext). log4j:WARN Please initialize the log4j system properly. Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authWrapperJbossLoginFactory' defined in file [/home/howard/projects/issues/229464-1456/testclient/conf/spring-contexts/testClientConnectivityContext.xml]: Cannot resolve reference to bean 'clusteredConnectionFactory' while setting bean property 'targetConnectionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusteredConnectionFactory' defined in file [/home/howard/projects/issues/229464-1456/testclient/conf/spring-contexts/testClientConnectivityContext.xml]: Invocation of init method failed; nested exception is javax.naming.CommunicationException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused] Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'clusteredConnectionFactory' defined in file [/home/howard/projects/issues/229464-1456/testclient/conf/spring-contexts/testClientConnectivityContext.xml]: Invocation of init method failed; nested exception is javax.naming.CommunicationException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused] Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:780) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627) at javax.naming.InitialContext.lookup(InitialContext.java:351) at org.springframework.jndi.JndiTemplate$1.doInContext(JndiTemplate.java:123) at org.springframework.jndi.JndiTemplate.execute(JndiTemplate.java:85) at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:121) at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:146) at org.springframework.jndi.JndiLocatorSupport.lookup(JndiLocatorSupport.java:93) at org.springframework.jndi.JndiObjectLocator.lookup(JndiObjectLocator.java:105) at org.springframework.jndi.JndiObjectFactoryBean.lookupWithFallback(JndiObjectFactoryBean.java:197) at org.springframework.jndi.JndiObjectFactoryBean.afterPropertiesSet(JndiObjectFactoryBean.java:184) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1201) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1171) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:425) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:251) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:156) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:248) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:261) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:109) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1099) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:861) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:421) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:251) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:156) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:248) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:287) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:352) at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:93) at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:77) at org.springframework.context.support.FileSystemXmlApplicationContext.<init>(FileSystemXmlApplicationContext.java:67) at com.ssn.jms.integration.test.bootstrap.SimpleSpringContextBootStrap.run(SimpleSpringContextBootStrap.java:29) at com.ssn.jms.integration.test.bootstrap.SimpleSpringContextBootStrap.main(SimpleSpringContextBootStrap.java:20) Caused by: java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:574) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:185) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:94) at org.jnp.server.NamingServer_Stub.lookup(Unknown Source) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:667) ... 33 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:520) at java.net.Socket.connect(Socket.java:470) at java.net.Socket.<init>(Socket.java:367) at java.net.Socket.<init>(Socket.java:180) at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22) at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:569) ... 38 more Please do help me. Thanks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Thanks Zach. I finally got it up and running. :) The jndi problem is gone once I use -b option for the run.sh. Now I'll see what's going to happen with my test. Thanks for the kind help. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao updated JBMESSAGING-1456: ------------------------------------ Attachment: (was: BaseProducerThread.java) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao updated JBMESSAGING-1456: ------------------------------------ Comment: was deleted -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao updated JBMESSAGING-1456: ------------------------------------ Comment: was deleted -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao updated JBMESSAGING-1456: ------------------------------------ Comment: was deleted -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- The reason the message got stuck: 1. Message A arrived at Queue on node0. 2. Client creates conn1/sess1/cons1 to receive messages. 3. Message A is delivered to cons1, but the message delivery failed at remoting layer due to timeout. So node0 will mark this cons1 dead and deliver messages to other consumers. However the timeout didn't cause the remoting to think of the connection being broken (maybe the connection was just occasionally slow), so JBM node0 didn't get connection failure notification, therefore the conn1/sess1/cons1 at the server node0 were not cleared (that means message A was not put back to queue for re-delivery). 4. before client closes conn1/sess1/cons1, client ping timout happened and it will trigger client side failover. The result is that internally a new connection to node1 (conn2/sess2/cons2) was created. To client this happened transparently, so it looks like messages continue coming in to the same consumer, but internally the consumer has changed. 5. client tried to close conn1/sess1/cons1 (actually it is closing conn2/sess2/cons2 now). node1 will close conn2/sess2/cons2 accordingly. 6. client created another set of conn/sess/cons and continue to work. 7. at this point, the conn1/sess1/cons1 at the server side keep open, so the message A will never be put back into queue, so stuck. 8. The message A will get redelivered if a) we shut down the client process and restart the client; or b) we shutdown the server node0 and restart the node0. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao updated JBMESSAGING-1456: ------------------------------------ Fix Version/s: 1.4.0.SP3.CP08 1.4.3.GA -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- In fact restarting client doesn't help relieve the stuck message. So point 8 a) is not valid. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Thanks, Ron. I saw that code in remoting too. So My reasoning/fix is not valid. I need to do more investigation. I have a question, if from client side we do the following: Client.setDisconnectTimeout(0) Client.removeListener(); Client.disconnect(); Is it guaranteed that the server side ConnectionListener.handleConnectionException() will be called after the lease timeout? Howard -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Hi Ron, that's great information in helping explanation the stuck situation. Thanks a lot! and That shows you are so indispensable. :) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Ron Sigal commented on JBMESSAGING-1456: ---------------------------------------- Laughing Out Loud. :) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Justin Bertram updated JBMESSAGING-1456: ---------------------------------------- Affects Version/s: 1.4.0.SP3.CP07 (was: 1.4.0.SP3.CP04) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Tim Fox commented on JBMESSAGING-1456: -------------------------------------- To clarify, the underlying reason here for the stuck messages is in a bug in JBoss remoting, for which we can provide a *workaround* in JBM until it is fixed in remoting. However, I would much rather see the underlying issue fixed in JBoss remoting. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Tim Fox updated JBMESSAGING-1456: --------------------------------- Priority: Blocker (was: Critical) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Kris Fraley commented on JBMESSAGING-1456: ------------------------------------------ For what it's worth ... I am also seeing this problem using a standalone (non-clustered) environment simply running the default server out-of-the-box. "My" JBoss EAP 4.3.0_CP02 contains JBoss Messaging 1.4.0.SP3-CP03 and is running on an XP SP2 platform using jdk1.5.0_18. Though intermittent, the "stuck" in the being-delivered state problem does not take very long (5 minutes) for me to observe. Unfortunately, without a solution to this problem (soon!), not only are messages stuck, but my customer is going to be stuck as well which in turn will stick me as well. Not whining (much), but for me resolution of this problem is becoming more urgent. Last time I looked the fix was scheduled to be available in JBoss Messaging 1.4.0.SP3.CP08 on 5/12/2009 ... today! I would be glad to assist in resolving this issue in any way practical. Kris -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Kris Fraley commented on JBMESSAGING-1456: ------------------------------------------ I'm game. Send me (or tell me where I can find) the pieces parts, let me know where to stick them (so to speak :o) ), and I'll attempt validation at least on the simple configuration I'm using. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Thanks guys. I'll upload the jars for you to test soon. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao updated JBMESSAGING-1456: ------------------------------------ Attachment: test-1456-jars.zip jars for test only -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Tim Fox commented on JBMESSAGING-1456: -------------------------------------- A note to anyone trying out these jars: You will need to update both remoting and JBM jars on both the client *and* the server side. Be particularly careful on the client side side if you are using jbossall-client.jar - this already contains remoting classes but of an earlier version, make sure JBM and Remoting jars are on the classpath *before* jbossall-client.jars (or preferably don't use jbossall-client.jar) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Martijn Verburg commented on JBMESSAGING-1456: ---------------------------------------------- OK, we've tried the patch and things certainly improved, but it subscribers slowly started to fail over time. I've attached a zip file with the logs, hopefully it helps! -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Martijn Verburg commented on JBMESSAGING-1456: ---------------------------------------------- Apologies for the Jira comment spam, I managed to upload the zip file to our corresponding customer portal case number 296348, I think Colin Mondesir was looking after it, hopefully he can forward the logs onto you. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Kris Fraley commented on JBMESSAGING-1456: ------------------------------------------ Howard (or others that might know), I'm sorry, I'm pretty much just a JBoss novice, so I have a basic question. Your readme.txt file packaged in test-1456-jars.zip says, "update the remoting configuration file to add the following params exactly as follows etc. etc.". Which file is the "remoting configuration file"? I'm guessing that it is the remoting-bisocket-service.xml (or http or sslbisocket) file. Please reply. Thanks. Kris -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Hi Victor, There are extra logs apart from the normal logs. They are all in your home directory (directories whose name begin with jms-). They are quite a log but they won't grow forever. It's for tests only. If you like to use the on without those logs, you can check out from Branch_1_4 and build the jars yourself. The remoting jar doesn't have the extra logs. Howard -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Thanks Victor! -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Kris Fraley commented on JBMESSAGING-1456: ------------------------------------------ Interestingly, after logging my test results (previous comment in this thread), I stopped, then restarted the default server. Using the JBoss JMX Management Console, I observed that the testQueue's MessageCount was still sitting at 143, but the Delivering Count was now back to 0. I connected my consumer to attempt to retrieve the 143 messages, but they were still stuck in the queue. Then, I attempted the testQueue's removeAllMessages() MBean operation. This caused the same exception as illustrated in the attached RemoveAllMessagesException.png. Then, I refreshed the MBean view and observed that the DeliveringCount was back to 143. Now, it appears the messages are really stuck! If you would like further details or other tests performed using my "simple" configuration, please contact me directly at kfraley(a)oldi.com. I would be happy to gather further information or try out other "fixes". Kris -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Victor Starenky commented on JBMESSAGING-1456: ---------------------------------------------- Ok, we ran test jars overnight. Results are not very good. While we don't see messages "stuck" in delivery we do see a bunch of nodes having tons of messages accumulated on some (not all) queues. That is MessageCount is not zero (rather in the order of hundreeds) but DeliveringCount is zero. Those nodes have ConsumerCount=0. Looks like message sucker is not working or something? It is configured as far as I can tell. I have actually already seen this problem as mentioned in the forum: http://www.jboss.org/index.html?module=bb&op=viewtopic&t=155317 Where would I send logs and config files? I'm afraid attaching here wouldn't work for the logs because of the size restriction. Our plan is to keep the test jars for now but go back to the config files with the vastly increased timeout values - that seemed to help as a work around. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Victor Starenky updated JBMESSAGING-1456: ----------------------------------------- Attachment: logs-and-config.zip Logs and config files -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Victor Starenky commented on JBMESSAGING-1456: ---------------------------------------------- Sorry for spamming JIRA, but one more potential problem with the test jars: once we've replaced the previous ones with them we're getting java.lang.OutOfMemoryError: GC overhead limit exceeded errors on the servers. This is not necessary related to these jars but that was the only difference in the environment config so the chances are it is. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Hi Vitor, I looked your logs, there are a lot of recorded 'stuck message' there, I inspected a few of them. I found: a message belongs to a transaction the message was not delivered to any consumer then session/connection is closed normally. So I think this is a different problem. The real 'stuck message' here means the message is in being delivered state but client connection was broken. Then the message won't be delivered forever even there is new consumers on that queue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao updated JBMESSAGING-1456: ------------------------------------ Comment: was deleted -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Hi Victor, I suggest you create another JIRA for your case and we can continue to investigate it. I noticed you probably have up to 7 JBM instances, and each have both server and clients. This may complicated the test, as for developers we don't have much resources. It would be better you narrow the issue down a bit, so we can have a good start. Thanks. Howard -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- That's great! Thanks Victor. H. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao closed JBMESSAGING-1456. ----------------------------------- Resolution: Done -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- As this long issue draws to an end. Let me give a brief summary in simple QA form: Why messages can got stuck? It is because the inconsistency of connection failure detection between server and client. Both client and server rely on Remoting's ConnectionListener mechanism to detect a connection failure. Remoting uses client ping to find if a connection is broken while it uses lease ping to detect a connection loss at server side. Client ping and Lease ping are working separately. When a client ping fails, Remoting notifies the client side listeners. When a lease ping fails, Remoting notifies the server side listeners. Before the fix, in Remoting the server side and client side by design are not necessarily 'bound together'. That is, a client ping failure is not always result in a server lease ping failure. In JBM, the server delivers messages to a client over the connection established by the client. If a client connection failure happens when some messages still in delivering mode, the client may stop working and it's not possible for client to issue the close() on the connection. Then the server side maintains all the state associated with the connection, including the sessions and the messages being delivered but not acked. If the server is not notified of the connection failure, the server won't clean up the connection and the associated states. That means the messages will be kept in delivering mode forever -- until the server node is shutdown and restarted. By 'binding' the client and server connection failure detection together, Remoting guarantees that whenever a client connection failure has been detected, there will always a corresponding notification on the server side. Therefore JBoss Messaging can correctly clean up the resources at the server side, including cancelling the messages in delivery state back to queue for redelivery. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Hi Tim, That's an issue in JBR. Basically what I understand is that the client ping object (ConnectionValidator) is shared among the remoting clients, and the ConnectionValidator is associated with a Lease Pinger for the server. Something wrong happens when new Client is created and ConnectionValidator fails to shutdown the correct Lease Pinger. Here I quote Ron's comments as I may not understand it in the very much details: -----------Ron's comment--------- 1. Client_A.connect() and Client_B.connect() register Client_A and Client_B with MicroRemoteClientInvoker_1's LeasePinger_1 and with the associated server side Lease. 2. Client_A.addConnectionListener() creates (or reuses) ConnectionValidator_1. 3. ConnectionValidator_1 sees a connection failure and notifies Client_A's listener(s), causing a JBM failover. ClientValidator_1 also shuts down MicroRemoteClientInvoker_1's LeasePinger_1, which causes the server side Leases for Client_A and Client_B to notify their listeners about a broken connection. 3. Client_B.addConnectionListener() reuses ConnectionValidator_1. 4. Client_C.connect() registers Client_C with a *new* LeasePinger_2 in MicroRemoteClientInvoker_1. 4. ConnectionValidator_1 detects a broken connection and notifies Client_B's listener(s) (causing a JBM failover). ConnectionValidator_1 also shuts down MicroRemoteClientInvoker_1's *new* LeasePinger_2, which results in all of the server side Leases associated with LeasePinger_2 to notify their listeners about a broken connection. BUT CLient_B WAS ASSOCIATED WITH LeasePinger_1's LEASE - IT"S NOT IN LeasePinger_2's LEASE. SO THERE IS NO NOTIFICATION ABOUT Client_B ON THE SERVER SIDE. Or something like that. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Hi Vitor, did you use the new parameters in the remoting config file? <attribute name="failureDisconnectTimeout" isParam="true">0</attribute> <attribute name="callbackErrorsAllowed">1</attribute> <attribute name="useClientConnectionIdentity" isParam="true">true</attribute> If you are seeing stuck messages, can you try the messaging jars included in test-1456-jars.zip and upload the logs? The logs can give us a proof of whether the message is stuck for the same reason. Thanks. Howard -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Hi Victor, You don't need to use the remoting jar in the zip, which may not be up to date and cause the mem leak. You only need to use the jboss-messaging.jar and jboss-messaging-client.jar. These two jars should be the same as CP08 exception those extra logs. Thanks Howard -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Hi Victor, the jars in the zip may not contain some other fixes in 1.4.4. But regarding to this fix, they should be the same. I wouldn't suggest you drop them to your production env. If you have a test env, you can try it. Or if you can tell me how to reproduce, I'll try myself. Thanks, Howard -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/JBMESSAGING-1456?page=com.atlassian.ji... ] Howard Gao commented on JBMESSAGING-1456: ----------------------------------------- Thanks Victor. Please note that: 1. If you have long live sessions, it is possible the messages got 'stuck' until the session is closed. 2. if session is not closed while failover is happening, the following parameter affects how long the messages will be canceled back to queue for redelivery: <attribute name="clientLeasePeriod" isParam="true">30000</attribute> For the above value, the 'stuck' message will not be canceled back until at least 30 sec after the client has finished failover. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JBMESSAGING-1456?page=com.atlassian.jira.... ] Justin Bertram commented on JBMESSAGING-1456: --------------------------------------------- This issue was resolved in JBoss Messaging versions 1.4.0.SP3.CP08 and 1.4.4.GA (as indicated by the JIRA). We have not seen a recurrence of this particular issue since that time (over 2.5 years ago). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/JBMESSAGING-1456?page=com.atlassian.jira.... ] Justin Bertram commented on JBMESSAGING-1456: --------------------------------------------- Despite how the comments may (or may not) look, this JIRA was resolved and all the users who were seeing the problem at the time reported the change fixed it. You may have an issue with similar symptoms, but it is almost certainly not this exact same issue. Please post your problem on the user forum (https://community.jboss.org/en/jbossmessaging?view=discussions). However, please note that JBoss Messaging is no longer under active development since all messaging effort is being put into HornetQ. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira