[jboss-user] [Clustering/JBoss] - Re: Clustering JBoss 4.0.5, not working, stuck

Tue Jul 8 10:06:45 EDT 2008

First off I turned org.jgroups to "TRACE", however I'm unsure how to read the results of this,but then I also started the 2 cluster members and attempted to probe the jgroup, as discussed at http://wiki.jboss.org/wiki/Probe:

root at oraclelinuxoracle11clusterA lib]# /opt/jdk1.6.0_02/bin/java -cp jgroups.jar  org.jgroups.tests.Probe -timeout 500
java.lang.NullPointerException: null buffer || null address
        at java.net.PlainDatagramSocketImpl.send(Native Method)
        at java.net.DatagramSocket.send(DatagramSocket.java:612)
        at org.jgroups.tests.Probe.start(Probe.java:33)
        at org.jgroups.tests.Probe.main(Probe.java:93)

Fails so lets give it all the possible options as suggested:
[root at oraclelinuxoracle11clusterA lib]# /opt/jdk1.6.0_02/bin/java -cp jgroups.jar:commons-logging.jar  org.jgroups.tests.Probe -timeout 500 -bind_addr 172.19.2.10 -port 7500 -addr 224.0.0.75

-- send probe on /224.0.0.75:7500

Find nothing, so lets change the port since the jboss log shows that  it is listening at> 1099, 1100 or 1102:
JBOSS log> 23:09:04,979 INFO  [DefaultPartition] I am (172.19.2.10:1099) received membershipChanged event:
JBOSS log> 23:09:04,991 INFO  [DefaultPartition] Dead members: 0 ([])
JBOSS log> 23:09:05,032 INFO  [DefaultPartition] New Members : 0 ([])
JBOSS log> 23:09:05,034 INFO  [DefaultPartition] All Members : 1 ([172.19.2.10:1099])
JBOSS log> 23:09:05,124 INFO  [HANamingService] Started ha-jndi bootstrap jnpPort=1100, backlog=50, bindAddress=/172.19.2.10
JBOSS log> 23:09:05,187 INFO  [DetachedHANamingService$AutomaticDiscovery] Listening on /172.19.2.10:1102, group=230.0.0.4, HA-JNDIaddress=172.19.2.10:1100

Ok testing on Ports 1099, 1100 or 1102 finds nothing, same as looking on port 7500, I also tried ports 1485, 1486 and 45571 since they showed up in the server.log file, this time (the 1485, 1486 and 45571 addresses change every time I restart JBoss so I'm assuming they are not what I want.

It would appear based on this information, that jgroups is not running at all, yet JBoss says its running and it shows in the server.log file when set to org.jgroup is set to TRACE.  I also thought that perhaps the multicast address is wrong, but then I could not find any place in JBoss where the jgroup multicast port is set so I have no idea what it is if not the default.  The only multicast set in the logs is "228.1.2.3",  yet it appears in a different section and even running the probe on that address fails to find any groups.

The following chunk of the JBoss server.log when I start the second cluster member. I am not sure, but it appears that there is no connections made from the second member (to this the first member), which makes sense as probing does not even show that the jgroup is started.

  | 2008-07-08 23:03:35,248 DEBUG [org.jgroups.protocols.PING] initial mbrs are []
  | 2008-07-08 23:03:35,248 DEBUG [org.jgroups.protocols.MERGE2] initial_mbrs=[]
  | 2008-07-08 23:03:37,172 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Looking for sessions that have expired ...
  | 2008-07-08 23:03:39,573 DEBUG [org.jgroups.protocols.PING] initial mbrs are []
  | 2008-07-08 23:03:39,575 DEBUG [org.jgroups.protocols.MERGE2] initial_mbrs=[]
  | 2008-07-08 23:03:45,511 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Creating an empty ClusteredSession
  | 2008-07-08 23:03:45,512 DEBUG [org.jboss.web.tomcat.tc5.session.SessionIDGenerator] getSessionId called: vTrmYfRCYsIAhMcEFPpSxA**
  | 2008-07-08 23:03:45,512 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] createSession(): useJK is true. Will append JvmRoute: node2
  | 2008-07-08 23:03:45,512 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Session with id=vTrmYfRCYsIAhMcEFPpSxA**.node2 added. Current active sessions 2
  | 2008-07-08 23:03:45,512 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Created a ClusteredSession with id: vTrmYfRCYsIAhMcEFPpSxA**.node2
  | 2008-07-08 23:03:45,559 DEBUG [org.jboss.web.tomcat.tc5.session.ClusteredSession] The session has expired with id: vTrmYfRCYsIAhMcEFPpSxA**.node2 -- is it local? false
  | 2008-07-08 23:03:45,561 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Removing session from store with id: vTrmYfRCYsIAhMcEFPpSxA**
  | 2008-07-08 23:03:45,561 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheService] Remove session from distributed store. Fqn: /JSESSION/localhost/cs/vTrmYfRCYsIAhMcEFPpSxA**
  | 2008-07-08 23:03:45,561 WARN  [org.jboss.cache.TreeCache] node /JSESSION/localhost/cs/vTrmYfRCYsIAhMcEFPpSxA** not found
  | 2008-07-08 23:03:47,176 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Looking for sessions that have expired ...
  | 2008-07-08 23:03:57,181 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Looking for sessions that have expired ...
  | 2008-07-08 23:03:59,779 DEBUG [org.jgroups.protocols.PING] initial mbrs are []
  | 2008-07-08 23:03:59,786 DEBUG [org.jgroups.protocols.MERGE2] initial_mbrs=[]
  | 2008-07-08 23:04:07,190 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Looking for sessions that have expired ...
  | 2008-07-08 23:04:15,522 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Creating an empty ClusteredSession
  | 2008-07-08 23:04:15,523 DEBUG [org.jboss.web.tomcat.tc5.session.SessionIDGenerator] getSessionId called: YgTiVX5lLdOoM63lthxGkQ**
  | 2008-07-08 23:04:15,523 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] createSession(): useJK is true. Will append JvmRoute: node2
  | 2008-07-08 23:04:15,540 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Session with id=YgTiVX5lLdOoM63lthxGkQ**.node2 added. Current active sessions 2
  | 2008-07-08 23:04:15,540 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Created a ClusteredSession with id: YgTiVX5lLdOoM63lthxGkQ**.node2
  | 2008-07-08 23:04:15,567 DEBUG [org.jboss.web.tomcat.tc5.session.ClusteredSession] The session has expired with id: YgTiVX5lLdOoM63lthxGkQ**.node2 -- is it local? false
  | 2008-07-08 23:04:15,568 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Removing session from store with id: YgTiVX5lLdOoM63lthxGkQ**
  | 2008-07-08 23:04:15,569 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheService] Remove session from distributed store. Fqn: /JSESSION/localhost/cs/YgTiVX5lLdOoM63lthxGkQ**
  | 2008-07-08 23:04:15,570 WARN  [org.jboss.cache.TreeCache] node /JSESSION/localhost/cs/YgTiVX5lLdOoM63lthxGkQ** not found
  | 2008-07-08 23:04:17,200 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Looking for sessions that have expired ...
  | 2008-07-08 23:04:20,494 DEBUG [org.jgroups.protocols.PING] initial mbrs are []
  | 2008-07-08 23:04:20,495 DEBUG [org.jgroups.protocols.MERGE2] initial_mbrs=[]
  | 2008-07-08 23:04:27,237 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Looking for sessions that have expired ...
  | 2008-07-08 23:04:32,946 DEBUG [org.jgroups.protocols.PING] initial mbrs are []
  | 2008-07-08 23:04:32,962 DEBUG [org.jgroups.protocols.MERGE2] initial_mbrs=[]
  | 2008-07-08 23:04:37,242 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Looking for sessions that have expired ...
  | 2008-07-08 23:04:45,519 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Creating an empty ClusteredSession
  | 2008-07-08 23:04:45,520 DEBUG [org.jboss.web.tomcat.tc5.session.SessionIDGenerator] getSessionId called: wUDdoVz3jmRKktZhk2HGJQ**
  | 2008-07-08 23:04:45,520 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] createSession(): useJK is true. Will append JvmRoute: node2
  | 2008-07-08 23:04:45,521 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Session with id=wUDdoVz3jmRKktZhk2HGJQ**.node2 added. Current active sessions 2
  | 2008-07-08 23:04:45,521 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Created a ClusteredSession with id: wUDdoVz3jmRKktZhk2HGJQ**.node2
  | 2008-07-08 23:04:45,551 DEBUG [org.jboss.web.tomcat.tc5.session.ClusteredSession] The session has expired with id: wUDdoVz3jmRKktZhk2HGJQ**.node2 -- is it local? false
  | 2008-07-08 23:04:45,561 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Removing session from store with id: wUDdoVz3jmRKktZhk2HGJQ**
  | 2008-07-08 23:04:45,562 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheService] Remove session from distributed store. Fqn: /JSESSION/localhost/cs/wUDdoVz3jmRKktZhk2HGJQ**
  | 2008-07-08 23:04:45,562 WARN  [org.jboss.cache.TreeCache] node /JSESSION/localhost/cs/wUDdoVz3jmRKktZhk2HGJQ** not found
  | 2008-07-08 23:04:47,247 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Looking for sessions that have expired ...
  | 2008-07-08 23:04:53,133 DEBUG [org.jgroups.protocols.PING] initial mbrs are []
  | 2008-07-08 23:04:53,135 DEBUG [org.jgroups.protocols.MERGE2] initial_mbrs=[]
  | 2008-07-08 23:04:53,591 DEBUG [org.jgroups.protocols.UDP]
  | sending msgs:
  | 230.1.2.7:45577: 1 msgs
  | 
  | 2008-07-08 23:04:55,562 DEBUG [org.jgroups.protocols.PING] initial mbrs are []
  | 2008-07-08 23:04:55,563 DEBUG [org.jgroups.protocols.MERGE2] initial_mbrs=[]
  | 2008-07-08 23:04:57,251 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Looking for sessions that have expired ...
  | 2008-07-08 23:05:06,152 DEBUG [org.jgroups.protocols.PING] initial mbrs are []
  | 2008-07-08 23:05:06,152 DEBUG [org.jgroups.protocols.MERGE2] initial_mbrs=[]
  | 2008-07-08 23:05:07,258 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Looking for sessions that have expired ...
  | 2008-07-08 23:05:15,508 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Creating an empty ClusteredSession
  | 2008-07-08 23:05:15,509 DEBUG [org.jboss.web.tomcat.tc5.session.SessionIDGenerator] getSessionId called: qt6x6A+8rq5zyFVD3EFJbg**
  | 2008-07-08 23:05:15,509 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] createSession(): useJK is true. Will append JvmRoute: node2
  | 2008-07-08 23:05:15,509 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Session with id=qt6x6A+8rq5zyFVD3EFJbg**.node2 added. Current active sessions 2
  | 2008-07-08 23:05:15,509 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Created a ClusteredSession with id: qt6x6A+8rq5zyFVD3EFJbg**.node2
  | 2008-07-08 23:05:15,519 DEBUG [org.jboss.web.tomcat.tc5.session.ClusteredSession] The session has expired with id: qt6x6A+8rq5zyFVD3EFJbg**.node2 -- is it local? false
  | 2008-07-08 23:05:15,520 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Removing session from store with id: qt6x6A+8rq5zyFVD3EFJbg**
  | 2008-07-08 23:05:15,521 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheService] Remove session from distributed store. Fqn: /JSESSION/localhost/cs/qt6x6A+8rq5zyFVD3EFJbg**
  | 2008-07-08 23:05:15,521 WARN  [org.jboss.cache.TreeCache] node /JSESSION/localhost/cs/qt6x6A+8rq5zyFVD3EFJbg** not found
  | 2008-07-08 23:05:17,263 DEBUG [org.jboss.web.tomcat.tc5.session.JBossCacheManager] Looking for sessions that have expired ...

I'm just at a loss here it appears that while using the "all" profile and having followed the instructions on setting up a cluster with Apache 2.2 and modproxyhttp or modproxyajp the required jgroup is not either starting (which the log says it is) or broadcasting its existance (which seems more likely) so that other nodes can find it.

Any ideas?
ERIC

View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4163069#4163069

Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4163069