[JBoss JIRA] Created: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[JBoss JIRA] Created: (JGRP-627)...

[JBoss JIRA] Created: (JBAS-4892)...

P Eger (JIRA)

Tuesday, 13 November 2007 Tue, 13 Nov '07

12:10 p.m.

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Attachments: stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

Show replies by date

P Eger (JIRA)

Tuesday, 13 November Tue, 13 Nov

12:10 p.m.

New subject: [JBoss JIRA] Updated: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=all ] P Eger updated JGRP-620: ------------------------ Attachment: stacktrace.txt

...

-- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

P Eger (JIRA)

12:10 p.m.

New subject: [JBoss JIRA] Updated: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=all ] P Eger updated JGRP-620: ------------------------ Attachment: tcp-config.xml

...

P Eger (JIRA)

4:52 p.m.

New subject: [JBoss JIRA] Commented: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=comments#action_12387395 ] P Eger commented on JGRP-620: ----------------------------- Browsing the code, the following code in GroupRequest.java looks suspect (no pun intended ;-): private void adjustMembership() { if(requests.isEmpty()) return; Address mbr; Rsp rsp; synchronized(members) { for(Map.Entry<Address,Rsp> entry: requests.entrySet()) { mbr=entry.getKey(); if((!this.members.contains(mbr)) || suspects.contains(mbr)) { //<-- should be !suspects.contains(mbr) ?? addSuspect(mbr); rsp=entry.getValue(); rsp.setValue(null); rsp.setSuspected(true); } } } } Also this code seems strange, though not necessarily incorrect: public void viewChange(View new_view) { ... if(modified || tmp != null) { //<-- should be just if(tmp != null)? "modified" variable appears useless / redundant synchronized(requests) { for(Address suspect: tmp) { addSuspect(suspect); } requests.notifyAll(); } } } Also, 2 additional notes on the code: 1) What's the deal with "max_suspects"? seems arbitrarily set to 40 2) Why is the GroupRequest locking so complicated? I see synchronization on "requests", "members", "this", and "suspects", which seems to complicate things. The fix for http://jira.jboss.com/jira/browse/JGRP-554 for instance. Would you accept a patch to unify everything under 1 lock? I don't see any callouts/callbacks that would result in deadlocks if this is done. I would doubt performace impact would be noticeable.

...

P Eger (JIRA)

5:01 p.m.

New subject: [JBoss JIRA] Commented: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=comments#action_12387396 ] P Eger commented on JGRP-620: ----------------------------- Never mind on the #2 above, i see you have done exactly this! http://javagroups.cvs.sourceforge.net/javagroups/JGroups/src/org/jgroups/... I will rereview 2.6 GA.

...

Bela Ban (JIRA)

Wednesday, 14 November Wed, 14 Nov

1:03 a.m.

New subject: [JBoss JIRA] Updated: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=all ] Bela Ban updated JGRP-620: -------------------------- Fix Version/s: 2.5.2 2.6.1 2.7

...

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Fix For: 2.5.2, 2.6.1, 2.7 Attachments: stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)

Bela Ban (JIRA)

Monday, 19 November Mon, 19 Nov

8:56 a.m.

New subject: [JBoss JIRA] Closed: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=all ] Bela Ban closed JGRP-620. ------------------------- Resolution: Cannot Reproduce Bug Note that it is better to use 'null' for the current membership, as the view you pass into callRemoteMethods() may be outdated. I'm closing this issue for now. If you have a small program that reproduces it, please re-open this case and attach the program.

...

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Fix For: 2.5.2, 2.6.1, 2.7 Attachments: stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)

P Eger (JIRA)

11:34 a.m.

New subject: [JBoss JIRA] Commented: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=comments#action_12388266 ] P Eger commented on JGRP-620: ----------------------------- Reproduction is difficult, since it was seen only during cluster churn. I will attempt to reproduce however. 3 Questions WRT this, i can't seem to find the answer in the documentation: 1) Would "infinite hang" be the expected behaviour for RPC to a set of members, some of whom potentially do not exist? Is this because the member may possibly rejoin? 2) Is there any way to adjust the above behaviour (i would like it to ERROR the members who have left / were not in the cluster immediately) 3) I need to pass in the list here, because at times the member list i am RPCing to is not necessarily all the members, it is a subset. Would a null for the list have additional "atomicity" guarantees over and above channel.getView().getMembers()?

...

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Fix For: 2.5.2, 2.6.1, 2.7 Attachments: stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)

Bela Ban (JIRA)

12:24 p.m.

New subject: [JBoss JIRA] Commented: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=comments#action_12388273 ] Bela Ban commented on JGRP-620: ------------------------------- Hi Patrick, please post questions to the mailing lists, not to a JIRA issue. Cheers, Bela

...

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Fix For: 2.7, 2.5.2, 2.6.1 Attachments: stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)

Bela Ban (JIRA)

Tuesday, 20 November Tue, 20 Nov

3:48 a.m.

New subject: [JBoss JIRA] Updated: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=all ] Bela Ban updated JGRP-620: -------------------------- Attachment: bla.java

...

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Fix For: 2.5.2, 2.6.1, 2.7 Attachments: bla.java, stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)

Bela Ban (JIRA)

3:48 a.m.

New subject: [JBoss JIRA] Reopened: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=all ] Bela Ban reopened JGRP-620: --------------------------- attached program bla reproduces the issue

...

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Fix For: 2.5.2, 2.6.1, 2.7 Attachments: bla.java, stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)

Bela Ban (JIRA)

3:50 a.m.

New subject: [JBoss JIRA] Commented: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=comments#action_12388367 ] Bela Ban commented on JGRP-620: ------------------------------- The problem is that when we invoke RpcDispatcher.callRemoteMethods() on {A,B,C}, but C left the cluster *before* the call, we wait forever on C. This is 50% the error of the user who should *not* cache member lists, but pass 'null' instead of a vector, but sometimes passing an explicit vector is useful. Solution: we have to weed out elements from the vector which are not in the current view anymore.

...

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Fix For: 2.5.2, 2.6.1, 2.7 Attachments: bla.java, stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)

Bela Ban (JIRA)

4:08 a.m.

New subject: [JBoss JIRA] Updated: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=all ] Bela Ban updated JGRP-620: -------------------------- Fix Version/s: 2.4.2

...

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Fix For: 2.7, 2.4.2, 2.5.2, 2.6.1 Attachments: bla.java, stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)

Bela Ban (JIRA)

4:57 a.m.

New subject: [JBoss JIRA] Commented: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=comments#action_12388377 ] Bela Ban commented on JGRP-620: ------------------------------- Unit test is RpcDispatcherTest.testMethodInvocationToNonExistingMembers()

...

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Fix For: 2.7, 2.4.2, 2.5.2, 2.6.1 Attachments: bla.java, stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)

Bela Ban (JIRA)

4:57 a.m.

New subject: [JBoss JIRA] Commented: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=comments#action_12388378 ] Bela Ban commented on JGRP-620: ------------------------------- Fixed in CVS head

...

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Fix For: 2.7, 2.4.2, 2.5.2, 2.6.1 Attachments: bla.java, stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)

Bela Ban (JIRA)

5:10 a.m.

New subject: [JBoss JIRA] Commented: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=comments#action_12388382 ] Bela Ban commented on JGRP-620: ------------------------------- fixed on 2.6 branch

...

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Fix For: 2.7, 2.4.2, 2.5.2, 2.6.1 Attachments: bla.java, stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)

Bela Ban (JIRA)

5:25 a.m.

New subject: [JBoss JIRA] Commented: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=comments#action_12388385 ] Bela Ban commented on JGRP-620: ------------------------------- fixed on 2.5 and 2.4 branches too

...

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Fix For: 2.7, 2.4.2, 2.5.2, 2.6.1 Attachments: bla.java, stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)

Bela Ban (JIRA)

5:25 a.m.

New subject: [JBoss JIRA] Resolved: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs

[ http://jira.jboss.com/jira/browse/JGRP-620?page=all ] Bela Ban resolved JGRP-620. --------------------------- Resolution: Done fixed by weeding out members of a destination list which are not current members

...

RpcDispatcher.callRemoteMethods() hangs --------------------------------------- Key: JGRP-620 URL: http://jira.jboss.com/jira/browse/JGRP-620 Project: JGroups Issue Type: Bug Affects Versions: 2.5.1 Environment: RHEL4 64 bit: Linux svr5 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 17:13:42 EST 2007 x86_64 x86_64 x86_64 GNU/Linux SUN JDK 1.5.0_12-b04 64 bit Reporter: P Eger Assigned To: Bela Ban Fix For: 2.4.2, 2.5.2, 2.6.1, 2.7 Attachments: bla.java, stacktrace.txt, tcp-config.xml RpcDispatcher.callRemoteMethods() hangs, while there is a lot of member churn at the time (4 servers starting into a cluster of 4). 1 of the 4 server starting up is hung with the attached stack trace. ------------------------------------------------------------------- channel=new JChannel(jgroups_config_file); channel.setOpt(Channel.AUTO_RECONNECT, Boolean.TRUE); channel.addChannelListener(this); //TODO: verify these startup params //NOTE: deadlock detection leaks memory as of 2.5b2, do not use disp=new RpcDispatcher(channel, null, this, this, false, true); //force connect channel.connect(clusterName); MethodCall mc = new MethodCall("remoteBroadcastAvailability", new Object[]{peer,sequence,serviceStatus,rotationStatus}, new Class[]{Address.class,Long.class,ServiceStatus.class,RotationStatus.class}); disp.callRemoteMethods(channel.getView().getMembers(), mc, GroupRequest.GET_ALL, 0); ------------------------------------------------------------------- Name: main State: WAITING on java.util.HashMap@2d95bbec Total blocked: 53 Total waited: 12 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:479) org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:190) org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:430) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:199) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:167) org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:163) utils.cluster.PeerClusterManager.broadcastAvailability(PeerClusterManager.java:1110) utils.cluster.PeerClusterManager.broadcastMyAvailability(PeerClusterManager.java:428) init.InitManager.ensureInitialized(InitManager.java:552) init.InitManager.__init(InitManager.java:409) init.InitManager.init(InitManager.java:300) init.ServletListener.contextInitialized(ServletListener.java:23) org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3669) org.apache.catalina.core.StandardContext.start(StandardContext.java:4104) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardHost.start(StandardHost.java:718) org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1012) org.apache.catalina.core.StandardEngine.start(StandardEngine.java:442) org.apache.catalina.core.StandardService.start(StandardService.java:450) org.apache.catalina.core.StandardServer.start(StandardServer.java:683) org.apache.catalina.startup.Catalina.start(Catalina.java:537) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:585) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:271) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:409)

6260

days inactive

6267

days old

jboss-jira@lists.jboss.org

Manage subscription

17 comments

2 participants

tags (0)

participants (2)

Bela Ban (JIRA)
P Eger (JIRA)

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

[JBoss JIRA] Created: (JGRP-620) RpcDispatcher.callRemoteMethods() hangs