JGroup crashes JBOSS applications with different error messages --------------------------------------------------------------- Key: JGRP-691 URL: http://jira.jboss.com/jira/browse/JGRP-691 Project: JGroups Issue Type: Bug Affects Versions: 2.4.1 SP3 Environment: Redhat Linux Enterprise Application Platform 5, JDK 1.5.0_13-b05, JBOSS AS 4.2 Reporter: Shubham Nagar Assigned To: Bela Ban These are the various JGROUPS errors received in my month old Logs. When these error occur the java Heap Space errors cause the system to crash and the only way is to reboot the system. Most common are the [failed unmarshelling message] or [org.jgroups] uncaught exception in Thread messages. Any clues anyone on what is really going on. We have run tests provided in JGROUPS to run a send and reciever to test the JGROUP settings and it all seems OK. We are not using a cluster and instead have a standalone web appliation server running a Java application. ./server.log:2008-02-13 13:57:20,161 WARN [org.jgroups.util.TimeScheduler] exception executing task org.jgroups.protocols.pbcast.STABLE$StableTask@cca07b ./server.log: at org.jgroups.protocols.pbcast.STABLE$StableTask.run(STABLE.java:783) ./server.log: at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:204) ./server.log.2008-02-04:2008-02-04 18:17:53,145 WARN [org.jgroups.protocols.pbcast.NAKACK] failure passing message down ./server.log.2008-02-04:2008-02-04 18:17:53,145 WARN [org.jgroups.util.TimeScheduler] exception executing task org.jgroups.protocols.pbcast.STABLE$StableTask@906563 ./server.log.2008-02-04:2008-02-04 18:17:56,302 WARN [org.jgroups.util.TimeScheduler] exception executing task org.jgroups.protocols.pbcast.STABLE$StabilitySendTask@231e35 ./server.log.2008-02-04: at org.jgroups.protocols.pbcast.STABLE$StabilitySendTask.run(STABLE.java:845) ./server.log.2008-02-04: at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:204) ./server.log.2008-02-04:2008-02-04 18:17:56,302 ERROR [org.jgroups.protocols.UDP] failed unmarshalling message ./server.log.2008-02-04:java.lang.NoClassDefFoundError: org/jgroups/stack/Retransmitter$Entry ./server.log.2008-02-04: at org.jgroups.stack.Retransmitter.add(Retransmitter.java:104) ./server.log.2008-02-04: at org.jgroups.stack.NakReceiverWindow.add(NakReceiverWindow.java:228) ./server.log.2008-02-04: at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:672) ./server.log.2008-02-04: at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:517) ./server.log.2008-02-04: at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470) ./server.log.2008-02-04: at org.jgroups.stack.Protocol.passUp(Protocol.java:520) ./server.log.2008-02-04: at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:170) ./server.log.2008-02-04: at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470) ./server.log.2008-02-04: at org.jgroups.stack.Protocol.passUp(Protocol.java:520) ./server.log.2008-02-04: at org.jgroups.protocols.FD.up(FD.java:300) ./server.log.2008-02-04: at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470) ./server.log.2008-02-04: at org.jgroups.stack.Protocol.passUp(Protocol.java:520) ./server.log.2008-02-04: at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301) ./server.log.2008-02-04: at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470) ./server.log.2008-02-04: at org.jgroups.stack.Protocol.passUp(Protocol.java:520) ./server.log.2008-02-04: at org.jgroups.protocols.MERGE2.up(MERGE2.java:162) ./server.log.2008-02-04: at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470) ./server.log.2008-02-04: at org.jgroups.stack.Protocol.passUp(Protocol.java:520) ./server.log.2008-02-04: at org.jgroups.protocols.Discovery.up(Discovery.java:225) ./server.log.2008-02-04: at org.jgroups.stack.Protocol.receiveUpEvent(Protocol.java:470) ./server.log.2008-02-04: at org.jgroups.stack.Protocol.passUp(Protocol.java:520) ./server.log.2008-02-04: at org.jgroups.protocols.TP.handleIncomingMessage(TP.java:908) ./server.log.2008-02-04: at org.jgroups.protocols.TP.handleIncomingPacket(TP.java:850) ./server.log.2008-02-04: at org.jgroups.protocols.TP.access$400(TP.java:45) ./server.log.2008-02-04: at org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1296) ./boot.log:13:31:32,940 DEBUG [ServerInfo] jgroups.bind_addr: yesbank-crm ./boot.log:13:31:33,467 DEBUG [SARDeployer] deployed classes for file:/var/lib/jbossas/server/production/lib/jgroups.jar ./boot.log:13:31:34,749 DEBUG [RepositoryClassLoader] Added url: file:/var/lib/jbossas/server/production/lib/jgroups.jar, to ucl: org.jboss.mx.loading.UnifiedClassLoader3@1950198{ url=null ,addedOrder=2} ./server.log.2008-02-01:2008-02-01 11:16:31,113 ERROR [org.jgroups.protocols.UDP] failed unmarshalling message ./server.log.2008-02-01: at org.jgroups.protocols.TP.bufferToMessage(TP.java:972) ./server.log.2008-02-01: at org.jgroups.protocols.TP.handleIncomingPacket(TP.java:829) ./server.log.2008-02-01: at org.jgroups.protocols.TP.access$400(TP.java:45) ./server.log.2008-02-01: at org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1296) ./server.log.2008-02-01:2008-02-01 11:17:24,743 ERROR [org.jgroups] uncaught exception in Thread[MERGE2.FindSubgroups thread (channel=DefaultPartition),5,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] ) ./server.log.2008-02-01: at org.jgroups.protocols.MERGE2$FindSubgroups.findInitialMembers(MERGE2.java:331) ./server.log.2008-02-01: at org.jgroups.protocols.MERGE2$FindSubgroups.run(MERGE2.java:291) ./server.log.2008-02-01:2008-02-01 11:17:24,745 WARN [org.jgroups.util.TimeScheduler] exception executing task org.jgroups.protocols.pbcast.STABLE$StableTask@f801c4 ./server.log.2008-02-01: at org.jgroups.protocols.pbcast.STABLE$StableTask.run(STABLE.java:783) ./server.log.2008-02-01: at org.jgroups.util.TimeScheduler$TaskWrapper.run(TimeScheduler.java:204) ./server.log.2008-02-01:2008-02-01 11:17:52,045 ERROR [org.jgroups] uncaught exception in Thread[MERGE2.FindSubgroups thread (channel=DefaultPartition-EntityCache),5,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] ) ./server.log.2008-02-01:2008-02-01 11:17:52,048 ERROR [org.jgroups] uncaught exception in Thread[UDP mcast receiver (channel=Tomcat-Cluster),10,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] ) ./server.log.2008-02-01:2008-02-01 15:19:54,898 ERROR [org.jgroups] uncaught exception in Thread[PingSender,5,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] ) ./server.log.2008-01-31:2008-01-31 17:08:07,530 ERROR [org.jgroups.protocols.UDP] failed unmarshalling message ./server.log.2008-01-31: at org.jgroups.Message.readFrom(Message.java:611) ./server.log.2008-01-31: at org.jgroups.protocols.TP.bufferToMessage(TP.java:973) ./server.log.2008-01-31: at org.jgroups.protocols.TP.handleIncomingPacket(TP.java:829) ./server.log.2008-01-31: at org.jgroups.protocols.TP.access$400(TP.java:45) ./server.log.2008-01-31: at org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1296) ./server.log.2008-01-31:2008-01-31 17:50:40,866 ERROR [org.jgroups.protocols.UDP] failed unmarshalling message ./server.log.2008-01-31:2008-01-31 18:11:23,638 WARN [org.jgroups.util.TimeScheduler] exception executing task org.jgroups.protocols.pbcast.STABLE$StableTask@1010435 ./server.log.2008-01-31:2008-01-31 18:11:26,715 ERROR [org.jgroups] uncaught exception in Thread[PingSender,5,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] ) ./server.log.2008-01-31:2008-01-31 18:11:26,715 ERROR [org.jgroups] uncaught exception in Thread[UDP mcast receiver (channel=Tomcat-Cluster),10,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] ) ./server.log.2008-01-31:2008-01-31 18:11:26,715 ERROR [org.jgroups] uncaught exception in Thread[MERGE2.FindSubgroups thread (channel=Tomcat-Cluster),5,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] ) ./server.log.2008-01-31:2008-01-31 18:11:26,710 WARN [org.jgroups.util.TimeScheduler] exception executing task org.jgroups.protocols.pbcast.STABLE$StableTask@1bb73b0 ./server.log.2008-01-31:2008-01-31 18:11:26,710 ERROR [org.jgroups.protocols.UDP] failed unmarshalling message ./server.log.2008-01-31:2008-01-31 20:08:47,129 ERROR [org.jgroups] uncaught exception in Thread[PingSender,5,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] ) ./server.log.2008-01-31:2008-01-31 20:09:59,780 ERROR [org.jgroups] uncaught exception in Thread[UDP mcast receiver (channel=DefaultPartition-SFSBCache),10,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] ) ./server.log.2008-02-12:2008-02-12 18:49:13,465 ERROR [org.jgroups] uncaught exception in Thread[PingWaiter,5,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] ) ./server.log.2008-02-12: at org.jgroups.protocols.PingSender.start(PingSender.java:32) ./server.log.2008-02-12: at org.jgroups.protocols.PingWaiter.findInitialMembers(PingWaiter.java:116) ./server.log.2008-02-12: at org.jgroups.protocols.PingWaiter.run(PingWaiter.java:99) ./server.log.2008-02-12:2008-02-12 19:32:50,839 WARN [org.jgroups.util.TimeScheduler] exception executing task org.jgroups.protocols.pbcast.STABLE$StabilitySendTask@1bae104 ./server.log.2008-02-12:2008-02-12 22:07:02,384 ERROR [org.jgroups] uncaught exception in Thread[PingWaiter,5,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] ) ./server.log.2008-02-12:2008-02-12 23:52:29,248 ERROR [org.jgroups] uncaught exception in Thread[PingWaiter,5,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] ) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira