[jboss-jira] [JBoss JIRA] Resolved: (JGRP-876) ConcurrentModificationException during startup of JBoss AS 5.0.0.GA

ConcurrentModificationException during startup of JBoss AS 5.0.0.GA ------------------------------------------------------------------- Key: JGRP-876 URL: https://jira.jboss.org/jira/browse/JGRP-876 Project: JGroups Issue Type: Bug Affects Versions: 2.6.7 Environment: JBoss AS 5.0.0.GA Reporter: Paul Ferraro Assignee: Bela Ban Fix For: 2.6.8, 2.7 I encountered a rare ConcurrentModificationException while starting the all profile of JBoss AS 5.0.0.GA: 10:34:45,077 ERROR [AbstractKernelController] Error installing to Start: name=HAPartition state=Create org.jgroups.ChannelException: connect() failed at org.jgroups.JChannel.connect(JChannel.java:374) at org.jboss.ha.framework.server.ClusterPartition$ChannelConnectTask.run(ClusterPartition.java:167) at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:147) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) Caused by: java.util.ConcurrentModificationException at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:449) at java.util.AbstractList$Itr.next(AbstractList.java:420) at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1010) at java.util.AbstractCollection.addAll(AbstractCollection.java:318) at org.jgroups.protocols.TP.handleDownEvent(TP.java:1575) at org.jgroups.protocols.TP.down(TP.java:1148) at org.jgroups.protocols.TP$ProtocolAdapter.down(TP.java:2308) at org.jgroups.protocols.Discovery.down(Discovery.java:354) at org.jgroups.protocols.MERGE2.down(MERGE2.java:153) at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:370) at org.jgroups.protocols.FD.down(FD.java:294) at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:95) at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:672) at org.jgroups.protocols.UNICAST.down(UNICAST.java:462) at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:299) at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:545) at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:453) at org.jgroups.protocols.pbcast.ClientGmsImpl.becomeSingletonMember(ClientGmsImpl.java:356) at org.jgroups.protocols.pbcast.ClientGmsImpl.join(ClientGmsImpl.java:86) at org.jgroups.protocols.pbcast.ClientGmsImpl.join(ClientGmsImpl.java:39) at org.jgroups.protocols.pbcast.GMS.down(GMS.java:832) at org.jgroups.protocols.FC.down(FC.java:430) at org.jgroups.protocols.FRAG2.down(FRAG2.java:158) at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:200) at org.jgroups.protocols.pbcast.FLUSH.down(FLUSH.java:272) at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:475) at org.jgroups.JChannel.downcall(JChannel.java:1478) at org.jgroups.JChannel.connect(JChannel.java:370) Looking at the source, here are the problematic lines: 1573: ProtocolAdapter ad=(ProtocolAdapter)prot; 1574: List<Address> tmp=ad.getMembers(); 1575: members.addAll(tmp); Looking now at ProtocolAdapter.getMembers(), I see that it does this: public List<Address> getMembers() { return Collections.unmodifiableList(members); } While Collections.unmodifiableList(...) ensures that the caller of getMembers() cannot modify the list, it can still iterate over it (e.g. line 1575). The offending mutation code is probably in ProtocolAdapter.down(Event): case Event.VIEW_CHANGE: View view=(View)evt.getArg(); Vector<Address> tmp=view.getMembers(); members.clear(); members.addAll(tmp); break; There are several ways fix this, including, but not limited to: * Make ProtocolAdapter.members volatile, and create a new member list on view change * Make ProcotolAdapter.members a CopyOnWriteArrayList