[ https://issues.jboss.org/browse/AS7-5393?page=com.atlassian.jira.plugin.s... ] Radoslav Husar commented on AS7-5393: ------------------------------------- https://raw.github.com/gist/3378550/7071060c2430a5065d27b5c4efd0d1d77ad3d... (in case the run gets deleted) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-5393?page=com.atlassian.jira.plugin.s... ] Radoslav Husar commented on AS7-5393: ------------------------------------- @Robby, thanks for the comment. Are you able to reproduce it consistently? Please share any information that might help analyze the root cause of the issue. Thanks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-5393?page=com.atlassian.jira.plugin.s... ] Robby Grossman commented on AS7-5393: ------------------------------------- @Radoslav I can reproduce it consistently as in TORQUE-908. Next time I catch it in our production setup I'll try to capture the logs. Also, if ISPN-2208 has been resolved, does that imply this is fixed elsewhere? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-5393?page=com.atlassian.jira.plugin.s... ] Brian Stansberry commented on AS7-5393: --------------------------------------- Apologies, Robby, my last comment was incorrect. ISPN-2208 was just a component upgrade to JBoss Marshalling 3.1.15, and that upgrade has been made in both the AS7 master and 7.1 branches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-5393?page=com.atlassian.jira.plugin.s... ] Robby Grossman commented on AS7-5393: ------------------------------------- Attempting to provide a little more context: {noformat} 20:44:57,924 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-2,ip-10-86-135-8/web) ISPN000172: Failed to prepare view CacheView{viewId=5, members=[ip-10-86-135-8/web, domu-12-31-39-0f-d8-91/web, ip-10-202-205-187/web]} for cache default-host/backstage, rolling back to view CacheView{viewId=4, members=[ip-10-86-135-8/web, domu-12-31-39-0f-d8-91/web]}: java.util.concurrent.TimeoutException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258) [rt.jar:1.6.0_24] at java.util.concurrent.FutureTask.get(FutureTask.java:119) [rt.jar:1.6.0_24] at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:318) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:249) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:875) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.6.0_24] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.6.0_24] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_24] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_24] at java.lang.Thread.run(Thread.java:679) [rt.jar:1.6.0_24] 20:44:57,928 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) JBAS010282: Stopped default-host/backstage cache from web container 20:44:57,962 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment torquebox-backstage-knob.yml in 43148ms 20:44:57,964 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015876: Starting deployment of "torquebox-backstage-knob.yml" 20:44:58,605 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-3) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. 20:44:58,607 INFO [org.jboss.as.osgi] (MSC service thread 1-3) JBAS011907: Register module: Module "deployment.torquebox-backstage-knob.yml:main" from Service Module Loader 20:44:58,612 INFO [org.torquebox.core.runtime] (MSC service thread 1-1) Creating ruby runtime (ruby_version: RUBY1_9, compile_mode: JIT, app: torquebox-backstage, context: web) 20:44:58,645 INFO [org.infinispan.jmx.CacheJmxRegistration] (CacheService lifecycle - 1) ISPN000031: MBeans were successfully registered to the platform mbean server. 20:44:59,271 ERROR [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-2,ip-10-86-135-8/web) ISPN000172: Failed to prepare view CacheView{viewId=7, members=[domu-12-31-39-0f-d8-91/web, ip-10-202-205-187/web]} for cache default-host/backstage, rolling back to view CacheView{viewId=6, members=[ip-10-86-135-8/web, domu-12-31-39-0f-d8-91/web]}: java.util.concurrent.ExecutionException: org.infinispan.CacheException: Problems invoking command. at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262) [rt.jar:1.6.0_24] at java.util.concurrent.FutureTask.get(FutureTask.java:119) [rt.jar:1.6.0_24] at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterPrepareView(CacheViewsManagerImpl.java:318) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:249) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:875) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.6.0_24] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.6.0_24] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_24] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_24] at java.lang.Thread.run(Thread.java:679) [rt.jar:1.6.0_24] Caused by: org.infinispan.CacheException: Problems invoking command. at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:215) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.RSVP.up(RSVP.java:179) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.stack.Protocol.up(Protocol.java:363) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) [jgroups-3.0.10.Final.jar:3.0.10.Final] ... 3 more Caused by: java.io.StreamCorruptedException: Unexpected byte found when reading an object: 50 at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:750) [jboss-marshalling-river-1.3.14.GA.jar:1.3.14.GA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [jboss-marshalling-river-1.3.14.GA.jar:1.3.14.GA] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA.jar:1.3.14.GA] at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:99) [jboss-marshalling-1.3.14.GA.jar:1.3.14.GA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:935) [jboss-marshalling-river-1.3.14.GA.jar:1.3.14.GA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1205) [jboss-marshalling-river-1.3.14.GA.jar:1.3.14.GA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) [jboss-marshalling-river-1.3.14.GA.jar:1.3.14.GA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [jboss-marshalling-river-1.3.14.GA.jar:1.3.14.GA] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA.jar:1.3.14.GA] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:162) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:66) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:395) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:224) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) [jboss-marshalling-river-1.3.14.GA.jar:1.3.14.GA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) [jboss-marshalling-river-1.3.14.GA.jar:1.3.14.GA] at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA.jar:1.3.14.GA] at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:104) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] ... 29 more 20:44:59,367 WARN [org.infinispan.cacheviews.CacheViewsManagerImpl] (CacheViewInstaller-2,ip-10-86-135-8/web) ISPN000168: Error rolling back to cache view 6 for cache default-host/backstage: java.lang.NullPointerException at org.infinispan.statetransfer.BaseStateTransferManagerImpl.rollbackView(BaseStateTransferManagerImpl.java:359) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.cacheviews.CacheViewsManagerImpl.handleRollbackView(CacheViewsManagerImpl.java:538) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterRollbackView(CacheViewsManagerImpl.java:354) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.cacheviews.CacheViewsManagerImpl.clusterInstallView(CacheViewsManagerImpl.java:274) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at org.infinispan.cacheviews.CacheViewsManagerImpl$ViewInstallationTask.call(CacheViewsManagerImpl.java:875) [infinispan-core-5.1.5.FINAL.jar:5.1.5.FINAL] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.6.0_24] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.6.0_24] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_24] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_24] at java.lang.Thread.run(Thread.java:679) [rt.jar:1.6.0_24] {noformat} Occasionally I see this beforehand: {noformat} 20:43:27,486 WARN [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (OOB-11,null) Problems unmarshalling remote command from byte buffer: java.lang.ClassNotFoundException: org.infinispan.util.ImmutableListCopy from [Module "org.jboss.as.clustering.jgroups:main" from local module loader @a94884d (roots: /opt/torquebox/current/jboss/modules)] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190) at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468) at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456) at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:423) at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398) at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120) at java.lang.Class.forName0(Native Method) [rt.jar:1.6.0_24] at java.lang.Class.forName(Class.java:264) [rt.jar:1.6.0_24] at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:135) [jboss-marshalling-1.3.14.GA.jar:1.3.14.GA] at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:116) [jboss-marshalling-1.3.14.GA.jar:1.3.14.GA] at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:935) at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1205) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA.jar:1.3.14.GA] at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:162) at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:66) at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:395) at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:224) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351) at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209) at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA.jar:1.3.14.GA] at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163) at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114) at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:104) at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.RSVP.up(RSVP.java:179) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:706) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:566) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.stack.Protocol.up(Protocol.java:363) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) [jgroups-3.0.10.Final.jar:3.0.10.Final] at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) [jgroups-3.0.10.Final.jar:3.0.10.Final] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_24] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_24] at java.lang.Thread.run(Thread.java:679) [rt.jar:1.6.0_24] {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-5393?page=com.atlassian.jira.plugin.s... ] Radoslav Husar commented on AS7-5393: ------------------------------------- Adding one more logging snippet, afterwards on node shutdown the exception on the leaving node is the following {noformat} 16:38:53,974 WARN [org.infinispan.commands.control.StateTransferControlCommand] ISPN000071: Caught exception when handling command StateTransferControlCommand{cache=default-host/container-0_deployment-0_distributable, type=APPLY_STATE, sender=dhcp-29-244/web, viewId=4, state=0, locks=N/A}: java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) [rt.jar:1.6.0_35] at org.infinispan.statetransfer.BaseStateTransferManagerImpl.waitForStateTransferToStart(BaseStateTransferManagerImpl.java:202) at org.infinispan.statetransfer.BaseStateTransferManagerImpl.applyState(BaseStateTransferManagerImpl.java:222) at org.infinispan.commands.control.StateTransferControlCommand.perform(StateTransferControlCommand.java:115) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162) at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:226) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:203) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601) at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) at org.jgroups.JChannel.up(JChannel.java:716) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) at org.jgroups.protocols.RSVP.up(RSVP.java:179) at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602) at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) at org.jgroups.protocols.FD.up(FD.java:273) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) at org.jgroups.protocols.Discovery.up(Discovery.java:359) at org.jgroups.stack.Protocol.up(Protocol.java:363) at org.jgroups.protocols.TP.passMessageUp(TP.java:1185) at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1733) at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1715) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_35] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_35] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_35] 16:38:54,076 INFO [org.infinispan.eviction.PassivationManagerImpl] ISPN000029: Passivating all entries to disk 16:38:54,076 INFO [org.infinispan.eviction.PassivationManagerImpl] ISPN000030: Passivated 0 entries in 0 milliseconds 16:38:54,102 INFO [org.jboss.as.clustering.infinispan] JBAS010282: Stopped local-query cache from hibernate container 16:38:54,104 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] ISPN000082: Stopping the RpcDispatcher 16:38:54,150 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] ISPN000082: Stopping the RpcDispatcher {noformat} plus the server hangs indefinitely in: {noformat} "MSC service thread 1-3" prio=10 tid=0x00007fd0e84fb800 nid=0x41a waiting on condition [0x00007fd0c1251000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000ec208108> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116) at org.jgroups.util.Promise.doWait(Promise.java:116) at org.jgroups.util.Promise._getResultWithTimeout(Promise.java:72) at org.jgroups.util.Promise.getResultWithTimeout(Promise.java:41) at org.jgroups.util.AckCollector.waitForAllAcks(AckCollector.java:93) at org.jgroups.protocols.RSVP$Entry.block(RSVP.java:275) at org.jgroups.protocols.RSVP.down(RSVP.java:114) at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:1033) at org.jgroups.JChannel.down(JChannel.java:730) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:617) at org.jgroups.blocks.RequestCorrelator.sendUnicastRequest(RequestCorrelator.java:193) at org.jgroups.blocks.UnicastRequest.sendRequest(UnicastRequest.java:44) at org.jgroups.blocks.Request.execute(Request.java:83) at org.jgroups.blocks.MessageDispatcher.sendMessageWithFuture(MessageDispatcher.java:418) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:329) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:132) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:492) at org.infinispan.cacheviews.CacheViewsManagerImpl.leave(CacheViewsManagerImpl.java:234) at org.infinispan.statetransfer.BaseStateTransferManagerImpl.stop(BaseStateTransferManagerImpl.java:168) at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:236) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:889) at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:675) at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:553) at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:199) at org.infinispan.CacheImpl.stop(CacheImpl.java:539) at org.infinispan.CacheImpl.stop(CacheImpl.java:531) at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:742) - locked <0x00000000f94ddb28> (a org.infinispan.manager.DefaultCacheManager) at org.infinispan.manager.AbstractDelegatingEmbeddedCacheManager.stop(AbstractDelegatingEmbeddedCacheManager.java:179) at org.jboss.as.clustering.infinispan.subsystem.EmbeddedCacheManagerService.stop(EmbeddedCacheManagerService.java:76) at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-5393?page=com.atlassian.jira.plugin.s... ] Benjamin Browning commented on AS7-5393: ---------------------------------------- Interesting - just this morning I came up with a workaround for TorqueBox users that so far appears to work. Part of that workaround involves removing start='EAGER' from the web cache-container (TorqueBox modified the default config to have start='EAGER' to workaround an earlier issue). The other part of the workaround involves removing an alias from the web cache-container and instead creating a new cache-container with the alias name. I think something about having a TorqueBox MSC dependency on the aliased web cache in combination with start='EAGER' was making this issue crop up reliably. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-5393?page=com.atlassian.jira.plugin.s... ] Doug Mayer commented on AS7-5393: --------------------------------- Having tried removing start='EAGER' with only temporary relief, I can confirm that deploying a JBoss app to a 2- or 3-node cluster in quick succession will cause this error to occur and not be able to recover on its own. We deploy to machines serially, waiting until the .dodeploy is cleaned up before moving onto the next server. When we pause for ~10 seconds AFTER dodeploy is gone before moving onto the next server, we only see the error once (if at all). The same can be seen when restarting JBoss (TorqueBox) servers - they log this error indefinitely when restarted too quickly, and are generally healthy when giving them some space between restarts. This almost always comes after a "ISPN000172: Failed to prepare view CacheView" error for us. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-5393?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-5393: ----------------------------------- In summary on the marshalling side, we are using a ContextClassResolver instead of a ModularClassResolver. As mentioned in the above email thread, I believe this is the problem but haven't proved it conclusively yet. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-5393?page=com.atlassian.jira.plugin.s... ] Radoslav Husar commented on AS7-5393: ------------------------------------- AS branch 7.2 (master) will be fixed with upgrade to Infinispan 5.2.0.CR1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira