<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>The tests work for me:</div><div><br></div><div><div>Running org.jboss.test.capedwarf.cluster.DatastoreTestCase</div><div>Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.924 sec</div></div><div><br></div><div>And the Lucene* caches start fine; no stderr, etc.</div><div><br></div><div>But I do get warnings and sometime even some NPEs (on AS boot),</div><div>which I guess is the problem of brand new NBST:</div><div><br></div><div>-- node1</div><div><br></div><div><div>08:20:32,574 WARN [org.infinispan.topology.CacheTopologyControlCommand] (transport-thread-3) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=LuceneIndexesData, type=LEAVE, sender=node-b/capedwarf, joinInfo=null, topologyId=0, currentCH=null, pendingCH=null, throwable=null, viewId=-1}: org.infinispan.CacheException: The cache has been stopped and invocations are not allowed!</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.remoting.transport.jgroups.JGroupsTransport.getViewId(JGroupsTransport.java:229)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.ClusterTopologyManagerImpl.updateConsistentHash(ClusterTopologyManagerImpl.java:121)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.DefaultRebalancePolicy.updateConsistentHash(DefaultRebalancePolicy.java:131)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.DefaultRebalancePolicy.updateCacheMembers(DefaultRebalancePolicy.java:238)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.DefaultRebalancePolicy.removeLeavers(DefaultRebalancePolicy.java:203)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.ClusterTopologyManagerImpl.handleLeave(ClusterTopologyManagerImpl.java:169)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:153)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:136)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.LocalTopologyManagerImpl$1.call(LocalTopologyManagerImpl.java:253)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.FutureTask.run(FutureTask.java:138) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.lang.Thread.run(Thread.java:680) [classes.jar:1.6.0_33]</div></div><div><br></div><div>-- node2</div><div><br></div><div><div>08:20:32,350 WARN [org.infinispan.topology.CacheTopologyControlCommand] (transport-thread-12) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=LuceneIndexesMetadata, type=LEAVE, sender=node-a/capedwarf, joinInfo=null, topologyId=0, currentCH=null, pendingCH=null, throwable=null, viewId=-1}: java.lang.InterruptedException</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1024) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.FutureTask.get(FutureTask.java:91) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:229)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.ClusterTopologyManagerImpl.updateConsistentHash(ClusterTopologyManagerImpl.java:124)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.DefaultRebalancePolicy.updateConsistentHash(DefaultRebalancePolicy.java:131)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.DefaultRebalancePolicy.updateCacheMembers(DefaultRebalancePolicy.java:238)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.DefaultRebalancePolicy.removeLeavers(DefaultRebalancePolicy.java:203)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.ClusterTopologyManagerImpl.handleLeave(ClusterTopologyManagerImpl.java:169)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:153)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:136)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.LocalTopologyManagerImpl$1.call(LocalTopologyManagerImpl.java:253)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.FutureTask.run(FutureTask.java:138) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.lang.Thread.run(Thread.java:680) [classes.jar:1.6.0_33]</div><div><br></div><div>08:20:32,366 WARN [org.infinispan.topology.CacheTopologyControlCommand] (transport-thread-8) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=dist, type=LEAVE, sender=node-a/capedwarf, joinInfo=null, topologyId=0, currentCH=null, pendingCH=null, throwable=null, viewId=1}: java.lang.InterruptedException</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1024) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:227) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.FutureTask.get(FutureTask.java:91) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterSync(ClusterTopologyManagerImpl.java:229)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.ClusterTopologyManagerImpl.updateConsistentHash(ClusterTopologyManagerImpl.java:124)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.DefaultRebalancePolicy.updateConsistentHash(DefaultRebalancePolicy.java:131)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.DefaultRebalancePolicy.updateCacheMembers(DefaultRebalancePolicy.java:238)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.DefaultRebalancePolicy.removeLeavers(DefaultRebalancePolicy.java:203)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.ClusterTopologyManagerImpl.handleLeave(ClusterTopologyManagerImpl.java:169)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:153)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:136)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at org.infinispan.topology.LocalTopologyManagerImpl$1.call(LocalTopologyManagerImpl.java:253)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.FutureTask.run(FutureTask.java:138) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [classes.jar:1.6.0_33]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>at java.lang.Thread.run(Thread.java:680) [classes.jar:1.6.0_33]</div><div><br></div></div><div>-Ales</div><br><div><div>On Sep 4, 2012, at 4:03 AM, Mircea Markus <<a href="mailto:mircea.markus@jboss.com">mircea.markus@jboss.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">The root cause of this, as suggested by Dan and Sanne is the fact that during a cache's creation another cache is being started.<div>I've made a fix candidate[1] which I assigned to Dan. I still get some failures on Ales' test, but with a different error message.</div><div><br></div><div>[1] <a href="https://github.com/infinispan/infinispan/pull/1281">https://github.com/infinispan/infinispan/pull/1281</a></div><div>Cheers,</div><div>Mircea </div><div><br><div><div>On 3 Sep 2012, at 10:30, Ales Justin wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html charset=us-ascii"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Also to add ...</div><div><br></div><div>With some more testing, it showed that it is *slave* node's queries that failed.</div><div><br></div><div>I setup a static master node selection: via jgroupsMaster / jgroupsSlave.</div><div>When I changed the node type, different test failed.</div><div><br></div><div>e.g. if node A was slave, <span style="color: rgb(20, 20, 20); font-family: 'Lucida Grande'; line-height: 14px; ">queryOnA, indexGenAndQueryInsertOnA, testContentOnA</span><span style="color: rgb(20, 20, 20); font-family: 'Lucida Grande'; line-height: 14px; "> failed.</span></div><div><span style="color: rgb(20, 20, 20); font-family: 'Lucida Grande'; line-height: 14px; ">if node B was slave, simply change A --> B in test names above</span></div><div><span style="color: rgb(20, 20, 20); font-family: 'Lucida Grande'; line-height: 14px; "><br></span></div><div><span style="color: rgb(20, 20, 20); font-family: 'Lucida Grande'; line-height: 14px; ">This is the full test:</span></div><div><span style="color: rgb(20, 20, 20); font-family: 'Lucida Grande'; line-height: 14px; ">* </span><a href="https://raw.github.com/capedwarf/capedwarf-blue/a3b33ffad60e69a4c67f6b0e4ebe299ffa37d350/cluster-tests/src/test/java/org/jboss/test/capedwarf/cluster/DatastoreTestCase.java">https://raw.github.com/capedwarf/capedwarf-blue/a3b33ffad60e69a4c67f6b0e4ebe299ffa37d350/cluster-tests/src/test/java/org/jboss/test/capedwarf/cluster/DatastoreTestCase.java</a></div><div><br></div><div>As Sanne mentioned, that locking issue on Lucene cache's looks suspicious ...</div><div><br></div><div>-Ales</div><br><div><div>On Sep 3, 2012, at 11:20 AM, Sanne Grinovero <<a href="mailto:sanne@infinispan.org">sanne@infinispan.org</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Team, I hope someone more familiar with state transfer and cache<br>starting logic can help Ales as I don't think this is a Query or<br>Hibernate Search related problem.<br>Even more as his test was working not so long ago, and there where no<br>Search releases nor mayor changes in Infinispan Query.<br><br>This problem and the locking errors he reported in a different post<br>are very likely related: if the background thread in Hibernate<br>Search's backend fails to update the index because of the locking<br>issue, it doesn't surprise me that the other nodes are unable to see<br>an updated index (as write operations failed).<br><br>Cheers,<br>Sanne<br><br>On 3 September 2012 01:00, Ales Justin <<a href="mailto:ales.justin@gmail.com">ales.justin@gmail.com</a>> wrote:<br><blockquote type="cite">After fixing that ISPN-2253:<br>*<br><a href="https://github.com/alesj/infinispan/commit/05229f426e829742902de0305488282b8283b8e5">https://github.com/alesj/infinispan/commit/05229f426e829742902de0305488282b8283b8e5</a><br>(Sanne is working on even cleaner solution)<br><br>It now also looks like our CapeDwarf clustering tests have been (almost)<br>fixed.<br>It now appears as if one node cannot do proper querying:<br><br>Deployment dep1 goes to node1, same deployment dep2 goes to node2.<br><br>"Failed tests:<br>queryOnA(org.jboss.test.capedwarf.cluster.DatastoreTestCase): Number of<br>entities: 0"<br><br>Where as "queryOnB" doesn't fail.<br><br>@Sanne -- do we have any query tests covering this kind of scenario?<br><br>I'm using jgroups' auto-master selection.<br><br>Tomorrow I'll try fixed jgroups master/slave selection,<br>and the new dynamic auto-master selection.<br><br>Any ideas still welcome. ;-)<br><br>-Ales<br><br>---<br><br> @InSequence(30)<br> @Test<br> @OperateOnDeployment("dep1")<br> public void putStoresEntityOnDepA() throws Exception {<br> Entity entity = createTestEntity("KIND", 1);<br> getService().put(entity);<br> assertStoreContains(entity);<br> }<br><br> @InSequence(31)<br> @Test<br> @OperateOnDeployment("dep2")<br> public void putStoresEntityOnDepB() throws Exception {<br> Entity entity = createTestEntity("KIND", 2);<br> getService().put(entity);<br> assertStoreContains(entity);<br> }<br><br> @InSequence(40)<br> @Test<br> @OperateOnDeployment("dep1")<br> public void getEntityOnDepA() throws Exception {<br> waitForSync();<br><br> Key key = KeyFactory.createKey("KIND", 1);<br> Entity lookup = getService().get(key);<br><br> Assert.assertNotNull(lookup);<br><br> Entity entity = createTestEntity("KIND", 1);<br> Assert.assertEquals(entity, lookup);<br> }<br><br> @InSequence(50)<br> @Test<br> @OperateOnDeployment("dep2")<br> public void getEntityOnDepB() throws Exception {<br> waitForSync();<br><br> Entity entity = createTestEntity("KIND", 1);<br> assertStoreContains(entity);<br> }<br><br> @InSequence(52)<br> @Test<br> @OperateOnDeployment("dep1")<br> public void queryOnA() throws Exception {<br> waitForSync();<br><br> int count = getService().prepare(new<br>Query("KIND")).countEntities(Builder.withDefaults());<br> Assert.assertTrue("Number of entities: " + count, count == 2);<br> }<br><br> @InSequence(53)<br> @Test<br> @OperateOnDeployment("dep2")<br> public void queryOnB() throws Exception {<br> waitForSync();<br><br> int count = getService().prepare(new<br>Query("KIND")).countEntities(Builder.withDefaults());<br> Assert.assertTrue("Number of entities: " + count, count == 2);<br> }<br><br></blockquote></blockquote></div><br></div>_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br><a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote></div><br><div apple-content-edited="true">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>-- <br>Mircea Markus</div><div>Infinispan lead (<a href="http://www.infinispan.org/">www.infinispan.org</a>)</div><div><br></div></div></span></div></span></div><br class="Apple-interchange-newline">
</div>
<br></div></div>_______________________________________________<br>infinispan-dev mailing list<br><a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>https://lists.jboss.org/mailman/listinfo/infinispan-dev</blockquote></div><br></body></html>