[infinispan-issues] [JBoss JIRA] Updated: (ISPN-1255) RequestIgnoredException if a node

[Erik Salter (JIRA)]

     [ https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Salter updated ISPN-1255:
------------------------------

    Attachment: cacheTest.zip


I added a new test in the "group" package that illustrate my desired usage:

1.  I get a set of keys.  I have to eagerly lock them all.
2.  I do an operation on one key and then save it in another cache.
3.  I return the cache key to the end user.

I make liberal use of the new Group API (previously I was screwing with the key's hash code.)  

The test fails with:

org.infinispan.CacheException: Unable to invoke method private void org.infinispan.distribution.DistributionManagerImpl.join() throws java.lang.Exception on object 
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:175)
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:918)
	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:711)
	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:611)
	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:180)
	at org.infinispan.CacheImpl.start(CacheImpl.java:366)
	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:559)
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:455)
	at net.beaumaris.cachetest.group.GroupingRehashTest$GroupStressingThread.initExecutor(GroupingRehashTest.java:103)
	at net.beaumaris.cachetest.BaseStressingThread.<init>(BaseStressingThread.java:24)
	at net.beaumaris.cachetest.group.GroupingRehashTest$GroupStressingThread.<init>(GroupingRehashTest.java:60)
	at net.beaumaris.cachetest.group.GroupingRehashTest.createStressingThread(GroupingRehashTest.java:54)
	at net.beaumaris.cachetest.BaseCacheTest.doRehashJoin(BaseCacheTest.java:66)
	at net.beaumaris.cachetest.group.GroupingRehashTest.testRehashJoin(GroupingRehashTest.java:49)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
	at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
	at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
	at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
	at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
	at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
	at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
	at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
	at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)
	at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
	at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
	at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
	... 35 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out after 30 seconds waiting for a response from esalter-lt-19341
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:267)
	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:116)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:437)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:132)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:156)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:265)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:252)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:248)
	at org.infinispan.distribution.DistributionManagerImpl.notifyCoordinatorPushCompleted(DistributionManagerImpl.java:469)
	at org.infinispan.distribution.DistributionManagerImpl.join(DistributionManagerImpl.java:187)
	... 40 more


> RequestIgnoredException if a node 
> ----------------------------------
>
>                 Key: ISPN-1255
>                 URL: https://issues.jboss.org/browse/ISPN-1255
>             Project: Infinispan
>          Issue Type: Bug
>            Reporter: Erik Salter
>            Assignee: Manik Surtani
>         Attachments: cacheTest.zip, server_node1.log, server_node2.log
>
>
> My application exposes its distributed operations via a REST-based infrastructure.  To minimize the delta between JBoss starting and the cache starting, I used the new Distributed Executor to "sticky" a task to the data owner of a set of keys (with the same hash code). 
> NOTE:  Rehash still causes problems seen in ISPN-1106.  (Attached new logs)
> I see a lot of the following error from the DistributedExecutorService when the new node's cache doesn't start in a timely manner: 
> Reason: java.lang.IllegalStateException: Invalid response {Satriani-52149(PHL)=RequestIgnoredResponse}
> In addition, I see:
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
> It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover.  For high throughput rate, the cluster doesn't recover. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira