2:42 p.m.
RequestIgnoredException if a node
----------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Reporter: Erik Salter
Assignee: Manik Surtani
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For high
throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:08 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-1255) RequestIgnoredException if a node
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Erik Salter updated ISPN-1255:
------------------------------
Attachment: server_node1.log
server_node2.log
Here are logs of the system after the first node starts, executes txs, and then a second
node starts. The caches are in DIST mode with numOwners == 2, which means that all data
in the system must rehash.
RequestIgnoredException if a node
----------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Reporter: Erik Salter
Assignee: Manik Surtani
Attachments: server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:42 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-1255) RequestIgnoredException if a node
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Erik Salter updated ISPN-1255:
------------------------------
Attachment: cacheTest.zip
I added a new test in the "group" package that illustrate my desired usage:
1. I get a set of keys. I have to eagerly lock them all.
2. I do an operation on one key and then save it in another cache.
3. I return the cache key to the end user.
I make liberal use of the new Group API (previously I was screwing with the key's hash
code.)
The test fails with:
org.infinispan.CacheException: Unable to invoke method private void
org.infinispan.distribution.DistributionManagerImpl.join() throws java.lang.Exception on
object
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:175)
at
org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:918)
at
org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:711)
at
org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:611)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:180)
at org.infinispan.CacheImpl.start(CacheImpl.java:366)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:559)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:455)
at
net.beaumaris.cachetest.group.GroupingRehashTest$GroupStressingThread.initExecutor(GroupingRehashTest.java:103)
at net.beaumaris.cachetest.BaseStressingThread.<init>(BaseStressingThread.java:24)
at
net.beaumaris.cachetest.group.GroupingRehashTest$GroupStressingThread.<init>(GroupingRehashTest.java:60)
at
net.beaumaris.cachetest.group.GroupingRehashTest.createStressingThread(GroupingRehashTest.java:54)
at net.beaumaris.cachetest.BaseCacheTest.doRehashJoin(BaseCacheTest.java:66)
at
net.beaumaris.cachetest.group.GroupingRehashTest.testRehashJoin(GroupingRehashTest.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
at
org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
at
org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)
at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
... 35 more
Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out after 30 seconds
waiting for a response from esalter-lt-19341
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:267)
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:116)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:437)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:132)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:156)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:265)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:252)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:248)
at
org.infinispan.distribution.DistributionManagerImpl.notifyCoordinatorPushCompleted(DistributionManagerImpl.java:469)
at
org.infinispan.distribution.DistributionManagerImpl.join(DistributionManagerImpl.java:187)
... 40 more
RequestIgnoredException if a node
----------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Reporter: Erik Salter
Assignee: Manik Surtani
Attachments: cacheTest.zip, server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:07 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-1255) RequestIgnoredException if a node
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Manik Surtani updated ISPN-1255:
--------------------------------
Assignee: Vladimir Blagojevic (was: Manik Surtani)
Fix Version/s: 5.0.0.FINAL
Affects Version/s: 5.0.0.CR7
RequestIgnoredException if a node
----------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.0.0.CR7
Reporter: Erik Salter
Assignee: Vladimir Blagojevic
Fix For: 5.0.0.FINAL
Attachments: cacheTest.zip, server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:10 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-1255) RequestIgnoredException if a node
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Manik Surtani commented on ISPN-1255:
-------------------------------------
This could be because a single-threaded retry queue is used to handle queued RPCs while a
node is starting and cannot handle RPCs. It would make sense to minimise the number of
RPCs enqueued by only enqueueing updates (and not things like remote executions), but this
needs more thought.
In the meanwhile, does increasing your sync repl timeout help?
RequestIgnoredException if a node
----------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.0.0.CR7
Reporter: Erik Salter
Assignee: Vladimir Blagojevic
Fix For: 5.0.0.FINAL
Attachments: cacheTest.zip, server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:32 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-1255) RequestIgnoredException if a node
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Erik Salter commented on ISPN-1255:
-----------------------------------
Are you referring to the above error? This was produced with a unit test. In production,
the syncReplTimeout is also 30 seconds. Changing it to 60 doesn't seem to help much,
even at low throughput rates (30 tx/sec).
RequestIgnoredException if a node
----------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.0.0.CR7
Reporter: Erik Salter
Assignee: Vladimir Blagojevic
Fix For: 5.0.0.FINAL
Attachments: cacheTest.zip, server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:56 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-1255) RequestIgnoredException on rehash using the Distributed Executor Service
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Erik Salter updated ISPN-1255:
------------------------------
Summary: RequestIgnoredException on rehash using the Distributed Executor Service
(was: RequestIgnoredException if a node )
RequestIgnoredException on rehash using the Distributed Executor
Service
------------------------------------------------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.0.0.CR7
Reporter: Erik Salter
Assignee: Vladimir Blagojevic
Fix For: 5.0.0.FINAL
Attachments: cacheTest.zip, server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:49 a.m.
New subject: [JBoss JIRA] Commented: (ISPN-1255) RequestIgnoredException on rehash using the Distributed Executor Service
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-1255:
------------------------------------
I decreased the GMS timeouts and the RequestIgnoredResponses disappeared from my log:
{code:xml}
<pbcast.GMS print_local_addr="true"
join_timeout="3000"
leave_timeout="3000"
merge_timeout="60000"
view_ack_collection_timeout="2000"
view_bundling="true"
max_bundling_time="1000"/>
{code}
I'm pretty sure that this is because GMS delays the new node's start procedure by
{{join_timeout}} milliseconds, and
RequestIgnoredException on rehash using the Distributed Executor
Service
------------------------------------------------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.0.0.CR7
Reporter: Erik Salter
Assignee: Vladimir Blagojevic
Fix For: 5.0.0.FINAL
Attachments: cacheTest.zip, server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:51 a.m.
New subject: [JBoss JIRA] Issue Comment Edited: (ISPN-1255) RequestIgnoredException on rehash using the Distributed Executor Service
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Dan Berindei edited comment on ISPN-1255 at 7/22/11 7:49 AM:
-------------------------------------------------------------
I decreased the GMS timeouts and the RequestIgnoredResponses disappeared from my log:
{code:xml}
<pbcast.GMS print_local_addr="true"
join_timeout="3000"
leave_timeout="3000"
merge_timeout="60000"
view_ack_collection_timeout="2000"
view_bundling="true"
max_bundling_time="1000"/>
{code}
InboundInvocationHandler will send back a {{RequestIgnoredResponse}} if the cache startup
takes more than 30 seconds, so I'm pretty sure that GMS somehow delays the new
node's startup procedure by {{join_timeout}} milliseconds.
was (Author: dan.berindei):
I decreased the GMS timeouts and the RequestIgnoredResponses disappeared from my log:
{code:xml}
<pbcast.GMS print_local_addr="true"
join_timeout="3000"
leave_timeout="3000"
merge_timeout="60000"
view_ack_collection_timeout="2000"
view_bundling="true"
max_bundling_time="1000"/>
{code}
I'm pretty sure that this is because GMS delays the new node's start procedure by
{{join_timeout}} milliseconds, and
RequestIgnoredException on rehash using the Distributed Executor
Service
------------------------------------------------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.0.0.CR7
Reporter: Erik Salter
Assignee: Vladimir Blagojevic
Fix For: 5.0.0.FINAL
Attachments: cacheTest.zip, server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:36 p.m.
New subject: [JBoss JIRA] Commented: (ISPN-1255) RequestIgnoredException on rehash using the Distributed Executor Service
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Dan Berindei commented on ISPN-1255:
------------------------------------
I never saw the timeout in join(), but I think I know how it coul happen. Could you get a
stack dump immediately afterwards?
I've also seen a TimeoutException during AllocateTask.toTx(), but I fixed that by
sorting the keys in the constructor.
RequestIgnoredException on rehash using the Distributed Executor
Service
------------------------------------------------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.0.0.CR7
Reporter: Erik Salter
Assignee: Vladimir Blagojevic
Fix For: 5.0.0.FINAL
Attachments: cacheTest.zip, server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
3:48 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-1255) RequestIgnoredException on rehash using the Distributed Executor Service
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Erik Salter updated ISPN-1255:
------------------------------
Attachment: ispn1255.log
I still see the timeout in join() and the cache never recovers. It happens
intermittently, and it's difficult to reproduce reliably.
I updated the unit test to sort the keys as well (in the stressing thread) and updated the
key class to use the new Grouping API (it can only use strings), and I get timeout
exceptions. I get them AFTER rehashing -- maybe some locks are not cleaned up?
I attached a log after the rehash.
RequestIgnoredException on rehash using the Distributed Executor
Service
------------------------------------------------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.0.0.CR7
Reporter: Erik Salter
Assignee: Vladimir Blagojevic
Fix For: 5.0.0.FINAL
Attachments: cacheTest.zip, ispn1255.log, server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:17 p.m.
New subject: [JBoss JIRA] Commented: (ISPN-1255) RequestIgnoredException on rehash using the Distributed Executor Service
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Erik Salter commented on ISPN-1255:
-----------------------------------
As an update, changing the GMS timeout values to Dan's values didn't really reduce
the number of these occurrences.
RequestIgnoredException on rehash using the Distributed Executor
Service
------------------------------------------------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.0.0.CR7
Reporter: Erik Salter
Assignee: Vladimir Blagojevic
Fix For: 5.0.0.FINAL
Attachments: cacheTest.zip, ispn1255.log, server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:29 a.m.
New subject: [JBoss JIRA] Assigned: (ISPN-1255) RequestIgnoredException on rehash using the Distributed Executor Service
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Manik Surtani reassigned ISPN-1255:
-----------------------------------
Assignee: Dan Berindei (was: Vladimir Blagojevic)
RequestIgnoredException on rehash using the Distributed Executor
Service
------------------------------------------------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.0.0.CR7
Reporter: Erik Salter
Assignee: Dan Berindei
Fix For: 5.0.0.FINAL
Attachments: cacheTest.zip, ispn1255.log, server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
10:24 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-1255) RequestIgnoredException on rehash using the Distributed Executor Service
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Erik Salter updated ISPN-1255:
------------------------------
Attachment: join_failure_thread_dump
I reproduced the join() failure.
--- MBeans waiting for other MBeans ---
ObjectName: net.beaumaris:InfinispanCacheDeployer=InfinispanCacheDeployer
State: FAILED
Reason: org.infinispan.CacheException: Unable to invoke method private void
org.infinispan.distribution.DistributionManagerImpl.join() throws java.lang.Exception on
object
I Depend On:
jboss:name=SystemProperties,type=Service
Depends On Me:
jboss.web.deployment:war=dht.billing-1.1.1.war,id=-961963574
jboss.web.deployment:war=dht.poilist-1.1.1.war,id=568459511
jboss.web.deployment:war=dht.session-1.1.1.war,id=1470944293
jboss.web.deployment:war=odrm-resource-1.1.1.war,id=-986086344
ObjectName: jboss.web.deployment:war=dht.billing-1.1.1.war,id=-961963574
State: NOTYETINSTALLED
I Depend On:
jboss.esb:deployment=spring.esb
net.beaumaris:InfinispanCacheDeployer=InfinispanCacheDeployer
ObjectName: jboss.web.deployment:war=dht.poilist-1.1.1.war,id=568459511
State: NOTYETINSTALLED
I Depend On:
jboss.esb:deployment=spring.esb
net.beaumaris:InfinispanCacheDeployer=InfinispanCacheDeployer
ObjectName: jboss.web.deployment:war=dht.session-1.1.1.war,id=1470944293
State: NOTYETINSTALLED
I Depend On:
jboss.esb:deployment=spring.esb
net.beaumaris:InfinispanCacheDeployer=InfinispanCacheDeployer
ObjectName: jboss.web.deployment:war=odrm-resource-1.1.1.war,id=-986086344
State: NOTYETINSTALLED
I Depend On:
jboss.esb:deployment=spring.esb
net.beaumaris:InfinispanCacheDeployer=InfinispanCacheDeployer
--- MBEANS THAT ARE THE ROOT CAUSE OF THE PROBLEM ---
ObjectName: net.beaumaris:InfinispanCacheDeployer=InfinispanCacheDeployer
State: FAILED
Reason: org.infinispan.CacheException: Unable to invoke method private void
org.infinispan.distribution.DistributionManagerImpl.join() throws java.lang.Exception on
object
I Depend On:
jboss:name=SystemProperties,type=Service
Depends On Me:
jboss.web.deployment:war=dht.billing-1.1.1.war,id=-961963574
jboss.web.deployment:war=dht.poilist-1.1.1.war,id=568459511
jboss.web.deployment:war=dht.session-1.1.1.war,id=1470944293
jboss.web.deployment:war=odrm-resource-1.1.1.war,id=-986086344
RequestIgnoredException on rehash using the Distributed Executor
Service
------------------------------------------------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.0.0.CR7
Reporter: Erik Salter
Assignee: Dan Berindei
Fix For: 5.0.0.FINAL
Attachments: cacheTest.zip, ispn1255.log, join_failure_thread_dump,
server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
9:26 a.m.
New subject: [JBoss JIRA] Updated: (ISPN-1255) RequestIgnoredException on rehash using the Distributed Executor Service
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Anonymous updated ISPN-1255:
----------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/491
RequestIgnoredException on rehash using the Distributed Executor
Service
------------------------------------------------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.0.0.CR7
Reporter: Erik Salter
Assignee: Dan Berindei
Fix For: 5.0.0.FINAL
Attachments: cacheTest.zip, ispn1255.log, join_failure_thread_dump,
server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
1:39 p.m.
New subject: [JBoss JIRA] Updated: (ISPN-1255) RequestIgnoredException on rehash using the Distributed Executor Service
[
https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin....
]
Manik Surtani updated ISPN-1255:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
RequestIgnoredException on rehash using the Distributed Executor
Service
------------------------------------------------------------------------
Key: ISPN-1255
URL: https://issues.jboss.org/browse/ISPN-1255
Project: Infinispan
Issue Type: Bug
Affects Versions: 5.0.0.CR7
Reporter: Erik Salter
Assignee: Dan Berindei
Fix For: 5.0.0.FINAL
Attachments: cacheTest.zip, ispn1255.log, join_failure_thread_dump,
server_node1.log, server_node2.log
My application exposes its distributed operations via a REST-based infrastructure. To
minimize the delta between JBoss starting and the cache starting, I used the new
Distributed Executor to "sticky" a task to the data owner of a set of keys (with
the same hash code).
NOTE: Rehash still causes problems seen in ISPN-1106. (Attached new logs)
I see a lot of the following error from the DistributedExecutorService when the new
node's cache doesn't start in a timely manner:
Reason: java.lang.IllegalStateException: Invalid response
{Satriani-52149(PHL)=RequestIgnoredResponse}
In addition, I see:
org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover. For
high throughput rate, the cluster doesn't recover.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
4658
days inactive
4675
days old
infinispan-issues@lists.jboss.org
15 comments
4 participants
participants (4)
-
Anonymous (JIRA) -
Dan Berindei (JIRA)
-
Erik Salter (JIRA)
-
Manik Surtani (JIRA)