[infinispan-issues] [JBoss JIRA] Updated: (ISPN-1255) RequestIgnoredException on rehash using the Distributed Executor Service

[Erik Salter (JIRA)]

     [ https://issues.jboss.org/browse/ISPN-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Salter updated ISPN-1255:
------------------------------

    Attachment: join_failure_thread_dump


I reproduced the join() failure.

--- MBeans waiting for other MBeans ---
ObjectName: net.beaumaris:InfinispanCacheDeployer=InfinispanCacheDeployer
  State: FAILED
  Reason: org.infinispan.CacheException: Unable to invoke method private void org.infinispan.distribution.DistributionManagerImpl.join() throws java.lang.Exception on object
  I Depend On:
    jboss:name=SystemProperties,type=Service
  Depends On Me:
    jboss.web.deployment:war=dht.billing-1.1.1.war,id=-961963574
    jboss.web.deployment:war=dht.poilist-1.1.1.war,id=568459511
    jboss.web.deployment:war=dht.session-1.1.1.war,id=1470944293
    jboss.web.deployment:war=odrm-resource-1.1.1.war,id=-986086344

ObjectName: jboss.web.deployment:war=dht.billing-1.1.1.war,id=-961963574
  State: NOTYETINSTALLED
  I Depend On:
    jboss.esb:deployment=spring.esb
    net.beaumaris:InfinispanCacheDeployer=InfinispanCacheDeployer

ObjectName: jboss.web.deployment:war=dht.poilist-1.1.1.war,id=568459511
  State: NOTYETINSTALLED
  I Depend On:
    jboss.esb:deployment=spring.esb
    net.beaumaris:InfinispanCacheDeployer=InfinispanCacheDeployer

ObjectName: jboss.web.deployment:war=dht.session-1.1.1.war,id=1470944293
  State: NOTYETINSTALLED
  I Depend On:
    jboss.esb:deployment=spring.esb
    net.beaumaris:InfinispanCacheDeployer=InfinispanCacheDeployer

ObjectName: jboss.web.deployment:war=odrm-resource-1.1.1.war,id=-986086344
  State: NOTYETINSTALLED
  I Depend On:
    jboss.esb:deployment=spring.esb
    net.beaumaris:InfinispanCacheDeployer=InfinispanCacheDeployer

--- MBEANS THAT ARE THE ROOT CAUSE OF THE PROBLEM ---
ObjectName: net.beaumaris:InfinispanCacheDeployer=InfinispanCacheDeployer
  State: FAILED
  Reason: org.infinispan.CacheException: Unable to invoke method private void org.infinispan.distribution.DistributionManagerImpl.join() throws java.lang.Exception on object
  I Depend On:
    jboss:name=SystemProperties,type=Service
  Depends On Me:
    jboss.web.deployment:war=dht.billing-1.1.1.war,id=-961963574
    jboss.web.deployment:war=dht.poilist-1.1.1.war,id=568459511
    jboss.web.deployment:war=dht.session-1.1.1.war,id=1470944293
    jboss.web.deployment:war=odrm-resource-1.1.1.war,id=-986086344


> RequestIgnoredException on rehash using the Distributed Executor Service
> ------------------------------------------------------------------------
>
>                 Key: ISPN-1255
>                 URL: https://issues.jboss.org/browse/ISPN-1255
>             Project: Infinispan
>          Issue Type: Bug
>    Affects Versions: 5.0.0.CR7
>            Reporter: Erik Salter
>            Assignee: Dan Berindei
>             Fix For: 5.0.0.FINAL
>
>         Attachments: cacheTest.zip, ispn1255.log, join_failure_thread_dump, server_node1.log, server_node2.log
>
>
> My application exposes its distributed operations via a REST-based infrastructure.  To minimize the delta between JBoss starting and the cache starting, I used the new Distributed Executor to "sticky" a task to the data owner of a set of keys (with the same hash code). 
> NOTE:  Rehash still causes problems seen in ISPN-1106.  (Attached new logs)
> I see a lot of the following error from the DistributedExecutorService when the new node's cache doesn't start in a timely manner: 
> Reason: java.lang.IllegalStateException: Invalid response {Satriani-52149(PHL)=RequestIgnoredResponse}
> In addition, I see:
> org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
> It takes the cache about 2+ minutes at low throughput rate (30 tx/s) to recover.  For high throughput rate, the cluster doesn't recover. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira