[ https://issues.jboss.org/browse/ISPN-1106?page=com.atlassian.jira.plugin.... ] Erik Salter edited comment on ISPN-1106 at 5/15/11 10:55 PM: ------------------------------------------------------------- As an update, I tested this behavior with the latest rehash changes from master. It's much better, but there are still issues with a DIST-mode cluster, eager lock single node, and 2 owners. For instance: - Start nodes 1 and 2. - Run a series of transactions against the cluster. - Start node 3. Repeat tests. There are some errors, but well within the expected behavior (i.e. not handling rollback exceptions in application code). - Stop node 2. Tests are still running. - Restart node 2. Here, I see that a bunch of failures happen due to locks held open on node 1. Node 2 was now the primary data owner. These locks are never released by node 1, even after the rehash. I'll see if I can't build a unit test to reproduce this. was (Author: an1310): As an update, I tested this behavior with the latest rehash changes from master. It's much better, but there are still issues with a DIST-mode cluster, eager lock single node, and 2 owners. For instance: - Start nodes 1 and 2. - Run a series of transactions against the cluster. - Start node 3. Repeat tests. There are some errors, but well within the expected behavior (i.e. not handling rollback exceptions in application code). - Stop node 2. Tests are still running. - Restart node 2. Here, I see that a bunch of failures happen due to locks held open on node 1. Node 2 was now the primary data owner. These locks are never released by node 1, even after the rehash. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1106?page=com.atlassian.jira.plugin.... ] Erik Salter commented on ISPN-1106: ----------------------------------- Per IRC chat, I used the new Distributed Executor to "sticky" a task to the data owner of a set of keys (with the same hash code). I then try to rehash. I'm seeing a tremendous number of timeout errors, including lock cleanup issues. In my production environment (with a JTA transaction manager), I get an inordinate number of the following errors after the new node joins: - org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses! The rehashing takes less than 20 seconds, but the cluster doesn't recover until 3-4 minutes outward. Attached is a unit test that simulates my production environment. It can easily reproduce the lock timeout errors. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1106?page=com.atlassian.jira.plugin.... ] Erik Salter edited comment on ISPN-1106 at 5/22/11 9:33 AM: ------------------------------------------------------------ Per IRC chat, I used the new Distributed Executor to "sticky" a task to the data owner of a set of keys (with the same hash code). I then try to rehash. I'm seeing a tremendous number of timeout errors, including lock cleanup issues. In my production environment (with a JTA transaction manager), I get an inordinate number of the following errors after the new node joins: - org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses! The rehashing takes less than 20 seconds, but the cluster doesn't recover until 3-4 minutes outward. Attached is a unit test that simulates my production environment. It can easily reproduce the lock timeout errors. NOTE: It was built from master, which was 5.0.0.CR3. Just replace that in the pom.xml was (Author: an1310): Per IRC chat, I used the new Distributed Executor to "sticky" a task to the data owner of a set of keys (with the same hash code). I then try to rehash. I'm seeing a tremendous number of timeout errors, including lock cleanup issues. In my production environment (with a JTA transaction manager), I get an inordinate number of the following errors after the new node joins: - org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses! The rehashing takes less than 20 seconds, but the cluster doesn't recover until 3-4 minutes outward. Attached is a unit test that simulates my production environment. It can easily reproduce the lock timeout errors. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1106?page=com.atlassian.jira.plugin.... ] Dan Berindei reassigned ISPN-1106: ---------------------------------- Assignee: Dan Berindei (was: Manik Surtani) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1106?page=com.atlassian.jira.plugin.... ] Dan Berindei commented on ISPN-1106: ------------------------------------ The cache probably became unusable because of the NPE in JGroupsTransport. The transaction cleanup was broken after ISPN-1000, and I had to make some changes to the transaction logger to ensure modifications are blocked *before* they happen. Even after these fixes the attached test case still got a lot of errors in the distributed executor because a new node would get a lot of requests while still starting up (the old nodes would update their CH the moment they saw the new node). I changed the configuration to use the jgroups-udp.xml JGroups configuration supplied with Infinispan and the cache startup got a lot faster, so I'm not seeing any errors any more. I think it was because the timeouts in the test's udp.xml configuration were too high. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1106?page=com.atlassian.jira.plugin.... ] Erik Salter commented on ISPN-1106: ----------------------------------- Good to hear. I'm a little concerned about the JGroups usage. I used TCP in production since my DIST node cluster was going to be < 100 nodes, per the wiki (http://community.jboss.org/wiki/ClusteredConfigurationQuickStart). Do you think the wiki may need to have more detail about using JGroups in these high-throughput situations? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1106?page=com.atlassian.jira.plugin.... ] Erik Salter reopened ISPN-1106: ------------------------------- While developing a UT for ISPN-1255, I re-ran this unit test built against the latest in master (circa-5.0.0-CR8). Attached is the log showing lock cleanup errors and HeuristicMixedExceptions(!) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1106?page=com.atlassian.jira.plugin.... ] Dan Berindei commented on ISPN-1106: ------------------------------------ Erik, I'm not seeing the TimeoutExceptions on my machine, could you also take a stack dump while this is happening and attach it here? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1106?page=com.atlassian.jira.plugin.... ] Anonymous updated ISPN-1106: ---------------------------- Status: Pull Request Sent (was: Reopened) Git Pull Request: https://github.com/infinispan/infinispan/pull/491 (was: https://github.com/infinispan/infinispan/pull/353) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira