[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] Manik Surtani updated ISPN-1508: -------------------------------- Fix Version/s: 5.1.0.BETA4 Priority: Critical (was: Major) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] Galder Zamarreño updated ISPN-1508: ----------------------------------- Fix Version/s: 5.1.0.BETA5 (was: 5.1.0.BETA4) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] Dan Berindei commented on ISPN-1508: ------------------------------------ The root of the problem is that the JGroups channel is created outside of Infinispan and injected with a ChannelLookup. Between the time the channel is started externally and the time JGroupsTransport attaches the RpcDispatcher there is a small window where incoming messages will be silently discarded. In replication mode a put operation would wait for a success response from all the members of the cluster, and if the RPC was initiated during this time window it would never get a response, failing with a TimeoutException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] Manik Surtani resolved ISPN-1508. --------------------------------- Resolution: Done -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] Michal Linhard commented on ISPN-1508: -------------------------------------- in the subsequent run, the cluster won't even form in 2 minute limit: http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-QE/job/edg-60-experi... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] Dan Berindei commented on ISPN-1508: ------------------------------------ Ok, looks like there was something else going on: the state transfer for a distributed cache may start before the state transfer for the topology cache. I'm not sure if it's because all the caches are started in parallel or if it only happens when the cluster forms through a merge (because getCache() will return on the 2nd node before doing any state transfer). The topology cache needs to be fully populated before we can create consistent hashes for the other caches, so if the topology cache's state transfer didn't run yet all the other (distributed) caches will fail to install the new cache view. We do retry to install the cache view, so this should only have caused some error messages on startup, but unfortunately there is another problem with the way cache view installation is cancelled. It allows the PREPARE_VIEW and ROLLBACK_VIEW commands to run in the reverse order on the remote nodes and seems to lead to a cycle of failed cache view installation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] Dan Berindei commented on ISPN-1508: ------------------------------------ It looks like the AS7 Infinispan subsystem can indeed start other caches before the topology cache has finished starting. I have fixed the ordering of the PREPARE_VIEW and ROLLBACK_VIEW commands and while there are still a few cache view installation failures on startup they stop after the topology cache has finished starting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] RH Bugzilla Integration updated ISPN-1508: ------------------------------------------ Git Pull Request: https://github.com/infinispan/infinispan/pull/655 (was: https://github.com/infinispan/infinispan/pull/655) Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=751382, https://bugzilla.redhat.com/show_bug.cgi?id=758218 (was: https://bugzilla.redhat.com/show_bug.cgi?id=751382) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] RH Bugzilla Integration commented on ISPN-1508: ----------------------------------------------- Michal Linhard <misoli(a)centrum.cz&gt; made a comment on [bug 758218|https://bugzilla.redhat.com/show_bug.cgi?id=758218] EDG build: http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-QE/job/edg-60-build-... Infinispan version: 5.1.0.BETA5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] Michal Linhard edited comment on ISPN-1508 at 11/30/11 1:58 PM: ---------------------------------------------------------------- I have this special case when I use jprofiler on node1 in 4 node cluster, where the cluster won't form even for 4 minutes: https://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-PERF/job/ed... (the node01 never ends joining and state transfer) I've set up my test to ignore following error patterns: {code} ERROR \\[org\\.infinispan\\.cacheviews\\.CacheViewsManagerImpl\\].*Failed to prepare view .* ERROR \\[org\\.infinispan\\.cacheviews\\.CacheViewsManagerImpl\\].*ISPN000166.* ERROR \\[org\\.infinispan\\.cacheviews\\.CacheViewInfo\\].*ISPN000164.* ERROR \\[org\\.infinispan\\.cacheviews\\.CacheViewsManagerImpl\\].*Error recovering views from the cluster members.* {code} When I don't use jprofiler (and the ignore patterns are the same) the cluster forms well: https://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-PERF/job/ed... was (Author: mlinhard): I have this special case when I use jprofiler on node1 in 4 node cluster, where the cluster won't form even for 4 minutes: https://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-PERF/job/ed... (the node01 never ends joining and state transfer) I've set up my test to ignore following error patterns: ERROR \\[org\\.infinispan\\.cacheviews\\.CacheViewsManagerImpl\\].*Failed to prepare view .* ERROR \\[org\\.infinispan\\.cacheviews\\.CacheViewsManagerImpl\\].*ISPN000166.* ERROR \\[org\\.infinispan\\.cacheviews\\.CacheViewInfo\\].*ISPN000164.* ERROR \\[org\\.infinispan\\.cacheviews\\.CacheViewsManagerImpl\\].*Error recovering views from the cluster members.* When I don't use jprofiler (and the ignore patterns are the same) the cluster forms well: https://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-PERF/job/ed... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] RH Bugzilla Integration commented on ISPN-1508: ----------------------------------------------- Michal Linhard <misoli(a)centrum.cz&gt; made a comment on [bug 758218|https://bugzilla.redhat.com/show_bug.cgi?id=758218] Raising severity. This is a blocker for EDG QE performance tests, currently no workaround seems to exist. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] Michal Linhard commented on ISPN-1508: -------------------------------------- IMO this should be reopened cause it no longer holds that the initial failures during cluster formation go away after a while. However I only saw this in EDG tests, therefore I reported only EDG BugZilla. (Initial issue was found with pure Infinispan instances) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] Dan Berindei commented on ISPN-1508: ------------------------------------ Manik, Michal, the fix for this issue will not be 100% complete until we remove the dependency of distributed caches on the HotRod server startup. Galder is working on that. The fix I committed for 5.1.0.BETA5 only ensured that the initial failure isn't fatal and the cache view installation and succeed the second time. However that didn't work all the time either, hence ISPN-1571. Michal, the logs you posted seem to indicate a slightly different problem. The cache views recovery after a merge fails with a TimeoutException, and we don't retry cache view recovery at the moment. Can you try again with the latest master and DEBUG logging enabled for org.infinispan and org.jgroups? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] Michal Linhard commented on ISPN-1508: -------------------------------------- run with EDG build with new Infinispan snapshot, server side DEBUG enabled, cluster size 4: build: http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-QE/job/edg-60-build-... versions: http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-QE/job/edg-60-build-... http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-PERF/job/edg... errors still present but cluster forms eventually -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] Michal Linhard commented on ISPN-1508: -------------------------------------- another TRACE log attempt: http://hudson.qa.jboss.com/hudson/view/EDG6/view/EDG-REPORTS-PERF/job/edg... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] RH Bugzilla Integration commented on ISPN-1508: ----------------------------------------------- Tristan Tarrant <ttarrant(a)redhat.com&gt; changed the Status of [bug 751382|https://bugzilla.redhat.com/show_bug.cgi?id=751382] from NEW to ON_QA -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] RH Bugzilla Integration commented on ISPN-1508: ----------------------------------------------- Martin Gencur <mgencur(a)redhat.com&gt; made a comment on [bug 751382|https://bugzilla.redhat.com/show_bug.cgi?id=751382] This problem does not occur anymore. Verified with ER2 build. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-1508?page=com.atlassian.jira.plugin.... ] RH Bugzilla Integration commented on ISPN-1508: ----------------------------------------------- Tristan Tarrant <ttarrant(a)redhat.com&gt; changed the Status of [bug 751382|https://bugzilla.redhat.com/show_bug.cgi?id=751382] from VERIFIED to CLOSED -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira