[ https://issues.jboss.org/browse/ISPN-937?page=com.atlassian.jira.plugin.s... ] Michal Linhard updated ISPN-937: -------------------------------- Attachment: hotrod-server-stdout.txt infinispan.xml hotrodconsole-0.0.1-SNAPSHOT-bin.zip -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-937?page=com.atlassian.jira.plugin.s... ] Galder Zamarreño updated ISPN-937: ---------------------------------- Fix Version/s: 4.2.1.Final -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-937?page=com.atlassian.jira.plugin.s... ] Michal Linhard updated ISPN-937: -------------------------------- Attachment: server1.log server2.log server2-stdout.log Same thing for 4.2.1.CR3 this time the servers produced logs (infinispan.log) that are attached. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-937?page=com.atlassian.jira.plugin.s... ] Galder Zamarreño commented on ISPN-937: --------------------------------------- Thanks checking CR3 and logs, will look into it. Indeed, the 1st beta of the python client was only released on Monday, so fair point on starting on the java one. Indeed as well, if you start using the python client, it serves as a way to round the edges that the python client might have. Please note the blog post about it (http://infinispan.blogspot.com/2011/02/infinispan-python-client-beta-for-...) and how some stuff are not there yet, but they'll come :) Great work Michal btw! -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-937?page=com.atlassian.jira.plugin.s... ] Galder Zamarreño commented on ISPN-937: --------------------------------------- Michal, this is really odd stuff. Since CR1 was working fine, the only culprit I can think of in CR2 that would cause these are the rehashing changes (that also affected the state transfer code) that Manik did as part of: ISPN-902, ISPN-493, ISPN-914. The state transfer that the exceptions are complaining about is the state transfer of the view cache which is configured with REPL_SYNC and state transfer. Let's wait for Manik to come back and ask him to have a look to this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-937?page=com.atlassian.jira.plugin.s... ] Galder Zamarreño commented on ISPN-937: --------------------------------------- Michal, just saw this and sounds a lot like what you're encountering - http://lists.jboss.org/pipermail/infinispan-dev/2011-February/007456.html It seems like the root cause is the removal of FLUSH in the JGroups stacks (ISPN-83). Can you take you use this https://github.com/infinispan/infinispan/blob/4.2.1.CR1/core/src/main/res... file instead of the one shipped with Infinispan for the transport config? Just download it and point to it via -Djboss.infinispan.jgroups.config= -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-937?page=com.atlassian.jira.plugin.s... ] Galder Zamarreño updated ISPN-937: ---------------------------------- Summary: FLUSH removal causing state transfer timeouts (was: hotrod server state transfer problem during startup) Assignee: Vladimir Blagojevic (was: Galder Zamarreño) Description: FLUSH removal (ISPN-83) is causing issues with state transfer as shown below. Vladimir, mind looking into it? --------- This is something that happend to me when playing with hotrod servers. I've built infinispan distribution from source (mvn install -P distribution -DskipTests=true) unzipped infinispan-4.2.1.CR2-all.zip 2x to folders distro1 distro2 created 2 virtual ifaces 192.168.11.101 test1 192.168.11.102 test2 ran distro1/bin/startServer.sh -r hotrod -l test1 -c /home/mlinhard/dev/projects/infinispan/issues/infinispan.xml distro2/bin/startServer.sh -r hotrod -l test2 -c /home/mlinhard/dev/projects/infinispan/issues/infinispan.xml ran demo-hotrod-console hotrodconsole-0.0.1-SNAPSHOT/bin/console.sh test1 entered the servers waited a bit and then server 2 wrote some errors to stdout (I guess after some timeout expired) client was still blocked, after some time produced ERROR: Invalid magic number. Expected a1 and received 50 attaching - binary version of demo-hotrod-console (source: https://github.com/mlinhard/demo-hotrod-console) - server 2 stdout - infinispan.xml I tried this with 4.2.1.CR1 and it worked. was: This is something that happend to me when playing with hotrod servers. I've built infinispan distribution from source (mvn install -P distribution -DskipTests=true) unzipped infinispan-4.2.1.CR2-all.zip 2x to folders distro1 distro2 created 2 virtual ifaces 192.168.11.101 test1 192.168.11.102 test2 ran distro1/bin/startServer.sh -r hotrod -l test1 -c /home/mlinhard/dev/projects/infinispan/issues/infinispan.xml distro2/bin/startServer.sh -r hotrod -l test2 -c /home/mlinhard/dev/projects/infinispan/issues/infinispan.xml ran demo-hotrod-console hotrodconsole-0.0.1-SNAPSHOT/bin/console.sh test1 entered the servers waited a bit and then server 2 wrote some errors to stdout (I guess after some timeout expired) client was still blocked, after some time produced ERROR: Invalid magic number. Expected a1 and received 50 attaching - binary version of demo-hotrod-console (source: https://github.com/mlinhard/demo-hotrod-console) - server 2 stdout - infinispan.xml I tried this with 4.2.1.CR1 and it worked. > FLUSH removal causing state transfer timeouts > --------------------------------------------- > > Key: ISPN-937 > URL: https://issues.jboss.org/browse/ISPN-937 > Project: Infinispan > Issue Type: Bug > Components: Cache Server > Affects Versions: 4.2.1.CR2 > Reporter: Michal Linhard > Assignee: Vladimir Blagojevic > Fix For: 4.2.1.FINAL > > Attachments: hotrod-server-stdout.txt, hotrodconsole-0.0.1-SNAPSHOT-bin.zip, infinispan.xml, server1.log, server2-stdout.log, server2.log > > > FLUSH removal (ISPN-83) is causing issues with state transfer as shown below. Vladimir, mind looking into it? > --------- > This is something that happend to me when playing with hotrod servers. > I've built infinispan distribution from source (mvn install -P distribution -DskipTests=true) > unzipped infinispan-4.2.1.CR2-all.zip 2x to folders > distro1 > distro2 > created 2 virtual ifaces > 192.168.11.101 test1 > 192.168.11.102 test2 > ran > distro1/bin/startServer.sh -r hotrod -l test1 -c /home/mlinhard/dev/projects/infinispan/issues/infinispan.xml > distro2/bin/startServer.sh -r hotrod -l test2 -c /home/mlinhard/dev/projects/infinispan/issues/infinispan.xml > ran demo-hotrod-console > hotrodconsole-0.0.1-SNAPSHOT/bin/console.sh test1 > entered > > the servers waited a bit and then server 2 wrote some errors to stdout (I guess after some timeout expired) > client was still blocked, after some time produced > ERROR: Invalid magic number. Expected a1 and received 50 > attaching > - binary version of demo-hotrod-console (source: https://github.com/mlinhard/demo-hotrod-console) > - server 2 stdout > - infinispan.xml > I tried this with 4.2.1.CR1 and it worked. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-937?page=com.atlassian.jira.plugin.s... ] Vladimir Blagojevic commented on ISPN-937: ------------------------------------------ Michal, just to reiterate, you first discovered this problem with 4.2.1CR2? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-937?page=com.atlassian.jira.plugin.s... ] Paul Ferraro updated ISPN-937: ------------------------------ Attachment: test.log Attach TRACE log demonstrating timeout. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-937?page=com.atlassian.jira.plugin.s... ] Manik Surtani commented on ISPN-937: ------------------------------------ @Vladimir have we got to the bottom of this? Or are we reinstating flush in 4.2.x for now? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/ISPN-937?page=com.atlassian.jira.plugin.s... ] Manik Surtani updated ISPN-937: ------------------------------- I'm closing this as we've reinstated FLUSH on 4.2.1. But we've added a system flag to bypass the FLUSH check, so you can still use a non-FLUSH protocol stack. More investigation into actually removing FLUSH will be done for 5.0 and is tracked by ISPN-83 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira