I'm using that "jgroups" backend, with auto-master election. No idea. btw: didn't you say you had some new backend mechanism? Off Infinispan's channel. -Ales

I believe this new backend is WIP in Hibernate Search. Sanne, didn't you have a prototype in Infinispan's codebase though? On 5 Apr 2013, at 15:28, Ales Justin <ales.justin(a)gmail.com&gt; wrote: -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Platform Architect, JBoss Data Grid http://red.ht/data-grid

On 8 Apr 2013, at 11:28, Ales Justin <ales.justin(a)gmail.com&gt; wrote: Any thoughts on this, Sanne? -- Manik Surtani manik(a)jboss.org twitter.com/maniksurtani Platform Architect, JBoss Data Grid http://red.ht/data-grid

No node failure that I'm aware of. We did get some unexpected NPE in DataNucleus framework, but, imo, that shouldn't completely kill the app. We'll re-try. And then also re-try with no locking. @Bela, Sanne: how would this look like? As this looks like the best workaround for now -- if this is really the issue. -Ales On Apr 8, 2013, at 1:19 PM, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote: > There should be no locking contention at all, that is the whole point > of using such a backend and forwarding changes to a single node: that > only a single node ever attempts to acquire this lock. Hence the error > is a simptom of some previous error, I primarily suspect cluster view > stability. > > I indeed have committed an experimental alternative backend in > Infinispan Query (included in master) and another one in Hibernate > Search (not master but a branch I'm working on); > the one in Hibernate Search is meant to superseed the others but it's > not working yet as I need CAS to be working in Infinispan, and this is > still failing my tests. > > The result of failing CAS is the master election: multiple nodes elect > themselves, which results in the locking error. > > Ales is this error happening after a node failure? AFAIK the missing > feature of the JGroups based backend is that it doesn't cleanup stale > index locks when a master fails; each master node releases the lock as > soon as possible (as you have set exclusive_index_use=false) but if > the node is disconnected exactly during the write operation the lock > will need to be cleaned up forcefully. I would normally expect this to > be very unlikely but it could be triggered if you have view stability > problems. > We could try integrating some kind of force-lock clean operation but > it's quite tricky to make sure this happens safely.. there is of > course a purpose for this lock. > > You could try turning off the seatbelt by setting > > default.locking_strategy = none > > > I do have a cleaner solution with proper lock cleanup routines, but > these are based on the CAS operation too.. they are failing stress > tests so I won't commit them yet. > > Sanne > > On 8 April 2013 11:38, Manik Surtani <msurtani(a)redhat.com&gt; wrote: >> >> On 8 Apr 2013, at 11:28, Ales Justin <ales.justin(a)gmail.com&gt; wrote: >> >> This "jgroups" backend was there "long" ago. >> And it was actually us - CD - that fixed it and made use of it. >> It's no different from static JGroups backed, the only diff that this one >> elects master automatically. >> >> I can change to Sanne's new Ispn based prototype if it will help. >> >> But - with my limited cluster knowledge - the issue doesn't look to be >> there. >> I mean, the msgs get properly routed to indexing master, which just cannot >> handle locking contention. >> >> >> Any thoughts on this, Sanne? >> >> >> -Ales >> >> I believe this new backend is WIP in Hibernate Search. Sanne, didn't you >> have a prototype in Infinispan's codebase though? >> >> On 5 Apr 2013, at 15:28, Ales Justin <ales.justin(a)gmail.com&gt; wrote: >> >> are you not using the JGroups backend anymore? >> >> >> I'm using that "jgroups" backend, with auto-master election. >> >> these Lock acquisitions are on the index lock, and make me suspect your >> configuration is no longer applying the pattern we discussed a while back, >> when you contributed the fixed to the JGroups indexing backend. >> >> Or is it the "Replication timeout for mstruk/capedwarf" which is causing >> those locking errors? >> >> >> No idea. >> >> btw: didn't you say you had some new backend mechanism? >> Off Infinispan's channel. >> >> -Ales >> >> On 5 April 2013 14:56, Ales Justin <ales.justin(a)gmail.com&gt; wrote: >>> >>> We're running a GAE HelloOrm2 example app on 3 nodes (3 laptops). >>> >>> Very soon after deploy, we get a never-ending stack of timeouts, >>> which completely kills the app: >>> * https://gist.github.com/alesj/5319414 >>> >>> I then need to kill the AS in order to get it shutdown. >>> >>> How can this be tuned / fixed? >>> >>> -Ales >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> -- >> Manik Surtani >> manik(a)jboss.org >> twitter.com/maniksurtani >> >> Platform Architect, JBoss Data Grid >> http://red.ht/data-grid >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> -- >> Manik Surtani >> manik(a)jboss.org >> twitter.com/maniksurtani >> >> Platform Architect, JBoss Data Grid >> http://red.ht/data-grid >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev

That's a ReplicationTimeout, I don't think Search has anything to do with it.. On 8 April 2013 13:25, Ales Justin <ales.justin(a)gmail.com&gt; wrote:

Steps to re-produce: (1) checkout JBossAS 7.2.0.Final tag --> JBOSS_HOME (2) build CapeDwarf Shared https://github.com/capedwarf/capedwarf-shared (3) build CapeDwarf Blue https://github.com/capedwarf/capedwarf-blue (4) build CapeDwarf AS https://github.com/capedwarf/capedwarf-jboss-as mvn clean install -Djboss.dir=<JBOSS_HOME> -Pupdate-as This will install CapeDwarf Subsystem into previous AS 7.2.0.Final (5) grab GAE 1.7.6 SDK http://googleappengine.googlecode.com/files/appengine-java-sdk-1.7.6.zip (6) Build GAE demos/helloorm2 ant cd war/ zip -r ROOT.war . This will zip the demo app as ROOT.war, which you then deploy to AS. (7) start CapeDwarf JBOSS_HOME/bin ./standalone.sh -c standalone-capedwarf.xml -b <IP> -Djboss.node.name=some_name (8) deploy the app / ROOT.war --- Deploy this on a few nodes, goto browser: http://<IP:8080>, add a few flights and see how it works. It now runs a bit better, where we changed mstruk's laptop with luksa's. But we still get replication locks ... Also, the problem is that query on indexing slave takes waaay tooo long. Anyway, you'll see. ;-) Ping me for any issues. -Ales

On 8 Apr 2013, at 14:38, Ales Justin wrote: what does a lot better mean? Does it run as expected? Would be good to see the amount of time spent in logging. Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org)

Querying data on any node returns in ~30ms. And it returns correct data. ;-) I doubt it's a lot. Imo, it's the amount of stuff that gets put into cache that's the problem. And we need to index it all as well -- for GAE log queries. But it's still not enormous amount of data. e.g. I'm yet to try GridFS (Ispn's GFS), in a real cluster, for our GAE Blobstore support ... -Ales

Nah, that would also show with standalone testing then, and it doesn't. Marko found out that DataNucleus is pushing DEBUG logs in, dunno yet why. But, imo, that still shouldn't totally cripple app's performance, yet alone kill it at the end. e.g. I can imagine users trying to debug issues with FINE, in a cluster -- that's the whole point of CD And atm, this is not possible. -Ales On Apr 8, 2013, at 4:43 PM, Sanne Grinovero <sanne(a)infinispan.org&gt; wrote:

Ah, ok. But from Marko's observations, this is not the case. -Ales

Hey Dan, first, thanks for looking into this! Yeah, with trace you get insane number of log lines. Although, this seems to show that again our logging doesn't really work as expected. * https://github.com/capedwarf/capedwarf-jboss-as/blob/master/extension/src... As we explicitly exclude logs from the frameworks/libs we use internally in CapeDwarf. @Luksa: we need to check this, again ... :-) @James: do you see any obvious mistakes in our logging deployer? That's why we excluded the internal frmwrks/libs in the first place. See the url above -- we define a new handler: org.jboss.capedwarf.shared.log.Logger As I said, it should already work that way -- but unfortunately it looks like it doesn't. But, hopefully, this should work: * you disable org.infinispan explicitly in GAE log file --> https://developers.google.com/appengine/docs/java/runtime#Logging Add WEB-INF/logging.properties file with JUL logging config / properties to helloorm2.war (aka our ROOT.war). HTH Ping me on irc for any other issues. -Ales

Yes, afaik, Dan got a ton of trace logs into CD for Ispn. But as you can see we explicitly exclude org.infinispan by default. I also asked Marko to investigate this. -Ales On Apr 9, 2013, at 19:25, "James R. Perkins" <jperkins(a)redhat.com&gt; wrote:

On 8 Apr 2013, at 12:19, Sanne Grinovero wrote: Do you have a branch with a failing CAS test? I could take a look. Cheers, -- Mircea Markus Infinispan lead (www.infinispan.org)