Hi, I've extended the original DLD design to also support deadlock detection on local caches and updated design forum [1].

This, together with the replicated deadlock detection is implemented in trunk (some minor stuff to do still: DLD for aggregation methods like clear and addAll + unit test). I've also created a benchmark to test what's the throughput (tx/min) between caches running with and without DLD. You can find full test description within test class: http://anonsvn.jboss.org/repos/infinispan/trunk/core/src/test/java//org/i... Local DLD does good job (cca 5.5 times better) but replicated DLD does extraordinary: cca 101 better throughput (see attached).

I think DLD is cool stuff and differentiates us a bit from competition, afaik none of them have a DLD. One more thing that's worth mentioning: while running DLD tests I've noticed that if all tx acquire locks on keys in same order, then no deadlocks exists. This is logic and might seem obvious, but is not stated anywhere and the performance increase by doing this might be very dramatic. I think we should document this as a best practice when it comes to transactions - any suggestion where? I also intend to blog about it shortly. Cheers, Mircea [1] http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4244838#... < DLD_local .JPG ><DLD_replicated.JPG>_______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

> So you do force a lot of deadlocks as a result. And the time taken > with the non-DLD case would depend on the TM's transaction timeout > configuration which again would vary. The Dummy TM does not force rollback based on tx timeout, indeed. Another important factor it depends on is the lockAcquisitionTimeout. Still, it's up to the user to benchmark against its very specific scenario.

> So as a result I'd be careful about quoting performance increase > factors in a public blog (although you should definitely blog about > this as a feature and how it *could* speed up transactions that > would otherwise timeout). Point taken. Are you also thinking not to bring up the diagrams? After all, they the numbers there are real: so by mentioning the context (intended high collision) and no tx timeout, they are relevant.

Manik Surtani wrote: > <SNIP /> > >>> So you do force a lot of deadlocks as a result. And the time >>> taken with the non-DLD case would depend on the TM's transaction >>> timeout configuration which again would vary. >> The Dummy TM does not force rollback based on tx timeout, indeed. >> Another important factor it depends on is the >> lockAcquisitionTimeout. Still, it's up to the user to benchmark >> against its very specific scenario. > > Sorry yes, I meant lockAcquisitionTimeout in my prev. comment. In this case lowering lock acquisition timeout would result in rollbacks of non-deadlocked txs as well, so not sure the overall outcome is better.

> >>> So as a result I'd be careful about quoting performance increase >>> factors in a public blog (although you should definitely blog >>> about this as a feature and how it *could* speed up transactions >>> that would otherwise timeout). >> Point taken. Are you also thinking not to bring up the diagrams? >> After all, they the numbers there are real: so by mentioning the >> context (intended high collision) and no tx timeout, they are >> relevant. > > Well, the numbers are only real given the rate of colliding > transactions (versus non-colliding ones) and the lock acquisition > timeout that you used. Which is artificial, or at best specific to > a certain use case. But either way, not generic enough to publish > since it would lead people to believe that they can expect a 500% > performance boost when using deadlock detection, and get really > pissed off when they *only* get a 25% performance boost since most > of their transactions don't collide anyway. :-) What about a pool of 50 objects? Or 100?

Manik Surtani wrote: > > On 22 Jul 2009, at 10:42, Mircea Markus wrote: > >> Manik Surtani wrote: >>> <SNIP /> >>> >>>>> So you do force a lot of deadlocks as a result. And the time >>>>> taken with the non-DLD case would depend on the TM's >>>>> transaction timeout configuration which again would vary. >>>> The Dummy TM does not force rollback based on tx timeout, >>>> indeed. Another important factor it depends on is the >>>> lockAcquisitionTimeout. Still, it's up to the user to benchmark >>>> against its very specific scenario. >>> >>> Sorry yes, I meant lockAcquisitionTimeout in my prev. comment. >> In this case lowering lock acquisition timeout would result in >> rollbacks of non-deadlocked txs as well, so not sure the overall >> outcome is better. > > My point is that any gain (measured as a percentage or a factor) is > directly related to the lock acquisition timeout. Which is, in > turn, hugely application-specific. :) > >>> >>>>> So as a result I'd be careful about quoting performance >>>>> increase factors in a public blog (although you should >>>>> definitely blog about this as a feature and how it *could* >>>>> speed up transactions that would otherwise timeout). >>>> Point taken. Are you also thinking not to bring up the diagrams? >>>> After all, they the numbers there are real: so by mentioning the >>>> context (intended high collision) and no tx timeout, they are >>>> relevant. >>> >>> Well, the numbers are only real given the rate of colliding >>> transactions (versus non-colliding ones) and the lock acquisition >>> timeout that you used. Which is artificial, or at best specific >>> to a certain use case. But either way, not generic enough to >>> publish since it would lead people to believe that they can >>> expect a 500% performance boost when using deadlock detection, >>> and get really pissed off when they *only* get a 25% performance >>> boost since most of their transactions don't collide anyway. :-) >> What about a pool of 50 objects? Or 100? > > Again, this is entirely artificial. How many of these should > collide? Is it realistic? :-) I think these numbers give a better understanding to the user than saying "DLD might/will have a significant performance increase in an environment where you have a significant collision". Also the test environment would be clearly described, so no user would expect an 500% increase out of the blue.

Manik Surtani wrote: > > On 22 Jul 2009, at 11:06, Mircea Markus wrote: > >> Manik Surtani wrote: >>> >>> On 22 Jul 2009, at 10:42, Mircea Markus wrote: >>> >>>> Manik Surtani wrote: >>>>> <SNIP /> >>>>> >>>>>>> So you do force a lot of deadlocks as a result. And the time >>>>>>> taken with the non-DLD case would depend on the TM's >>>>>>> transaction timeout configuration which again would vary. >>>>>> The Dummy TM does not force rollback based on tx timeout, >>>>>> indeed. Another important factor it depends on is the >>>>>> lockAcquisitionTimeout. Still, it's up to the user to >>>>>> benchmark against its very specific scenario. >>>>> >>>>> Sorry yes, I meant lockAcquisitionTimeout in my prev. comment. >>>> In this case lowering lock acquisition timeout would result in >>>> rollbacks of non-deadlocked txs as well, so not sure the overall >>>> outcome is better. >>> >>> My point is that any gain (measured as a percentage or a factor) >>> is directly related to the lock acquisition timeout. Which is, >>> in turn, hugely application-specific. :) >>> >>>>> >>>>>>> So as a result I'd be careful about quoting performance >>>>>>> increase factors in a public blog (although you should >>>>>>> definitely blog about this as a feature and how it *could* >>>>>>> speed up transactions that would otherwise timeout). >>>>>> Point taken. Are you also thinking not to bring up the >>>>>> diagrams? After all, they the numbers there are real: so by >>>>>> mentioning the context (intended high collision) and no tx >>>>>> timeout, they are relevant. >>>>> >>>>> Well, the numbers are only real given the rate of colliding >>>>> transactions (versus non-colliding ones) and the lock >>>>> acquisition timeout that you used. Which is artificial, or at >>>>> best specific to a certain use case. But either way, not >>>>> generic enough to publish since it would lead people to believe >>>>> that they can expect a 500% performance boost when using >>>>> deadlock detection, and get really pissed off when they *only* >>>>> get a 25% performance boost since most of their transactions >>>>> don't collide anyway. :-) >>>> What about a pool of 50 objects? Or 100? >>> >>> Again, this is entirely artificial. How many of these should >>> collide? Is it realistic? :-) >> I think these numbers give a better understanding to the user than >> saying "DLD might/will have a significant performance increase in >> an environment where you have a significant collision". Also the >> test environment would be clearly described, so no user would >> expect an 500% increase out of the blue. > > I think the real solution in a system that has a lot of deadlocking > transactions is to try and redesign the app so that these > deadlocking transactions do not occur. :-) Are you just telling me that this functionality is useless ??? :)

Yes, that's what one needs (and should do), where possible. Even so, it was a bit difficult (not anymore) to determine weather there are deadlocks in the system, or just regular timeouts. I've also exposed the deadlock detection info through JMX - I'll make sure this appears as a not in the blog I'll write. > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > >

Manik Surtani wrote: > > On 21 Jul 2009, at 11:19, Mircea Markus wrote: > >> Hi, >> >> I've extended the original DLD design to also support deadlock >> detection on local caches and updated design forum [1]. > > Does it also support async transports (1-phase commits)? I've just finished implementing 1PC. > >> This, together with the replicated deadlock detection is >> implemented in trunk (some minor stuff to do still: DLD for >> aggregation methods like clear and addAll + unit test). >> >> I've also created a benchmark to test what's the throughput (tx/ >> min) between caches running with and without DLD. >> You can find full test description within test class: http://anonsvn.jboss.org/repos/infinispan/trunk/core/src/test/java//org/i... >> Local DLD does good job (cca 5.5 times better) but replicated DLD >> does extraordinary: cca 101 better throughput (see attached). > > This is very interesting, but perhaps a little artificial since > your key pool size is only 10. So you do force a lot of deadlocks > as a result. And the time taken with the non-DLD case would depend > on the TM's transaction timeout configuration which again would > vary. So as a result I'd be careful about quoting performance > increase factors in a public blog (although you should definitely > blog about this as a feature and how it *could* speed up > transactions that would otherwise timeout). > > Also, it would be interesting to see how the cache fares with and > without DLD, in a test where there are absolutely no deadlocks. > E.g., each thread's access patterns access the same keys, but in a > way that would never deadlock. I'd like to see if DLD adds much of > an overhead. I've also run such a test (is the same test as previously, but the key set is ordered now -> no deadlocks). The average performance decrease is approx 7%. (See attached).

> > Overall though, very cool stuff! :) > > Cheers > Manik > >> I think DLD is cool stuff and differentiates us a bit from >> competition, afaik none of them have a DLD. >> >> One more thing that's worth mentioning: while running DLD tests >> I've noticed that if all tx acquire locks on keys in same order, >> then no deadlocks exists. This is logic and might seem obvious, >> but is not stated anywhere and the performance increase by doing >> this might be very dramatic. I think we should document this as a >> best practice when it comes to transactions - any suggestion where? >> >> I also intend to blog about it shortly. >> >> Cheers, >> Mircea >> >> [1] http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4244838#... >> >> >> < >> DLD_local >> .JPG >> ><DLD_replicated.JPG>_______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > <DLD_enabling_overhead.JPG>

Hi, I've extended the original DLD design to also support deadlock detection on local caches and updated design forum [1]. This, together with the replicated deadlock detection is implemented in trunk (some minor stuff to do still: DLD for aggregation methods like clear and addAll + unit test). I've also created a benchmark to test what's the throughput (tx/min) between caches running with and without DLD. You can find full test description within test class: http://anonsvn.jboss.org/repos/infinispan/trunk/core/src/test/java//org/i... Local DLD does good job (cca 5.5 times better) but replicated DLD does extraordinary: cca 101 better throughput (see attached). I think DLD is cool stuff and differentiates us a bit from competition, afaik none of them have a DLD. One more thing that's worth mentioning: while running DLD tests I've noticed that if all tx acquire locks on keys in same order, then no deadlocks exists. This is logic and might seem obvious, but is not stated anywhere and the performance increase by doing this might be very dramatic. I think we should document this as a best practice when it comes to transactions - any suggestion where? I also intend to blog about it shortly. Cheers, Mircea [1] http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4244838#... < DLD_local .JPG ><DLD_replicated.JPG>_______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Manik Surtani wrote: > BTW I just thought of something (implementation detail): you have a > new LockManager impl here, right? > DeadlockDetectingLockManagerImpl? The default LockManagerImpl > exposes stuff via JMX (and JOPR) using annotations on the impl > class. Can we move these annotations onto the LockManager > interface so that we don't need to repeat this for any arbitrary > interface? Should test that these annotations are still picked up > even if they are on the interface. Not sure if this is the case. > > Cheers > Manik Well, even if we would have this annotations on the interface, the different implementations would still need to take care of implementing them, and hence the advantage would be in not repeating the annotation itself only. DeadlockDetectingLockManager is inheriting from LockMnagerImpl and hence the problem is already solved for current setup.