Could we also model and benchmark Hibernate's 2nd level cache as an access pattern?

On 29 November 2011 10:26, Mircea Markus <mircea.markus(a)jboss.com&gt; wrote: > > On 29 Nov 2011, at 10:08, Manik Surtani wrote: > >> Could we also model and benchmark Hibernate's 2nd level cache as an access pattern? > Galder had some very convincing arguments about not doing it with Radargun the last few times when we discussed about this :-) Maybe not with Radargun, but is sounds like a use case that we should test in some way.

_______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi, Now that all the 5.1 major work is done I plan to run some benchmarks comparing 5.1 with 5.0. It's not only tx stuff I'd like to compare, as some other were made, so here's my plan of action. Each of the following benchmarks will be run on local, distributed and replicated caches: 1. non transactional: web session replication[1] 2. transactional (both optimistic and pessimistic): web session replication[2] 3. transactional (both optimistic and pessimistic): tpcc [3] The difference between 2 and 3 is the fact that 3 induces some key contention between transactions. Any other suggestion for benchmarking?

Cheers, Mircea [1] http://sourceforge.net/apps/trac/radargun/wiki/WebSessionBenchmark [2] http://sourceforge.net/apps/trac/radargun/wiki/BenchmarkingTransactions [3] tpcc is Radargun extension developed by Sebastiano Peluso from the CloudTM team. It's now integrated in Radargun, I'm working on adding some documentation around it atm. _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi, On Tue, Nov 29, 2011 at 09:00, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > > On Nov 28, 2011, at 5:00 PM, Mircea Markus wrote: > >> Hi, >> >> Now that all the 5.1 major work is done I plan to run some benchmarks comparing 5.1 with 5.0. It's not only tx stuff I'd like to compare, as some other were made, so here's my plan of action. >> >> Each of the following benchmarks will be run on local, distributed and replicated caches: >> >> 1. non transactional: web session replication[1] >> 2. transactional (both optimistic and pessimistic): web session replication[2] >> 3. transactional (both optimistic and pessimistic): tpcc [3] >> >> The difference between 2 and 3 is the fact that 3 induces some key contention between transactions. >> >> Any other suggestion for benchmarking? > > Could it be interesting to see what the autoCommit penalty is? i.e. comparing: non-transactional cache vs transactional cache with autoCommit Pure Transactional and autoCommit 'off' does not work in BETA 5. I am compiling a list of changes I made to make it happen. To save you the trouble of going through what I did and give you a solution that 'works'.

Hi, O.K. so here is what I found out (this is linked with the thread titled "Re: [infinispan-dev] PutForExternalRead and autoCommit", a copy of this message will be sent on it, though it does not close the discussion on it). First, let me remind everyone, that I am using: - Atomikos as a transaction manager - Hibernate 3.5.5 - infinispan 5.1.0 BETA5. - I have the 2 nodes set up for invalidation on all caches except for the timestamp cache which is in replication mode (for hibernate). So, as discussed on the other thread, when autoCommit is set to false in a transactional cache environment, an exception occurs since in a few cases, the infinispan system suspends the ongoing transaction. In this case, when it comes to effectively putting data (whether it is "forExternalRead" or not, I will not get into that here) it show a valid exception stating that a transaction is needed for a transactional cache. So here is what I had to do to move forward on that. Since I have a smaller use case that what infinispan can handle, I could safely remove the code that suspends the transaction where needed to test it out. Here is what I did: 1. In CacheImpl, I removed the logic in "final void putForExternalRead(K key, V value, EnumSet<Flag> explicitFlags, ClassLoader explicitClassLoader)" on lines 355, 357-359, 376-385 (removed all code linked with the suspension that I did not want given that I wanted to ensure the autoCommit to false). That worked and got a HUGE performance improvement, factor of 5 in the execution/replication/invalidation time (as in literally 5 times faster). 2. Then I realized that the BaseRegion class also had some logic that I was running into that was suspending transactions. I changed the methods "suspend" and "resume(Transaction tx)" to do nothing (suspends returns systematically null and resume expect always null as an input with my changes). This change was what brought me to realize that there was a MAJOR issue with hibernate (at least in 3.5.5) when doing that. The timestamp cache/region has a special behaviour in "ActionQueue". If you look at "AfterTransactionCompletionProcessQueue" the method "public void afterTransactionCompletion(boolean success)" once all is done, checks if there is a cache and pushes a set of invalidations. The main issue here is that: 1. my transaction was already successfully committed by hibernate and these operations are done outside of the scope of the transaction. 2. consequence of #1: I had set autoCommit to false, so no transactions were auto started for me, so the invalidations failed systematically (I think it was correct for the invalidations to fail). (The changes to "fix" that are enclosed to the email, they are simple). Thus, this leads me to say that a pure transactional cache with autoCommit set to false cannot work at this point without the changes I made to my own. I correctly works for me and after other changes I got it to perform really well. On that note, since we are talking performance, I explored the batching functionality of infinispan (which is great!!). It does not quite perform as is for all regions/cache. Indeed, if you run into the issue I describe above in the timestamp cache/region, and your autoCommit is set to true, no batching happens (you are indeed outside of the transaction which started the batch) and the invalidation/replication is super expensive: 2 commands/messages per invalidation (one for prepare and one for commit). Implementing the changes I described allowed the batching to work for my timestamp region and that is when I got an additional 2x factor and everything is lightening fast. When I say 2x factor, I mean a total of 10x faster with all the changes I put in versus, the 5.1.0 BETA5 out of the box (from 1 sec to 100ms per http transaction on my API !!). This is for asynchronous invalidation, it is slower with synchronous. While I am at it, I was wondering if it would be logical to have infinispan automatically start batches in a transactional context. It made sense to me and I have activated in my own code. Please find enclosed the files changed. Let me know if anything I said does not make sense or is wrong. I tested it and so far it works well but again my use case is a small subset of what infinispan can do. Regards, L.P. PS: also, org/hibernate/cache/infinispan/util/AddressAdapterImpl.java and org/hibernate/cache/infinispan/util/CacheHelper.java need public no-arg constructors for the de serialization to work, they did not have them in hibernate 3.5.5. On Tue, Nov 29, 2011 at 10:15, Mircea Markus <mircea.markus(a)jboss.com&gt; wrote: > > On 29 Nov 2011, at 14:03, Slorg1 wrote: > > Hi, > > On Tue, Nov 29, 2011 at 09:00, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > > > On Nov 28, 2011, at 5:00 PM, Mircea Markus wrote: > > > Hi, > > > Now that all the 5.1 major work is done I plan to run some benchmarks > comparing 5.1 with 5.0. It's not only tx stuff I'd like to compare, as some > other were made, so here's my plan of action. > > > Each of the following benchmarks will be run on local, distributed and > replicated caches: > > > 1. non transactional: web session replication[1] > > 2. transactional (both optimistic and pessimistic): web session > replication[2] > > 3. transactional (both optimistic and pessimistic): tpcc [3] > > > The difference between 2 and 3 is the fact that 3 induces some key > contention between transactions. > > > Any other suggestion for benchmarking? > > > Could it be interesting to see what the autoCommit penalty is? i.e. > comparing: non-transactional cache vs transactional cache with autoCommit > > > > Pure Transactional and autoCommit 'off' does not work in BETA 5. I am > compiling a list of changes I made to make it happen. > > > To save you the trouble of going through what I did and give you a > solution that 'works'. > > What do you mean by do not work? > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Please consider the environment - do you really need to print this email ?

Hey Slorg1, I have not forgotten this thread by any means. I think you make some interesting points which I need to look into more deeply. I'm also in the process of trying to integrate 5.1.0.CR1 with Hibernate 2LC and I'm having problems with one unit test. I'll get back to you wrt to this email asap. Thanks for your input! On Dec 7, 2011, at 6:06 PM, Slorg1 wrote: > Hi, > > Has anyone given any thoughts to that? > > Thank you. > > Regards, > > Slorg1. > > On Tue, Nov 29, 2011 at 12:34, Slorg1 <slorg1(a)gmail.com&gt; wrote: >> Hi, >> >> O.K. so here is what I found out (this is linked with the thread >> titled "Re: [infinispan-dev] PutForExternalRead and autoCommit", a >> copy of this message will be sent on it, though it does not close the >> discussion on it). >> >> First, let me remind everyone, that I am using: >> - Atomikos as a transaction manager >> - Hibernate 3.5.5 >> - infinispan 5.1.0 BETA5. >> - I have the 2 nodes set up for invalidation on all caches except for >> the timestamp cache which is in replication mode (for hibernate). >> >> So, as discussed on the other thread, when autoCommit is set to false >> in a transactional cache environment, an exception occurs since in a >> few cases, the infinispan system suspends the ongoing transaction. In >> this case, when it comes to effectively putting data (whether it is >> "forExternalRead" or not, I will not get into that here) it show a >> valid exception stating that a transaction is needed for a >> transactional cache. >> >> So here is what I had to do to move forward on that. Since I have a >> smaller use case that what infinispan can handle, I could safely >> remove the code that suspends the transaction where needed to test it >> out. >> >> Here is what I did: >> >> 1. In CacheImpl, I removed the logic in "final void >> putForExternalRead(K key, V value, EnumSet<Flag> explicitFlags, >> ClassLoader explicitClassLoader)" >> on lines 355, 357-359, 376-385 (removed all code linked with the >> suspension that I did not want given that I wanted to ensure the >> autoCommit to false). >> >> That worked and got a HUGE performance improvement, factor of 5 in the >> execution/replication/invalidation time (as in literally 5 times >> faster). >> >> 2. Then I realized that the BaseRegion class also had some logic that >> I was running into that was suspending transactions. I changed the >> methods "suspend" and "resume(Transaction tx)" to do nothing (suspends >> returns systematically null and resume expect always null as an input >> with my changes). >> >> This change was what brought me to realize that there was a MAJOR >> issue with hibernate (at least in 3.5.5) when doing that. >> >> The timestamp cache/region has a special behaviour in "ActionQueue". >> If you look at "AfterTransactionCompletionProcessQueue" the method >> "public void afterTransactionCompletion(boolean success)" once all is >> done, checks if there is a cache and pushes a set of invalidations. >> The main issue here is that: >> 1. my transaction was already successfully committed by hibernate and >> these operations are done outside of the scope of the transaction. >> 2. consequence of #1: I had set autoCommit to false, so no >> transactions were auto started for me, so the invalidations failed >> systematically (I think it was correct for the invalidations to fail). >> >> (The changes to "fix" that are enclosed to the email, they are simple). >> >> Thus, this leads me to say that a pure transactional cache with >> autoCommit set to false cannot work at this point without the changes >> I made to my own. I correctly works for me and after other changes I >> got it to perform really well. >> >> On that note, since we are talking performance, I explored the >> batching functionality of infinispan (which is great!!). It does not >> quite perform as is for all regions/cache. Indeed, if you run into the >> issue I describe above in the timestamp cache/region, and your >> autoCommit is set to true, no batching happens (you are indeed outside >> of the transaction which started the batch) and the >> invalidation/replication is super expensive: 2 commands/messages per >> invalidation (one for prepare and one for commit). Implementing the >> changes I described allowed the batching to work for my timestamp >> region and that is when I got an additional 2x factor and everything >> is lightening fast. When I say 2x factor, I mean a total of 10x faster >> with all the changes I put in versus, the 5.1.0 BETA5 out of the box >> (from 1 sec to 100ms per http transaction on my API !!). This is for >> asynchronous invalidation, it is slower with synchronous. >> >> While I am at it, I was wondering if it would be logical to have >> infinispan automatically start batches in a transactional context. It >> made sense to me and I have activated in my own code. >> >> Please find enclosed the files changed. >> >> Let me know if anything I said does not make sense or is wrong. I >> tested it and so far it works well but again my use case is a small >> subset of what infinispan can do. >> >> Regards, >> >> L.P. >> >> PS: also, org/hibernate/cache/infinispan/util/AddressAdapterImpl.java >> and org/hibernate/cache/infinispan/util/CacheHelper.java need public >> no-arg constructors for the de serialization to work, they did not >> have them in hibernate 3.5.5. >> >> On Tue, Nov 29, 2011 at 10:15, Mircea Markus <mircea.markus(a)jboss.com&gt; wrote: >>> >>> On 29 Nov 2011, at 14:03, Slorg1 wrote: >>> >>> Hi, >>> >>> On Tue, Nov 29, 2011 at 09:00, Galder Zamarreño <galder(a)redhat.com&gt; wrote: >>> >>> >>> On Nov 28, 2011, at 5:00 PM, Mircea Markus wrote: >>> >>> >>> Hi, >>> >>> >>> Now that all the 5.1 major work is done I plan to run some benchmarks >>> comparing 5.1 with 5.0. It's not only tx stuff I'd like to compare, as some >>> other were made, so here's my plan of action. >>> >>> >>> Each of the following benchmarks will be run on local, distributed and >>> replicated caches: >>> >>> >>> 1. non transactional: web session replication[1] >>> >>> 2. transactional (both optimistic and pessimistic): web session >>> replication[2] >>> >>> 3. transactional (both optimistic and pessimistic): tpcc [3] >>> >>> >>> The difference between 2 and 3 is the fact that 3 induces some key >>> contention between transactions. >>> >>> >>> Any other suggestion for benchmarking? >>> >>> >>> Could it be interesting to see what the autoCommit penalty is? i.e. >>> comparing: non-transactional cache vs transactional cache with autoCommit >>> >>> >>> >>> Pure Transactional and autoCommit 'off' does not work in BETA 5. I am >>> compiling a list of changes I made to make it happen. >>> >>> >>> To save you the trouble of going through what I did and give you a >>> solution that 'works'. >>> >>> What do you mean by do not work? >>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> >> >> -- >> Please consider the environment - do you really need to print this email ? > > > > -- > Please consider the environment - do you really need to print this email ? > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

Hi, O.K. so here is what I found out (this is linked with the thread titled "Re: [infinispan-dev] PutForExternalRead and autoCommit", a copy of this message will be sent on it, though it does not close the discussion on it). First, let me remind everyone, that I am using: - Atomikos as a transaction manager - Hibernate 3.5.5 - infinispan 5.1.0 BETA5. - I have the 2 nodes set up for invalidation on all caches except for the timestamp cache which is in replication mode (for hibernate). So, as discussed on the other thread, when autoCommit is set to false in a transactional cache environment, an exception occurs since in a few cases, the infinispan system suspends the ongoing transaction. In this case, when it comes to effectively putting data (whether it is "forExternalRead" or not, I will not get into that here) it show a valid exception stating that a transaction is needed for a transactional cache. So here is what I had to do to move forward on that. Since I have a smaller use case that what infinispan can handle, I could safely remove the code that suspends the transaction where needed to test it out. Here is what I did: 1. In CacheImpl, I removed the logic in "final void putForExternalRead(K key, V value, EnumSet<Flag> explicitFlags, ClassLoader explicitClassLoader)" on lines 355, 357-359, 376-385 (removed all code linked with the suspension that I did not want given that I wanted to ensure the autoCommit to false).

That worked and got a HUGE performance improvement, factor of 5 in the execution/replication/invalidation time (as in literally 5 times faster).

2. Then I realized that the BaseRegion class also had some logic that I was running into that was suspending transactions. I changed the methods "suspend" and "resume(Transaction tx)" to do nothing (suspends returns systematically null and resume expect always null as an input with my changes). This change was what brought me to realize that there was a MAJOR issue with hibernate (at least in 3.5.5) when doing that. The timestamp cache/region has a special behaviour in "ActionQueue". If you look at "AfterTransactionCompletionProcessQueue" the method "public void afterTransactionCompletion(boolean success)" once all is done, checks if there is a cache and pushes a set of invalidations. The main issue here is that: 1. my transaction was already successfully committed by hibernate and these operations are done outside of the scope of the transaction. 2. consequence of #1: I had set autoCommit to false, so no transactions were auto started for me, so the invalidations failed systematically (I think it was correct for the invalidations to fail). (The changes to "fix" that are enclosed to the email, they are simple).

Thus, this leads me to say that a pure transactional cache with autoCommit set to false cannot work at this point without the changes I made to my own. I correctly works for me and after other changes I got it to perform really well. On that note, since we are talking performance, I explored the batching functionality of infinispan (which is great!!). It does not quite perform as is for all regions/cache. Indeed, if you run into the issue I describe above in the timestamp cache/region, and your autoCommit is set to true, no batching happens (you are indeed outside of the transaction which started the batch) and the invalidation/replication is super expensive: 2 commands/messages per invalidation (one for prepare and one for commit). Implementing the changes I described allowed the batching to work for my timestamp region and that is when I got an additional 2x factor and everything is lightening fast. When I say 2x factor, I mean a total of 10x faster with all the changes I put in versus, the 5.1.0 BETA5 out of the box (from 1 sec to 100ms per http transaction on my API !!). This is for asynchronous invalidation, it is slower with synchronous. While I am at it, I was wondering if it would be logical to have infinispan automatically start batches in a transactional context. It made sense to me and I have activated in my own code.

Please find enclosed the files changed. Let me know if anything I said does not make sense or is wrong. I tested it and so far it works well but again my use case is a small subset of what infinispan can do. Regards, L.P. PS: also, org/hibernate/cache/infinispan/util/AddressAdapterImpl.java and org/hibernate/cache/infinispan/util/CacheHelper.java need public no-arg constructors for the de serialization to work, they did not have them in hibernate 3.5.5.

On Tue, Nov 29, 2011 at 10:15, Mircea Markus <mircea.markus(a)jboss.com&gt; wrote: > > On 29 Nov 2011, at 14:03, Slorg1 wrote: > > Hi, > > On Tue, Nov 29, 2011 at 09:00, Galder Zamarreño <galder(a)redhat.com&gt; wrote: > > > On Nov 28, 2011, at 5:00 PM, Mircea Markus wrote: > > > Hi, > > > Now that all the 5.1 major work is done I plan to run some benchmarks > comparing 5.1 with 5.0. It's not only tx stuff I'd like to compare, as some > other were made, so here's my plan of action. > > > Each of the following benchmarks will be run on local, distributed and > replicated caches: > > > 1. non transactional: web session replication[1] > > 2. transactional (both optimistic and pessimistic): web session > replication[2] > > 3. transactional (both optimistic and pessimistic): tpcc [3] > > > The difference between 2 and 3 is the fact that 3 induces some key > contention between transactions. > > > Any other suggestion for benchmarking? > > > Could it be interesting to see what the autoCommit penalty is? i.e. > comparing: non-transactional cache vs transactional cache with autoCommit > > > > Pure Transactional and autoCommit 'off' does not work in BETA 5. I am > compiling a list of changes I made to make it happen. > > > To save you the trouble of going through what I did and give you a > solution that 'works'. > > What do you mean by do not work? > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Please consider the environment - do you really need to print this email ? <changedFiles.tar.bz2>_______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev