The main problem remains unresolved: creating cache managers, using them, and not cleaning up. Have a look at ChannelLookupTest. Are you familiar with this one :)? On 09-12-24 3:18 PM, Manik Surtani wrote: > Guys, > > as far as possible, please extend either SingleCacheManagerTest [1] or MultipleCacheManagersTest [2]. and use the corresponding helper methods there to create a new CacheManager.  Alternatively, if you have a unit test that needs to create its own CacheManager directly, please use TestCacheManagerFactory [3].  Do *not* use "new DefaultCacheManager()" directly since this means the framework does not have a chance to alter config settings to work within the framework, including clustering settings to make sure the test does not interfere with other tests, threadpool settings to prevent unnecessary OOMs when running the entire suite with thousands of cache managers, etc. > > I have just fixed a bunch of offending tests [4], and updated the wiki page [5] on writing tests accordingly.  Please follow these guidelines. > > Cheers&  Happy X'mas! > Manik > > [1] http://fisheye.jboss.org/browse/Infinispan/trunk/core/src/test/java/org/i... > [2] http://fisheye.jboss.org/browse/Infinispan/trunk/core/src/test/java/org/i... > [3] http://fisheye.jboss.org/browse/Infinispan/trunk/core/src/test/java/org/i... > [4] http://fisheye.jboss.org/changelog/Infinispan/trunk?cs=1327 > [5] http://community.jboss.org/wiki/ParallelTestSuite > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

I've found that still some tests are not properly killing the CacheManager, making other random tests fail, so I wrote an extension to Infinispan's TestNG listener to identify the guilty implementations. I opened ISPN-323 to track this. To identify a non-cleaning test the testNG listener will create a new CacheManager and verify the number of members, so this is slowing down tests; to not annoy you it's only enabled in a new profile "debug-tests": mvn -Pdebug-tests clean test Also keep in mind that, being a listener, it can't make the test fail: to find the tests you'll have to grep the build output for "CacheManagers alive after test!". We still need to fix the identified tests, I'll create subtasks of ISPN-323 for that. Cheers, Sanne

Hi Vladimir, thanks for the tips, please have a look into what I've already done, suggestions on improving are welcome: http://fisheye.jboss.org/changelog/Infinispan?cs=1335 The limitation is that it can't make tests fail, it can only print warnings to the logs. I didn't use @afterclass because it relies on the test itself to verify it's correctly written: it doesn't solve the problem of spotting an incorrectly written test. (I could as well open them all to have to verify they all extend the same parent test...) I believe we can run the manual test when we spot "random behavior" that make you think some test is misbehaving, and keep it disabled other times. As discussed on a testng forum thread (*1) it doesn't appear to be possible to change the test result from a listener, I tried even with a RuntimeException but it's ignoring them. Anyway the results of my effort uncovered just one misbehaving test in the query module; the tests currently failing in core are failing for other reasons. Hudson (the public one I can see *2) is stuck since 22 December, so I can't really see how stable the current testing results are. Could someone start it? Cheers, Sanne *1 http://groups.google.com/group/testng-users/browse_thread/thread/c07dee91... *2 http://hudson.jboss.org/hudson/view/Infinispan/job/Infinispan-trunk-JDK6-... 2010/1/2 Vladimir Blagojevic<vblagoje(a)redhat.com>: > Sanne, we can use probe utility from JGroups to ping if there are any > remaining jgroups members alive ( and therefore CacheManagers). We can > do this verification from @afterClass in superclass of all test classes. > We just have to somehow record which ports were used by particular > TestNG running thread. I will look into this problem as well. We have to > do verification somehow automatically, anything else is a futile > nightmare.... > > On 09-12-29 8:43 PM, Sanne Grinovero wrote: > >> I've found that still some tests are not properly killing the >> CacheManager, making other random tests fail, >> so I wrote an extension to Infinispan's TestNG listener to identify >> the guilty implementations. >> >> I opened ISPN-323 to track this. >> >> To identify a non-cleaning test the testNG listener will create a new >> CacheManager and verify the number of members, >> so this is slowing down tests; to not annoy you it's only enabled in a >> new profile "debug-tests": >> >> mvn -Pdebug-tests clean test >> >> Also keep in mind that, being a listener, it can't make the test fail: >> to find the tests you'll have to grep the build output >> for "CacheManagers alive after test!". >> >> We still need to fix the identified tests, I'll create subtasks of >> ISPN-323 for that. >> >> Cheers, >> Sanne >> >> >> > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

below: 2010/1/4 Vladimir Blagojevic <vblagoje(a)redhat.com&gt;: > Sanne, > > I've looked at your listener approach. I think this is the best we can > do for now but is there a potential problem with listener approach > because we sometimes create cache managers on per test method and > sometimes on per test class basis? Can listener handle cases where each > test method creates a few cache managers and does not close them properly? no the test still needs to be written using the methods explained by Manik in first post of this thread, this listener is only useful to detect if some test is forgetting to cleanup (kill the cachemanager), which was happening and is quite annoying to find. If the test is going to create a cachemanager on the wrong port (not the one dedicated to running thread), it won't detect it :-(

Cheers, Sanne > > Cheers > > On 10-01-02 11:47 PM, Sanne Grinovero wrote: >> Hi Vladimir, >> thanks for the tips, please have a look into what I've already done, >> suggestions on improving are welcome: >> http://fisheye.jboss.org/changelog/Infinispan?cs=1335 >> >> The limitation is that it can't make tests fail, it can only print >> warnings to the logs. >> I didn't use @afterclass because it relies on the test itself to >> verify it's correctly written: it doesn't solve the problem of >> spotting an incorrectly written test. >> (I could as well open them all to have to verify they all extend the >> same parent test...) >> >> I believe we can run the manual test when we spot "random behavior" >> that make you think some test is misbehaving, and keep it disabled >> other times. >> >> As discussed on a testng forum thread (*1) it doesn't appear to be >> possible to change the test result from a listener, I tried even with >> a RuntimeException but it's ignoring them. >> >> Anyway the results of my effort uncovered just one misbehaving test in >> the query module; the tests currently failing in core are failing for >> other reasons. >> Hudson (the public one I can see *2) is stuck since 22 December, so I >> can't really see how stable the current testing results are. Could >> someone start it? >> >> Cheers, >> Sanne >> >> *1 http://groups.google.com/group/testng-users/browse_thread/thread/c07dee91... >> *2 http://hudson.jboss.org/hudson/view/Infinispan/job/Infinispan-trunk-JDK6-... >> >> >> >> 2010/1/2 Vladimir Blagojevic&lt;vblagoje(a)redhat.com&gt;: >> >>> Sanne, we can use probe utility from JGroups to ping if there are any >>> remaining jgroups members alive ( and therefore CacheManagers). We can >>> do this verification from @afterClass in superclass of all test classes. >>> We just have to somehow record which ports were used by particular >>> TestNG running thread. I will look into this problem as well. We have to >>> do verification somehow automatically, anything else is a futile >>> nightmare.... >>> >>> On 09-12-29 8:43 PM, Sanne Grinovero wrote: >>> >>>> I've found that still some tests are not properly killing the >>>> CacheManager, making other random tests fail, >>>> so I wrote an extension to Infinispan's TestNG listener to identify >>>> the guilty implementations. >>>> >>>> I opened ISPN-323 to track this. >>>> >>>> To identify a non-cleaning test the testNG listener will create a new >>>> CacheManager and verify the number of members, >>>> so this is slowing down tests; to not annoy you it's only enabled in a >>>> new profile "debug-tests": >>>> >>>> mvn -Pdebug-tests clean test >>>> >>>> Also keep in mind that, being a listener, it can't make the test fail: >>>> to find the tests you'll have to grep the build output >>>> for "CacheManagers alive after test!". >>>> >>>> We still need to fix the identified tests, I'll create subtasks of >>>> ISPN-323 for that. >>>> >>>> Cheers, >>>> Sanne >>>> >>>> >>>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>> >>> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

The main problem remains unresolved: creating cache managers, using them, and not cleaning up.

Have a look at ChannelLookupTest. Are you familiar with this one :)? On 09-12-24 3:18 PM, Manik Surtani wrote: > Guys, > > as far as possible, please extend either SingleCacheManagerTest [1] or MultipleCacheManagersTest [2]. and use the corresponding helper methods there to create a new CacheManager. Alternatively, if you have a unit test that needs to create its own CacheManager directly, please use TestCacheManagerFactory [3]. Do *not* use "new DefaultCacheManager()" directly since this means the framework does not have a chance to alter config settings to work within the framework, including clustering settings to make sure the test does not interfere with other tests, threadpool settings to prevent unnecessary OOMs when running the entire suite with thousands of cache managers, etc. > > I have just fixed a bunch of offending tests [4], and updated the wiki page [5] on writing tests accordingly. Please follow these guidelines. > > Cheers& Happy X'mas! > Manik > > [1] http://fisheye.jboss.org/browse/Infinispan/trunk/core/src/test/java/org/i... > [2] http://fisheye.jboss.org/browse/Infinispan/trunk/core/src/test/java/org/i... > [3] http://fisheye.jboss.org/browse/Infinispan/trunk/core/src/test/java/org/i... > [4] http://fisheye.jboss.org/changelog/Infinispan/trunk?cs=1327 > [5] http://community.jboss.org/wiki/ParallelTestSuite > > > -- > Manik Surtani > manik(a)jboss.org > Lead, Infinispan > Lead, JBoss Cache > http://www.infinispan.org > http://www.jbosscache.org > > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev > _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev

On 2010-01-04, at 1:02 PM, Mircea Markus wrote: > > As long as the CM are created through TestCacheManagersFactory, this should will soon not be an issue: https://jira.jboss.org/jira/browse/ISPN-314 It still is an issue when tests do not kill cache managers, is it not!

However, we can still leverage the fact that we use TestCacheManagersFactory throughout. We can track all DefaultCacheManager(s) we create from TestCacheManagersFactory. We can use hashmap keyed by test class name

and a value which is a list of all created DefaultCacheManager (in all test methods of a test class). Since lifecycle of DefaultCacheManager never spans across test classes we can find these cache manager and reclaim them in AbstractInfinispanTest @afterClass method. _______________________________________________ infinispan-dev mailing list infinispan-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev