I've been getting intermittent ConcurrencyTest failures. Tests in error: ConcurrencyTest.testAllConcurrently:57->concurrentTest:49-> AbstractConcurrencyTest.run:53->AbstractConcurrencyTest.run:96 » Runtime I'm unable to reliably replicate it but it never happens when running ConcurrencyTest alone (i.e. -Dtest=ConcurrencyTest) but always as part of full testsuite 30 mins into the run. I propose to add: static boolean runIntermittentlyFailingTests() { return "true".equals(System.getProperty("test.intermittent")); } in AbstractKeycloakTest.java and check at the beginning of ConcurrencyTest.java#testAllConcurrently(): if (!runIntermittentlyFailingTests()) { System.out.println("TEST SKIPPED - This test currently suffers from intermittent failures. Use -Dtest.intermittent=true to run it."); return; } _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev

But now that we know it's failing, and I need to run four full builds with slightly different options, that will take twice as long unless I disable this test that is unrelated to my PR :) On Oct 3, 2017 06:22, "Stian Thorgersen" <sthorger(a)redhat.com&gt; wrote: > -1000 The tests needs fixing not ignoring. Can you create a JIRA for the > test that is unstable please? > > On 2 October 2017 at 18:32, Marko Strukelj <mstrukel(a)redhat.com&gt; wrote: > >> I've been getting intermittent ConcurrencyTest failures. >> >> Tests in error: >> >> ConcurrencyTest.testAllConcurrently:57->concurrentTest:49->A >> bstractConcurrencyTest.run:53->AbstractConcurrencyTest.run:96 >> » Runtime >> >> >> >> I'm unable to reliably replicate it but it never happens when running >> ConcurrencyTest alone (i.e. -Dtest=ConcurrencyTest) but always as part of >> full testsuite 30 mins into the run. >> >> I propose to add: >> >> static boolean runIntermittentlyFailingTests() { >> return "true".equals(System.getProperty("test.intermittent")); >> } >> >> >> in AbstractKeycloakTest.java and check at the beginning of >> ConcurrencyTest.java#testAllConcurrently(): >> >> if (!runIntermittentlyFailingTests()) { >> System.out.println("TEST SKIPPED - This test currently suffers >> from intermittent failures. Use -Dtest.intermittent=true to run it."); >> return; >> } >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > > >

Created a JIRA: https://issues.jboss.org/browse/KEYCLOAK-5617 It's not as black and white as you put it. My assumption is that this test will not get fixed for a forseable future because concurrency issues are very hard to debug and fix. During that time everyone to whom this issue manifests will stumble upon it multiple times unless disabling the test locally manually, and taking extra care to not commit those changes. On Tue, Oct 3, 2017 at 10:43 AM, Stian Thorgersen <sthorger(a)redhat.com&gt; wrote: > Or, why not just ignore the fact that the test is failing? > > On 3 October 2017 at 10:40, Marko Strukelj <mstrukel(a)redhat.com&gt; wrote: > >> But now that we know it's failing, and I need to run four full builds >> with slightly different options, that will take twice as long unless I >> disable this test that is unrelated to my PR :) >> >> >> On Oct 3, 2017 06:22, "Stian Thorgersen" <sthorger(a)redhat.com&gt; wrote: >> >>> -1000 The tests needs fixing not ignoring. Can you create a JIRA for >>> the test that is unstable please? >>> >>> On 2 October 2017 at 18:32, Marko Strukelj <mstrukel(a)redhat.com&gt; wrote: >>> >>>> I've been getting intermittent ConcurrencyTest failures. >>>> >>>> Tests in error: >>>> >>>> ConcurrencyTest.testAllConcurrently:57->concurrentTest:49->A >>>> bstractConcurrencyTest.run:53->AbstractConcurrencyTest.run:96 >>>> » Runtime >>>> >>>> >>>> >>>> I'm unable to reliably replicate it but it never happens when running >>>> ConcurrencyTest alone (i.e. -Dtest=ConcurrencyTest) but always as part >>>> of >>>> full testsuite 30 mins into the run. >>>> >>>> I propose to add: >>>> >>>> static boolean runIntermittentlyFailingTests() { >>>> return "true".equals(System.getProperty("test.intermittent")); >>>> } >>>> >>>> >>>> in AbstractKeycloakTest.java and check at the beginning of >>>> ConcurrencyTest.java#testAllConcurrently(): >>>> >>>> if (!runIntermittentlyFailingTests()) { >>>> System.out.println("TEST SKIPPED - This test currently suffers >>>> from intermittent failures. Use -Dtest.intermittent=true to run it."); >>>> return; >>>> } >>>> _______________________________________________ >>>> keycloak-dev mailing list >>>> keycloak-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>> >>> >>> >