[keycloak-dev] ConcurrencyTest failures

[Stian Thorgersen]

If we want to have stable CI and run CI as part of PRs the tests have to be
stable. So we just have to put the effort into stabilizing tests.

On 3 October 2017 at 12:22, Marko Strukelj <mstrukel at redhat.com> wrote:

> Created a JIRA: https://issues.jboss.org/browse/KEYCLOAK-5617
>
> It's not as black and white as you put it. My assumption is that this test
> will not get fixed for a forseable future because concurrency issues are
> very hard to debug and fix. During that time everyone to whom this issue
> manifests will stumble upon it multiple times unless disabling the test
> locally manually, and taking extra care to not commit those changes.
>
>
> On Tue, Oct 3, 2017 at 10:43 AM, Stian Thorgersen <sthorger at redhat.com>
> wrote:
>
>> Or, why not just ignore the fact that the test is failing?
>>
>> On 3 October 2017 at 10:40, Marko Strukelj <mstrukel at redhat.com> wrote:
>>
>>> But now that we know it's failing, and I need to run four full builds
>>> with slightly different options, that will take twice as long unless I
>>> disable this test that is unrelated to my PR :)
>>>
>>>
>>> On Oct 3, 2017 06:22, "Stian Thorgersen" <sthorger at redhat.com> wrote:
>>>
>>>> -1000 The tests needs fixing not ignoring. Can you create a JIRA for
>>>> the test that is unstable please?
>>>>
>>>> On 2 October 2017 at 18:32, Marko Strukelj <mstrukel at redhat.com> wrote:
>>>>
>>>>> I've been getting intermittent ConcurrencyTest failures.
>>>>>
>>>>> Tests in error:
>>>>>
>>>>>   ConcurrencyTest.testAllConcurrently:57->concurrentTest:49->A
>>>>> bstractConcurrencyTest.run:53->AbstractConcurrencyTest.run:96
>>>>> » Runtime
>>>>>
>>>>>
>>>>>
>>>>> I'm unable to reliably replicate it but it never happens when running
>>>>> ConcurrencyTest alone (i.e. -Dtest=ConcurrencyTest) but always as part
>>>>> of
>>>>> full testsuite 30 mins into the run.
>>>>>
>>>>> I propose to add:
>>>>>
>>>>> static boolean runIntermittentlyFailingTests() {
>>>>>     return "true".equals(System.getProperty("test.intermittent"));
>>>>> }
>>>>>
>>>>>
>>>>> in AbstractKeycloakTest.java and check at the beginning of
>>>>> ConcurrencyTest.java#testAllConcurrently():
>>>>>
>>>>> if (!runIntermittentlyFailingTests()) {
>>>>>     System.out.println("TEST SKIPPED - This test currently suffers
>>>>> from intermittent failures. Use -Dtest.intermittent=true to run it.");
>>>>>     return;
>>>>> }
>>>>> _______________________________________________
>>>>> keycloak-dev mailing list
>>>>> keycloak-dev at lists.jboss.org
>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>
>>>>
>>>>
>>
>