[keycloak-dev] local build failing, travis passing

Stian Thorgersen sthorger at redhat.com
Tue Dec 13 02:55:51 EST 2016 

On 13 December 2016 at 08:50, Marek Posolda <mposolda at redhat.com> wrote:

> On 13/12/16 08:26, Stian Thorgersen wrote:
>
>
>
> On 12 December 2016 at 21:44, Marek Posolda <mposolda at redhat.com> wrote:
>
>> On 12/12/16 17:51, Stian Thorgersen wrote:
>>
>> It should run at least what it used to run before I broke it up into
>> multiple parallell jobs (seems I did a poor job when I did that and messed
>> some stuff up). That can be fixed straight away, but adding more tests (new
>> adapter tests, clustering tests, console tests, etc.) we should probably
>> review later as it may impact stability and time to test a PR on Travis.
>>
>> +1 for better review. Few things:
>>
>> * Unit tests in the modules outside testsuite are not executed during
>> travis build. For example tests under core/src/test/java .
>>
>> * Everything in testsuite/integration and testsuite/integration-arquillian/tests/base
>> is running in Travis build ATM. However if we add new package with tests
>> into the testsuite (For example org.keycloak.testsuite.zzz.NewTest), it
>> is possible that it won't be running due to missing filter in the
>> travis-run-tests.sh for the package starting with "z" . I hope we can have
>> the list of packages by travis-run-tests.sh to be more dynamic.
>>
>> * IMO it is sufficient that travis will run all the tests (including
>> adapters and cluster) on embedded undertow. Running the tests on Wildfly is
>> unecessary overhead, which will slightly increase the travis build time (as
>> it will need to build distribution, start server etc) . There is not much
>> added value as 95 % of regressions are catched with undertow too.
>>
>
> -1 We've seen this many times where Travis is happy, but the distribution
> is broken. This happened several times last year so we need to prevent this.
>
> Isn't it better to have travis run just on "simple" environment for each
> PR and then rely on Central CI daily jobs to run the tests on real servers?
> Even if we setup travis to run adapter tests on Wildfly, it won't catch all
> the other broken adapters (EAP6, Tomcat, Jetty, ...).
>
> IMO it is generally better to have CI for each PR, which will be able to
> catch 95% of regressions (not necessarily 100%) and doesn't take hours to
> wait for every PR. And then separate daily jobs to test everything.
>

Yes, we can't run it all on Travis, but I don't think making it run with WF
is going to add that much overhead (I think we're talking a few min). If
it's a big overhead then we should keep it on Undertow for sure.


>
>
> Marek
>
>
>
>>
>> * I've added some undertow adapter tests to be executed during default
>> travis build. They are all in the package org.keycloak.testsuite.adapter
>> .undertow.servlet . Problem is, that not all adapter tests are executed
>> as it will require creating subclass for every "Abstract" class with test
>> and I just didn't create all the classes for embedded undertow. I've
>> created JIRA https://issues.jboss.org/browse/KEYCLOAK-4069 to omprove
>> adapter tests to avoid "dead" modules and many classes needed for single
>> adapter test.
>>
>> Marek
>>
>> On 12 December 2016 at 17:46, Bill Burke <bburke at redhat.com> wrote:
>>
>>> This wasn't just the adapter tests, initially it was the
>>> PartialImportsTest which lives under org.keycloak.testsuite.admin somewhere
>>> and should have run and should have failed.
>>>
>>> https://issues.jboss.org/browse/KEYCLOAK-4068
>>>
>>>
>>> On 12/12/16 8:24 AM, Stian Thorgersen wrote:
>>>
>>> Let's review the tests ran by Travis after the new year, but with Travis
>>> having the option to run groups of tests in parallel we should be able to
>>> tests more including adapters and console. Travis should also be changed to
>>> tests with the full KC server and WildFly for adapters rather than embedded
>>> Undertow.
>>>
>>> On 5 December 2016 at 16:32, Bill Burke <bburke at redhat.com> wrote:
>>>
>>>> These broke because they weren't part of the main build.  I thought they
>>>> were just dead code because when I did a "Find Usages" for them, nothing
>>>> came up.  Minimally, things should at least be compiled with the main
>>>> build, that way when refactorings happen, somebody doesn't delete a test
>>>> dependency by accident.
>>>>
>>>>
>>>> On 12/5/16 6:53 AM, Hynek Mlnarik wrote:
>>>> > Speaking of the tests, we seem not to run any of the adapter test
>>>> > suites. I believe that running at least adapter tests for wildfly
>>>> > would be beneficial, preventing e.g. [1]. WDYT?
>>>> >
>>>> > [1] https://issues.jboss.org/browse/KEYCLOAK-4017
>>>> >
>>>> > On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda <mposolda at redhat.com
>>>> > <mailto:mposolda at redhat.com>> wrote:
>>>> >
>>>> >     Fyi. On Friday afternoon, I've found the issue that travis didn't
>>>> run
>>>> >     all the tests from the testsuite. And PartialImportTest was the
>>>> one,
>>>> >     which wasn't executed. See
>>>> >     https://issues.jboss.org/browse/KEYCLOAK-4021
>>>> >     <https://issues.jboss.org/browse/KEYCLOAK-4021>
>>>> >
>>>> >     However with the travis fix, all the tests were passing (including
>>>> >     PartialImportTest). So it seems this test was just failing
>>>> >     randomly (not
>>>> >     always)?
>>>> >
>>>> >     Now I can see in latest travis build that PartialImportTest
>>>> passes.
>>>> >
>>>> >     Marek
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >     On 03/12/16 17:36, Bill Burke wrote:
>>>> >     > I just noticed that my local build fails while travis passes.
>>>> >     The bug is
>>>> >     > really something travis should have picked up, specifically the
>>>> >     > PartialImportsTest was removing an identity provider.  The JPA
>>>> >     > removeIdenittyProviderByAlias method was wrong as it was trying
>>>> >     to load
>>>> >     > an IdentityProviderModel after it was removed thus resulting in
>>>> a
>>>> >     > Hibernate error.  Travis did not pick this up which makes me
>>>> >     wonder if
>>>> >     > the test is even running.
>>>> >     >
>>>> >     > FYI, i have a pull request that fixes this that is incoming. The
>>>> >     bug,
>>>> >     > not travis.
>>>> >     >
>>>> >     > Bill
>>>> >     >
>>>> >     >
>>>> >     > _______________________________________________
>>>> >     > keycloak-dev mailing list
>>>> >     > keycloak-dev at lists.jboss.org <mailto:keycloak-dev at lists.jbo
>>>> ss.org>
>>>> >     > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>> >     <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>>>> >
>>>> >
>>>> >     _______________________________________________
>>>> >     keycloak-dev mailing list
>>>> >     keycloak-dev at lists.jboss.org <mailto:keycloak-dev at lists.jboss.org
>>>> >
>>>> >     https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>> >     <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> >
>>>> > --Hynek
>>>>
>>>> _______________________________________________
>>>> keycloak-dev mailing list
>>>> keycloak-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>
>>>
>>>
>>>
>>
>>
>
>

More information about the keycloak-dev mailing list