[keycloak-dev] local build failing, travis passing
Marek Posolda
mposolda at redhat.com
Tue Dec 13 02:50:13 EST 2016
On 13/12/16 08:26, Stian Thorgersen wrote:
>
>
> On 12 December 2016 at 21:44, Marek Posolda <mposolda at redhat.com
> <mailto:mposolda at redhat.com>> wrote:
>
> On 12/12/16 17:51, Stian Thorgersen wrote:
>> It should run at least what it used to run before I broke it up
>> into multiple parallell jobs (seems I did a poor job when I did
>> that and messed some stuff up). That can be fixed straight away,
>> but adding more tests (new adapter tests, clustering tests,
>> console tests, etc.) we should probably review later as it may
>> impact stability and time to test a PR on Travis.
> +1 for better review. Few things:
>
> * Unit tests in the modules outside testsuite are not executed
> during travis build. For example tests under core/src/test/java .
>
> * Everything in testsuite/integration and
> testsuite/integration-arquillian/tests/base is running in Travis
> build ATM. However if we add new package with tests into the
> testsuite (For example org.keycloak.testsuite.zzz.NewTest), it is
> possible that it won't be running due to missing filter in the
> travis-run-tests.sh for the package starting with "z" . I hope we
> can have the list of packages by travis-run-tests.sh to be more
> dynamic.
>
> * IMO it is sufficient that travis will run all the tests
> (including adapters and cluster) on embedded undertow. Running the
> tests on Wildfly is unecessary overhead, which will slightly
> increase the travis build time (as it will need to build
> distribution, start server etc) . There is not much added value as
> 95 % of regressions are catched with undertow too.
>
>
> -1 We've seen this many times where Travis is happy, but the
> distribution is broken. This happened several times last year so we
> need to prevent this.
Isn't it better to have travis run just on "simple" environment for each
PR and then rely on Central CI daily jobs to run the tests on real
servers? Even if we setup travis to run adapter tests on Wildfly, it
won't catch all the other broken adapters (EAP6, Tomcat, Jetty, ...).
IMO it is generally better to have CI for each PR, which will be able to
catch 95% of regressions (not necessarily 100%) and doesn't take hours
to wait for every PR. And then separate daily jobs to test everything.
Marek
>
>
> * I've added some undertow adapter tests to be executed during
> default travis build. They are all in the package
> org.keycloak.testsuite.adapter.undertow.servlet . Problem is, that
> not all adapter tests are executed as it will require creating
> subclass for every "Abstract" class with test and I just didn't
> create all the classes for embedded undertow. I've created JIRA
> https://issues.jboss.org/browse/KEYCLOAK-4069
> <https://issues.jboss.org/browse/KEYCLOAK-4069> to omprove adapter
> tests to avoid "dead" modules and many classes needed for single
> adapter test.
>
> Marek
>> On 12 December 2016 at 17:46, Bill Burke <bburke at redhat.com
>> <mailto:bburke at redhat.com>> wrote:
>>
>> This wasn't just the adapter tests, initially it was the
>> PartialImportsTest which lives under
>> org.keycloak.testsuite.admin somewhere and should have run
>> and should have failed.
>>
>> https://issues.jboss.org/browse/KEYCLOAK-4068
>> <https://issues.jboss.org/browse/KEYCLOAK-4068>
>>
>>
>> On 12/12/16 8:24 AM, Stian Thorgersen wrote:
>>> Let's review the tests ran by Travis after the new year, but
>>> with Travis having the option to run groups of tests in
>>> parallel we should be able to tests more including adapters
>>> and console. Travis should also be changed to tests with the
>>> full KC server and WildFly for adapters rather than embedded
>>> Undertow.
>>>
>>> On 5 December 2016 at 16:32, Bill Burke <bburke at redhat.com
>>> <mailto:bburke at redhat.com>> wrote:
>>>
>>> These broke because they weren't part of the main
>>> build. I thought they
>>> were just dead code because when I did a "Find Usages"
>>> for them, nothing
>>> came up. Minimally, things should at least be compiled
>>> with the main
>>> build, that way when refactorings happen, somebody
>>> doesn't delete a test
>>> dependency by accident.
>>>
>>>
>>> On 12/5/16 6:53 AM, Hynek Mlnarik wrote:
>>> > Speaking of the tests, we seem not to run any of the
>>> adapter test
>>> > suites. I believe that running at least adapter tests
>>> for wildfly
>>> > would be beneficial, preventing e.g. [1]. WDYT?
>>> >
>>> > [1] https://issues.jboss.org/browse/KEYCLOAK-4017
>>> <https://issues.jboss.org/browse/KEYCLOAK-4017>
>>> >
>>> > On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda
>>> <mposolda at redhat.com <mailto:mposolda at redhat.com>
>>> > <mailto:mposolda at redhat.com
>>> <mailto:mposolda at redhat.com>>> wrote:
>>> >
>>> > Fyi. On Friday afternoon, I've found the issue
>>> that travis didn't run
>>> > all the tests from the testsuite. And
>>> PartialImportTest was the one,
>>> > which wasn't executed. See
>>> > https://issues.jboss.org/browse/KEYCLOAK-4021
>>> <https://issues.jboss.org/browse/KEYCLOAK-4021>
>>> > <https://issues.jboss.org/browse/KEYCLOAK-4021
>>> <https://issues.jboss.org/browse/KEYCLOAK-4021>>
>>> >
>>> > However with the travis fix, all the tests were
>>> passing (including
>>> > PartialImportTest). So it seems this test was just
>>> failing
>>> > randomly (not
>>> > always)?
>>> >
>>> > Now I can see in latest travis build that
>>> PartialImportTest passes.
>>> >
>>> > Marek
>>> >
>>> >
>>> >
>>> >
>>> > On 03/12/16 17:36, Bill Burke wrote:
>>> > > I just noticed that my local build fails while
>>> travis passes.
>>> > The bug is
>>> > > really something travis should have picked up,
>>> specifically the
>>> > > PartialImportsTest was removing an identity
>>> provider. The JPA
>>> > > removeIdenittyProviderByAlias method was wrong
>>> as it was trying
>>> > to load
>>> > > an IdentityProviderModel after it was removed
>>> thus resulting in a
>>> > > Hibernate error. Travis did not pick this up
>>> which makes me
>>> > wonder if
>>> > > the test is even running.
>>> > >
>>> > > FYI, i have a pull request that fixes this that
>>> is incoming. The
>>> > bug,
>>> > > not travis.
>>> > >
>>> > > Bill
>>> > >
>>> > >
>>> > > _______________________________________________
>>> > > keycloak-dev mailing list
>>> > > keycloak-dev at lists.jboss.org
>>> <mailto:keycloak-dev at lists.jboss.org>
>>> <mailto:keycloak-dev at lists.jboss.org
>>> <mailto:keycloak-dev at lists.jboss.org>>
>>> > >
>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>>> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>>
>>> >
>>> >
>>> > _______________________________________________
>>> > keycloak-dev mailing list
>>> > keycloak-dev at lists.jboss.org
>>> <mailto:keycloak-dev at lists.jboss.org>
>>> <mailto:keycloak-dev at lists.jboss.org
>>> <mailto:keycloak-dev at lists.jboss.org>>
>>> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>>> >
>>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>>
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > --Hynek
>>>
>>> _______________________________________________
>>> keycloak-dev mailing list
>>> keycloak-dev at lists.jboss.org
>>> <mailto:keycloak-dev at lists.jboss.org>
>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>>>
>>>
>>
>>
>
>
More information about the keycloak-dev
mailing list