10:36 a.m.
I just noticed that my local build fails while travis passes. The bug is
really something travis should have picked up, specifically the
PartialImportsTest was removing an identity provider. The JPA
removeIdenittyProviderByAlias method was wrong as it was trying to load
an IdentityProviderModel after it was removed thus resulting in a
Hibernate error. Travis did not pick this up which makes me wonder if
the test is even running.
FYI, i have a pull request that fixes this that is incoming. The bug,
not travis.
Bill
1:39 a.m.
Fyi. On Friday afternoon, I've found the issue that travis didn't run
all the tests from the testsuite. And PartialImportTest was the one,
which wasn't executed. See https://issues.jboss.org/browse/KEYCLOAK-4021
However with the travis fix, all the tests were passing (including
PartialImportTest). So it seems this test was just failing randomly (not
always)?
Now I can see in latest travis build that PartialImportTest passes.
Marek
On 03/12/16 17:36, Bill Burke wrote:
I just noticed that my local build fails while travis passes. The bug
is
really something travis should have picked up, specifically the
PartialImportsTest was removing an identity provider. The JPA
removeIdenittyProviderByAlias method was wrong as it was trying to load
an IdentityProviderModel after it was removed thus resulting in a
Hibernate error. Travis did not pick this up which makes me wonder if
the test is even running.
FYI, i have a pull request that fixes this that is incoming. The bug,
not travis.
Bill
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
5:53 a.m.
Speaking of the tests, we seem not to run any of the adapter test suites. I
believe that running at least adapter tests for wildfly would be
beneficial, preventing e.g. [1]. WDYT?
[1] https://issues.jboss.org/browse/KEYCLOAK-4017
On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda <mposolda(a)redhat.com> wrote:
Fyi. On Friday afternoon, I've found the issue that travis
didn't run
all the tests from the testsuite. And PartialImportTest was the one,
which wasn't executed. See https://issues.jboss.org/browse/KEYCLOAK-4021
However with the travis fix, all the tests were passing (including
PartialImportTest). So it seems this test was just failing randomly (not
always)?
Now I can see in latest travis build that PartialImportTest passes.
Marek
On 03/12/16 17:36, Bill Burke wrote:
> I just noticed that my local build fails while travis passes. The bug is
> really something travis should have picked up, specifically the
> PartialImportsTest was removing an identity provider. The JPA
> removeIdenittyProviderByAlias method was wrong as it was trying to load
> an IdentityProviderModel after it was removed thus resulting in a
> Hibernate error. Travis did not pick this up which makes me wonder if
> the test is even running.
>
> FYI, i have a pull request that fixes this that is incoming. The bug,
> not travis.
>
> Bill
>
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
--
--Hynek
8:52 a.m.
Since last week, I've added the support for running the adapter tests on
embedded undertow and some are run during default travis build too.
Basically everything in package
"org.keycloak.testsuite.adapter.undertow.servlet" for now. This also
allows running adapter tests from IDE, which is great for development
and debugging. Previously you either needed to develop adapters in the
old testsuite or develop against wildfly server, which sucks (constantly
rebuilding stuff with maven, copying the jars to modules etc).
This should help to avoid most of regressions in adapters, which should
be catched during travis build now. Unfortunately it won't help with the
issue like KEYCLOAK-4017 because the modules inside
testsuite/integration-arquillian/tests/other/adapters/jboss/ are
"detached" from the main project, so removing the class in IDE won't
automatically remove the subclasses in those detached modules. I wonder
if we can improve to avoid those detached modules and abstract classes
with tests and instead have just the single class with particular
adapter test, which will just work with any app server profile (
-Papp-server-wildfly, -Pauth-server-eap etc)?
Marek
On 05/12/16 12:53, Hynek Mlnarik wrote:
Speaking of the tests, we seem not to run any of the adapter test
suites. I believe that running at least adapter tests for wildfly
would be beneficial, preventing e.g. [1]. WDYT?
[1] https://issues.jboss.org/browse/KEYCLOAK-4017
On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
Fyi. On Friday afternoon, I've found the issue that travis didn't run
all the tests from the testsuite. And PartialImportTest was the one,
which wasn't executed. See
https://issues.jboss.org/browse/KEYCLOAK-4021
<https://issues.jboss.org/browse/KEYCLOAK-4021>
However with the travis fix, all the tests were passing (including
PartialImportTest). So it seems this test was just failing
randomly (not
always)?
Now I can see in latest travis build that PartialImportTest passes.
Marek
On 03/12/16 17:36, Bill Burke wrote:
> I just noticed that my local build fails while travis passes.
The bug is
> really something travis should have picked up, specifically the
> PartialImportsTest was removing an identity provider. The JPA
> removeIdenittyProviderByAlias method was wrong as it was trying
to load
> an IdentityProviderModel after it was removed thus resulting in a
> Hibernate error. Travis did not pick this up which makes me
wonder if
> the test is even running.
>
> FYI, i have a pull request that fixes this that is incoming. The
bug,
> not travis.
>
> Bill
>
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
<https://lists.jboss.org/mailman/listinfo/keycloak-dev>
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-dev
<https://lists.jboss.org/mailman/listinfo/keycloak-dev>
--
--Hynek
9:32 a.m.
These broke because they weren't part of the main build. I thought they
were just dead code because when I did a "Find Usages" for them, nothing
came up. Minimally, things should at least be compiled with the main
build, that way when refactorings happen, somebody doesn't delete a test
dependency by accident.
On 12/5/16 6:53 AM, Hynek Mlnarik wrote:
Speaking of the tests, we seem not to run any of the adapter test
suites. I believe that running at least adapter tests for wildfly
would be beneficial, preventing e.g. [1]. WDYT?
[1] https://issues.jboss.org/browse/KEYCLOAK-4017
On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
Fyi. On Friday afternoon, I've found the issue that travis didn't run
all the tests from the testsuite. And PartialImportTest was the one,
which wasn't executed. See
https://issues.jboss.org/browse/KEYCLOAK-4021
<https://issues.jboss.org/browse/KEYCLOAK-4021>
However with the travis fix, all the tests were passing (including
PartialImportTest). So it seems this test was just failing
randomly (not
always)?
Now I can see in latest travis build that PartialImportTest passes.
Marek
On 03/12/16 17:36, Bill Burke wrote:
> I just noticed that my local build fails while travis passes.
The bug is
> really something travis should have picked up, specifically the
> PartialImportsTest was removing an identity provider. The JPA
> removeIdenittyProviderByAlias method was wrong as it was trying
to load
> an IdentityProviderModel after it was removed thus resulting in a
> Hibernate error. Travis did not pick this up which makes me
wonder if
> the test is even running.
>
> FYI, i have a pull request that fixes this that is incoming. The
bug,
> not travis.
>
> Bill
>
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
<https://lists.jboss.org/mailman/listinfo/keycloak-dev>
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-dev
<https://lists.jboss.org/mailman/listinfo/keycloak-dev>
--
--Hynek
7:24 a.m.
Let's review the tests ran by Travis after the new year, but with Travis
having the option to run groups of tests in parallel we should be able to
tests more including adapters and console. Travis should also be changed to
tests with the full KC server and WildFly for adapters rather than embedded
Undertow.
On 5 December 2016 at 16:32, Bill Burke <bburke(a)redhat.com> wrote:
These broke because they weren't part of the main build. I
thought they
were just dead code because when I did a "Find Usages" for them, nothing
came up. Minimally, things should at least be compiled with the main
build, that way when refactorings happen, somebody doesn't delete a test
dependency by accident.
On 12/5/16 6:53 AM, Hynek Mlnarik wrote:
> Speaking of the tests, we seem not to run any of the adapter test
> suites. I believe that running at least adapter tests for wildfly
> would be beneficial, preventing e.g. [1]. WDYT?
>
> [1] https://issues.jboss.org/browse/KEYCLOAK-4017
>
> On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda <mposolda(a)redhat.com
> <mailto:mposolda@redhat.com>> wrote:
>
> Fyi. On Friday afternoon, I've found the issue that travis didn't run
> all the tests from the testsuite. And PartialImportTest was the one,
> which wasn't executed. See
> https://issues.jboss.org/browse/KEYCLOAK-4021
> <https://issues.jboss.org/browse/KEYCLOAK-4021>
>
> However with the travis fix, all the tests were passing (including
> PartialImportTest). So it seems this test was just failing
> randomly (not
> always)?
>
> Now I can see in latest travis build that PartialImportTest passes.
>
> Marek
>
>
>
>
> On 03/12/16 17:36, Bill Burke wrote:
> > I just noticed that my local build fails while travis passes.
> The bug is
> > really something travis should have picked up, specifically the
> > PartialImportsTest was removing an identity provider. The JPA
> > removeIdenittyProviderByAlias method was wrong as it was trying
> to load
> > an IdentityProviderModel after it was removed thus resulting in a
> > Hibernate error. Travis did not pick this up which makes me
> wonder if
> > the test is even running.
> >
> > FYI, i have a pull request that fixes this that is incoming. The
> bug,
> > not travis.
> >
> > Bill
> >
> >
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jboss.org>
> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>
>
>
>
> --
>
> --Hynek
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/keycloak-dev
10:46 a.m.
This wasn't just the adapter tests, initially it was the
PartialImportsTest which lives under org.keycloak.testsuite.admin
somewhere and should have run and should have failed.
https://issues.jboss.org/browse/KEYCLOAK-4068
On 12/12/16 8:24 AM, Stian Thorgersen wrote:
Let's review the tests ran by Travis after the new year, but with
Travis having the option to run groups of tests in parallel we should
be able to tests more including adapters and console. Travis should
also be changed to tests with the full KC server and WildFly for
adapters rather than embedded Undertow.
On 5 December 2016 at 16:32, Bill Burke <bburke(a)redhat.com
<mailto:bburke@redhat.com>> wrote:
These broke because they weren't part of the main build. I
thought they
were just dead code because when I did a "Find Usages" for them,
nothing
came up. Minimally, things should at least be compiled with the main
build, that way when refactorings happen, somebody doesn't delete
a test
dependency by accident.
On 12/5/16 6:53 AM, Hynek Mlnarik wrote:
> Speaking of the tests, we seem not to run any of the adapter test
> suites. I believe that running at least adapter tests for wildfly
> would be beneficial, preventing e.g. [1]. WDYT?
>
> [1] https://issues.jboss.org/browse/KEYCLOAK-4017
<https://issues.jboss.org/browse/KEYCLOAK-4017>
>
> On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda
<mposolda(a)redhat.com <mailto:mposolda@redhat.com>
> <mailto:mposolda@redhat.com <mailto:mposolda@redhat.com>>>
wrote:
>
> Fyi. On Friday afternoon, I've found the issue that travis
didn't run
> all the tests from the testsuite. And PartialImportTest was
the one,
> which wasn't executed. See
> https://issues.jboss.org/browse/KEYCLOAK-4021
<https://issues.jboss.org/browse/KEYCLOAK-4021>
> <https://issues.jboss.org/browse/KEYCLOAK-4021
<https://issues.jboss.org/browse/KEYCLOAK-4021>>
>
> However with the travis fix, all the tests were passing
(including
> PartialImportTest). So it seems this test was just failing
> randomly (not
> always)?
>
> Now I can see in latest travis build that PartialImportTest
passes.
>
> Marek
>
>
>
>
> On 03/12/16 17:36, Bill Burke wrote:
> > I just noticed that my local build fails while travis passes.
> The bug is
> > really something travis should have picked up,
specifically the
> > PartialImportsTest was removing an identity provider. The JPA
> > removeIdenittyProviderByAlias method was wrong as it was
trying
> to load
> > an IdentityProviderModel after it was removed thus
resulting in a
> > Hibernate error. Travis did not pick this up which makes me
> wonder if
> > the test is even running.
> >
> > FYI, i have a pull request that fixes this that is
incoming. The
> bug,
> > not travis.
> >
> > Bill
> >
> >
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev(a)lists.jboss.org
<mailto:keycloak-dev@lists.jboss.org>
<mailto:keycloak-dev@lists.jboss.org
<mailto:keycloak-dev@lists.jboss.org>>
> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
<https://lists.jboss.org/mailman/listinfo/keycloak-dev>
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev
<https://lists.jboss.org/mailman/listinfo/keycloak-dev>>
>
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org
<mailto:keycloak-dev@lists.jboss.org>
<mailto:keycloak-dev@lists.jboss.org
<mailto:keycloak-dev@lists.jboss.org>>
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
<https://lists.jboss.org/mailman/listinfo/keycloak-dev>
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev
<https://lists.jboss.org/mailman/listinfo/keycloak-dev>>
>
>
>
>
> --
>
> --Hynek
_______________________________________________
keycloak-dev mailing list
keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/keycloak-dev
<https://lists.jboss.org/mailman/listinfo/keycloak-dev>
10:51 a.m.
It should run at least what it used to run before I broke it up into
multiple parallell jobs (seems I did a poor job when I did that and messed
some stuff up). That can be fixed straight away, but adding more tests (new
adapter tests, clustering tests, console tests, etc.) we should probably
review later as it may impact stability and time to test a PR on Travis.
On 12 December 2016 at 17:46, Bill Burke <bburke(a)redhat.com> wrote:
This wasn't just the adapter tests, initially it was the
PartialImportsTest which lives under org.keycloak.testsuite.admin somewhere
and should have run and should have failed.
https://issues.jboss.org/browse/KEYCLOAK-4068
On 12/12/16 8:24 AM, Stian Thorgersen wrote:
Let's review the tests ran by Travis after the new year, but with Travis
having the option to run groups of tests in parallel we should be able to
tests more including adapters and console. Travis should also be changed to
tests with the full KC server and WildFly for adapters rather than embedded
Undertow.
On 5 December 2016 at 16:32, Bill Burke <bburke(a)redhat.com> wrote:
> These broke because they weren't part of the main build. I thought they
> were just dead code because when I did a "Find Usages" for them, nothing
> came up. Minimally, things should at least be compiled with the main
> build, that way when refactorings happen, somebody doesn't delete a test
> dependency by accident.
>
>
> On 12/5/16 6:53 AM, Hynek Mlnarik wrote:
> > Speaking of the tests, we seem not to run any of the adapter test
> > suites. I believe that running at least adapter tests for wildfly
> > would be beneficial, preventing e.g. [1]. WDYT?
> >
> > [1] https://issues.jboss.org/browse/KEYCLOAK-4017
> >
> > On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda <mposolda(a)redhat.com
> > <mailto:mposolda@redhat.com>> wrote:
> >
> > Fyi. On Friday afternoon, I've found the issue that travis didn't
> run
> > all the tests from the testsuite. And PartialImportTest was the one,
> > which wasn't executed. See
> > https://issues.jboss.org/browse/KEYCLOAK-4021
> > <https://issues.jboss.org/browse/KEYCLOAK-4021>
> >
> > However with the travis fix, all the tests were passing (including
> > PartialImportTest). So it seems this test was just failing
> > randomly (not
> > always)?
> >
> > Now I can see in latest travis build that PartialImportTest passes.
> >
> > Marek
> >
> >
> >
> >
> > On 03/12/16 17:36, Bill Burke wrote:
> > > I just noticed that my local build fails while travis passes.
> > The bug is
> > > really something travis should have picked up, specifically the
> > > PartialImportsTest was removing an identity provider. The JPA
> > > removeIdenittyProviderByAlias method was wrong as it was trying
> > to load
> > > an IdentityProviderModel after it was removed thus resulting in a
> > > Hibernate error. Travis did not pick this up which makes me
> > wonder if
> > > the test is even running.
> > >
> > > FYI, i have a pull request that fixes this that is incoming. The
> > bug,
> > > not travis.
> > >
> > > Bill
> > >
> > >
> > > _______________________________________________
> > > keycloak-dev mailing list
> > > keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jboss.org
> >
> > > https://lists.jboss.org/mailman/listinfo/keycloak-dev
> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
> >
> >
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jboss.org>
> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
> >
> >
> >
> >
> > --
> >
> > --Hynek
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>
10:54 a.m.
The filter you have looks like the test should have run which is why the
review needs to happen.
On 12/12/16 11:51 AM, Stian Thorgersen wrote:
It should run at least what it used to run before I broke it up into
multiple parallell jobs (seems I did a poor job when I did that and
messed some stuff up). That can be fixed straight away, but adding
more tests (new adapter tests, clustering tests, console tests, etc.)
we should probably review later as it may impact stability and time to
test a PR on Travis.
On 12 December 2016 at 17:46, Bill Burke <bburke(a)redhat.com
<mailto:bburke@redhat.com>> wrote:
This wasn't just the adapter tests, initially it was the
PartialImportsTest which lives under org.keycloak.testsuite.admin
somewhere and should have run and should have failed.
https://issues.jboss.org/browse/KEYCLOAK-4068
<https://issues.jboss.org/browse/KEYCLOAK-4068>
On 12/12/16 8:24 AM, Stian Thorgersen wrote:
> Let's review the tests ran by Travis after the new year, but with
> Travis having the option to run groups of tests in parallel we
> should be able to tests more including adapters and console.
> Travis should also be changed to tests with the full KC server
> and WildFly for adapters rather than embedded Undertow.
>
> On 5 December 2016 at 16:32, Bill Burke <bburke(a)redhat.com
> <mailto:bburke@redhat.com>> wrote:
>
> These broke because they weren't part of the main build. I
> thought they
> were just dead code because when I did a "Find Usages" for
> them, nothing
> came up. Minimally, things should at least be compiled with
> the main
> build, that way when refactorings happen, somebody doesn't
> delete a test
> dependency by accident.
>
>
> On 12/5/16 6:53 AM, Hynek Mlnarik wrote:
> > Speaking of the tests, we seem not to run any of the
> adapter test
> > suites. I believe that running at least adapter tests for
> wildfly
> > would be beneficial, preventing e.g. [1]. WDYT?
> >
> > [1] https://issues.jboss.org/browse/KEYCLOAK-4017
> <https://issues.jboss.org/browse/KEYCLOAK-4017>
> >
> > On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda
> <mposolda(a)redhat.com <mailto:mposolda@redhat.com>
> > <mailto:mposolda@redhat.com
<mailto:mposolda@redhat.com>>>
> wrote:
> >
> > Fyi. On Friday afternoon, I've found the issue that
> travis didn't run
> > all the tests from the testsuite. And PartialImportTest
> was the one,
> > which wasn't executed. See
> > https://issues.jboss.org/browse/KEYCLOAK-4021
> <https://issues.jboss.org/browse/KEYCLOAK-4021>
> > <https://issues.jboss.org/browse/KEYCLOAK-4021
> <https://issues.jboss.org/browse/KEYCLOAK-4021>>
> >
> > However with the travis fix, all the tests were passing
> (including
> > PartialImportTest). So it seems this test was just failing
> > randomly (not
> > always)?
> >
> > Now I can see in latest travis build that
> PartialImportTest passes.
> >
> > Marek
> >
> >
> >
> >
> > On 03/12/16 17:36, Bill Burke wrote:
> > > I just noticed that my local build fails while travis
> passes.
> > The bug is
> > > really something travis should have picked up,
> specifically the
> > > PartialImportsTest was removing an identity
> provider. The JPA
> > > removeIdenittyProviderByAlias method was wrong as it
> was trying
> > to load
> > > an IdentityProviderModel after it was removed thus
> resulting in a
> > > Hibernate error. Travis did not pick this up which
> makes me
> > wonder if
> > > the test is even running.
> > >
> > > FYI, i have a pull request that fixes this that is
> incoming. The
> > bug,
> > > not travis.
> > >
> > > Bill
> > >
> > >
> > > _______________________________________________
> > > keycloak-dev mailing list
> > > keycloak-dev(a)lists.jboss.org
> <mailto:keycloak-dev@lists.jboss.org>
> <mailto:keycloak-dev@lists.jboss.org
> <mailto:keycloak-dev@lists.jboss.org>>
> > > https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>>
> >
> >
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev(a)lists.jboss.org
> <mailto:keycloak-dev@lists.jboss.org>
> <mailto:keycloak-dev@lists.jboss.org
> <mailto:keycloak-dev@lists.jboss.org>>
> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>>
> >
> >
> >
> >
> > --
> >
> > --Hynek
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org
> <mailto:keycloak-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>
>
10:56 a.m.
There was an issue before that the script didn't return the exit code so it
would look like it passed even though the tests failed. That's fixed though:
https://github.com/keycloak/keycloak/commit/a52f7f9baa5972fb606714d0f370f...
On 12 December 2016 at 17:54, Bill Burke <bburke(a)redhat.com> wrote:
The filter you have looks like the test should have run which is why
the
review needs to happen.
On 12/12/16 11:51 AM, Stian Thorgersen wrote:
It should run at least what it used to run before I broke it up into
multiple parallell jobs (seems I did a poor job when I did that and messed
some stuff up). That can be fixed straight away, but adding more tests (new
adapter tests, clustering tests, console tests, etc.) we should probably
review later as it may impact stability and time to test a PR on Travis.
On 12 December 2016 at 17:46, Bill Burke <bburke(a)redhat.com> wrote:
> This wasn't just the adapter tests, initially it was the
> PartialImportsTest which lives under org.keycloak.testsuite.admin somewhere
> and should have run and should have failed.
>
> https://issues.jboss.org/browse/KEYCLOAK-4068
>
>
> On 12/12/16 8:24 AM, Stian Thorgersen wrote:
>
> Let's review the tests ran by Travis after the new year, but with Travis
> having the option to run groups of tests in parallel we should be able to
> tests more including adapters and console. Travis should also be changed to
> tests with the full KC server and WildFly for adapters rather than embedded
> Undertow.
>
> On 5 December 2016 at 16:32, Bill Burke <bburke(a)redhat.com> wrote:
>
>> These broke because they weren't part of the main build. I thought they
>> were just dead code because when I did a "Find Usages" for them,
nothing
>> came up. Minimally, things should at least be compiled with the main
>> build, that way when refactorings happen, somebody doesn't delete a test
>> dependency by accident.
>>
>>
>> On 12/5/16 6:53 AM, Hynek Mlnarik wrote:
>> > Speaking of the tests, we seem not to run any of the adapter test
>> > suites. I believe that running at least adapter tests for wildfly
>> > would be beneficial, preventing e.g. [1]. WDYT?
>> >
>> > [1] https://issues.jboss.org/browse/KEYCLOAK-4017
>> >
>> > On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda <mposolda(a)redhat.com
>> > <mailto:mposolda@redhat.com>> wrote:
>> >
>> > Fyi. On Friday afternoon, I've found the issue that travis
didn't
>> run
>> > all the tests from the testsuite. And PartialImportTest was the
>> one,
>> > which wasn't executed. See
>> > https://issues.jboss.org/browse/KEYCLOAK-4021
>> > <https://issues.jboss.org/browse/KEYCLOAK-4021>
>> >
>> > However with the travis fix, all the tests were passing (including
>> > PartialImportTest). So it seems this test was just failing
>> > randomly (not
>> > always)?
>> >
>> > Now I can see in latest travis build that PartialImportTest passes.
>> >
>> > Marek
>> >
>> >
>> >
>> >
>> > On 03/12/16 17:36, Bill Burke wrote:
>> > > I just noticed that my local build fails while travis passes.
>> > The bug is
>> > > really something travis should have picked up, specifically the
>> > > PartialImportsTest was removing an identity provider. The JPA
>> > > removeIdenittyProviderByAlias method was wrong as it was trying
>> > to load
>> > > an IdentityProviderModel after it was removed thus resulting in a
>> > > Hibernate error. Travis did not pick this up which makes me
>> > wonder if
>> > > the test is even running.
>> > >
>> > > FYI, i have a pull request that fixes this that is incoming. The
>> > bug,
>> > > not travis.
>> > >
>> > > Bill
>> > >
>> > >
>> > > _______________________________________________
>> > > keycloak-dev mailing list
>> > > keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jbo
>> ss.org>
>> > > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>> >
>> >
>> > _______________________________________________
>> > keycloak-dev mailing list
>> > keycloak-dev(a)lists.jboss.org
<mailto:keycloak-dev@lists.jboss.org>
>> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>> >
>> >
>> >
>> >
>> > --
>> >
>> > --Hynek
>>
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
>
>
>
2:05 p.m.
FYI. There was also an issue that some tests weren't executed due to the
filter, which didn't include sub-packages of specified packages. See
commit https://github.com/keycloak/keycloak/pull/3596/files and JIRA
https://issues.jboss.org/browse/KEYCLOAK-4021 . PartialImportTest was
one of the tests, which weren't executed, but now it is.
Marek
On 12/12/16 17:56, Stian Thorgersen wrote:
There was an issue before that the script didn't return the exit
code
so it would look like it passed even though the tests failed. That's
fixed though:
https://github.com/keycloak/keycloak/commit/a52f7f9baa5972fb606714d0f370f...
On 12 December 2016 at 17:54, Bill Burke <bburke(a)redhat.com
<mailto:bburke@redhat.com>> wrote:
The filter you have looks like the test should have run which is
why the review needs to happen.
On 12/12/16 11:51 AM, Stian Thorgersen wrote:
> It should run at least what it used to run before I broke it up
> into multiple parallell jobs (seems I did a poor job when I did
> that and messed some stuff up). That can be fixed straight away,
> but adding more tests (new adapter tests, clustering tests,
> console tests, etc.) we should probably review later as it may
> impact stability and time to test a PR on Travis.
>
> On 12 December 2016 at 17:46, Bill Burke <bburke(a)redhat.com
> <mailto:bburke@redhat.com>> wrote:
>
> This wasn't just the adapter tests, initially it was the
> PartialImportsTest which lives under
> org.keycloak.testsuite.admin somewhere and should have run
> and should have failed.
>
> https://issues.jboss.org/browse/KEYCLOAK-4068
> <https://issues.jboss.org/browse/KEYCLOAK-4068>
>
>
> On 12/12/16 8:24 AM, Stian Thorgersen wrote:
>> Let's review the tests ran by Travis after the new year, but
>> with Travis having the option to run groups of tests in
>> parallel we should be able to tests more including adapters
>> and console. Travis should also be changed to tests with the
>> full KC server and WildFly for adapters rather than embedded
>> Undertow.
>>
>> On 5 December 2016 at 16:32, Bill Burke <bburke(a)redhat.com
>> <mailto:bburke@redhat.com>> wrote:
>>
>> These broke because they weren't part of the main
>> build. I thought they
>> were just dead code because when I did a "Find Usages"
>> for them, nothing
>> came up. Minimally, things should at least be compiled
>> with the main
>> build, that way when refactorings happen, somebody
>> doesn't delete a test
>> dependency by accident.
>>
>>
>> On 12/5/16 6:53 AM, Hynek Mlnarik wrote:
>> > Speaking of the tests, we seem not to run any of the
>> adapter test
>> > suites. I believe that running at least adapter tests
>> for wildfly
>> > would be beneficial, preventing e.g. [1]. WDYT?
>> >
>> > [1] https://issues.jboss.org/browse/KEYCLOAK-4017
>> <https://issues.jboss.org/browse/KEYCLOAK-4017>
>> >
>> > On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda
>> <mposolda(a)redhat.com <mailto:mposolda@redhat.com>
>> > <mailto:mposolda@redhat.com
>> <mailto:mposolda@redhat.com>>> wrote:
>> >
>> > Fyi. On Friday afternoon, I've found the issue
>> that travis didn't run
>> > all the tests from the testsuite. And
>> PartialImportTest was the one,
>> > which wasn't executed. See
>> > https://issues.jboss.org/browse/KEYCLOAK-4021
>> <https://issues.jboss.org/browse/KEYCLOAK-4021>
>> > <https://issues.jboss.org/browse/KEYCLOAK-4021
>> <https://issues.jboss.org/browse/KEYCLOAK-4021>>
>> >
>> > However with the travis fix, all the tests were
>> passing (including
>> > PartialImportTest). So it seems this test was just
>> failing
>> > randomly (not
>> > always)?
>> >
>> > Now I can see in latest travis build that
>> PartialImportTest passes.
>> >
>> > Marek
>> >
>> >
>> >
>> >
>> > On 03/12/16 17:36, Bill Burke wrote:
>> > > I just noticed that my local build fails while
>> travis passes.
>> > The bug is
>> > > really something travis should have picked up,
>> specifically the
>> > > PartialImportsTest was removing an identity
>> provider. The JPA
>> > > removeIdenittyProviderByAlias method was wrong
>> as it was trying
>> > to load
>> > > an IdentityProviderModel after it was removed
>> thus resulting in a
>> > > Hibernate error. Travis did not pick this up
>> which makes me
>> > wonder if
>> > > the test is even running.
>> > >
>> > > FYI, i have a pull request that fixes this that
>> is incoming. The
>> > bug,
>> > > not travis.
>> > >
>> > > Bill
>> > >
>> > >
>> > > _______________________________________________
>> > > keycloak-dev mailing list
>> > > keycloak-dev(a)lists.jboss.org
>> <mailto:keycloak-dev@lists.jboss.org>
>> <mailto:keycloak-dev@lists.jboss.org
>> <mailto:keycloak-dev@lists.jboss.org>>
>> > >
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>>
>> >
>> >
>> > _______________________________________________
>> > keycloak-dev mailing list
>> > keycloak-dev(a)lists.jboss.org
>> <mailto:keycloak-dev@lists.jboss.org>
>> <mailto:keycloak-dev@lists.jboss.org
>> <mailto:keycloak-dev@lists.jboss.org>>
>> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>> >
>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>>
>> >
>> >
>> >
>> >
>> > --
>> >
>> > --Hynek
>>
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev(a)lists.jboss.org
>> <mailto:keycloak-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>>
>>
>
>
2:44 p.m.
On 12/12/16 17:51, Stian Thorgersen wrote:
It should run at least what it used to run before I broke it up into
multiple parallell jobs (seems I did a poor job when I did that and
messed some stuff up). That can be fixed straight away, but adding
more tests (new adapter tests, clustering tests, console tests, etc.)
we should probably review later as it may impact stability and time to
test a PR on Travis.
+1 for better review. Few things:
* Unit tests in the modules outside testsuite are not executed during
travis build. For example tests under core/src/test/java .
* Everything in testsuite/integration and
testsuite/integration-arquillian/tests/base is running in Travis build
ATM. However if we add new package with tests into the testsuite (For
example org.keycloak.testsuite.zzz.NewTest), it is possible that it
won't be running due to missing filter in the travis-run-tests.sh for
the package starting with "z" . I hope we can have the list of packages
by travis-run-tests.sh to be more dynamic.
* IMO it is sufficient that travis will run all the tests (including
adapters and cluster) on embedded undertow. Running the tests on Wildfly
is unecessary overhead, which will slightly increase the travis build
time (as it will need to build distribution, start server etc) . There
is not much added value as 95 % of regressions are catched with undertow
too.
* I've added some undertow adapter tests to be executed during default
travis build. They are all in the package
org.keycloak.testsuite.adapter.undertow.servlet . Problem is, that not
all adapter tests are executed as it will require creating subclass for
every "Abstract" class with test and I just didn't create all the
classes for embedded undertow. I've created JIRA
https://issues.jboss.org/browse/KEYCLOAK-4069 to omprove adapter tests
to avoid "dead" modules and many classes needed for single adapter test.
Marek
On 12 December 2016 at 17:46, Bill Burke <bburke(a)redhat.com
<mailto:bburke@redhat.com>> wrote:
This wasn't just the adapter tests, initially it was the
PartialImportsTest which lives under org.keycloak.testsuite.admin
somewhere and should have run and should have failed.
https://issues.jboss.org/browse/KEYCLOAK-4068
<https://issues.jboss.org/browse/KEYCLOAK-4068>
On 12/12/16 8:24 AM, Stian Thorgersen wrote:
> Let's review the tests ran by Travis after the new year, but with
> Travis having the option to run groups of tests in parallel we
> should be able to tests more including adapters and console.
> Travis should also be changed to tests with the full KC server
> and WildFly for adapters rather than embedded Undertow.
>
> On 5 December 2016 at 16:32, Bill Burke <bburke(a)redhat.com
> <mailto:bburke@redhat.com>> wrote:
>
> These broke because they weren't part of the main build. I
> thought they
> were just dead code because when I did a "Find Usages" for
> them, nothing
> came up. Minimally, things should at least be compiled with
> the main
> build, that way when refactorings happen, somebody doesn't
> delete a test
> dependency by accident.
>
>
> On 12/5/16 6:53 AM, Hynek Mlnarik wrote:
> > Speaking of the tests, we seem not to run any of the
> adapter test
> > suites. I believe that running at least adapter tests for
> wildfly
> > would be beneficial, preventing e.g. [1]. WDYT?
> >
> > [1] https://issues.jboss.org/browse/KEYCLOAK-4017
> <https://issues.jboss.org/browse/KEYCLOAK-4017>
> >
> > On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda
> <mposolda(a)redhat.com <mailto:mposolda@redhat.com>
> > <mailto:mposolda@redhat.com
<mailto:mposolda@redhat.com>>>
> wrote:
> >
> > Fyi. On Friday afternoon, I've found the issue that
> travis didn't run
> > all the tests from the testsuite. And PartialImportTest
> was the one,
> > which wasn't executed. See
> > https://issues.jboss.org/browse/KEYCLOAK-4021
> <https://issues.jboss.org/browse/KEYCLOAK-4021>
> > <https://issues.jboss.org/browse/KEYCLOAK-4021
> <https://issues.jboss.org/browse/KEYCLOAK-4021>>
> >
> > However with the travis fix, all the tests were passing
> (including
> > PartialImportTest). So it seems this test was just failing
> > randomly (not
> > always)?
> >
> > Now I can see in latest travis build that
> PartialImportTest passes.
> >
> > Marek
> >
> >
> >
> >
> > On 03/12/16 17:36, Bill Burke wrote:
> > > I just noticed that my local build fails while travis
> passes.
> > The bug is
> > > really something travis should have picked up,
> specifically the
> > > PartialImportsTest was removing an identity
> provider. The JPA
> > > removeIdenittyProviderByAlias method was wrong as it
> was trying
> > to load
> > > an IdentityProviderModel after it was removed thus
> resulting in a
> > > Hibernate error. Travis did not pick this up which
> makes me
> > wonder if
> > > the test is even running.
> > >
> > > FYI, i have a pull request that fixes this that is
> incoming. The
> > bug,
> > > not travis.
> > >
> > > Bill
> > >
> > >
> > > _______________________________________________
> > > keycloak-dev mailing list
> > > keycloak-dev(a)lists.jboss.org
> <mailto:keycloak-dev@lists.jboss.org>
> <mailto:keycloak-dev@lists.jboss.org
> <mailto:keycloak-dev@lists.jboss.org>>
> > > https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>>
> >
> >
> > _______________________________________________
> > keycloak-dev mailing list
> > keycloak-dev(a)lists.jboss.org
> <mailto:keycloak-dev@lists.jboss.org>
> <mailto:keycloak-dev@lists.jboss.org
> <mailto:keycloak-dev@lists.jboss.org>>
> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>>
> >
> >
> >
> >
> > --
> >
> > --Hynek
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org
> <mailto:keycloak-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>
>
1:26 a.m.
On 12 December 2016 at 21:44, Marek Posolda <mposolda(a)redhat.com> wrote:
On 12/12/16 17:51, Stian Thorgersen wrote:
It should run at least what it used to run before I broke it up into
multiple parallell jobs (seems I did a poor job when I did that and messed
some stuff up). That can be fixed straight away, but adding more tests (new
adapter tests, clustering tests, console tests, etc.) we should probably
review later as it may impact stability and time to test a PR on Travis.
+1 for better review. Few things:
* Unit tests in the modules outside testsuite are not executed during
travis build. For example tests under core/src/test/java .
* Everything in testsuite/integration and testsuite/integration-arquillian/tests/base
is running in Travis build ATM. However if we add new package with tests
into the testsuite (For example org.keycloak.testsuite.zzz.NewTest), it
is possible that it won't be running due to missing filter in the
travis-run-tests.sh for the package starting with "z" . I hope we can have
the list of packages by travis-run-tests.sh to be more dynamic.
* IMO it is sufficient that travis will run all the tests (including
adapters and cluster) on embedded undertow. Running the tests on Wildfly is
unecessary overhead, which will slightly increase the travis build time (as
it will need to build distribution, start server etc) . There is not much
added value as 95 % of regressions are catched with undertow too.
-1 We've seen this many times where Travis is happy, but the distribution
is broken. This happened several times last year so we need to prevent this.
* I've added some undertow adapter tests to be executed during default
travis build. They are all in the package org.keycloak.testsuite.
adapter.undertow.servlet . Problem is, that not all adapter tests are
executed as it will require creating subclass for every "Abstract" class
with test and I just didn't create all the classes for embedded undertow.
I've created JIRA https://issues.jboss.org/browse/KEYCLOAK-4069 to
omprove adapter tests to avoid "dead" modules and many classes needed for
single adapter test.
Marek
On 12 December 2016 at 17:46, Bill Burke <bburke(a)redhat.com> wrote:
> This wasn't just the adapter tests, initially it was the
> PartialImportsTest which lives under org.keycloak.testsuite.admin somewhere
> and should have run and should have failed.
>
> https://issues.jboss.org/browse/KEYCLOAK-4068
>
>
> On 12/12/16 8:24 AM, Stian Thorgersen wrote:
>
> Let's review the tests ran by Travis after the new year, but with Travis
> having the option to run groups of tests in parallel we should be able to
> tests more including adapters and console. Travis should also be changed to
> tests with the full KC server and WildFly for adapters rather than embedded
> Undertow.
>
> On 5 December 2016 at 16:32, Bill Burke <bburke(a)redhat.com> wrote:
>
>> These broke because they weren't part of the main build. I thought they
>> were just dead code because when I did a "Find Usages" for them,
nothing
>> came up. Minimally, things should at least be compiled with the main
>> build, that way when refactorings happen, somebody doesn't delete a test
>> dependency by accident.
>>
>>
>> On 12/5/16 6:53 AM, Hynek Mlnarik wrote:
>> > Speaking of the tests, we seem not to run any of the adapter test
>> > suites. I believe that running at least adapter tests for wildfly
>> > would be beneficial, preventing e.g. [1]. WDYT?
>> >
>> > [1] https://issues.jboss.org/browse/KEYCLOAK-4017
>> >
>> > On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda <mposolda(a)redhat.com
>> > <mailto:mposolda@redhat.com>> wrote:
>> >
>> > Fyi. On Friday afternoon, I've found the issue that travis
didn't
>> run
>> > all the tests from the testsuite. And PartialImportTest was the
>> one,
>> > which wasn't executed. See
>> > https://issues.jboss.org/browse/KEYCLOAK-4021
>> > <https://issues.jboss.org/browse/KEYCLOAK-4021>
>> >
>> > However with the travis fix, all the tests were passing (including
>> > PartialImportTest). So it seems this test was just failing
>> > randomly (not
>> > always)?
>> >
>> > Now I can see in latest travis build that PartialImportTest passes.
>> >
>> > Marek
>> >
>> >
>> >
>> >
>> > On 03/12/16 17:36, Bill Burke wrote:
>> > > I just noticed that my local build fails while travis passes.
>> > The bug is
>> > > really something travis should have picked up, specifically the
>> > > PartialImportsTest was removing an identity provider. The JPA
>> > > removeIdenittyProviderByAlias method was wrong as it was trying
>> > to load
>> > > an IdentityProviderModel after it was removed thus resulting in a
>> > > Hibernate error. Travis did not pick this up which makes me
>> > wonder if
>> > > the test is even running.
>> > >
>> > > FYI, i have a pull request that fixes this that is incoming. The
>> > bug,
>> > > not travis.
>> > >
>> > > Bill
>> > >
>> > >
>> > > _______________________________________________
>> > > keycloak-dev mailing list
>> > > keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jbo
>> ss.org>
>> > > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>> >
>> >
>> > _______________________________________________
>> > keycloak-dev mailing list
>> > keycloak-dev(a)lists.jboss.org
<mailto:keycloak-dev@lists.jboss.org>
>> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>> >
>> >
>> >
>> >
>> > --
>> >
>> > --Hynek
>>
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
>
>
>
1:50 a.m.
On 13/12/16 08:26, Stian Thorgersen wrote:
On 12 December 2016 at 21:44, Marek Posolda <mposolda(a)redhat.com
<mailto:mposolda@redhat.com>> wrote:
On 12/12/16 17:51, Stian Thorgersen wrote:
> It should run at least what it used to run before I broke it up
> into multiple parallell jobs (seems I did a poor job when I did
> that and messed some stuff up). That can be fixed straight away,
> but adding more tests (new adapter tests, clustering tests,
> console tests, etc.) we should probably review later as it may
> impact stability and time to test a PR on Travis.
+1 for better review. Few things:
* Unit tests in the modules outside testsuite are not executed
during travis build. For example tests under core/src/test/java .
* Everything in testsuite/integration and
testsuite/integration-arquillian/tests/base is running in Travis
build ATM. However if we add new package with tests into the
testsuite (For example org.keycloak.testsuite.zzz.NewTest), it is
possible that it won't be running due to missing filter in the
travis-run-tests.sh for the package starting with "z" . I hope we
can have the list of packages by travis-run-tests.sh to be more
dynamic.
* IMO it is sufficient that travis will run all the tests
(including adapters and cluster) on embedded undertow. Running the
tests on Wildfly is unecessary overhead, which will slightly
increase the travis build time (as it will need to build
distribution, start server etc) . There is not much added value as
95 % of regressions are catched with undertow too.
-1 We've seen this many times where Travis is happy, but the
distribution is broken. This happened several times last year so we
need to prevent this.
Isn't it better to have travis run just on
"simple" environment for each
PR and then rely on Central CI daily jobs to run the tests on real
servers? Even if we setup travis to run adapter tests on Wildfly, it
won't catch all the other broken adapters (EAP6, Tomcat, Jetty, ...).
IMO it is generally better to have CI for each PR, which will be able to
catch 95% of regressions (not necessarily 100%) and doesn't take hours
to wait for every PR. And then separate daily jobs to test everything.
Marek
* I've added some undertow adapter tests to be executed during
default travis build. They are all in the package
org.keycloak.testsuite.adapter.undertow.servlet . Problem is, that
not all adapter tests are executed as it will require creating
subclass for every "Abstract" class with test and I just didn't
create all the classes for embedded undertow. I've created JIRA
https://issues.jboss.org/browse/KEYCLOAK-4069
<https://issues.jboss.org/browse/KEYCLOAK-4069> to omprove adapter
tests to avoid "dead" modules and many classes needed for single
adapter test.
Marek
> On 12 December 2016 at 17:46, Bill Burke <bburke(a)redhat.com
> <mailto:bburke@redhat.com>> wrote:
>
> This wasn't just the adapter tests, initially it was the
> PartialImportsTest which lives under
> org.keycloak.testsuite.admin somewhere and should have run
> and should have failed.
>
> https://issues.jboss.org/browse/KEYCLOAK-4068
> <https://issues.jboss.org/browse/KEYCLOAK-4068>
>
>
> On 12/12/16 8:24 AM, Stian Thorgersen wrote:
>> Let's review the tests ran by Travis after the new year, but
>> with Travis having the option to run groups of tests in
>> parallel we should be able to tests more including adapters
>> and console. Travis should also be changed to tests with the
>> full KC server and WildFly for adapters rather than embedded
>> Undertow.
>>
>> On 5 December 2016 at 16:32, Bill Burke <bburke(a)redhat.com
>> <mailto:bburke@redhat.com>> wrote:
>>
>> These broke because they weren't part of the main
>> build. I thought they
>> were just dead code because when I did a "Find Usages"
>> for them, nothing
>> came up. Minimally, things should at least be compiled
>> with the main
>> build, that way when refactorings happen, somebody
>> doesn't delete a test
>> dependency by accident.
>>
>>
>> On 12/5/16 6:53 AM, Hynek Mlnarik wrote:
>> > Speaking of the tests, we seem not to run any of the
>> adapter test
>> > suites. I believe that running at least adapter tests
>> for wildfly
>> > would be beneficial, preventing e.g. [1]. WDYT?
>> >
>> > [1] https://issues.jboss.org/browse/KEYCLOAK-4017
>> <https://issues.jboss.org/browse/KEYCLOAK-4017>
>> >
>> > On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda
>> <mposolda(a)redhat.com <mailto:mposolda@redhat.com>
>> > <mailto:mposolda@redhat.com
>> <mailto:mposolda@redhat.com>>> wrote:
>> >
>> > Fyi. On Friday afternoon, I've found the issue
>> that travis didn't run
>> > all the tests from the testsuite. And
>> PartialImportTest was the one,
>> > which wasn't executed. See
>> > https://issues.jboss.org/browse/KEYCLOAK-4021
>> <https://issues.jboss.org/browse/KEYCLOAK-4021>
>> > <https://issues.jboss.org/browse/KEYCLOAK-4021
>> <https://issues.jboss.org/browse/KEYCLOAK-4021>>
>> >
>> > However with the travis fix, all the tests were
>> passing (including
>> > PartialImportTest). So it seems this test was just
>> failing
>> > randomly (not
>> > always)?
>> >
>> > Now I can see in latest travis build that
>> PartialImportTest passes.
>> >
>> > Marek
>> >
>> >
>> >
>> >
>> > On 03/12/16 17:36, Bill Burke wrote:
>> > > I just noticed that my local build fails while
>> travis passes.
>> > The bug is
>> > > really something travis should have picked up,
>> specifically the
>> > > PartialImportsTest was removing an identity
>> provider. The JPA
>> > > removeIdenittyProviderByAlias method was wrong
>> as it was trying
>> > to load
>> > > an IdentityProviderModel after it was removed
>> thus resulting in a
>> > > Hibernate error. Travis did not pick this up
>> which makes me
>> > wonder if
>> > > the test is even running.
>> > >
>> > > FYI, i have a pull request that fixes this that
>> is incoming. The
>> > bug,
>> > > not travis.
>> > >
>> > > Bill
>> > >
>> > >
>> > > _______________________________________________
>> > > keycloak-dev mailing list
>> > > keycloak-dev(a)lists.jboss.org
>> <mailto:keycloak-dev@lists.jboss.org>
>> <mailto:keycloak-dev@lists.jboss.org
>> <mailto:keycloak-dev@lists.jboss.org>>
>> > >
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>>
>> >
>> >
>> > _______________________________________________
>> > keycloak-dev mailing list
>> > keycloak-dev(a)lists.jboss.org
>> <mailto:keycloak-dev@lists.jboss.org>
>> <mailto:keycloak-dev@lists.jboss.org
>> <mailto:keycloak-dev@lists.jboss.org>>
>> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>> >
>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>>
>> >
>> >
>> >
>> >
>> > --
>> >
>> > --Hynek
>>
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev(a)lists.jboss.org
>> <mailto:keycloak-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>> <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>>
>>
>
>
1:55 a.m.
On 13 December 2016 at 08:50, Marek Posolda <mposolda(a)redhat.com> wrote:
On 13/12/16 08:26, Stian Thorgersen wrote:
On 12 December 2016 at 21:44, Marek Posolda <mposolda(a)redhat.com> wrote:
> On 12/12/16 17:51, Stian Thorgersen wrote:
>
> It should run at least what it used to run before I broke it up into
> multiple parallell jobs (seems I did a poor job when I did that and messed
> some stuff up). That can be fixed straight away, but adding more tests (new
> adapter tests, clustering tests, console tests, etc.) we should probably
> review later as it may impact stability and time to test a PR on Travis.
>
> +1 for better review. Few things:
>
> * Unit tests in the modules outside testsuite are not executed during
> travis build. For example tests under core/src/test/java .
>
> * Everything in testsuite/integration and
testsuite/integration-arquillian/tests/base
> is running in Travis build ATM. However if we add new package with tests
> into the testsuite (For example org.keycloak.testsuite.zzz.NewTest), it
> is possible that it won't be running due to missing filter in the
> travis-run-tests.sh for the package starting with "z" . I hope we can have
> the list of packages by travis-run-tests.sh to be more dynamic.
>
> * IMO it is sufficient that travis will run all the tests (including
> adapters and cluster) on embedded undertow. Running the tests on Wildfly is
> unecessary overhead, which will slightly increase the travis build time (as
> it will need to build distribution, start server etc) . There is not much
> added value as 95 % of regressions are catched with undertow too.
>
-1 We've seen this many times where Travis is happy, but the distribution
is broken. This happened several times last year so we need to prevent this.
Isn't it better to have travis run just on "simple" environment for each
PR and then rely on Central CI daily jobs to run the tests on real servers?
Even if we setup travis to run adapter tests on Wildfly, it won't catch all
the other broken adapters (EAP6, Tomcat, Jetty, ...).
IMO it is generally better to have CI for each PR, which will be able to
catch 95% of regressions (not necessarily 100%) and doesn't take hours to
wait for every PR. And then separate daily jobs to test everything.
Yes, we can't run it all on Travis, but I don't think making it run with WF
is going to add that much overhead (I think we're talking a few min). If
it's a big overhead then we should keep it on Undertow for sure.
Marek
>
> * I've added some undertow adapter tests to be executed during default
> travis build. They are all in the package org.keycloak.testsuite.adapter
> .undertow.servlet . Problem is, that not all adapter tests are executed
> as it will require creating subclass for every "Abstract" class with test
> and I just didn't create all the classes for embedded undertow. I've
> created JIRA https://issues.jboss.org/browse/KEYCLOAK-4069 to omprove
> adapter tests to avoid "dead" modules and many classes needed for single
> adapter test.
>
> Marek
>
> On 12 December 2016 at 17:46, Bill Burke <bburke(a)redhat.com> wrote:
>
>> This wasn't just the adapter tests, initially it was the
>> PartialImportsTest which lives under org.keycloak.testsuite.admin somewhere
>> and should have run and should have failed.
>>
>> https://issues.jboss.org/browse/KEYCLOAK-4068
>>
>>
>> On 12/12/16 8:24 AM, Stian Thorgersen wrote:
>>
>> Let's review the tests ran by Travis after the new year, but with Travis
>> having the option to run groups of tests in parallel we should be able to
>> tests more including adapters and console. Travis should also be changed to
>> tests with the full KC server and WildFly for adapters rather than embedded
>> Undertow.
>>
>> On 5 December 2016 at 16:32, Bill Burke <bburke(a)redhat.com> wrote:
>>
>>> These broke because they weren't part of the main build. I thought they
>>> were just dead code because when I did a "Find Usages" for them,
nothing
>>> came up. Minimally, things should at least be compiled with the main
>>> build, that way when refactorings happen, somebody doesn't delete a test
>>> dependency by accident.
>>>
>>>
>>> On 12/5/16 6:53 AM, Hynek Mlnarik wrote:
>>> > Speaking of the tests, we seem not to run any of the adapter test
>>> > suites. I believe that running at least adapter tests for wildfly
>>> > would be beneficial, preventing e.g. [1]. WDYT?
>>> >
>>> > [1] https://issues.jboss.org/browse/KEYCLOAK-4017
>>> >
>>> > On Mon, Dec 5, 2016 at 8:39 AM, Marek Posolda <mposolda(a)redhat.com
>>> > <mailto:mposolda@redhat.com>> wrote:
>>> >
>>> > Fyi. On Friday afternoon, I've found the issue that travis
didn't
>>> run
>>> > all the tests from the testsuite. And PartialImportTest was the
>>> one,
>>> > which wasn't executed. See
>>> > https://issues.jboss.org/browse/KEYCLOAK-4021
>>> > <https://issues.jboss.org/browse/KEYCLOAK-4021>
>>> >
>>> > However with the travis fix, all the tests were passing (including
>>> > PartialImportTest). So it seems this test was just failing
>>> > randomly (not
>>> > always)?
>>> >
>>> > Now I can see in latest travis build that PartialImportTest
>>> passes.
>>> >
>>> > Marek
>>> >
>>> >
>>> >
>>> >
>>> > On 03/12/16 17:36, Bill Burke wrote:
>>> > > I just noticed that my local build fails while travis passes.
>>> > The bug is
>>> > > really something travis should have picked up, specifically
the
>>> > > PartialImportsTest was removing an identity provider. The JPA
>>> > > removeIdenittyProviderByAlias method was wrong as it was
trying
>>> > to load
>>> > > an IdentityProviderModel after it was removed thus resulting
in
>>> a
>>> > > Hibernate error. Travis did not pick this up which makes me
>>> > wonder if
>>> > > the test is even running.
>>> > >
>>> > > FYI, i have a pull request that fixes this that is incoming.
The
>>> > bug,
>>> > > not travis.
>>> > >
>>> > > Bill
>>> > >
>>> > >
>>> > > _______________________________________________
>>> > > keycloak-dev mailing list
>>> > > keycloak-dev(a)lists.jboss.org <mailto:keycloak-dev@lists.jbo
>>> ss.org>
>>> > > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>>> >
>>> >
>>> > _______________________________________________
>>> > keycloak-dev mailing list
>>> > keycloak-dev(a)lists.jboss.org
<mailto:keycloak-dev@lists.jboss.org
>>> >
>>> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>> > <https://lists.jboss.org/mailman/listinfo/keycloak-dev>
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > --Hynek
>>>
>>> _______________________________________________
>>> keycloak-dev mailing list
>>> keycloak-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>
>>
>>
>>
>
>
9:27 a.m.
Yeah, I fixed a bug in RealmAdapter that was causing the failure.
On 12/5/16 2:39 AM, Marek Posolda wrote:
Fyi. On Friday afternoon, I've found the issue that travis
didn't run
all the tests from the testsuite. And PartialImportTest was the one,
which wasn't executed. See https://issues.jboss.org/browse/KEYCLOAK-4021
However with the travis fix, all the tests were passing (including
PartialImportTest). So it seems this test was just failing randomly
(not always)?
Now I can see in latest travis build that PartialImportTest passes.
Marek
On 03/12/16 17:36, Bill Burke wrote:
> I just noticed that my local build fails while travis passes. The bug is
> really something travis should have picked up, specifically the
> PartialImportsTest was removing an identity provider. The JPA
> removeIdenittyProviderByAlias method was wrong as it was trying to load
> an IdentityProviderModel after it was removed thus resulting in a
> Hibernate error. Travis did not pick this up which makes me wonder if
> the test is even running.
>
> FYI, i have a pull request that fixes this that is incoming. The bug,
> not travis.
>
> Bill
>
>
> _______________________________________________
> keycloak-dev mailing list
> keycloak-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/keycloak-dev
2691
days inactive
2701
days old
15 comments
4 participants
participants (4)
-
Bill Burke -
Hynek Mlnarik -
Marek Posolda -
Stian Thorgersen