[keycloak-dev] Why aren't tomcat and jetty tests run!?

[Marek Posolda]

IMO ideal is, if we can have both possibilities. Test with embedded 
container during development, but also has possibility to run the same 
test against external container in different JVM. Arquillian testsuite 
can support both approaches, but the old testsuite doesn't support the 
external...

I agree some tests shouldn't use REST endpoints. Like the model tests 
from the old testsuite. But they still need to be able to run on 
external environment though. I guess we can have something like the 
"bridge" test, which will just deploy the model test to the REST 
endpoint and you will be able to test things inside this REST endpoint 
on server side? So you will be able to use all the stuff like model 
classes etc.

Marek

On 08/12/16 17:47, Marek Posolda wrote:
> Arquillian can support that. All we need is just different 
> implementations of arquillian DeployableContainer interface to 
> represent various containers. I've managed to have adapter tests 
> already running on embedded undertow and everything is running in 
> single JVM. Development and debugging is fine with that.
>
> Having the test+server+adapter in single JVM just rocks during 
> development. Problem is, that it won't handle all type of errors. For 
> example if Jetty adapter ZIP distribution is broken (eg. some JAR file 
> is missing in it), the Jetty adapter tests, which uses embedded Jetty 
> won't catch that. For those reasons, also QA team needs to test with 
> "real" containers, not with the embedded ones.
>
>
>
> On 08/12/16 15:05, Bill Burke wrote:
>>
>>
>> On 12/8/16 6:48 AM, Marek Posolda wrote:
>>> They are not included in the script running travis 
>>> https://github.com/keycloak/keycloak/blob/master/travis-run-tests.sh#L4
>>>
>>> It seems wee should add them here.
>>>
>>> From the long run, it will be better to remove them (as they rely on 
>>> the old testsuite) and rather have them running on the 
>>> integration-arquillian testsuite. I know it sucks during development 
>>> that Jetty itself doesn't run in same JVM like the test (debugging 
>>> etc). But I hope we can have the solution, which will allow to 
>>> choose whether Jetty (or hopefully some other servers) are running 
>>> in same JVM or different JVM.
>>>
>> Unless arquillian can support running these adapter tests within an 
>> IDE, they need to stay. 
> Arquillian can support that. All we need is just different 
> implementations of arquillian DeployableContainer interface to 
> represent various containers. I've managed to have adapter tests 
> already running on embedded undertow and everything is running in 
> single JVM. Development and debugging is fine with that.
>> In general, More often than not, I don't like the arquillian style 
>> we've enforced because it takes a lot longer to write tests, is often 
>> nearly impossible to do everything through a REST interface, and is 
>> often really hard to debug.  Maybe its just the types of things I'm 
>> testing.  Some of the tests should not have been ported to 
>> arquillian, or, at least shouldn't have been ported to using a REST 
>> interface for the entire test.
> Yeah, having the test+server+adapter in single JVM just rocks during 
> development. Problem is, that it won't handle all type of errors. For 
> example if Jetty adapter ZIP distribution is broken (eg. some JAR file 
> is missing), the Jetty adapter tests, which uses embedded Jetty, won't 
> catch that. For those and similar reasons, also QA team needs to test 
> with "real" containers, not with the embedded ones.
>
> IMO ideal is, if we can have both possibilities. Test with embedded 
> container during development, but also has possibility to run the same 
> test against external container in different JVM. Arquillian testsuite 
> can support both approaches, but the old testsuite doesn't support the 
> external...
>
> I agree some tests shouldn't use REST endpoints. Like the model tests 
> from the old testsuite. But they still need to be able to run on 
> external environment though. I guess we can have something like the 
> "bridge" test, which will just deploy the model test to the REST 
> endpoint and you will be able to test things inside this REST endpoint 
> on server side? So you will be able to use all the stuff like model 
> classes etc.
>
> Marek
>>
>> Bill
>
>