[keycloak-dev] Why aren't tomcat and jetty tests run!?

[Marek Posolda]

Arquillian can support that. All we need is just different 
implementations of arquillian DeployableContainer interface to represent 
various containers. I've managed to have adapter tests already running 
on embedded undertow and everything is running in single JVM. 
Development and debugging is fine with that.

Having the test+server+adapter in single JVM just rocks during 
development. Problem is, that it won't handle all type of errors. For 
example if Jetty adapter ZIP distribution is broken (eg. some JAR file 
is missing in it), the Jetty adapter tests, which uses embedded Jetty 
won't catch that. For those reasons, also QA team needs to test with 
"real" containers, not with the embedded ones.



On 08/12/16 15:05, Bill Burke wrote:
>
>
> On 12/8/16 6:48 AM, Marek Posolda wrote:
>> They are not included in the script running travis 
>> https://github.com/keycloak/keycloak/blob/master/travis-run-tests.sh#L4
>>
>> It seems wee should add them here.
>>
>> From the long run, it will be better to remove them (as they rely on 
>> the old testsuite) and rather have them running on the 
>> integration-arquillian testsuite. I know it sucks during development 
>> that Jetty itself doesn't run in same JVM like the test (debugging 
>> etc). But I hope we can have the solution, which will allow to choose 
>> whether Jetty (or hopefully some other servers) are running in same 
>> JVM or different JVM.
>>
> Unless arquillian can support running these adapter tests within an 
> IDE, they need to stay. 
Arquillian can support that. All we need is just different 
implementations of arquillian DeployableContainer interface to represent 
various containers. I've managed to have adapter tests already running 
on embedded undertow and everything is running in single JVM. 
Development and debugging is fine with that.
> In general, More often than not, I don't like the arquillian style 
> we've enforced because it takes a lot longer to write tests, is often 
> nearly impossible to do everything through a REST interface, and is 
> often really hard to debug.  Maybe its just the types of things I'm 
> testing.  Some of the tests should not have been ported to arquillian, 
> or, at least shouldn't have been ported to using a REST interface for 
> the entire test.
Yeah, having the test+server+adapter in single JVM just rocks during 
development. Problem is, that it won't handle all type of errors. For 
example if Jetty adapter ZIP distribution is broken (eg. some JAR file 
is missing), the Jetty adapter tests, which uses embedded Jetty, won't 
catch that. For those and similar reasons, also QA team needs to test 
with "real" containers, not with the embedded ones.

IMO ideal is, if we can have both possibilities. Test with embedded 
container during development, but also has possibility to run the same 
test against external container in different JVM. Arquillian testsuite 
can support both approaches, but the old testsuite doesn't support the 
external...

I agree some tests shouldn't use REST endpoints. Like the model tests 
from the old testsuite. But they still need to be able to run on 
external environment though. I guess we can have something like the 
"bridge" test, which will just deploy the model test to the REST 
endpoint and you will be able to test things inside this REST endpoint 
on server side? So you will be able to use all the stuff like model 
classes etc.

Marek
>
> Bill