[keycloak-dev] WebAuthn testing

[Marek Posolda]

We have community PR for adding initial webauthn support [1]. The PR is 
great work for the start IMO (anyone welcome to review PR :)

But among other things, it touches one of the challenges for webauthn 
support, which is automated testing. So far, the PR introduces testing 
webauthn with the Chrome WebDriver and the Chrome testing API [2] . The 
Chrome Testing API allows chrome browser to "simulate" the webauthn 
authenticator without having the real webauthn authenticator device 
available.

IMO it is fine to use the Chrome Testing API for the parts, which are 
more focus on the Keycloak integration itself rather than on the details 
of WebAuthn configurations. From one point of view, it may be sufficient 
to test just with Chrome Testing API as Keycloak is just WebAuthn 
Relying party and it doesn't need to care which underlying WebAuthn 
authenticator was used - that's more the responsibilty of the platform 
and web browser, which sends HTTP requests to Keycloak. However I afraid 
that there may be some cases where this is not sufficient and we may 
need some more proper interoperability testing. Especially for testing 
the details of webauthn specification. For example if Keycloak sends the 
WebAuthn registration request with "userVerification: required", we may 
need to ensure that authentication will fail if userVerification flag is 
not provided by the authenticator etc.

So what are possible options?

(a) Use just testing with chrome testing API and rely on the underlying 
webauthn library (webauthn4j) to hide the webauthn specification 
interoperability testing. This may be the easy option, but not sure 
whether sufficient?

(b) Look at WebDriver Authenticator Extension and check if/when it is 
possible to use it. Details here [2] (also see the open PR linked from 
there). I am not sure when this will be available for us to be able to 
consume it, so not sure if this is an option...

(c) Look at FIDO Alliance certification conformance [3] and investigate 
if it provides some automated tests, which can be somehow 
used/forked/duplicated to our testsuite

(d) Check how webauthn4j library (and eventually some other libraries) 
are doing automated tests and inspire from them if possible.

(e) Test manually with some defined set of webauthn authenticators on 
some defined set of platforms/browsers. Then investigate which HTTP 
requests were sent to Keycloak from the browser for various 
configuration options and use those manual tests to create some 
"client", which will just use Apache HTTP Client under the covers to 
directly send HTTP requests to Keycloak. In other words, just simulate 
the browser+OS+Authenticator. This is not ideal, however possibly useful 
as a fallback option?

(f) any better option?

[1] https://github.com/keycloak/keycloak/pull/6248/files

[2] 
https://chromium.googlesource.com/chromium/src/+/a82bdb76536b5fce185734225128ca3ccb5a2e8c

[3] https://github.com/w3c/webauthn/issues/1236

[4] 
https://fidoalliance.org/certification/functional-certification/conformance/

Marek