[keycloak-dev] WebAuthn testing

Mon Sep 9 07:36:11 EDT 2019

Hi Noji,

Thanks a lot for your feedback and sorry for late response. Some notes 
inline.

On 22. 08. 19 18:20, Yoshikazu Nojima wrote:
> Hello Marek,
>
> I'm Noji, main author of WebAuthn4J library.
>
> It is webauthn library's responsibility to properly interpret the
> WebAuthn request and validate it in the way defined in the
> specification.
> WebAuthn4J will do its best.
> In order to ensure conformance with the specifications, we check
> WebAuthn4J pass the FIDO Alliance conformance test tool.
>
> ## (a) Chrome Testing API / (b) WebDriver Authenticator Extension
>
> I suggests WebAuthn4J library caller like KeyCloak to put a weight
> caller/WebAuthn4J integration tests or E2E tests.
>  From the perspective, using the Chrome Testing API is a good option,
> and if circumstance allows, WebDriver Authenticator Extension would be
> great as it can test client side javascript part.

Yes, I agree to rely mostly on webauthn4j. However in Keycloak, we may 
still need to test some integration scenarios somehow.

For example ensure that when some error scenario happens, (for example 
"platform" authenticator requested, but not having any "platform" 
authenticator available), there is proper error page displayed by 
Keycloak UI and not some broken page with the exception stacktrace on it 
:) During my manual testing, I didn't simulate any error scenario. But 
in few cases, chrome just crashed for me when used some unsupported option.

Another example could be to support passwordless scenario in Keycloak 
when WebAuthn is used without previously known userId sent by the 
Keycloak. I've tried to manually test with chrome testing API with 
"requireResidentKey" switched to true. Also tried to send request 
"navigator.credentials.get" without any "allowCredentials" in the list. 
Both those calls result in chrome browser crash ;)

I may probably submit bug report to chrome. Do you accidentally know 
some more about known limitations of chrome web authentication testing 
API or about submitting bug reports to it etc? I have chrome 
76.0.3809.87 on ubuntu 18.04.

>
> ## (c) FIDO Alliance conformance tool
>
> FIDO Alliance conformance tool is an official test suite and has good
> test cases, but it is Electron based desktop tool.
> It cannot be fit in CI environment. I execute the tool manually in my laptop.
Good to know, Thanks!
>
> ## (d) how WebAuthn4J library do
>
> WebAuthn4J has unit tests per classes, which is executed in the CI.
> FIDO Alliance conformance tool is also executed, but the execution is
> not automated.
> In order to increase the test coverage in CI, I'm developing
> authenticator emulator (webauthn4j-test) in Java, and WebAuthn4J use
> it for integration test.
> ex.
> https://github.com/webauthn4j/webauthn4j/blob/master/webauthn4j-core/src/test/java/integration/scenario/FIDOU2FAuthenticatorRegistrationValidationTest.java#L82-L95
>
> authenticator emulator (webauthn4j-test) is still in development and
> focusing on internal use for now, but it might be good option when it
> finished.

Thanks again.

Marek

>
> Regards, Noji
>
> 2019年8月20日(火) 22:10 Marek Posolda <mposolda at redhat.com>:
>> We have community PR for adding initial webauthn support [1]. The PR is
>> great work for the start IMO (anyone welcome to review PR :)
>>
>> But among other things, it touches one of the challenges for webauthn
>> support, which is automated testing. So far, the PR introduces testing
>> webauthn with the Chrome WebDriver and the Chrome testing API [2] . The
>> Chrome Testing API allows chrome browser to "simulate" the webauthn
>> authenticator without having the real webauthn authenticator device
>> available.
>>
>> IMO it is fine to use the Chrome Testing API for the parts, which are
>> more focus on the Keycloak integration itself rather than on the details
>> of WebAuthn configurations. From one point of view, it may be sufficient
>> to test just with Chrome Testing API as Keycloak is just WebAuthn
>> Relying party and it doesn't need to care which underlying WebAuthn
>> authenticator was used - that's more the responsibilty of the platform
>> and web browser, which sends HTTP requests to Keycloak. However I afraid
>> that there may be some cases where this is not sufficient and we may
>> need some more proper interoperability testing. Especially for testing
>> the details of webauthn specification. For example if Keycloak sends the
>> WebAuthn registration request with "userVerification: required", we may
>> need to ensure that authentication will fail if userVerification flag is
>> not provided by the authenticator etc.
>>
>> So what are possible options?
>>
>> (a) Use just testing with chrome testing API and rely on the underlying
>> webauthn library (webauthn4j) to hide the webauthn specification
>> interoperability testing. This may be the easy option, but not sure
>> whether sufficient?
>>
>> (b) Look at WebDriver Authenticator Extension and check if/when it is
>> possible to use it. Details here [2] (also see the open PR linked from
>> there). I am not sure when this will be available for us to be able to
>> consume it, so not sure if this is an option...
>>
>> (c) Look at FIDO Alliance certification conformance [3] and investigate
>> if it provides some automated tests, which can be somehow
>> used/forked/duplicated to our testsuite
>>
>> (d) Check how webauthn4j library (and eventually some other libraries)
>> are doing automated tests and inspire from them if possible.
>>
>> (e) Test manually with some defined set of webauthn authenticators on
>> some defined set of platforms/browsers. Then investigate which HTTP
>> requests were sent to Keycloak from the browser for various
>> configuration options and use those manual tests to create some
>> "client", which will just use Apache HTTP Client under the covers to
>> directly send HTTP requests to Keycloak. In other words, just simulate
>> the browser+OS+Authenticator. This is not ideal, however possibly useful
>> as a fallback option?
>>
>> (f) any better option?
>>
>> [1] https://github.com/keycloak/keycloak/pull/6248/files
>>
>> [2]
>> https://chromium.googlesource.com/chromium/src/+/a82bdb76536b5fce185734225128ca3ccb5a2e8c
>>
>> [3] https://github.com/w3c/webauthn/issues/1236
>>
>> [4]
>> https://fidoalliance.org/certification/functional-certification/conformance/
>>
>> Marek
>>
>> _______________________________________________
>> keycloak-dev mailing list
>> keycloak-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/keycloak-dev