Hi all,

Wei recommended me to start further discussion about viability of making E2E tests for regression testing of our SDKs. There was some debate around it, so I'm putting it also to the mailing list.

I have done Appium UI test of the example app included in AG Android SDK [1]. It has been done for Auth functionality. This test checks happy path of login and logout. 

Is it worth doing it this way? Is it worth doing it at least for some functionality? 

My take TLDR:

Pros:
+ it tests real end-user/developer usage of the SDK in the real device environment

In the middle:

= it requires running emulator or connected device to the test server

Cons:
- test stability (it can go wrong because of load on the test server)

- test speed

- UI change volatility can break the test - this is partially avoided by using page objects [2] pattern

I think the real environment is clear benefit, for OpenID authentication, it's difficult to do it other way than from UI and using real browser. I don't know if this is true for other functionality. But writing tests at least for happy paths can be beneficial.

refs:
[1] https://github.com/aerogear/aerogear-sdk-e2e-tests/pull/1
[2] https://martinfowler.com/bliki/PageObject.html

--

VOJTĚCH SÁZEL

SENIOR SOFTWARE ENGINEER, RED HAT MOBILE

Red Hat 

Remote Czech Republic

vsazel@redhat.com    IM: vsazel