2:14 a.m.
Hello,
Today we have a situation with the integration tests that is starting to be a concern.
Basically, there is a lack of integration testing that makes some potential bugs escapes
preliminary controls and it is showed in the ManageIQ.
I guess we could increase the integration test coverage overall and define some additional
processes to improve this.
These are some ideas commented with the team:
- Today, we have some coverage per component, but not at hawkular-services level. It could
be good if we can add more end-to-end tests at hawkular-services level.
- Before a component is released, it could be good if component can test the
hawkular-services integration suite to validate that there is nothing broken, or if it is,
start a discussion to get a consensus/tradeoff. Probably from isolated component view
everything is ok, but in the hawkular-services context it could be a potential problem.
(Not sure if the CI can help us here, extending running the new component against the
hawkular-services itests).
- This also happens with the clients, in the hawkular-client-ruby, we have a recorded set
of http calls (VCR cassettes). This works fine and it is great, but today we have some mix
of versions recorded (at least for the hawkular-services part). So, it would be great if
that can be normalized. For example, I think that for a new release of the
hawkular-services version, the CI could run the tests to validate if the recording are
still valid or not.
- The same situation happens at ManageIQ side, there are recordings of VCR cassettes for
different versions of hawkular-services (we have started to annotate the version), but it
happens that the tests can pass but that doesnt mean that the last hawkular-services
version will pass, so an action tasks could be that if the hawkular-client-ruby version is
changed, VCR tests could be re-recorded against for version validated for
hawkular-services.
These are just some ideas to start discussing.
I guess that the CI (travis or internally using torii) can help in some degree, but I have
lack of experience with them.
In any case, the goal of this is not slow any development, but just to have an early
indicator that if there is some change in the component, the overall system is notified
and we can address it better than if this is happening during a demo/final stage of a QE
task, for example.
Any thoughts about these ideas are welcome.
Lucas