Hi. I would like to discuss how to extend AS7 testsuite, my suggestion is to create 2 new modules: -- management Tests for management APIs, probably separate submodules for cli, http api and java api tests. -- multi-node Tests with several instances (managed by DC or several standalone servers) on one machine. Cluster, failover, session replication tests should be present in this module. ARQ with DC support is needed, start/stop/kill support for instances managed by DC is needed too (ARQ-336 is designed for standalone), maybe start/stop/kill for DC will be required too. Module structure should be similar to DOC-69049 [1] with submodules to share tests, prepare servers and to execute tests. What do you think about suggested changes? Rosta [1] https://docspace.corp.redhat.com/docs/DOC-69049 _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

Btw, the console test suite contains all the bits and pieces to test the HTTP interface already: https://github.com/heiko-braun/as7-console/tree/master/gui/src/test/java/... It pretty similar to the default tests that rely on the java API and wire format. Just that this one uses the base64 encoded protocol and the HTTP endpoint. Maybe you want to build upon that. Ike On Jul 19, 2011, at 2:12 PM, Rostislav Svoboda wrote: > Hi. > > I would like to discuss how to extend AS7 testsuite, my suggestion is to create 2 new modules: > > -- management > Tests for management APIs, probably separate submodules for cli, http api and java api tests. > > -- multi-node > Tests with several instances (managed by DC or several standalone servers) on one machine. > Cluster, failover, session replication tests should be present in this module. > ARQ with DC support is needed, start/stop/kill support for instances managed by DC is needed too (ARQ-336 is designed for standalone), maybe start/stop/kill for DC will be required too. > > Module structure should be similar to DOC-69049 [1] with submodules to share tests, prepare servers and to execute tests. > > > What do you think about suggested changes? > > Rosta > > [1] https://docspace.corp.redhat.com/docs/DOC-69049 > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

Hi. I would like to discuss how to extend AS7 testsuite, my suggestion is to create 2 new modules: -- management Tests for management APIs, probably separate submodules for cli, http api and java api tests.

-- multi-node Tests with several instances (managed by DC or several standalone servers) on one machine. Cluster, failover, session replication tests should be present in this module.

ARQ with DC support is needed, start/stop/kill support for instances managed by DC is needed too (ARQ-336 is designed for standalone), maybe start/stop/kill for DC will be required too.

Module structure should be similar to DOC-69049 [1] with submodules to share tests, prepare servers and to execute tests.

What do you think about suggested changes? Rosta [1] https://docspace.corp.redhat.com/docs/DOC-69049 _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

Hi Brian, thank you for your reply, comments are inline. > On 7/19/11 7:12 AM, Rostislav Svoboda wrote: >> Hi. >> >> I would like to discuss how to extend AS7 testsuite, my suggestion >> is to create 2 new modules: >> >> -- management >> Tests for management APIs, probably separate submodules for cli, >> http api and java api tests. >> > > Makes sense. The existing testsuite/domain module could be moved into > here. It's essentially testing management of a domain via the java > api. > > IMO management tests of standalone servers should be segregated from > tests of a managed domain. Agree, submodules should be named domain-http, domain-cli, domain-java, standalone-http, standalone-cli and standalone-java. Maybe one submodules for shared tests and resources. > >> -- multi-node >> Tests with several instances (managed by DC or several standalone >> servers) on one machine. >> Cluster, failover, session replication tests should be present in >> this module. > > The testsuite/clustering module is for this purpose. So we should propose some changes for this module, it's currently running against one standalone server and contains only one test.

I think testsuite/clustering will have several submodules to cover different scenarios, positive test can have server execution automatically managed by ARQ. Negative tests (failovers) must have server execution managed by ARQ but controlled from testscases (ARQ-336 [1], FailoverTestCase.java [2]).

> >> ARQ with DC support is needed, start/stop/kill support for instances >> managed by DC is needed too (ARQ-336 is designed for standalone), >> maybe start/stop/kill for DC will be required too. >> > > I regard that as a nice to have, not a requirement. HA functionality > is > orthogonal to how the servers are managed; a testsuite of Paul's HA > features based on launching multiple standalone servers is ok. The > actual HA services inside the servers have no clue whether they are > running in a standalone server or a managed domain. OK, I agree with that, DC is only about server management, no effect on AS7 functionality. My point about ARQ with DC support is more about message to users, we really recommend to use DC, but we don't use it in our tests ...

Several standalone server in cluster are sufficient to verify that clustering functionality is working properly. > Note also that ARQ with DC support is not needed for management tests of domain. But I steel need ARQ to start DC (using managed mode). Could you please explain how do you mean your note?

>> Module structure should be similar to DOC-69049 [1] with submodules >> to share tests, prepare servers and to execute tests. >> > > Hopefully in 95%+ of cases "prepare servers" can consist of altering > the > server/domain launch command to point at the appropriate writable > directory, config file and module path. We don't want to store configuration xml files in repository, we would like to prepare necessary configuration using management API. We have quite bad experience with configuration stored in xml files for testing, at least when development is still going on. Plan is to keep "prepare servers" as simple as possible. But still I want to have appropriate prepare module for each test module to keep it straightforward when searching for possible problem - in one module is configured server and in second is test which should work.

If I'm not wrong, pointing ARQ to custom configuration xml file can be done in arquillian.xml by defining element property named serverConfig under configuration element. >> >> What do you think about suggested changes? >> >> Rosta >> >> [1] https://docspace.corp.redhat.com/docs/DOC-69049 >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > > -- > Brian Stansberry > Principal Software Engineer > JBoss by Red Hat > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev -- Rostislav Svoboda JBoss QA, Brno [1] https://issues.jboss.org/browse/ARQ-336 [2] https://github.com/mgencur/edg-tests/blob/master/failover/src/test/java/o...

> On 7/21/11 4:08 AM, Rostislav Svoboda wrote: > > Hi Brian, thank you for your reply, comments are inline. > > > >> On 7/19/11 7:12 AM, Rostislav Svoboda wrote: > >>> Hi. > >>> > >>> I would like to discuss how to extend AS7 testsuite, my suggestion > >>> is to create 2 new modules: > >>> > >>> -- management > >>> Tests for management APIs, probably separate submodules for cli, > >>> http api and java api tests. > >>> > >> > >> Makes sense. The existing testsuite/domain module could be moved > >> into > >> here. It's essentially testing management of a domain via the java > >> api. > >> > >> IMO management tests of standalone servers should be segregated > >> from > >> tests of a managed domain. > > > > Agree, submodules should be named domain-http, domain-cli, > > domain-java, standalone-http, standalone-cli and standalone-java. > > Maybe one submodules for shared tests and resources. > > > >> > >>> -- multi-node > >>> Tests with several instances (managed by DC or several standalone > >>> servers) on one machine. > >>> Cluster, failover, session replication tests should be present in > >>> this module. > >> > >> The testsuite/clustering module is for this purpose. > > > > So we should propose some changes for this module, it's currently > > running against one standalone server and contains only one test. > > No argument. And I may have spoken out of turn. Perhaps Paul Ferraro > had > some other intent for this module than the kind of multi-server > testing > you're talking about (but I don't think so. ;) ) > > > I think testsuite/clustering will have several submodules to cover > > different scenarios, positive test can have server execution > > automatically managed by ARQ. Negative tests (failovers) must have > > server execution managed by ARQ but controlled from testscases > > (ARQ-336 [1], FailoverTestCase.java [2]). > > > > Paul's your man for input on this. I just wanted to point out there's > an existing module. Let's wait for Paul's reply.

> >> > >>> ARQ with DC support is needed, start/stop/kill support for > >>> instances > >>> managed by DC is needed too (ARQ-336 is designed for standalone), > >>> maybe start/stop/kill for DC will be required too. > >>> > >> > >> I regard that as a nice to have, not a requirement. HA > >> functionality > >> is > >> orthogonal to how the servers are managed; a testsuite of Paul's HA > >> features based on launching multiple standalone servers is ok. The > >> actual HA services inside the servers have no clue whether they are > >> running in a standalone server or a managed domain. > > > > OK, I agree with that, DC is only about server management, no effect > > on AS7 functionality. My point about ARQ with DC support is more > > about message to users, we really recommend to use DC, but we don't > > use it in our tests ... > > I agree with the message to users point, but I just want to emphasize > we > have limited time and a large number of hard requirements and AFAIK > this > is not one of them. So always be cautious about adding new > requirements, > particularly ones that become blockers earlier in the process. > > We have a nice testing story already with AS 7 and Arquillian. And > it's > improving rapidly. So if support for using a managed domain is on the > roadmap but doesn't get in AS 7.1 or comes in too late to be the basis > of our HA testsuite, I don't think that's too bad of a message. ARQ without DC support is not blocker for us in this case, we can do majority of clustering/HA testing with several standalone servers. For complicated tests we wanted to use DC for simpler control of the cluster. There is motivation to keep everything Java based and located in one testsuite. We have other tools like SmartFrog (SF) which we use for HA testing but we planned to drop SF usage for AS7 tests. It's not easy to develop components and test templates, debugging is really painful. Maintenance of SF tests and components is resource consuming, not speaking about learning curve. That's our main motivation for ARQ with DC support. > > Several standalone server in cluster are sufficient to verify that > > clustering functionality is working properly. > > > >> Note also that ARQ with DC support is not needed for management > >> tests of domain. > > > > But I steel need ARQ to start DC (using managed mode). Could you > > please explain how do you mean your note? > > > > No, you don't. :) See the testsuite/domain module. It uses a couple > classes to control the DC (and a slave HostController.) They are in > the > arquillian/container-managed-domain module but putting them there was > a > mistake on my part. They don't actually have anything really to do > with > ARQ at this point. I put them there because I dreamed I'd have time to > also use them as utility classes in a real ARQ container for a managed > domain. OK, see your point. > A bit more on domain management tests and ARQ. I see ARQ as being > oriented toward testing deployments. Most domain management testing is > not about testing deployments. Like the HA point above, how the > deployments behave inside a server is orthogonal to managed domain vs > standalone. All the deep testing of how deployments behave can be done > with a standalone server. > > When testing the domain management, I don't want to have to deal with > deployments unless there is a real need. I want to start/stop domain > processes and interact with them via the management API. For sure I > don't want to have to deploy fake deployments to a domain just to > satisfy an ARQ requirement. (We actually do that in some standalone > tests of the management API.) Testing against standalone server is sufficient in first phase, we need to make sure that management for DC is working too. So we need a way how to start DC before our tests, ARQ support this task seems to be the proper way. I found filled JIRA for this usage (https://issues.jboss.org/browse/ARQ-498), maybe it should be cloned to AS7. > >>> Module structure should be similar to DOC-69049 [1] with > >>> submodules > >>> to share tests, prepare servers and to execute tests. > >>> > >> > >> Hopefully in 95%+ of cases "prepare servers" can consist of > >> altering > >> the > >> server/domain launch command to point at the appropriate writable > >> directory, config file and module path. > > > > We don't want to store configuration xml files in repository, we > > would like to prepare necessary configuration using management API. > > We have quite bad experience with configuration stored in xml files > > for testing, at least when development is still going on. > > Plan is to keep "prepare servers" as simple as possible. But still I > > want to have appropriate prepare module for each test module to keep > > it straightforward when searching for possible problem - in one > > module is configured server and in second is test which should work. > > > > Sounds good. I didn't mean at all to imply that the "prepare" module > idea is wrong. > > The biggest thing to avoid is copying the > build/target/jboss-xxx/modules > multiple times into different testsuite dirs. We were doing that for a > couple weeks before CR1 and it wasted a LOT of time (and disk.) Good point, that will speed up testsuite. > Re: preparing necessary config using the management API, keep an eye > on https://issues.jboss.org/browse/AS7-346 and subtasks. Added as watcher :) > > > If I'm not wrong, pointing ARQ to custom configuration xml file can > > be done in arquillian.xml by defining element property named > > serverConfig under configuration element. > > > >>> > >>> What do you think about suggested changes? > >>> > >>> Rosta > >>> > >>> [1] https://docspace.corp.redhat.com/docs/DOC-69049 > >>> _______________________________________________ > >>> jboss-as7-dev mailing list > >>> jboss-as7-dev(a)lists.jboss.org > >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > >> > >> > >> -- > >> Brian Stansberry > >> Principal Software Engineer > >> JBoss by Red Hat > >> _______________________________________________ > >> jboss-as7-dev mailing list > >> jboss-as7-dev(a)lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > > > -- > > Rostislav Svoboda > > JBoss QA, Brno > > > > [1] https://issues.jboss.org/browse/ARQ-336 > > [2] > > https://github.com/mgencur/edg-tests/blob/master/failover/src/test/java/o... > > -- > Brian Stansberry > Principal Software Engineer > JBoss by Red Hat Thanks for all comments. -- Rostislav Svoboda JBoss QA, Brno

On 7/22/11 2:34 AM, Rostislav Svoboda wrote: <snip/> >>>>> ARQ with DC support is needed, start/stop/kill support for >>>>> instances >>>>> managed by DC is needed too (ARQ-336 is designed for standalone), >>>>> maybe start/stop/kill for DC will be required too. >>>>> >>>> I regard that as a nice to have, not a requirement. HA >>>> functionality >>>> is >>>> orthogonal to how the servers are managed; a testsuite of Paul's HA >>>> features based on launching multiple standalone servers is ok. The >>>> actual HA services inside the servers have no clue whether they are >>>> running in a standalone server or a managed domain. >>> OK, I agree with that, DC is only about server management, no effect >>> on AS7 functionality. My point about ARQ with DC support is more >>> about message to users, we really recommend to use DC, but we don't >>> use it in our tests ... >> I agree with the message to users point, but I just want to emphasize >> we >> have limited time and a large number of hard requirements and AFAIK >> this >> is not one of them. So always be cautious about adding new >> requirements, >> particularly ones that become blockers earlier in the process. >> >> We have a nice testing story already with AS 7 and Arquillian. And >> it's >> improving rapidly. So if support for using a managed domain is on the >> roadmap but doesn't get in AS 7.1 or comes in too late to be the basis >> of our HA testsuite, I don't think that's too bad of a message. > ARQ without DC support is not blocker for us in this case, we can do majority of clustering/HA testing with several standalone servers. > For complicated tests we wanted to use DC for simpler control of the cluster. There is motivation to keep everything Java based and located in one testsuite. We have other tools like SmartFrog (SF) which we use for HA testing but we planned to drop SF usage for AS7 tests. It's not easy to develop components and test templates, debugging is really painful. Maintenance of SF tests and components is resource consuming, not speaking about learning curve. That's our main motivation for ARQ with DC support. > I certainly don't object to using the DC for this kind of stuff. :-) That is what it's for. Keep in mind that the DC can't be used to hard kill servers; it only exposes operations for controlled shutdown. I know that one of the things the QE smartfrog stuff does in some HA tests is hard kill servers.