[jboss-as7-dev] AS7 testsuite extension
Brian Stansberry
brian.stansberry at redhat.com
Thu Jul 21 11:23:06 EDT 2011
On 7/21/11 4:08 AM, Rostislav Svoboda wrote:
> Hi Brian, thank you for your reply, comments are inline.
>
>> On 7/19/11 7:12 AM, Rostislav Svoboda wrote:
>>> Hi.
>>>
>>> I would like to discuss how to extend AS7 testsuite, my suggestion
>>> is to create 2 new modules:
>>>
>>> -- management
>>> Tests for management APIs, probably separate submodules for cli,
>>> http api and java api tests.
>>>
>>
>> Makes sense. The existing testsuite/domain module could be moved into
>> here. It's essentially testing management of a domain via the java
>> api.
>>
>> IMO management tests of standalone servers should be segregated from
>> tests of a managed domain.
>
> Agree, submodules should be named domain-http, domain-cli, domain-java, standalone-http, standalone-cli and standalone-java.
> Maybe one submodules for shared tests and resources.
>
>>
>>> -- multi-node
>>> Tests with several instances (managed by DC or several standalone
>>> servers) on one machine.
>>> Cluster, failover, session replication tests should be present in
>>> this module.
>>
>> The testsuite/clustering module is for this purpose.
>
> So we should propose some changes for this module, it's currently running against one standalone server and contains only one test.
No argument. And I may have spoken out of turn. Perhaps Paul Ferraro had
some other intent for this module than the kind of multi-server testing
you're talking about (but I don't think so. ;) )
> I think testsuite/clustering will have several submodules to cover different scenarios, positive test can have server execution automatically managed by ARQ. Negative tests (failovers) must have server execution managed by ARQ but controlled from testscases (ARQ-336 [1], FailoverTestCase.java [2]).
>
Paul's your man for input on this. I just wanted to point out there's an
existing module.
>>
>>> ARQ with DC support is needed, start/stop/kill support for instances
>>> managed by DC is needed too (ARQ-336 is designed for standalone),
>>> maybe start/stop/kill for DC will be required too.
>>>
>>
>> I regard that as a nice to have, not a requirement. HA functionality
>> is
>> orthogonal to how the servers are managed; a testsuite of Paul's HA
>> features based on launching multiple standalone servers is ok. The
>> actual HA services inside the servers have no clue whether they are
>> running in a standalone server or a managed domain.
>
> OK, I agree with that, DC is only about server management, no effect on AS7 functionality. My point about ARQ with DC support is more about message to users, we really recommend to use DC, but we don't use it in our tests ...
I agree with the message to users point, but I just want to emphasize we
have limited time and a large number of hard requirements and AFAIK this
is not one of them. So always be cautious about adding new requirements,
particularly ones that become blockers earlier in the process.
We have a nice testing story already with AS 7 and Arquillian. And it's
improving rapidly. So if support for using a managed domain is on the
roadmap but doesn't get in AS 7.1 or comes in too late to be the basis
of our HA testsuite, I don't think that's too bad of a message.
> Several standalone server in cluster are sufficient to verify that clustering functionality is working properly.
>
>> Note also that ARQ with DC support is not needed for management tests of domain.
>
> But I steel need ARQ to start DC (using managed mode). Could you please explain how do you mean your note?
>
No, you don't. :) See the testsuite/domain module. It uses a couple
classes to control the DC (and a slave HostController.) They are in the
arquillian/container-managed-domain module but putting them there was a
mistake on my part. They don't actually have anything really to do with
ARQ at this point. I put them there because I dreamed I'd have time to
also use them as utility classes in a real ARQ container for a managed
domain.
A bit more on domain management tests and ARQ. I see ARQ as being
oriented toward testing deployments. Most domain management testing is
not about testing deployments. Like the HA point above, how the
deployments behave inside a server is orthogonal to managed domain vs
standalone. All the deep testing of how deployments behave can be done
with a standalone server.
When testing the domain management, I don't want to have to deal with
deployments unless there is a real need. I want to start/stop domain
processes and interact with them via the management API. For sure I
don't want to have to deploy fake deployments to a domain just to
satisfy an ARQ requirement. (We actually do that in some standalone
tests of the management API.)
>>> Module structure should be similar to DOC-69049 [1] with submodules
>>> to share tests, prepare servers and to execute tests.
>>>
>>
>> Hopefully in 95%+ of cases "prepare servers" can consist of altering
>> the
>> server/domain launch command to point at the appropriate writable
>> directory, config file and module path.
>
> We don't want to store configuration xml files in repository, we would like to prepare necessary configuration using management API. We have quite bad experience with configuration stored in xml files for testing, at least when development is still going on.
> Plan is to keep "prepare servers" as simple as possible. But still I want to have appropriate prepare module for each test module to keep it straightforward when searching for possible problem - in one module is configured server and in second is test which should work.
>
Sounds good. I didn't mean at all to imply that the "prepare" module
idea is wrong.
The biggest thing to avoid is copying the build/target/jboss-xxx/modules
multiple times into different testsuite dirs. We were doing that for a
couple weeks before CR1 and it wasted a LOT of time (and disk.)
Re: preparing necessary config using the management API, keep an eye on
https://issues.jboss.org/browse/AS7-346 and subtasks.
> If I'm not wrong, pointing ARQ to custom configuration xml file can be done in arquillian.xml by defining element property named serverConfig under configuration element.
>
>>>
>>> What do you think about suggested changes?
>>>
>>> Rosta
>>>
>>> [1] https://docspace.corp.redhat.com/docs/DOC-69049
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> jboss-as7-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>>
>> --
>> Brian Stansberry
>> Principal Software Engineer
>> JBoss by Red Hat
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> --
> Rostislav Svoboda
> JBoss QA, Brno
>
> [1] https://issues.jboss.org/browse/ARQ-336
> [2] https://github.com/mgencur/edg-tests/blob/master/failover/src/test/java/org/arquillian/edg/failover/FailoverTestCase.java
--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
More information about the jboss-as7-dev
mailing list