[jboss-as7-dev] AS7 testsuite extension

Thu Jul 21 11:23:06 EDT 2011

On 7/21/11 4:08 AM, Rostislav Svoboda wrote:
> Hi Brian, thank you for your reply, comments are inline.
>
>> On 7/19/11 7:12 AM, Rostislav Svoboda wrote:
>>> Hi.
>>>
>>> I would like to discuss how to extend AS7 testsuite, my suggestion
>>> is to create 2 new modules:
>>>
>>> -- management
>>> Tests for management APIs, probably separate submodules for cli,
>>> http api and java api tests.
>>>
>>
>> Makes sense. The existing testsuite/domain module could be moved into
>> here. It's essentially testing management of a domain via the java
>> api.
>>
>> IMO management tests of standalone servers should be segregated from
>> tests of a managed domain.
>
> Agree, submodules should be named domain-http, domain-cli, domain-java, standalone-http, standalone-cli and standalone-java.
> Maybe one submodules for shared tests and resources.
>
>>
>>> -- multi-node
>>> Tests with several instances (managed by DC or several standalone
>>> servers) on one machine.
>>> Cluster, failover, session replication tests should be present in
>>> this module.
>>
>> The testsuite/clustering module is for this purpose.
>
> So we should propose some changes for this module, it's currently running against one standalone server and contains only one test.

No argument. And I may have spoken out of turn. Perhaps Paul Ferraro had 
some other intent for this module than the kind of multi-server testing 
you're talking about (but I don't think so. ;) )

> I think testsuite/clustering will have several submodules to cover different scenarios, positive test can have server execution automatically  managed by ARQ. Negative tests (failovers) must have server execution managed by ARQ but controlled from testscases (ARQ-336 [1], FailoverTestCase.java [2]).
>

Paul's your man for input on this. I just wanted to point out there's an 
existing module.

>>
>>> ARQ with DC support is needed, start/stop/kill support for instances
>>> managed by DC is needed too (ARQ-336 is designed for standalone),
>>> maybe start/stop/kill for DC will be required too.
>>>
>>
>> I regard that as a nice to have, not a requirement. HA functionality
>> is
>> orthogonal to how the servers are managed; a testsuite of Paul's HA
>> features based on launching multiple standalone servers is ok. The
>> actual HA services inside the servers have no clue whether they are
>> running in a standalone server or a managed domain.
>
> OK, I agree with that, DC is only about server management, no effect on AS7 functionality. My point about ARQ with DC support is more about message to users, we really recommend to use DC, but we don't use it in our tests ...

I agree with the message to users point, but I just want to emphasize we 
have limited time and a large number of hard requirements and AFAIK this 
is not one of them. So always be cautious about adding new requirements, 
particularly ones that become blockers earlier in the process.

We have a nice testing story already with AS 7 and Arquillian. And it's 
improving rapidly. So if support for using a managed domain is on the 
roadmap but doesn't get in AS 7.1 or comes in too late to be the basis 
of our HA testsuite, I don't think that's too bad of a message.

> Several standalone server in cluster are sufficient to verify that clustering functionality is working properly.
>
>> Note also that ARQ with DC support is not needed for management tests of domain.
>
> But I steel need ARQ to start DC (using managed mode). Could you please explain how do you mean your note?
>

No, you don't. :) See the testsuite/domain module. It uses a couple 
classes to control the DC (and a slave HostController.) They are in the 
arquillian/container-managed-domain module but putting them there was a 
mistake on my part. They don't actually have anything really to do with 
ARQ at this point. I put them there because I dreamed I'd have time to 
also use them as utility classes in a real ARQ container for a managed 
domain.

A bit more on domain management tests and ARQ. I see ARQ as being 
oriented toward testing deployments. Most domain management testing is 
not about testing deployments. Like the HA point above, how the 
deployments behave inside a server is orthogonal to managed domain vs 
standalone. All the deep testing of how deployments behave can be done 
with a standalone server.

When testing the domain management, I don't want to have to deal with 
deployments unless there is a real need. I want to start/stop domain 
processes and interact with them via the management API. For sure I 
don't want to have to deploy fake deployments to a domain just to 
satisfy an ARQ requirement. (We actually do that in some standalone 
tests of the management API.)

>>> Module structure should be similar to DOC-69049 [1] with submodules
>>> to share tests, prepare servers and to execute tests.
>>>
>>
>> Hopefully in 95%+ of cases "prepare servers" can consist of altering
>> the
>> server/domain launch command to point at the appropriate writable
>> directory, config file and module path.
>
> We don't want to store configuration xml files in repository, we would like to prepare necessary configuration using management API. We have quite bad experience with configuration stored in xml files for testing, at least when development is still going on.
> Plan is to keep "prepare servers" as simple as possible. But still I want to have appropriate prepare module for each test module to keep it straightforward when searching for possible problem - in one module is configured server and in second is test which should work.
>

Sounds good. I didn't mean at all to imply that the "prepare" module 
idea is wrong.

The biggest thing to avoid is copying the build/target/jboss-xxx/modules 
multiple times into different testsuite dirs. We were doing that for a 
couple weeks before CR1 and it wasted a LOT of time (and disk.)

Re: preparing necessary config using the management API, keep an eye on 
https://issues.jboss.org/browse/AS7-346 and subtasks.

> If I'm not wrong, pointing ARQ to custom configuration xml file can be done in arquillian.xml by defining element property named serverConfig under configuration element.
>
>>>
>>> What do you think about suggested changes?
>>>
>>> Rosta
>>>
>>> [1] https://docspace.corp.redhat.com/docs/DOC-69049
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> jboss-as7-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>>
>> --
>> Brian Stansberry
>> Principal Software Engineer
>> JBoss by Red Hat
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> --
> Rostislav Svoboda
> JBoss QA, Brno
>
> [1] https://issues.jboss.org/browse/ARQ-336
> [2] https://github.com/mgencur/edg-tests/blob/master/failover/src/test/java/org/arquillian/edg/failover/FailoverTestCase.java

-- 
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat