On Friday 18 March 2011 12:51 PM, Andrew Lee Rubinger wrote: > > I suspect this breaks down into two categories, which may be modelled as > separate modules under the existing "testsuite" aggregator parent: > > * Specification > * AS-specific APIs On a slightly different categorization, we also have to come up with tests for stress testing. They can either be for "Specification" related tests or even for "AS-specific APIs". Perhaps test "profiles" (need _not_ be Maven profile) within those 2 modules would be better instead of a separate module? > > [End Goal] > > 1) No compile-time dependencies in the module except for what's > absolutely necessary. > > For the spec suite, this means: JDK and EE Spec APIs only in the > compilation classpath. We might also have to include JBoss specific APIs which are exposed to the users (for ex: org.jboss.ejb3.annotation which currently isn't there but might be introduced soon in AS7). > Testable asset sources and resources (ie. EJBs, > Servlets, etc) would live under src/main/* to enforce that. Only the > tests themselves would be located under "src/test/*". Sounds fine. Although, do we want to end up with all tech specific tests (like for EJB3, web, etc...) all in one source path? i.e. spec-tests/src/main/org/jboss/spec/<ejb3, web, jsf>....? Or should we plan to have different maven modules under the "Spec" tests for each individual tech? > > 2) Every single new test created is to have an associated JIRA. I'm not quite sure that this will work out. I'm not against it, but it might not be practical. > > By linking to JIRA we get history of intent, which acts as a > nice record even in the case that the test isn't so well-documented. > I'd argue that tests are a bigger asset than our code, and we should be > thinking about these in terms of long-term maintenance to outlive any > specific impl. I agree about the well-documented tests, but JIRA linking shouldn't always be an absolute necessity. > > 3) Documentation > > Alongside the JIRA reference, a quick note about we're looking to > accomplish is something I find very helpful. I don't personally buy the > argument that code is self-documenting if written well. It gets > refactored and stale over time. +1000. A simple brief explanation on what the test is supposed to test is really useful. > > 4) Run-mode profiles > > Arquillian provides a wonderful abstraction such that we can get > coverage for AS in both remote managed *and* embedded modes without > changing the test itself. To certify that everything is working as > advertised no matter the runtime, we should be able to run the same > suite in standalone, domain, and embedded modes (generally speaking). Are these container specific modes that we are talking about? Or are these more of test profiles (a.k.a groups) wherein we could perhaps add a test under a "stress-test" profile? -Jaikiran _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

Looks like a lot of us have different ideas for what the AS7 Integration TestSuite should consist of, so I'll kickoff with what I believe is the first design proposal towards getting coverage focused on the end-user (not certifying our own internals). I suspect this breaks down into two categories, which may be modelled as separate modules under the existing "testsuite" aggregator parent: * Specification * AS-specific APIs This isn't difficult work, though I do think it's important we consider some hard rules. IMO we should be developing these suites as if we were application developers, not wearing our server dev hats. ---------------- [End Goal] 1) No compile-time dependencies in the module except for what's absolutely necessary. For the spec suite, this means: JDK and EE Spec APIs only in the compilation classpath. Testable asset sources and resources (ie. EJBs, Servlets, etc) would live under src/main/* to enforce that. Only the tests themselves would be located under "src/test/*". The AS-specific API suite may also add in our own APIs to the compilation classpath, but the line should end there. In "test" scope we can place all runtime dependencies. For the specification suite, AS-specific grammars like our own deployment descriptors are fine; these are in many ways equivalent to the TCK porting layer. We're not building a TCK; we're showing that our implementation supports the features advertised. 2) Every single new test created is to have an associated JIRA. We all remember the nightmare it was when the old AS4-6 suite would fall down. We'd comb through each test, at times trying to determine its purpose. By linking to JIRA we get history of intent, which acts as a nice record even in the case that the test isn't so well-documented. I'd argue that tests are a bigger asset than our code, and we should be thinking about these in terms of long-term maintenance to outlive any specific impl. 3) Documentation Alongside the JIRA reference, a quick note about we're looking to accomplish is something I find very helpful. I don't personally buy the argument that code is self-documenting if written well. It gets refactored and stale over time. 4) Run-mode profiles Arquillian provides a wonderful abstraction such that we can get coverage for AS in both remote managed *and* embedded modes without changing the test itself. To certify that everything is working as advertised no matter the runtime, we should be able to run the same suite in standalone, domain, and embedded modes (generally speaking). 5) Porting of AS6 Tests There's no discounting the value this coverage has given us, though I question the purpose of a lot of these tests. I think a great majority of these need to come into the new codebase, refactored to align if needed. ---------------- [Current State] Here[1] is an example of what I believe to be a simple, well-written test, with the exception that the tested Servlet and EJB are in the same test source folder. The current "testsuite" aggregator contains modules which mix our end-user certification stuff alongside internals, so I think these should be separated out. A lot of this is set up in some fashion already, but I would like to see us: 1) Agree upon a strict scope for each type of testsuite along the lines of my points above, once we reach agreement 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap 1.0.0-alpha-12), just released tonight. Currently AS is on a forked release of ARQ for OSGi purposes, and these changes, if necessary, need to get upstream so we can do upgrades. It's clear that AS7 has made full-steam-ahead progress since last summer, and with a little organization our testsuite can give us a great view of where we stand, from an end-user's perspective, with minimal investment. S, ALR [1] https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

Inline. On 03/19/2011 03:19 PM, Jason Greene wrote: > I really like the idea of our primary test suite covering API interaction and your typical unit test which is tightly coupled to internals. I also like your idea to separate spec Apis from JBoss aAPIs. > > However, I disagree on the Jira per test requirement. This is only because I worked with a similar model (all jiras require a test), and you ended up with lots of duplicate tests, and you always had to read Jira to understand what was being tested. Also, the tests ended up being too specific, typically an exact replication of the reporter's workflow. Consider this: If you had to read JIRA to understand what was being tested, that's a clear violation of point 3): Document the test. And failing good documentation, with no JIRA to turn to, you're flying blind. Also, how exactly does registering a JIRA encourage test duplication? Whether the tests are in an issue tracker or not seems to be unrelated to folks writing duplicate tests. > > IMO a better way to organize it would be by functional area, with generous descriptions. All spec tests should reference spec sections, for example. If we have a Jira that demonstrates lack of coverage, we should translate the problem into meaningful and reusable tests for the gaps we missed. Those tests should primarily refer to the API contract and not the Jira. Although commenting a list of jiras for extra info may be useful provided the Jira doesn't confuse what the correct behavior should be. I'm a fan of organizing by functional areas and spec references. We'd done that in EJB3 with success: some packages were even named for the spec. Regardless, every commit had a JIRA association. In the end, so long as it's perfectly clear what we're testing and why, I'll have no objections. But I do believe that JIRA is the obvious answer to that question, and requires minimal effort to make a new issue. Especially since every commit is isolated to one concern and linked back to JIRA anyway, right? ;) S, ALR > > Sent from my iPad > > On Mar 18, 2011, at 2:21 AM, Andrew Lee Rubinger&lt;andrew.rubinger(a)redhat.com&gt; wrote: > >> Looks like a lot of us have different ideas for what the AS7 Integration >> TestSuite should consist of, so I'll kickoff with what I believe is the >> first design proposal towards getting coverage focused on the end-user >> (not certifying our own internals). >> >> I suspect this breaks down into two categories, which may be modelled as >> separate modules under the existing "testsuite" aggregator parent: >> >> * Specification >> * AS-specific APIs >> >> This isn't difficult work, though I do think it's important we consider >> some hard rules. IMO we should be developing these suites as if we were >> application developers, not wearing our server dev hats. >> >> ---------------- >> >> [End Goal] >> >> 1) No compile-time dependencies in the module except for what's >> absolutely necessary. >> >> For the spec suite, this means: JDK and EE Spec APIs only in the >> compilation classpath. Testable asset sources and resources (ie. EJBs, >> Servlets, etc) would live under src/main/* to enforce that. Only the >> tests themselves would be located under "src/test/*". >> >> The AS-specific API suite may also add in our own APIs to the >> compilation classpath, but the line should end there. In "test" scope >> we can place all runtime dependencies. >> >> For the specification suite, AS-specific grammars like our own >> deployment descriptors are fine; these are in many ways equivalent to >> the TCK porting layer. We're not building a TCK; we're showing that our >> implementation supports the features advertised. >> >> 2) Every single new test created is to have an associated JIRA. >> >> We all remember the nightmare it was when the old AS4-6 suite would fall >> down. We'd comb through each test, at times trying to determine its >> purpose. By linking to JIRA we get history of intent, which acts as a >> nice record even in the case that the test isn't so well-documented. >> I'd argue that tests are a bigger asset than our code, and we should be >> thinking about these in terms of long-term maintenance to outlive any >> specific impl. >> >> 3) Documentation >> >> Alongside the JIRA reference, a quick note about we're looking to >> accomplish is something I find very helpful. I don't personally buy the >> argument that code is self-documenting if written well. It gets >> refactored and stale over time. >> >> 4) Run-mode profiles >> >> Arquillian provides a wonderful abstraction such that we can get >> coverage for AS in both remote managed *and* embedded modes without >> changing the test itself. To certify that everything is working as >> advertised no matter the runtime, we should be able to run the same >> suite in standalone, domain, and embedded modes (generally speaking). >> >> 5) Porting of AS6 Tests >> >> There's no discounting the value this coverage has given us, though I >> question the purpose of a lot of these tests. I think a great majority >> of these need to come into the new codebase, refactored to align if needed. >> >> ---------------- >> >> [Current State] >> >> Here[1] is an example of what I believe to be a simple, well-written >> test, with the exception that the tested Servlet and EJB are in the same >> test source folder. >> >> The current "testsuite" aggregator contains modules which mix our >> end-user certification stuff alongside internals, so I think these >> should be separated out. >> >> A lot of this is set up in some fashion already, but I would like to see us: >> >> 1) Agree upon a strict scope for each type of testsuite along the lines >> of my points above, once we reach agreement >> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap >> 1.0.0-alpha-12), just released tonight. Currently AS is on a forked >> release of ARQ for OSGi purposes, and these changes, if necessary, need >> to get upstream so we can do upgrades. >> >> It's clear that AS7 has made full-steam-ahead progress since last >> summer, and with a little organization our testsuite can give us a great >> view of where we stand, from an end-user's perspective, with minimal >> investment. >> >> S, >> ALR >> >> >> [1] >> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

> On 03/19/2011 03:19 PM, Jason Greene wrote: >> I really like the idea of our primary test suite covering API interaction and your typical unit test which is tightly coupled to internals. I also like your idea to separate spec Apis from JBoss aAPIs. >> >> However, I disagree on the Jira per test requirement. This is only because I worked with a similar model (all jiras require a test), and you ended up with lots of duplicate tests, and you always had to read Jira to understand what was being tested. Also, the tests ended up being too specific, typically an exact replication of the reporter's workflow. > > Consider this: If you had to read JIRA to understand what was being > tested, that's a clear violation of point 3): Document the test. And > failing good documentation, with no JIRA to turn to, you're flying blind. Ok so you agree that the jira is unnecessary :)

> > Also, how exactly does registering a JIRA encourage test duplication? > Whether the tests are in an issue tracker or not seems to be unrelated > to folks writing duplicate tests. For a simplified example, Let's say a deployment descriptor had a field that allows any valid string, but we incorrectly limited it to be lower case alphanumeric. Then we get 2 jiras one is "component attribute x does not allow capital letters". Another is "component attribute x does not allow spaces". We end up with two tests that are testing the same fundamental problem, but even so aren't exhaustive(special chars etc). >> >> IMO a better way to organize it would be by functional area, with generous descriptions. All spec tests should reference spec sections, for example. If we have a Jira that demonstrates lack of coverage, we should translate the problem into meaningful and reusable tests for the gaps we missed. Those tests should primarily refer to the API contract and not the Jira. Although commenting a list of jiras for extra info may be useful provided the Jira doesn't confuse what the correct behavior should be. > > I'm a fan of organizing by functional areas and spec references. We'd > done that in EJB3 with success: some packages were even named for the > spec. Regardless, every commit had a JIRA association. > > In the end, so long as it's perfectly clear what we're testing and why, > I'll have no objections. But I do believe that JIRA is the obvious > answer to that question, and requires minimal effort to make a new > issue. Especially since every commit is isolated to one concern and > linked back to JIRA anyway, right? ;) That's like reading commit history to understand the feature list of a software package. JIRA represents a snapshot of time, one that may not be relevant or complete at a later date.

> > S, > ALR > >> >> Sent from my iPad >> >> On Mar 18, 2011, at 2:21 AM, Andrew Lee Rubinger&lt;andrew.rubinger(a)redhat.com&gt; wrote: >> >>> Looks like a lot of us have different ideas for what the AS7 Integration >>> TestSuite should consist of, so I'll kickoff with what I believe is the >>> first design proposal towards getting coverage focused on the end-user >>> (not certifying our own internals). >>> >>> I suspect this breaks down into two categories, which may be modelled as >>> separate modules under the existing "testsuite" aggregator parent: >>> >>> * Specification >>> * AS-specific APIs >>> >>> This isn't difficult work, though I do think it's important we consider >>> some hard rules. IMO we should be developing these suites as if we were >>> application developers, not wearing our server dev hats. >>> >>> ---------------- >>> >>> [End Goal] >>> >>> 1) No compile-time dependencies in the module except for what's >>> absolutely necessary. >>> >>> For the spec suite, this means: JDK and EE Spec APIs only in the >>> compilation classpath. Testable asset sources and resources (ie. EJBs, >>> Servlets, etc) would live under src/main/* to enforce that. Only the >>> tests themselves would be located under "src/test/*". >>> >>> The AS-specific API suite may also add in our own APIs to the >>> compilation classpath, but the line should end there. In "test" scope >>> we can place all runtime dependencies. >>> >>> For the specification suite, AS-specific grammars like our own >>> deployment descriptors are fine; these are in many ways equivalent to >>> the TCK porting layer. We're not building a TCK; we're showing that our >>> implementation supports the features advertised. >>> >>> 2) Every single new test created is to have an associated JIRA. >>> >>> We all remember the nightmare it was when the old AS4-6 suite would fall >>> down. We'd comb through each test, at times trying to determine its >>> purpose. By linking to JIRA we get history of intent, which acts as a >>> nice record even in the case that the test isn't so well-documented. >>> I'd argue that tests are a bigger asset than our code, and we should be >>> thinking about these in terms of long-term maintenance to outlive any >>> specific impl. >>> >>> 3) Documentation >>> >>> Alongside the JIRA reference, a quick note about we're looking to >>> accomplish is something I find very helpful. I don't personally buy the >>> argument that code is self-documenting if written well. It gets >>> refactored and stale over time. >>> >>> 4) Run-mode profiles >>> >>> Arquillian provides a wonderful abstraction such that we can get >>> coverage for AS in both remote managed *and* embedded modes without >>> changing the test itself. To certify that everything is working as >>> advertised no matter the runtime, we should be able to run the same >>> suite in standalone, domain, and embedded modes (generally speaking). >>> >>> 5) Porting of AS6 Tests >>> >>> There's no discounting the value this coverage has given us, though I >>> question the purpose of a lot of these tests. I think a great majority >>> of these need to come into the new codebase, refactored to align if needed. >>> >>> ---------------- >>> >>> [Current State] >>> >>> Here[1] is an example of what I believe to be a simple, well-written >>> test, with the exception that the tested Servlet and EJB are in the same >>> test source folder. >>> >>> The current "testsuite" aggregator contains modules which mix our >>> end-user certification stuff alongside internals, so I think these >>> should be separated out. >>> >>> A lot of this is set up in some fashion already, but I would like to see us: >>> >>> 1) Agree upon a strict scope for each type of testsuite along the lines >>> of my points above, once we reach agreement >>> 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap >>> 1.0.0-alpha-12), just released tonight. Currently AS is on a forked >>> release of ARQ for OSGi purposes, and these changes, if necessary, need >>> to get upstream so we can do upgrades. >>> >>> It's clear that AS7 has made full-steam-ahead progress since last >>> summer, and with a little organization our testsuite can give us a great >>> view of where we stand, from an end-user's perspective, with minimal >>> investment. >>> >>> S, >>> ALR >>> >>> >>> [1] >>> https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

An additional note on the 'integration' tests. While [1] is a nice example for the API test-suite in the integration test-suite I want a no-bar, so I can do stuff like [2]. [1] also needs a spot of comment like [3]. The moment we go "what or why" on a test, it's a goner. Carlo [1] https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... [2] https://github.com/wolfc/jboss-as/blob/weld/testsuite/integration/src/tes... [3] https://github.com/wolfc/jboss-as/blob/weld/testsuite/integration/src/tes... On 03/18/2011 08:21 AM, Andrew Lee Rubinger wrote: > Looks like a lot of us have different ideas for what the AS7 Integration > TestSuite should consist of, so I'll kickoff with what I believe is the > first design proposal towards getting coverage focused on the end-user > (not certifying our own internals). > > I suspect this breaks down into two categories, which may be modelled as > separate modules under the existing "testsuite" aggregator parent: > > * Specification > * AS-specific APIs > > This isn't difficult work, though I do think it's important we consider > some hard rules. IMO we should be developing these suites as if we were > application developers, not wearing our server dev hats. > > ---------------- > > [End Goal] > > 1) No compile-time dependencies in the module except for what's > absolutely necessary. > > For the spec suite, this means: JDK and EE Spec APIs only in the > compilation classpath. Testable asset sources and resources (ie. EJBs, > Servlets, etc) would live under src/main/* to enforce that. Only the > tests themselves would be located under "src/test/*". > > The AS-specific API suite may also add in our own APIs to the > compilation classpath, but the line should end there. In "test" scope > we can place all runtime dependencies. > > For the specification suite, AS-specific grammars like our own > deployment descriptors are fine; these are in many ways equivalent to > the TCK porting layer. We're not building a TCK; we're showing that our > implementation supports the features advertised. > > 2) Every single new test created is to have an associated JIRA. > > We all remember the nightmare it was when the old AS4-6 suite would fall > down. We'd comb through each test, at times trying to determine its > purpose. By linking to JIRA we get history of intent, which acts as a > nice record even in the case that the test isn't so well-documented. > I'd argue that tests are a bigger asset than our code, and we should be > thinking about these in terms of long-term maintenance to outlive any > specific impl. > > 3) Documentation > > Alongside the JIRA reference, a quick note about we're looking to > accomplish is something I find very helpful. I don't personally buy the > argument that code is self-documenting if written well. It gets > refactored and stale over time. > > 4) Run-mode profiles > > Arquillian provides a wonderful abstraction such that we can get > coverage for AS in both remote managed *and* embedded modes without > changing the test itself. To certify that everything is working as > advertised no matter the runtime, we should be able to run the same > suite in standalone, domain, and embedded modes (generally speaking). > > 5) Porting of AS6 Tests > > There's no discounting the value this coverage has given us, though I > question the purpose of a lot of these tests. I think a great majority > of these need to come into the new codebase, refactored to align if > needed. > > ---------------- > > [Current State] > > Here[1] is an example of what I believe to be a simple, well-written > test, with the exception that the tested Servlet and EJB are in the same > test source folder. > > The current "testsuite" aggregator contains modules which mix our > end-user certification stuff alongside internals, so I think these > should be separated out. > > A lot of this is set up in some fashion already, but I would like to > see us: > > 1) Agree upon a strict scope for each type of testsuite along the lines > of my points above, once we reach agreement > 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap > 1.0.0-alpha-12), just released tonight. Currently AS is on a forked > release of ARQ for OSGi purposes, and these changes, if necessary, need > to get upstream so we can do upgrades. > > It's clear that AS7 has made full-steam-ahead progress since last > summer, and with a little organization our testsuite can give us a great > view of where we stand, from an end-user's perspective, with minimal > investment. > > S, > ALR > > > [1] > https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... > > _______________________________________________ > jboss-as7-dev mailing list > jboss-as7-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

Looks like a lot of us have different ideas for what the AS7 Integration TestSuite should consist of, so I'll kickoff with what I believe is the first design proposal towards getting coverage focused on the end-user (not certifying our own internals). I suspect this breaks down into two categories, which may be modelled as separate modules under the existing "testsuite" aggregator parent: * Specification * AS-specific APIs This isn't difficult work, though I do think it's important we consider some hard rules. IMO we should be developing these suites as if we were application developers, not wearing our server dev hats. ---------------- [End Goal] 1) No compile-time dependencies in the module except for what's absolutely necessary. For the spec suite, this means: JDK and EE Spec APIs only in the compilation classpath. Testable asset sources and resources (ie. EJBs, Servlets, etc) would live under src/main/* to enforce that. Only the tests themselves would be located under "src/test/*". The AS-specific API suite may also add in our own APIs to the compilation classpath, but the line should end there. In "test" scope we can place all runtime dependencies. For the specification suite, AS-specific grammars like our own deployment descriptors are fine; these are in many ways equivalent to the TCK porting layer. We're not building a TCK; we're showing that our implementation supports the features advertised. 2) Every single new test created is to have an associated JIRA. We all remember the nightmare it was when the old AS4-6 suite would fall down. We'd comb through each test, at times trying to determine its purpose. By linking to JIRA we get history of intent, which acts as a nice record even in the case that the test isn't so well-documented. I'd argue that tests are a bigger asset than our code, and we should be thinking about these in terms of long-term maintenance to outlive any specific impl. 3) Documentation Alongside the JIRA reference, a quick note about we're looking to accomplish is something I find very helpful. I don't personally buy the argument that code is self-documenting if written well. It gets refactored and stale over time. 4) Run-mode profiles Arquillian provides a wonderful abstraction such that we can get coverage for AS in both remote managed *and* embedded modes without changing the test itself. To certify that everything is working as advertised no matter the runtime, we should be able to run the same suite in standalone, domain, and embedded modes (generally speaking). 5) Porting of AS6 Tests There's no discounting the value this coverage has given us, though I question the purpose of a lot of these tests. I think a great majority of these need to come into the new codebase, refactored to align if needed. ---------------- [Current State] Here[1] is an example of what I believe to be a simple, well-written test, with the exception that the tested Servlet and EJB are in the same test source folder. The current "testsuite" aggregator contains modules which mix our end-user certification stuff alongside internals, so I think these should be separated out. A lot of this is set up in some fashion already, but I would like to see us: 1) Agree upon a strict scope for each type of testsuite along the lines of my points above, once we reach agreement 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap 1.0.0-alpha-12), just released tonight. Currently AS is on a forked release of ARQ for OSGi purposes, and these changes, if necessary, need to get upstream so we can do upgrades. It's clear that AS7 has made full-steam-ahead progress since last summer, and with a little organization our testsuite can give us a great view of where we stand, from an end-user's perspective, with minimal investment. S, ALR [1] https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

Andrew, I assume this testing will use the Arquillian API for deploying and that the arquillian API will use the yet-to-defined (AFAIK) programmatic deployment API directly to do the deployment.

On 03/24/2011 02:17 PM, Jason T. Greene wrote: > On 3/23/11 9:50 AM, Max Rydahl Andersen wrote: >> Andrew, I assume this testing will use the Arquillian API for >> deploying and that the arquillian API will use the yet-to-defined (AFAIK) >> programmatic deployment API directly to do the deployment. > > It's using the programmatic API right now, which is the better interface. Why is that better than the ARQ abstration?

On 03/24/2011 02:52 PM, David M. Lloyd wrote: > On 03/24/2011 01:51 PM, Andrew Lee Rubinger wrote: >> On 03/24/2011 02:17 PM, Jason T. Greene wrote: >>> On 3/23/11 9:50 AM, Max Rydahl Andersen wrote: >>>> Andrew, I assume this testing will use the Arquillian API for >>>> deploying and that the arquillian API will use the yet-to-defined (AFAIK) >>>> programmatic deployment API directly to do the deployment. >>> >>> It's using the programmatic API right now, which is the better interface. >> >> Why is that better than the ARQ abstration? > > I think you mis-parsed that exchange... Ah yes, ARQ is using the programmatic API. Depends on the definition of what "it" is. :P

>> programmatic deployment API directly to do the deployment. > > It's using the programmatic API right now, which is the better > interface. I think the definition of "better" depends on the eye of the beholder :) I'm just saying that being able to run the TCK tests in a "use the file deployment API which a huge number of users will still use since it works without running server" is a Good Thing to do.

In other words, you want us to sacrifice things like the ability to have a testsuite that works on domain mode and responds to deployment as fast as possible because you chose to use the FS API?

> > In other words, you want us to sacrifice things like the ability to have > a testsuite that works on domain mode and responds to deployment as fast > as possible because you chose to use the FS API? btw. i'm still in the blind on what alternative we have to FS api.... Where is the tool/way to deploy to AS7 without writing a custom client that uses the native api ?

>>> >>> In other words, you want us to sacrifice things like the ability to have >>> a testsuite that works on domain mode and responds to deployment as fast >>> as possible because you chose to use the FS API? >> >> >> btw. i'm still in the blind on what alternative we have to FS api.... >> >> Where is the tool/way to deploy to AS7 without writing a custom client that uses >> the native api ? > > Alexey is working on the CLI support right now. Ok, i'll ask him how he do it then. > A contributor also did this in like a couple of hours: > https://github.com/jamezp/jboss-as-deploy-plugin That code is using as far as I can see the Java API which I will *not* work across multiple versions unless you guys guarantee serialization and API will not change ? Last time we had the discussion (Devoxx) it was said to me that if I wanted portability I (and other tools) would need to use the HTTP exposed API (or the file API). Has that changed ?

Looks like a lot of us have different ideas for what the AS7 Integration TestSuite should consist of, so I'll kickoff with what I believe is the first design proposal towards getting coverage focused on the end-user (not certifying our own internals). I suspect this breaks down into two categories, which may be modelled as separate modules under the existing "testsuite" aggregator parent: * Specification * AS-specific APIs This isn't difficult work, though I do think it's important we consider some hard rules. IMO we should be developing these suites as if we were application developers, not wearing our server dev hats. ---------------- [End Goal] 1) No compile-time dependencies in the module except for what's absolutely necessary. For the spec suite, this means: JDK and EE Spec APIs only in the compilation classpath. Testable asset sources and resources (ie. EJBs, Servlets, etc) would live under src/main/* to enforce that. Only the tests themselves would be located under "src/test/*". The AS-specific API suite may also add in our own APIs to the compilation classpath, but the line should end there. In "test" scope we can place all runtime dependencies. For the specification suite, AS-specific grammars like our own deployment descriptors are fine; these are in many ways equivalent to the TCK porting layer. We're not building a TCK; we're showing that our implementation supports the features advertised. 2) Every single new test created is to have an associated JIRA. We all remember the nightmare it was when the old AS4-6 suite would fall down. We'd comb through each test, at times trying to determine its purpose. By linking to JIRA we get history of intent, which acts as a nice record even in the case that the test isn't so well-documented. I'd argue that tests are a bigger asset than our code, and we should be thinking about these in terms of long-term maintenance to outlive any specific impl. 3) Documentation Alongside the JIRA reference, a quick note about we're looking to accomplish is something I find very helpful. I don't personally buy the argument that code is self-documenting if written well. It gets refactored and stale over time. 4) Run-mode profiles Arquillian provides a wonderful abstraction such that we can get coverage for AS in both remote managed *and* embedded modes without changing the test itself. To certify that everything is working as advertised no matter the runtime, we should be able to run the same suite in standalone, domain, and embedded modes (generally speaking). 5) Porting of AS6 Tests There's no discounting the value this coverage has given us, though I question the purpose of a lot of these tests. I think a great majority of these need to come into the new codebase, refactored to align if needed. ---------------- [Current State] Here[1] is an example of what I believe to be a simple, well-written test, with the exception that the tested Servlet and EJB are in the same test source folder. The current "testsuite" aggregator contains modules which mix our end-user certification stuff alongside internals, so I think these should be separated out. A lot of this is set up in some fashion already, but I would like to see us: 1) Agree upon a strict scope for each type of testsuite along the lines of my points above, once we reach agreement 2) Upgrade to ARQ 1.0.0.Alpha5 (which implies ShrinkWrap 1.0.0-alpha-12), just released tonight. Currently AS is on a forked release of ARQ for OSGi purposes, and these changes, if necessary, need to get upstream so we can do upgrades. It's clear that AS7 has made full-steam-ahead progress since last summer, and with a little organization our testsuite can give us a great view of where we stand, from an end-user's perspective, with minimal investment. S, ALR [1] https://github.com/jbossas/jboss-as/blob/master/testsuite/integration/src... _______________________________________________ jboss-as7-dev mailing list jboss-as7-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-as7-dev

On 4/11/11 6:37 AM, Andrew Lee Rubinger wrote: > To follow up on this with some JIRAs: > > Nothing proposed here is too complex, and reading through the lengthy > responses to this topic I think the only real point of disagreement is > the "JIRA per test" thing. As there's no policy in AS right now to > reference a JIRA for each commit, we'll just skip this part. > > To that end, I've some tracking JIRAs: > > 1) (JBAS-9285) Create AS Specification TestSuite > > ...this to cover most user cases. No reliance on AS APIs or internals. > These are all good. However this one in particular should be broken up into a set of deliverables, and then assigned to specific releases. We only have 2 months left for the whole shebang.