[infinispan-issues] [JBoss JIRA] (ISPN-4317) Optimize server test suite

[Jakub Markos (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12975249#comment-12975249 ] 

Jakub Markos edited comment on ISPN-4317 at 6/11/14 8:58 AM:
-------------------------------------------------------------

Here are my observations:
Currently the server testsuite takes 30 minutes on my PC, and there are 130 server starts (with tests for example configs, which are currently marked as unstable in upstream). Server starts in 5-10 seconds on my machine (in teamcity it's 10-20), so around 15 minutes (30 in TC) are spent on that.
Here is a table of amount of server starts for various tests:
|| tests name || \# server starts || comment / what can be done ||
| jdbc tests | 36 | Most of the tests use memcached cache, which means they can't share the server configuration. Using hotrod instead could reduce this, I'll look into this next. |
| example configurations tests | 30 | Largest chunk is taken by server hinting tests (9 server starts, 3 for each machine/rack/site). Generally I don't think anything can be done about these tests, since their whole purpose is to take the example configuration and test it as is (we still need to do some transformations on them, though). |
| tests for suppressing the state transfer | 22 | I've cut this down to 6, which seems ok. |
| file/remote/custom/leveldb cache stores tests | ~10 | They all need to restart the server to see if entries are really stored. I'll look if they can be somehow grouped together. |
| remote query tests | 6 | Made them share the configuration (see option 2 below), so only 1 now. | 
| transport stack tests | 6 | Cut down to 2. |
| Rest security tests | 6 | Can't be reduced, they each need their own definition of rest endpoint. |
| client tests | 5 | They run across different failsafe executions. |

The rest of the tests uses around 10 starts. 
Reusing the same configuration for various tests is of course possible. There are 2 ways in doing this:
# put the tests into 1 class
# move the tests to separate failsafe execution so we can use arquillians 'suite' container mode, and create a new category for them (and also a new Unstable category).

The second option is OK, but I think it clutters the testsuite a bit and creates a barrier for writing tests by new people.
I'll look on the jdbc and cachestore tests next, because that promises the biggest time gains.

Other ways to minimize the time would be in the tests themselves (reducing jmx calls for example).


was (Author: jmarkos):
Here are my observations:
Currently the server testsuite takes 30 minutes on my PC, and there are 130 server starts (with tests for example configs, which are currently marked as unstable in upstream). Server starts in 5-10 seconds on my machine (in teamcity it's 10-20), so around 15 minutes are spent on that.
Here is a table of amount of server starts for various tests:
|| tests name || \# server starts || comment / what can be done ||
| jdbc tests | 36 | Most of the tests use memcached cache, which means they can't share the server configuration. Using hotrod instead could reduce this, I'll look into this next. |
| example configurations tests | 30 | Largest chunk is taken by server hinting tests (9 server starts, 3 for each machine/rack/site). Generally I don't think anything can be done about these tests, since their whole purpose is to take the example configuration and test it as is (we still need to do some transformations on them, though). |
| tests for suppressing the state transfer | 22 | I've cut this down to 6, which seems ok. |
| file/remote/custom/leveldb cache stores tests | ~10 | They all need to restart the server to see if entries are really stored. I'll look if they can be somehow grouped together. |
| remote query tests | 6 | Made them share the configuration (see option 2 below), so only 1 now. | 
| transport stack tests | 6 | Cut down to 2. |
| Rest security tests | 6 | Can't be reduced, they each need their own definition of rest endpoint. |
| client tests | 5 | They run across different failsafe executions. |

The rest of the tests uses around 10 starts. 
Reusing the same configuration for various tests is of course possible. There are 2 ways in doing this:
# put the tests into 1 class
# move the tests to separate failsafe execution so we can use arquillians 'suite' container mode, and create a new category for them (and also a new Unstable category).

The second option is OK, but I think it clutters the testsuite a bit and creates a barrier for writing tests by new people.
I'll look on the jdbc and cachestore tests next, because that promises the biggest time gains.

Other ways to minimize the time would be in the tests themselves (reducing jmx calls for example).

> Optimize server test suite
> --------------------------
>
>                 Key: ISPN-4317
>                 URL: https://issues.jboss.org/browse/ISPN-4317
>             Project: Infinispan
>          Issue Type: Task
>          Components: Test Suite - Server
>    Affects Versions: 7.0.0.Alpha4
>            Reporter: Martin Gencur
>            Assignee: Jakub Markos
>
> The goal is to make the test execution of server test suite shorter and minimize the number of maven profiles so that most of the tests run in a default profile.
> The first thing to do is to minimize the number of server restarts when running the test suite. This will require adding many different cache configurations to the same Infinispan subsystem.
> Also, look for other ways to minimize test execution time.



--
This message was sent by Atlassian JIRA
(v6.2.6#6264)