User development,
The document "CacheBenchmarkFramework", was updated Feb 22, 2010
by Manik Surtani.
To view the document, visit:
http://community.jboss.org/docs/DOC-9241#cf
Document:
--------------------------------------------------------------
h2. > NOTE: This page and this project is deprecated. It has been moved to a
standalone project on SourceForge -
http://cachebenchfwk.sourceforge.net - and has been
heavily modernized since moving. Please refer to the project on Sourceforge for best
results.
This page is still retained for historic purposes.
h2.
h2. What is the Cache Benchmark Framework?
It is a framework that allows running multiple performance tests against various cache
products. It was primarily conceived as a tool to benchmark different releases of JBoss
Cache.
Here are some useful features offered by the framework:
* Works on any caching product - simply write a CacheWrapper implementation, which
delegates calls like put(), get() and remove().
* pluggable tests - JUnit-like tests can be written to mimic your data access patterns and
cache usage scenarios, making benchmarks relevant and useful. We ship with a few typical
tests, such as SessionSimulatorTest which mimics storing and replicating of HTTP
sessions.
* pluggable reports - prefer excel spreadsheets to CSV? Or SVG images to PNGs? Write
your own report generators. Ships with a basic CSV generator.
* cluster barriers - guarantee that all running nodes synchronize during each stage of the
test, such as warm-up, test and report generation phases, guaranteeing that each of these
phases happens at the same time on each cluster node.
* centralized reports - each node of the cluster produces its own performance report. The
primary node (first node in the cluster) then collates these reports into a single report
file.
* cache warmup - triggers a sequence of operations on the cache before running any tests,
to allow for hotspot compilers to optimize.
* replication assertion - asserts that any values intended to be replicated have in fact
been replicated.
h2. Usage
h3. Download and install
h4. Prerequisites:
* Apache ANT version 1.7.x or higher is needed for building the binaries.
* A Java 5 JDK is needed to build and run the binaries.
* The framework ships with a number of shell scripts to enable proper classpath setting
and command execution.
* Cygwin can be used for running the test on windows; if so add the scripts from to .
Those scripts make an automatic conversion from unix CLASSPATH to win CLASSPATH, needed
when executing
* Running cluster-wide tests require SSH and passphrase-less public keys set up on all
servers, to enable cluster-wide execution.
h4. Download
There is no public release of the framework at the moment. However, it can be built from
sources. The Subversion repository is .
After getting the sources from Subversion,
1. Download the JARs for specific cache products; for details pertaining to specific
cache products.
2. Build the sources using in the source root directory.
h3. Configure and run a test
h4. Configuring
1. Before running a test configure . The configuration settings are documented within the
file. *IMPORTANT* please read this file carefully as it contains important information,
like where reports will be generated, etc.
2. Configure logging by editing . Recommended verbosity levels are WARN or ERROR, so that
benchmark measurements won't be affected by unnecessary logging. In order to have
individual log files for each node in the cluster, use rather than log4j's . This
extends (i.e. support its configurations) and prepend the node index on the node's
log file name.
3. Edit and ensure that the , and variables accurately describe the test configs you
intend to run. The framework will execute each config against each existing product, on
all specified cluster sizes.
4. Edit so it accurately points to the network interface to bind clustered caches to. It
is a good idea for this to point to an environment variable that is set individually by
each host.
h4. Running tests
* some cache products, such as Terracotta, might need additional setup, such as starting
of a master servers. All this information is described in an Readme.txt file in the
directory which coresponds to the benchmark product, e.g.
cache-products/terracotta-2.5.0/Readme.txt. Make sure you follow any steps described in
that file before running the tests.
* will kick a test run, according to the configuration you set when editing it.
* will kill all instances running all all nodes. If you are not sure whether the benchmark
exit gracefully at the previous run, better run this first to make sure that there are no
daemon nodes interfering with new ones.
h4. Results
Results will appear a format and in the location specified in . By default this is a CSV
file, in the project root directory.
h3. Local mode
The framework can now deal with non-clustered tests as well. Use the script instead of
the script which remotely runs . Note that you should use an appropriately tuned
configuration for your cache, so that it does not create unnecessary network resources.
Also, you should NOT use the when running tests in local mode - you won't get any
results!!
This is very useful for benchmarking non-clustered performance.
h2. Write your own tests
* Every newly created test should implement . For more details please refer to 's
javadoc. The framework ships with some tests already, see the package for details and
examples.
* Archive your new test as a JAR file and drop it in the directory so that it is picked
up by the framework.
* Specify your new test to be run in , within a element. Refer to how existing tests are
specified for examples.
h2. Benchmark a new cache product
To add to the cache benchmark framework, follow these steps:
1. Create a directory
2. Create for the cache product distribution jars.
3. Create and write a cache wrapper for the product. See for example.
4. Copy to , and modify the file as per the comments to suit your product. This script
builds necessary classpaths.
5. Your cache product is now ready to be benchmarked, as per the instructions above.
h2. TODOs
See
http://anonsvn.jboss.org/repos/jbosscache/benchmarks/benchmark-fwk/trunk/...
h2. Feedback
Please provide all feedback on this on the
http://www.jboss.com/index.html?module=bb&op=viewforum&f=157 for now, until a
separate mailing list can be set up.
--------------------------------------------------------------