GPLed software AS IS, sponsored by 1&1 Internet AG (www.1und1.de). Contact frank.liepold@1und1.de The test suite is work in progress. At the moment it is not in the state to be usable outside of the development team. Event this file is not complete! ============================================================================= Contents -------- 1. Running a test 1.1. Test output 2. Naming conventions 3. Error handling 4. Signal handling 5. Configuration of the local build environment 6. Configuration of the test hosts on which the tests are running 6.1. Access requirements 6.2. Installation kernel and mars module on the test hosts 7. Configuration of logical volumes 8. Resources 9. Starting the whole test suite via cronjob 10. Concurrent test runs 1. Running a test ----------------- Tests are executed by a call to scripts/start_test.sh from a subdirectory of one of the both directories build_test_environment and test_cases. The scope and configuration of a test is completely described by this subdirectory (we call it start directory) as follows: - All leaf subdirectories of the start directory define one test case. If the test specified by one leaf subdirectory fails the tests belonging to other leaf subdirectories of the start directory will not be executed. Though only tests which are likely not to fail should be executed by only one call to start_test.sh. The simple and recommended way is to call start_test.sh for each leaf directory once. - For each leaf subdirectory xyz there exists a configuration file named xyz.conf which contains (among others) in the variables prepare_list, setup_list, run_list, cleanup_list and finish_list names of shell functions defined in scripts/modules/*.sh. These functions are called in the order they appear in the mentioned *list variables. The file xyz.conf may reside in the leaf directory or any of it's parent directories (up to 20 levels higher in the directory tree). - The configuration of the test case belonging to a leaf directory is defined by a set of *.conf files which are included by start_test.sh in the following order: -- default-*.conf: These are the configuration files belonging to the scripts scripts/modules/*.sh which are also included bei start_test.sh and which define all functions executed by the various tests. -- user_modules.conf -- .conf where subdirname runs top down through the parent directories starting with the immediate subdirectory of the start directory (or the immediate subdirectory directory given by option --config_root_dir) to the leaf directory. If the start directory is a leaf directory the .conf is included. For example: 1. start directory = test_cases/perf leaf directory = test_cases/perf/apply/no_parallel_writer leads to including of apply.conf and no_parallel_writer.conf (in this order!) which must reside as mentioned above in the leaf directory or any of it's parent directories (up to 20 levels). 2. start directory = test_cases/perf/apply/no_parallel_writer --config_root_dir=.../test_cases/perf leaf directory = test_cases/perf/apply/no_parallel_writer leads to including the same *.conf files as above. 3. start directory = test_cases/perf/apply/no_parallel_writer leaf directory = test_cases/perf/apply/no_parallel_writer leads to including of no_parallel_writer.conf 1.1. Test output ---------------- The output fo start_test.sh consists of the following sections: - List titled "Sourcing libraries in " of included scripts from /lib*.sh - List titled "Sourcing modules and default configuration" of included scripts from scripts/modules/[0-9]*.sh and corresponding included default-*.conf files. - Line titled "Scanning subdirectories of " followed by a list of ignored or skipped subdirectories - Per leaf directory: - Line titled "Test directory " - List titled "Sourcing config files between and " of included *.conf files corresponding to the directory structure. - List titled "Configuration variables" of all configuration variables set via all included *.conf file - Section titled "Creating lock files" - Section titled "Starting " In this section lines starting with "calling ..." mark the call to one of the functions listed in the aforementioned variables prepare_list, setup_list, run_list, cleanup_list and finish_list. The main part of this section consists of output lines of the function lib_vmsg. These lines have the following format: [[]] The stack level which the bash call stack is printed from () is configurable via variable main_min_stack_level. - In case of an error: Subsection titled "Callstack" (to stderr) - Subsection titled "Deleting lock files" - Subsection titled "General checks of error and log files" - Line titled "Failure " (to stderr) in case of failure, "Finished " (to stdout) otherwise - If all tests in all leaf directories of the start directory terminated successfully: Line titled "Finished start directory " 2. Naming conventions --------------------- - All functions defined in scripts/modules/*.sh have the script's name as name prefix (e.g. resource_check_variables) - All variables used in *.conf files should be "defined" (the shell has no explicit definition) and explained in a default-*.conf file. If a variable is "defined" in default-xyz.conf it's name is prefixed by xyz. - WIP 3. Error handling ----------------- WIP 4. Signal handling ------------------ WIP 5. Configuration of the local build environment ----------------------------------------------- To build the mars module and userspace tools you need: - a directory containing a git repository which contains the kernel sources variable : checkout_mars_kernel_src_directory to be specified in file : default-checkout_mars.conf - the branch from which to take the kernel sources variable : checkout_mars_kernel_git_branch to be specified in file : default-checkout_mars.conf - a directory containing the kernel sources against which the mars module should be built (may be the same as checkout_mars_kernel_src_directory) variable : make_mars_kernel_src_directory to be specified in file : default-make_mars.conf - a directory containing a git repository which contains the mars sources variable : checkout_mars_src_directory to be specified in file : default-checkout_mars.conf - the branch from which to take the mars sources variable : checkout_mars_git_branch to be specified in file : default-checkout_mars.conf - a directory containing the mars sources which is linked in to the kernel source directory make_mars_kernel_src_directory for making the mars module. May coincide with checkout_mars_src_directory variable : make_mars_src_directory to be specified in file : default-make_mars.conf - a directory containing a git repository which contains the userspace tool marsview variable : checkout_mars_contrib_src_directory to be specified in file : default-checkout_mars.conf - the branch from which to take marsview variable : checkout_mars_contrib_git_branch to be specified in file : default-checkout_mars.conf 6. Configuration of the test hosts on which the tests are running ------------------------------------------------------------ The test hosts are given in main_host_list (default-main.conf). 6.1. Access requirements ------------------------ The tests are executed by ssh commands from your work station to the test hosts. These commands must not prompt for a password. Though a id file (for ssh -i ) with null password must be specified in main_ssh_idfile_opt (default-main.conf) All test hosts must have ssh access to each other without prompt for passwords (this is required by marsadm join-cluster). On your work station you must have sudo permissions for "make install", "cp". And root of your work station must have ssh access (as root) without prompt for password to all test hosts (this is required, because local files belonging to root must be copied to the test hosts). 6.2. Installation kernel and mars module on the test hosts ---------------------------------------------------------- This is done by executing the tests build_test_environment/checkout build_test_environment/make/make_mars/grub build_test_environment/install_mars It's a must to read default-checkout_mars.conf, default-make_mars.conf, default-install_mars.conf and the *.conf files in the directories above otherwise you risk damage of the boot information on your work station and/or the test hosts. After these three tests (which you should call one after the other!) you can reboot the test hosts with the new kernel. 7. Configuration of logical volumes ----------------------------------- This is done by the test build_test_environment/lv_config The configuration options are contained in default-lv_config.conf 8. Resources ------------ Cluster and Resources are created by build_test_environment/cluster build_test_environment/resource/create_resource Initially the first host in main_host_list is the primary, the following hosts secondaries. If you do not change any configuration variables after the build_test_environment/resource/create_resource Test you will have created one resource lv-1-2 on all hosts with underlying data devices /dev/vg-mars/lv-1-2. /mars will be mounted on /dev/vg-mars/lv-6-100. For further information please read default-cluster.conf and default-resource.conf. 9. Starting the whole test suite via cronjob -------------------------------------------- This can be done mars_test_cronjob.sh. The variable tests_to_execute contains all tests to be executed. 10. Concurrent test runs ------------------------ To avoid concurrent run of tests on the same host each run creates temporary files (/tmp/test-suite_on., see main_lock_file_list) and deletes them afterwards.