2013-07-04 08:05:39 +00:00
|
|
|
GPLed software AS IS, sponsored by 1&1 Internet AG (www.1und1.de).
|
|
|
|
|
|
|
|
Contact frank.liepold@1und1.de
|
|
|
|
|
|
|
|
|
|
|
|
The test suite is work in progress.
|
|
|
|
|
|
|
|
At the moment it is not in the state to be usable outside of the development
|
|
|
|
team.
|
2014-01-29 08:20:27 +00:00
|
|
|
Even this file is not complete!
|
2013-11-07 09:33:43 +00:00
|
|
|
|
|
|
|
|
|
|
|
=============================================================================
|
|
|
|
|
|
|
|
Contents
|
|
|
|
--------
|
|
|
|
1. Running a test
|
2013-12-30 09:28:30 +00:00
|
|
|
1.1. Test output
|
2013-11-07 09:33:43 +00:00
|
|
|
2. Naming conventions
|
|
|
|
3. Error handling
|
|
|
|
4. Signal handling
|
2013-12-10 14:32:40 +00:00
|
|
|
5. Configuration of the local build environment
|
|
|
|
6. Configuration of the test hosts on which the tests are running
|
|
|
|
6.1. Access requirements
|
|
|
|
6.2. Installation kernel and mars module on the test hosts
|
|
|
|
7. Configuration of logical volumes
|
|
|
|
8. Resources
|
|
|
|
9. Starting the whole test suite via cronjob
|
|
|
|
10. Concurrent test runs
|
2014-01-21 10:18:36 +00:00
|
|
|
11. Firewalls
|
2014-02-03 07:21:06 +00:00
|
|
|
12. Programming hints and conventions
|
2013-11-07 09:33:43 +00:00
|
|
|
|
|
|
|
1. Running a test
|
|
|
|
-----------------
|
|
|
|
|
|
|
|
Tests are executed by a call to scripts/start_test.sh from a subdirectory
|
|
|
|
of one of the both directories build_test_environment and test_cases.
|
|
|
|
|
|
|
|
The scope and configuration of a test is completely described by this
|
|
|
|
subdirectory (we call it start directory) as follows:
|
|
|
|
|
2014-01-29 08:20:27 +00:00
|
|
|
- All subdirectories of the start directory containing a (usually empty)
|
|
|
|
file named i_am_a_testdirectory define one test case. In the following
|
|
|
|
we call such a subdirectory a test directory.
|
|
|
|
If a test for one test directory below the start directory fails, the tests
|
|
|
|
belonging to the other test directories of the start directory will not be
|
|
|
|
executed.
|
2013-12-10 14:32:40 +00:00
|
|
|
Though only tests which are likely not to fail should be executed by only
|
|
|
|
one call to start_test.sh. The simple and recommended way is to call
|
2014-01-29 08:20:27 +00:00
|
|
|
start_test.sh for each test directory once (which means that the
|
|
|
|
start directory consists of only one test directory and coincides with it.
|
|
|
|
|
|
|
|
- The configuration of the test case is defined by a set of *.conf files which
|
|
|
|
are included by start_test.sh in the following order:
|
|
|
|
|
2013-11-07 09:33:43 +00:00
|
|
|
-- default-*.conf: These are the configuration files belonging to the
|
|
|
|
scripts scripts/modules/*.sh which are also included
|
2014-01-29 08:20:27 +00:00
|
|
|
bei start_test.sh and which define all functions which may
|
|
|
|
be used by the various tests.
|
|
|
|
|
2013-11-07 09:33:43 +00:00
|
|
|
-- user_modules.conf
|
2014-01-29 08:20:27 +00:00
|
|
|
|
2013-12-23 06:04:59 +00:00
|
|
|
-- <subdirname>.conf where subdirname runs top down through the parent
|
2014-01-29 08:20:27 +00:00
|
|
|
directories starting with the first subdirectory of a directory called
|
|
|
|
root config directory to the test directory.
|
|
|
|
This config root directory is set to the start directory (on the directory
|
|
|
|
branch start directory -> test directory) by default. It may be changed by
|
|
|
|
the option --config_root_dir=<my dir>.
|
|
|
|
If the start directory coincides with the test directory the file
|
|
|
|
<test directory name>.conf is included for conveniance (in the strict
|
|
|
|
sense there are no true subdirectories of the start directory residing
|
|
|
|
above the test directory).
|
|
|
|
The <subdirnam>.conf files may reside in the test directory or any of
|
|
|
|
it's parent directories (up to 20 levels).
|
2013-12-23 06:04:59 +00:00
|
|
|
|
2014-01-29 08:20:27 +00:00
|
|
|
|
|
|
|
Examples:
|
2013-11-07 09:33:43 +00:00
|
|
|
|
2013-12-23 06:04:59 +00:00
|
|
|
1. start directory = test_cases/perf
|
2014-01-29 08:20:27 +00:00
|
|
|
test directory = test_cases/perf/apply/no_parallel_writer
|
|
|
|
Option --config_root_dir not given
|
2013-12-23 06:04:59 +00:00
|
|
|
|
2013-12-30 09:28:30 +00:00
|
|
|
leads to including of apply.conf and no_parallel_writer.conf (in
|
|
|
|
this order!) which must reside as mentioned above in the leaf
|
|
|
|
directory or any of it's parent directories (up to 20 levels).
|
2013-12-23 06:04:59 +00:00
|
|
|
|
|
|
|
2. start directory = test_cases/perf/apply/no_parallel_writer
|
|
|
|
leaf directory = test_cases/perf/apply/no_parallel_writer
|
2014-01-29 08:20:27 +00:00
|
|
|
Option --config_root_dir=test_cases/perf
|
2013-11-07 09:33:43 +00:00
|
|
|
|
2013-12-23 06:04:59 +00:00
|
|
|
leads to including the same *.conf files as above.
|
2013-11-07 09:33:43 +00:00
|
|
|
|
2013-12-23 06:04:59 +00:00
|
|
|
3. start directory = test_cases/perf/apply/no_parallel_writer
|
|
|
|
leaf directory = test_cases/perf/apply/no_parallel_writer
|
2014-01-29 08:20:27 +00:00
|
|
|
Option --config_root_dir given or not
|
2013-11-07 09:33:43 +00:00
|
|
|
|
2013-12-23 06:04:59 +00:00
|
|
|
leads to including of no_parallel_writer.conf
|
2013-11-07 09:33:43 +00:00
|
|
|
|
2014-01-29 08:20:27 +00:00
|
|
|
- In one of the included <subdirnam>.conf files of a test one of the variables
|
|
|
|
prepare_list, setup_list, run_list, cleanup_list or finish_list must be set.
|
|
|
|
Each of these variables may be empty or contain a list of shell function
|
|
|
|
names of functions defined in scripts/modules/*.sh.
|
|
|
|
These functions are called by start_test.sh in the order they appear in the
|
|
|
|
mentioned *list variables. The *list variables are evaluated in the order
|
|
|
|
prepare_list, setup_list, run_list, cleanup_list, finish_list.
|
|
|
|
|
2014-01-09 06:31:18 +00:00
|
|
|
1.1. Test output
|
|
|
|
----------------
|
2013-12-30 09:28:30 +00:00
|
|
|
|
|
|
|
The output fo start_test.sh consists of the following sections:
|
|
|
|
|
2014-01-08 09:18:31 +00:00
|
|
|
- List titled "Sourcing libraries in <library directory>" of included scripts
|
|
|
|
from <libraries directory>/lib*.sh
|
|
|
|
|
2013-12-30 09:28:30 +00:00
|
|
|
- List titled "Sourcing modules and default configuration" of included scripts
|
2014-01-08 09:18:31 +00:00
|
|
|
from scripts/modules/[0-9]*.sh and corresponding included default-*.conf
|
|
|
|
files.
|
|
|
|
|
|
|
|
- Line titled "Scanning subdirectories of <start directory>" followed by a
|
|
|
|
list of ignored or skipped subdirectories
|
2013-12-30 09:28:30 +00:00
|
|
|
|
|
|
|
- Per leaf directory:
|
2014-01-09 06:31:18 +00:00
|
|
|
- Line titled "Test directory <leaf directory> <date and time>"
|
2013-12-30 09:28:30 +00:00
|
|
|
- List titled "Sourcing config files between <config_root_dir> and
|
|
|
|
<leaf directory>" of included *.conf files corresponding to the directory
|
|
|
|
structure.
|
|
|
|
|
|
|
|
- List titled "Configuration variables" of all configuration variables set
|
|
|
|
via all included *.conf file
|
|
|
|
|
|
|
|
- Section titled "Creating lock files"
|
|
|
|
|
2014-01-09 06:31:18 +00:00
|
|
|
- Section titled "Starting <leaf directory> <date and time>"
|
2013-12-30 09:28:30 +00:00
|
|
|
In this section lines starting with "calling ..." mark the call to one of
|
|
|
|
the functions listed in the aforementioned variables prepare_list,
|
2013-12-30 14:58:22 +00:00
|
|
|
setup_list, run_list, cleanup_list and finish_list.
|
|
|
|
The main part of this section consists of output lines of the function
|
|
|
|
lib_vmsg. These lines have the following format:
|
2014-01-13 08:35:19 +00:00
|
|
|
<date time> [[<callstack>]] <message>
|
2014-01-14 07:56:07 +00:00
|
|
|
The stack level which the bash call stack is printed from (<callstack>)
|
|
|
|
is configurable via variable main_min_stack_level.
|
2013-12-30 09:28:30 +00:00
|
|
|
|
|
|
|
- In case of an error: Subsection titled "Callstack" (to stderr)
|
|
|
|
|
|
|
|
- Subsection titled "Deleting lock files"
|
|
|
|
|
|
|
|
- Subsection titled "General checks of error and log files"
|
2014-01-09 06:31:18 +00:00
|
|
|
- Line titled "Failure <return code> <leaf directory> <date and time>"
|
|
|
|
(to stderr) in case of failure,
|
|
|
|
"Finished <leaf directory> <date and time>" (to stdout) otherwise
|
2013-12-30 09:28:30 +00:00
|
|
|
|
2013-12-30 10:19:21 +00:00
|
|
|
- If all tests in all leaf directories of the start directory terminated
|
2014-01-09 06:31:18 +00:00
|
|
|
successfully: Line titled "Finished start directory <start directory>"
|
2013-12-30 09:28:30 +00:00
|
|
|
|
2013-11-07 09:33:43 +00:00
|
|
|
|
|
|
|
2. Naming conventions
|
|
|
|
---------------------
|
|
|
|
|
|
|
|
- All functions defined in scripts/modules/*.sh have the script's name as
|
|
|
|
name prefix (e.g. resource_check_variables)
|
|
|
|
- All variables used in *.conf files should be "defined" (the shell has no
|
|
|
|
explicit definition) and explained in a default-*.conf file. If a variable is
|
|
|
|
"defined" in default-xyz.conf it's name is prefixed by xyz.
|
2013-12-10 14:32:40 +00:00
|
|
|
- WIP
|
2013-11-07 09:33:43 +00:00
|
|
|
|
|
|
|
|
|
|
|
3. Error handling
|
|
|
|
-----------------
|
|
|
|
|
|
|
|
WIP
|
|
|
|
|
|
|
|
4. Signal handling
|
|
|
|
------------------
|
|
|
|
|
|
|
|
WIP
|
|
|
|
|
2013-12-10 14:32:40 +00:00
|
|
|
5. Configuration of the local build environment
|
|
|
|
-----------------------------------------------
|
2013-11-07 09:33:43 +00:00
|
|
|
To build the mars module and userspace tools you need:
|
|
|
|
|
|
|
|
- a directory containing a git repository which contains the kernel sources
|
|
|
|
variable : checkout_mars_kernel_src_directory
|
|
|
|
to be specified in file : default-checkout_mars.conf
|
|
|
|
|
|
|
|
- the branch from which to take the kernel sources
|
|
|
|
variable : checkout_mars_kernel_git_branch
|
|
|
|
to be specified in file : default-checkout_mars.conf
|
|
|
|
|
|
|
|
- a directory containing the kernel sources against which the mars module
|
|
|
|
should be built (may be the same as checkout_mars_kernel_src_directory)
|
|
|
|
variable : make_mars_kernel_src_directory
|
|
|
|
to be specified in file : default-make_mars.conf
|
|
|
|
|
|
|
|
- a directory containing a git repository which contains the mars sources
|
|
|
|
variable : checkout_mars_src_directory
|
|
|
|
to be specified in file : default-checkout_mars.conf
|
|
|
|
|
|
|
|
- the branch from which to take the mars sources
|
|
|
|
variable : checkout_mars_git_branch
|
|
|
|
to be specified in file : default-checkout_mars.conf
|
|
|
|
|
|
|
|
- a directory containing the mars sources which is linked in to the
|
|
|
|
kernel source directory make_mars_kernel_src_directory for making the mars
|
|
|
|
module.
|
|
|
|
May coincide with checkout_mars_src_directory
|
2013-12-27 06:27:11 +00:00
|
|
|
variable : make_mars_src_directory
|
2013-11-07 09:33:43 +00:00
|
|
|
to be specified in file : default-make_mars.conf
|
|
|
|
|
|
|
|
- a directory containing a git repository which contains the userspace tool
|
|
|
|
marsview
|
|
|
|
variable : checkout_mars_contrib_src_directory
|
|
|
|
to be specified in file : default-checkout_mars.conf
|
|
|
|
|
|
|
|
- the branch from which to take marsview
|
|
|
|
variable : checkout_mars_contrib_git_branch
|
|
|
|
to be specified in file : default-checkout_mars.conf
|
|
|
|
|
|
|
|
|
2013-12-10 14:32:40 +00:00
|
|
|
6. Configuration of the test hosts on which the tests are running
|
2013-11-07 09:33:43 +00:00
|
|
|
------------------------------------------------------------
|
|
|
|
|
2013-12-10 14:32:40 +00:00
|
|
|
The test hosts are given in main_host_list (default-main.conf).
|
2013-11-07 09:33:43 +00:00
|
|
|
|
2013-12-10 14:32:40 +00:00
|
|
|
6.1. Access requirements
|
|
|
|
------------------------
|
|
|
|
The tests are executed by ssh commands from your work station to the
|
|
|
|
test hosts. These commands must not prompt for a password. Though a
|
|
|
|
id file (for ssh -i <id-file>) with null password must be specified in
|
|
|
|
main_ssh_idfile_opt (default-main.conf)
|
2013-11-07 09:33:43 +00:00
|
|
|
|
2013-12-10 14:32:40 +00:00
|
|
|
All test hosts must have ssh access to each other without prompt for passwords
|
|
|
|
(this is required by marsadm join-cluster).
|
2013-11-07 09:33:43 +00:00
|
|
|
|
|
|
|
|
2013-12-10 14:32:40 +00:00
|
|
|
On your work station you must have sudo permissions for "make install", "cp".
|
|
|
|
|
|
|
|
And root of your work station must have ssh access (as root) without prompt for
|
|
|
|
password to all test hosts (this is required, because local files belonging to
|
|
|
|
root must be copied to the test hosts).
|
|
|
|
|
|
|
|
|
|
|
|
6.2. Installation kernel and mars module on the test hosts
|
|
|
|
----------------------------------------------------------
|
|
|
|
This is done by executing the tests
|
|
|
|
|
|
|
|
build_test_environment/checkout
|
|
|
|
build_test_environment/make/make_mars/grub
|
|
|
|
build_test_environment/install_mars
|
|
|
|
|
|
|
|
It's a must to read default-checkout_mars.conf, default-make_mars.conf,
|
|
|
|
default-install_mars.conf and the *.conf files in the directories above
|
2014-02-03 07:21:06 +00:00
|
|
|
otherwise you risk damage of the boot information on your work station and/or
|
|
|
|
the test hosts.
|
2013-12-10 14:32:40 +00:00
|
|
|
|
|
|
|
After these three tests (which you should call one after the other!) you can
|
|
|
|
reboot the test hosts with the new kernel.
|
|
|
|
|
|
|
|
7. Configuration of logical volumes
|
|
|
|
-----------------------------------
|
|
|
|
This is done by the test
|
|
|
|
|
|
|
|
build_test_environment/lv_config
|
|
|
|
|
|
|
|
The configuration options are contained in default-lv_config.conf
|
|
|
|
|
|
|
|
|
|
|
|
8. Resources
|
|
|
|
------------
|
|
|
|
Cluster and Resources are created by
|
|
|
|
|
|
|
|
build_test_environment/cluster
|
|
|
|
build_test_environment/resource/create_resource
|
|
|
|
|
|
|
|
Initially the first host in main_host_list is the primary, the following hosts
|
|
|
|
secondaries.
|
|
|
|
If you do not change any configuration variables after the
|
|
|
|
build_test_environment/resource/create_resource Test you will have created one
|
|
|
|
resource lv-1-2 on all hosts with underlying data devices /dev/vg-mars/lv-1-2.
|
|
|
|
/mars will be mounted on /dev/vg-mars/lv-6-100.
|
|
|
|
|
|
|
|
For further information please read default-cluster.conf and
|
|
|
|
default-resource.conf.
|
|
|
|
|
|
|
|
|
|
|
|
9. Starting the whole test suite via cronjob
|
|
|
|
--------------------------------------------
|
|
|
|
This can be done mars_test_cronjob.sh. The variable tests_to_execute contains
|
2014-01-22 08:48:05 +00:00
|
|
|
all tests to be executed. See the documentation in the header.
|
2013-12-10 14:32:40 +00:00
|
|
|
|
|
|
|
10. Concurrent test runs
|
|
|
|
------------------------
|
|
|
|
To avoid concurrent run of tests on the same host each run creates temporary
|
|
|
|
files (/tmp/test-suite_on.<host>, see main_lock_file_list) and deletes them
|
|
|
|
afterwards.
|
|
|
|
|
2014-01-21 10:18:36 +00:00
|
|
|
11. Firewalls
|
|
|
|
-------------
|
|
|
|
Certain tests cut temporarily network connections by defining firewall rules.
|
|
|
|
Normally even in case of failure these connections are restored. To be sure
|
|
|
|
that no firewall rules prevent tests from running the flag
|
|
|
|
net_clear_iptables_in_prepare_phase is set to 1 in default-net.conf.
|
|
|
|
This flag leads to deletion of *all* iptable chains on the test hosts.
|
|
|
|
If you do not want this behaviour set net_clear_iptables_in_prepare_phase=0.
|
2013-11-07 09:33:43 +00:00
|
|
|
|
|
|
|
|
2014-02-03 07:21:06 +00:00
|
|
|
12. Programming hints and conventions
|
|
|
|
-------------------------------------
|
|
|
|
|
|
|
|
- all global variables should be defined and explained in a
|
|
|
|
default-<module_name>.conf file
|
|
|
|
- the names of all global variables resp. all functions should have as prefix the
|
|
|
|
module name of the default-*.conf file resp. of the <module>.sh file which they
|
|
|
|
are defined in.
|
|
|
|
|
|
|
|
- in case of an error certain cleanup functions may be called (e.g. if
|
|
|
|
a network connection is cut during a test case it should be restored).
|
|
|
|
The associotive array main_error_recovery_functions has function names as
|
|
|
|
index an function parameters as values. In lib_exit each function contained
|
|
|
|
in main_error_recovery_functions is called with it's corresponding arguments.
|
2013-11-07 09:33:43 +00:00
|
|
|
|
2013-07-04 08:05:39 +00:00
|
|
|
|