mirror of https://github.com/schoebel/mars
test_suite: separation of frame work and use case mars
The mars test cases are now located in .../test_suite/mars. The use case .../test_suite/example describes how the frame work works (please refer to README). mars_test_cronjob.sh have a second parameter which specifies the use case directory (.../test_suite/mars). The leading numbers in the module numbers are no longer needed.
This commit is contained in:
Frank Liepold
parent
f9d42097c8
commit
e38d873564
|
|
@ -5,36 +5,58 @@ Contact frank.liepold@1und1.de
|
|||
|
||||
The test suite is work in progress.
|
||||
|
||||
At the moment it is not in the state to be usable outside of the development
|
||||
team.
|
||||
Even this file is not complete!
|
||||
|
||||
|
||||
=============================================================================
|
||||
|
||||
Contents
|
||||
--------
|
||||
1. Running a test
|
||||
1.1. Test output
|
||||
2. Naming conventions
|
||||
3. Error handling
|
||||
4. Signal handling
|
||||
5. Configuration of the local build environment
|
||||
6. Configuration of the test hosts on which the tests are running
|
||||
6.1. Access requirements
|
||||
6.2. Installation kernel and mars module on the test hosts
|
||||
7. Configuration of logical volumes
|
||||
8. Resources
|
||||
9. Starting the whole test suite via cronjob
|
||||
10. Concurrent test runs
|
||||
11. Firewalls
|
||||
12. Programming hints and conventions
|
||||
1. Framework
|
||||
1.1. Global settings
|
||||
1.2. Running a test
|
||||
1.2.1. Test output
|
||||
1.3. Programming hints and conventions
|
||||
1.4. Error handling
|
||||
1.5. Signal handling and unexpected termination
|
||||
1.6. Example
|
||||
2. mars specific settings
|
||||
2.1. Configuration of the local build environment
|
||||
2.2. Configuration of the test hosts on which the tests are running
|
||||
2.2.1. Access requirements
|
||||
2.2.2. Installation kernel and mars module on the test hosts
|
||||
2.3. Configuration of logical volumes
|
||||
2.4. Resources
|
||||
2.5. Starting the whole test suite via cronjob
|
||||
2.6. Concurrent test runs
|
||||
2.7. Firewalls
|
||||
|
||||
1. Running a test
|
||||
-----------------
|
||||
1. Framework
|
||||
------------
|
||||
|
||||
Tests are executed by a call to scripts/start_test.sh from a subdirectory
|
||||
of one of the both directories build_test_environment and test_cases.
|
||||
1.1. Global settings
|
||||
------------------
|
||||
The directory where this README resides (normally <git-repo>/test_suite) is
|
||||
called base directory in the following. If relativ paths are given they refer to
|
||||
this base directory.
|
||||
|
||||
The frame work of the test suite consists of the files README,
|
||||
default-main.conf and the subdirectory scripts.
|
||||
|
||||
You specify your use case of the test suite by creating a subdirectory
|
||||
- e.g. my_use_case_dir - of the base directory.
|
||||
|
||||
In this subdirectory a file global.conf must be created, containing at lease
|
||||
the following two variables:
|
||||
global_user_dir must be set to .../test_suite/my_use_case_dir
|
||||
global_user_module_dir must be set to a subdirectory of global_user_dir.
|
||||
In global_user_module_dir the use case specific shell scripts (called modules)
|
||||
are to be located.
|
||||
We recommend global_user_module_dir=$global_user_dir/modules.
|
||||
|
||||
1.2. Running a test
|
||||
-------------------
|
||||
|
||||
Tests are executed by a call to scripts/start_test.sh from a subdirectory of
|
||||
$global_user_dir.
|
||||
|
||||
The scope and configuration of a test is completely described by this
|
||||
subdirectory (we call it start directory) as follows:
|
||||
|
|
@ -48,17 +70,31 @@ subdirectory (we call it start directory) as follows:
|
|||
Though only tests which are likely not to fail should be executed by only
|
||||
one call to start_test.sh. The simple and recommended way is to call
|
||||
start_test.sh for each test directory once (which means that the
|
||||
start directory consists of only one test directory and coincides with it.
|
||||
start directory consists of only one test directory and coincides with it).
|
||||
|
||||
- The function set usable is defined in the following set of shell scripts:
|
||||
-- scripts/lib/lib*.sh
|
||||
-- scripts/modules/*.sh
|
||||
-- $global_user_module_dir/*.sh
|
||||
|
||||
- The configuration of the test case is defined by a set of *.conf files which
|
||||
are included by start_test.sh in the following order:
|
||||
|
||||
-- default-*.conf: These are the configuration files belonging to the
|
||||
scripts scripts/modules/*.sh which are also included
|
||||
bei start_test.sh and which define all functions which may
|
||||
be used by the various tests.
|
||||
-- default-*.conf:
|
||||
These are the configuration files belonging to the scripts
|
||||
scripts/modules/*.sh which are also included bei start_test.sh and
|
||||
which define some library functions which are independent of the
|
||||
specific use case.
|
||||
|
||||
-- $global_user_dir/global.conf:
|
||||
Configuration file for use case specific global variables which cannot be
|
||||
assigned to a certain user module.
|
||||
|
||||
-- $global_user_dir/default-*.conf:
|
||||
These are the configuration files belonging to the scripts
|
||||
$global_user_module_dir/*.sh which are also included bei start_test.sh and
|
||||
which define all functions necessary for your use case.
|
||||
|
||||
-- user_modules.conf
|
||||
|
||||
-- <subdirname>.conf where subdirname runs top down through the parent
|
||||
directories starting with the first subdirectory of a directory called
|
||||
|
|
@ -96,25 +132,29 @@ subdirectory (we call it start directory) as follows:
|
|||
|
||||
leads to including of no_parallel_writer.conf
|
||||
|
||||
- In one of the included <subdirnam>.conf files of a test one of the variables
|
||||
prepare_list, setup_list, run_list, cleanup_list or finish_list must be set.
|
||||
- In at least one of the included <subdirname>.conf files of a test you must
|
||||
set one of the variables prepare_list, setup_list, run_list, cleanup_list or
|
||||
finish_list.
|
||||
Each of these variables may be empty or contain a list of shell function
|
||||
names of functions defined in scripts/modules/*.sh.
|
||||
names of functions defined in one of the set of shell scripts mentioned
|
||||
above.
|
||||
These functions are called by start_test.sh in the order they appear in the
|
||||
mentioned *list variables. The *list variables are evaluated in the order
|
||||
prepare_list, setup_list, run_list, cleanup_list, finish_list.
|
||||
|
||||
1.1. Test output
|
||||
----------------
|
||||
1.2.1. Test output
|
||||
------------------
|
||||
|
||||
The output fo start_test.sh consists of the following sections:
|
||||
The output of start_test.sh is very verbose to ease the research in case of any
|
||||
errors. You are encouraged to use grep & Co. to shrink it as desired.
|
||||
The output consists of the following sections:
|
||||
|
||||
- List titled "Sourcing libraries in <library directory>" of included scripts
|
||||
from <libraries directory>/lib*.sh
|
||||
|
||||
- List titled "Sourcing modules and default configuration" of included scripts
|
||||
from scripts/modules/[0-9]*.sh and corresponding included default-*.conf
|
||||
files.
|
||||
(= the set of shell scripts mentioned above) and corresponding included
|
||||
default-*.conf files.
|
||||
|
||||
- Line titled "Scanning subdirectories of <start directory>" followed by a
|
||||
list of ignored or skipped subdirectories
|
||||
|
|
@ -136,7 +176,7 @@ The output fo start_test.sh consists of the following sections:
|
|||
setup_list, run_list, cleanup_list and finish_list.
|
||||
The main part of this section consists of output lines of the function
|
||||
lib_vmsg. These lines have the following format:
|
||||
<date time> [[<callstack>]] <message>
|
||||
<date time> [[<callstack>]] <message>
|
||||
The stack level which the bash call stack is printed from (<callstack>)
|
||||
is configurable via variable main_min_stack_level.
|
||||
|
||||
|
|
@ -153,29 +193,102 @@ The output fo start_test.sh consists of the following sections:
|
|||
successfully: Line titled "Finished start directory <start directory>"
|
||||
|
||||
|
||||
2. Naming conventions
|
||||
---------------------
|
||||
1.3. Programming hints and conventions
|
||||
--------------------------------------
|
||||
|
||||
- All functions defined in scripts/modules/*.sh have the script's name as
|
||||
name prefix (e.g. resource_check_variables)
|
||||
- All variables used in *.conf files should be "defined" (the shell has no
|
||||
explicit definition) and explained in a default-*.conf file. If a variable is
|
||||
"defined" in default-xyz.conf it's name is prefixed by xyz.
|
||||
- WIP
|
||||
- all global variables should be defined and explained in a
|
||||
default-<module_name>.conf file
|
||||
|
||||
- the names of all global variables resp. all functions should have as prefix the
|
||||
module name of the default-*.conf file resp. of the *.sh file which they are
|
||||
defined in.
|
||||
|
||||
- in case of an error certain cleanup functions may be called (e.g. if
|
||||
a network connection is cut during a test case it should be restored).
|
||||
The associative array main_error_recovery_functions has function names as
|
||||
index and function parameters as values. In lib_exit each function contained
|
||||
in main_error_recovery_functions is called with it's corresponding arguments.
|
||||
|
||||
|
||||
3. Error handling
|
||||
-----------------
|
||||
1.4. Error handling
|
||||
-------------------
|
||||
|
||||
WIP
|
||||
Instead of directly calling exit you should use lib_exit from lib/lib.sh, which
|
||||
prints a call stack and does further checks.
|
||||
If the variable lib_general_checks_after_every_test_function is set in
|
||||
global_user_dir/global.conf to a shell function contained in the use case
|
||||
specific modules, this function is called by lib_exit.
|
||||
|
||||
4. Signal handling
|
||||
------------------
|
||||
|
||||
WIP
|
||||
|
||||
5. Configuration of the local build environment
|
||||
1.5. Signal handling and unexpected termination
|
||||
-----------------------------------------------
|
||||
|
||||
Normally signals given to start_test.sh should terminate all child processes but
|
||||
this can not be guaranteed. Thus you have to kill left over child processes
|
||||
manually.
|
||||
Thus we recommend that all scripts and programs you are starting within a test
|
||||
case should have a significant part of name - e.g. in the use case mars we used
|
||||
the string MARS as part of all script names.
|
||||
|
||||
1.6. Example
|
||||
------------
|
||||
|
||||
In the base directory's subdirectory example you find a simple use case. The
|
||||
directory tree looks as follows:
|
||||
|
||||
example/blue.conf
|
||||
example/color_tests.conf
|
||||
example/default-colors.conf
|
||||
example/default-lib_err.conf
|
||||
example/global.conf
|
||||
example/green.conf
|
||||
example/red.conf
|
||||
example/color_tests
|
||||
example/color_tests/blue
|
||||
example/color_tests/blue/i_am_a_testdirectory
|
||||
example/color_tests/green
|
||||
example/color_tests/green/i_am_a_testdirectory
|
||||
example/color_tests/red
|
||||
example/color_tests/red/i_am_a_testdirectory
|
||||
example/modules
|
||||
example/modules/colors.sh
|
||||
example/modules/lib_err.sh
|
||||
|
||||
Three test case are defined because there are three files i_am_a_testdirectory.
|
||||
The used shell functions are defined in the scripts colors.sh and lib_err.sh
|
||||
with corresponding *.conf files default-colors.conf resp. default-lib_err.conf.
|
||||
The test case configurations are given via the directory tree and the *.conf
|
||||
files red.conf, green.conf and blue.conf.
|
||||
In global.conf we defined global_user_dir, global_user_module_dir,
|
||||
main_min_stack_level and lib_general_checks_after_every_test_function.
|
||||
|
||||
To start all three tests do:
|
||||
|
||||
cd base directory/example
|
||||
../scripts/start_test.sh 2>&1 | tee /tmp/test_all.out
|
||||
|
||||
Thoroughly studying the output file /tmp/my_test.out will clarify the working of
|
||||
the test frame work.
|
||||
The third test (red) is of special interest because it fails.
|
||||
Thus in this case you can study the call stack, the call to
|
||||
lib_general_checks_after_every_test_function (in our example set to
|
||||
lib_err_general_checks_after_every_test_case) and the use of
|
||||
main_error_recovery_functions (used in colors.sh).
|
||||
|
||||
To start only the test case red do:
|
||||
|
||||
cd base directory/example/colors/red
|
||||
../../../scripts/start_test.sh --config_root_dir=../.. 2>&1 | tee /tmp/test_red.out
|
||||
|
||||
Note that the option --config_root_dir is important in this case because it
|
||||
tells start_test.sh up to which level it should regard parent directories and
|
||||
the corresponding *.conf files (see section 1.2).
|
||||
|
||||
|
||||
2. mars specific settings
|
||||
-------------------------
|
||||
|
||||
2.1. Configuration of the local build environment
|
||||
-------------------------------------------------
|
||||
To build the mars module and userspace tools you need:
|
||||
|
||||
- a directory containing a git repository which contains the kernel sources
|
||||
|
|
@ -216,13 +329,13 @@ To build the mars module and userspace tools you need:
|
|||
to be specified in file : default-checkout_mars.conf
|
||||
|
||||
|
||||
6. Configuration of the test hosts on which the tests are running
|
||||
------------------------------------------------------------
|
||||
2.2. Configuration of the test hosts on which the tests are running
|
||||
-------------------------------------------------------------------
|
||||
|
||||
The test hosts are given in main_host_list (default-main.conf).
|
||||
The test hosts are given in global_host_list (global.conf).
|
||||
|
||||
6.1. Access requirements
|
||||
------------------------
|
||||
2.2.1. Access requirements
|
||||
--------------------------
|
||||
The tests are executed by ssh commands from your work station to the
|
||||
test hosts. These commands must not prompt for a password. Though a
|
||||
id file (for ssh -i <id-file>) with null password must be specified in
|
||||
|
|
@ -239,8 +352,8 @@ password to all test hosts (this is required, because local files belonging to
|
|||
root must be copied to the test hosts).
|
||||
|
||||
|
||||
6.2. Installation kernel and mars module on the test hosts
|
||||
----------------------------------------------------------
|
||||
2.2.2. Installation kernel and mars module on the test hosts
|
||||
------------------------------------------------------------
|
||||
This is done by executing the tests
|
||||
|
||||
build_test_environment/checkout
|
||||
|
|
@ -255,8 +368,8 @@ the test hosts.
|
|||
After these three tests (which you should call one after the other!) you can
|
||||
reboot the test hosts with the new kernel.
|
||||
|
||||
7. Configuration of logical volumes
|
||||
-----------------------------------
|
||||
2.3. Configuration of logical volumes
|
||||
-------------------------------------
|
||||
This is done by the test
|
||||
|
||||
build_test_environment/lv_config
|
||||
|
|
@ -264,14 +377,14 @@ build_test_environment/lv_config
|
|||
The configuration options are contained in default-lv_config.conf
|
||||
|
||||
|
||||
8. Resources
|
||||
------------
|
||||
2.4. Resources
|
||||
--------------
|
||||
Cluster and Resources are created by
|
||||
|
||||
build_test_environment/cluster
|
||||
build_test_environment/resource/create_resource
|
||||
|
||||
Initially the first host in main_host_list is the primary, the following hosts
|
||||
Initially the first host in global_host_list is the primary, the following hosts
|
||||
secondaries.
|
||||
If you do not change any configuration variables after the
|
||||
build_test_environment/resource/create_resource Test you will have created one
|
||||
|
|
@ -282,40 +395,24 @@ For further information please read default-cluster.conf and
|
|||
default-resource.conf.
|
||||
|
||||
|
||||
9. Starting the whole test suite via cronjob
|
||||
--------------------------------------------
|
||||
2.5. Starting the whole test suite via cronjob
|
||||
----------------------------------------------
|
||||
This can be done mars_test_cronjob.sh. The variable tests_to_execute contains
|
||||
all tests to be executed. See the documentation in the header.
|
||||
|
||||
10. Concurrent test runs
|
||||
------------------------
|
||||
2.6. Concurrent test runs
|
||||
-------------------------
|
||||
To avoid concurrent run of tests on the same host each run creates temporary
|
||||
files (/tmp/test-suite_on.<host>, see main_lock_file_list) and deletes them
|
||||
afterwards.
|
||||
files (/tmp/test-suite_on.<host>, see main_lock_file_list in global.conf) and
|
||||
deletes them afterwards. It may happen that these files are left over
|
||||
unintentionally in some cases of unexpected termination. Then you must remove
|
||||
them manually before starting the next test run.
|
||||
|
||||
11. Firewalls
|
||||
-------------
|
||||
2.7. Firewalls
|
||||
--------------
|
||||
Certain tests cut temporarily network connections by defining firewall rules.
|
||||
Normally even in case of failure these connections are restored. To be sure
|
||||
that no firewall rules prevent tests from running the flag
|
||||
net_clear_iptables_in_prepare_phase is set to 1 in default-net.conf.
|
||||
This flag leads to deletion of *all* iptable chains on the test hosts.
|
||||
If you do not want this behaviour set net_clear_iptables_in_prepare_phase=0.
|
||||
|
||||
|
||||
12. Programming hints and conventions
|
||||
-------------------------------------
|
||||
|
||||
- all global variables should be defined and explained in a
|
||||
default-<module_name>.conf file
|
||||
- the names of all global variables resp. all functions should have as prefix the
|
||||
module name of the default-*.conf file resp. of the <module>.sh file which they
|
||||
are defined in.
|
||||
|
||||
- in case of an error certain cleanup functions may be called (e.g. if
|
||||
a network connection is cut during a test case it should be restored).
|
||||
The associotive array main_error_recovery_functions has function names as
|
||||
index an function parameters as values. In lib_exit each function contained
|
||||
in main_error_recovery_functions is called with it's corresponding arguments.
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,148 +0,0 @@
|
|||
#!/bin/sh
|
||||
|
||||
source ~/tools/shell/lib.sh || exit 1
|
||||
|
||||
function usage
|
||||
{
|
||||
echo usage: $(basename $0) config_file >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
function check_config_variables
|
||||
{
|
||||
local config_file=$1 varname
|
||||
|
||||
source $(pwd)/$config_file || lib_exit 1
|
||||
|
||||
for varname in "${!config_variables[@]}"; do
|
||||
local val
|
||||
eval val='$'$varname
|
||||
if [ -z "$val" ]; then
|
||||
lib_exit 1 "variable $varname not set in $config_file"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
function get_conf_var_value
|
||||
{
|
||||
local value=$1 varname=$2
|
||||
local ret=${config_variables[$varname]}
|
||||
if [ -z "$ret" ]; then
|
||||
lib_exit 1 "no entry for variable $varname in config_variables"
|
||||
fi
|
||||
case $value in # (((
|
||||
file) echo $src_dir/${ret##*:};;
|
||||
prompt) echo ${ret%%:*};;
|
||||
*) lib_exit 1 "invalid value $value for get_conf_var_value";;
|
||||
esac
|
||||
}
|
||||
|
||||
function replace_value
|
||||
{
|
||||
local config_varname=$1 value="$2" file prompt
|
||||
file=$(get_conf_var_value "file" $config_varname) || lib_exit 1
|
||||
prompt=$(get_conf_var_value "prompt" $config_varname) || lib_exit 1
|
||||
echo "replacing $prompt in $file"
|
||||
if ! grep "^$prompt" $file; then
|
||||
lib_exit 1 "cannot find prompt $prompt in $file"
|
||||
fi
|
||||
sed -i -e "s,^$prompt.*,$prompt=$value," $file || lib_exit 1
|
||||
grep "^$prompt" $file || lib_exit 1
|
||||
|
||||
}
|
||||
|
||||
function replace_config_values
|
||||
{
|
||||
local varname_list="$@" varname val
|
||||
for varname in ${varname_list[@]}; do
|
||||
eval val='$'$varname
|
||||
replace_value "$varname" "$val"
|
||||
done
|
||||
}
|
||||
|
||||
function delete_test_entry
|
||||
{
|
||||
local test_entry="$1" file=$2
|
||||
local pattern="^ *\<$test_entry\>"
|
||||
echo "deleting test $test_entry from $file"
|
||||
if ! grep "$pattern" $file; then
|
||||
lib_exit 1 "cannot find pattern $pattern in $file"
|
||||
fi
|
||||
sed -i -e "\,$pattern,d" $file || lib_exit 1
|
||||
}
|
||||
|
||||
function delete_some_tests
|
||||
{
|
||||
local file=$1 t
|
||||
[ -r $file ] || lib_exit 1 "echo file $file does not exist or is not readable"
|
||||
for t in $tests_to_skip; do
|
||||
delete_test_entry "$t" $file
|
||||
done
|
||||
}
|
||||
|
||||
function set_default_configs
|
||||
{
|
||||
tests_to_skip=" "
|
||||
}
|
||||
|
||||
function set_globals_depending_on_configs
|
||||
{
|
||||
cronjob_script=$src_dir/mars_test_cronjob.sh
|
||||
}
|
||||
|
||||
[ $# -ne 1 ] && usage
|
||||
|
||||
config_file=$1
|
||||
|
||||
[ -r $config_file ] || lib_exit 1 "echo file $config_file does not exist or is not readable"
|
||||
|
||||
src_origin=/home/fl/mars/mars-git-hub/test_suite
|
||||
|
||||
# index = name of config variable, value = prompt:file, where file is the default-*.conf where the
|
||||
# value of the config variable is defined
|
||||
declare -A config_variables
|
||||
config_variables=(\
|
||||
["src_dir"]=" : " \
|
||||
["host_list"]="main_host_list:default-main.conf" \
|
||||
["checkout_dir"]="checkout_mars_src_directory:default-checkout_mars.conf" \
|
||||
["branches_to_test"]="checkout_mars_git_branch:default-checkout_mars.conf" \
|
||||
["base_dir"]="main_base_directory:default-main.conf" \
|
||||
["install_mars_src_dir"]="install_mars_src_directory:default-install_mars.conf" \
|
||||
["make_mars_src_dir"]="make_mars_src_directory:default-make_mars.conf" \
|
||||
["mars_kernel_src_dir"]="checkout_mars_kernel_src_directory:default-checkout_mars.conf" \
|
||||
["make_mars_kernel_src_dir"]="make_mars_kernel_src_directory:default-make_mars.conf" \
|
||||
["tests_to_skip"]=" : " \
|
||||
)
|
||||
|
||||
set_default_configs
|
||||
|
||||
check_config_variables $config_file
|
||||
|
||||
set_globals_depending_on_configs
|
||||
|
||||
if [ "$src_origin" != "$src_dir" ]; then
|
||||
echo "fetching origin in $src_dir"
|
||||
cd $src_dir || lib_exit 1 "cannot cd $src_dir"
|
||||
git fetch origin || lib_exit 1
|
||||
git checkout master || lib_exit 1
|
||||
git reset --hard origin/master || lib_exit 1
|
||||
|
||||
replace_config_values "host_list" "checkout_dir" "base_dir" "install_mars_src_dir" \
|
||||
"make_mars_src_dir" "mars_kernel_src_dir" \
|
||||
"make_mars_kernel_src_dir"
|
||||
delete_some_tests $cronjob_script
|
||||
|
||||
|
||||
echo "fetching origin in $checkout_dir"
|
||||
cd $checkout_dir || lib_exit 1 "cannot cd $checkout_dir"
|
||||
git fetch origin || lib_exit 1
|
||||
fi
|
||||
|
||||
|
||||
for branch in $branches_to_test; do
|
||||
replace_value "branches_to_test" "$branch"
|
||||
|
||||
echo TESTING branch $branch
|
||||
|
||||
$src_dir/mars_test_cronjob.sh $src_dir
|
||||
done
|
||||
|
|
@ -1,62 +0,0 @@
|
|||
*1. sequence leave-resource create-resource does not work
|
||||
(test build_test_environment/resource/leave_resource)
|
||||
Fixed:
|
||||
Author: Thomas Schoebel-Theuer <tst@1und1.de> 2013-07-03 15:01:28
|
||||
Branches: WIP-unstable-06, remotes/fl-mars-testing/WIP-unstable-06, remotes/origin/WIP-unstable-06
|
||||
commit 295b0f4ada8cf96acf8b412c
|
||||
Follows: light0.1beta0.5
|
||||
Precedes: light0.1beta0.6
|
||||
|
||||
marsadm: relax precondition for create-resource
|
||||
|
||||
*2. replay symlink of secondary is not updated if end of
|
||||
logfile has errors (e.g. after marsadm disconnect)
|
||||
(test test_cases/admin/apply_fetch/apply)
|
||||
Fixed:
|
||||
commit 62b016d4ccc305356cd128e5abb
|
||||
Author: Frank Liepold <frank.liepold@1und1.de> 2013-06-27 08:14:40
|
||||
Branches: WIP-unstable-06, remotes/fl-mars-testing/WIP-unstable-06, remotes/origin/WIP-unstable-06
|
||||
Follows: light0.1beta0.5
|
||||
Precedes: light0.1beta0.6
|
||||
|
||||
marsadm: set syncstatus link on primary after marsadm resize
|
||||
*3. sync uses physical device size instead of
|
||||
mars size
|
||||
Fixed:
|
||||
commit d65577c9d5f261c4a19d1e83589e59b629f65c53
|
||||
Author: Thomas Schoebel-Theuer <schoebel@bell.site>
|
||||
Date: Thu Jun 27 14:30:05 2013 +0200
|
||||
|
||||
light: fix wrong target size of sync
|
||||
|
||||
*4. marsview indicates Inconsistent, synching after
|
||||
resize if physical size > mars size
|
||||
(test /home/fl/mars/test_suite/under_construction/admin/resizing)
|
||||
Fixed:
|
||||
commit fdc13bec9ccc80bdcf3cb59819ba
|
||||
Author: Daniel Hermann <daniel.hermann@1und1.de> 2013-06-28 14:43:46
|
||||
Branches: master, remotes/origin/master
|
||||
Follows: debian/0.9_ui60+3
|
||||
Precedes: debian/0.10_ui60+1
|
||||
|
||||
MARS::Resource: Fixed 'size' attribute semantics
|
||||
|
||||
- 'size' link is actual state
|
||||
- no target state
|
||||
|
||||
*5. memleak: marsadm secondary produces memleaks if executed during "flying writes"
|
||||
(test /home/fl/mars/test_suite/test_cases/bugs/memleak)
|
||||
Fixed:
|
||||
commit 281fa300b27d574b2c11066bf2c4f6114ade1325
|
||||
Author: Frank Liepold <frank.liepold@1und1.de> 2013-09-16 12:53:22
|
||||
Branch: master
|
||||
Follows: light0.1beta0.12
|
||||
Precedes:
|
||||
|
||||
light: workaround flying IO before reporting memory leaks
|
||||
|
||||
|
||||
*6. marsadm invalidate does not repair secondary after mars dir full
|
||||
(test /home/fl/mars/test_suite/test_cases/hardcore/mars_dir_full)
|
||||
Fixed:
|
||||
light0.1beta0.14
|
||||
|
|
@ -1,83 +0,0 @@
|
|||
src/bin.exe anlegen (lib.sh)
|
||||
alle function Variablen local
|
||||
|
||||
Konzept:
|
||||
Unterscheidung replay apply fetch durchgaengig
|
||||
siehe marsadm_pause_cmd
|
||||
Symlinknamen: apply fetch sync
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Fragen:
|
||||
Einstellen von hosts am besten dynamisch. Wie? Momentan mach ich das in .conf
|
||||
Dateien
|
||||
|
||||
|
||||
Doku:
|
||||
- Alle globalen Variablen in .conf - Dateien
|
||||
- Die Reihenfolge der default-*.conf - Dateien ist u.U. wichtig, wenn die
|
||||
gegenseitig Variable referenziert werden!! Daher sollte hier eine
|
||||
entsprechende <modul>_set_globals Funktion definiert sein (siehe z.B.
|
||||
resource_set_globals)
|
||||
- Alle globalen Variablen und alle Funktionen mit Modulname als Präfix
|
||||
- Globale Arrays muessen mit declare -g deklariert werden, da die Variablen via
|
||||
source innerhalb einer Funktion bekannt gemacht werden
|
||||
- Globale Arrays sollten im Namen _list als Postfix haben
|
||||
- Nur die _prepare, _run, _finish, _cleanup Funktionen duerfen mit globalen
|
||||
Variablen arbeiten.
|
||||
- Werden in einem Modul globale Variablen gebraucht, bekommen die das Präfix
|
||||
my_<modul_name>_
|
||||
- Standardfunktion _check_variables fuer Plausi der globalen Variablen
|
||||
- Funktionen die return-Wert auf stdout schreiben, duerfen nicht myexit
|
||||
aufrufen wg a=$(myfunc) || myexit 1
|
||||
- Die Pflege der run_list, prepare_list usw. sollte ausschließlich in den
|
||||
.conf - Dateien erfolgen. (wird in blkreplay nicht beachtet
|
||||
30_recreate_lvm.sh)
|
||||
- ueber verbose_script lib_vmsg ("echo if verbose") definieren
|
||||
- error recovery functions: Wann gesetzt, wann wieder aufgehoben:
|
||||
siehe Beispiel grub_restore_boot_configuration
|
||||
- Signale abfangen siehe main_error_recovery_functions
|
||||
- check - Funktionen sollten nicht lib_exit aufrufen, sondern
|
||||
eine Fehlermeldung ausspucken und 1 returnen, damit der
|
||||
Aufrufer noch reagieren kann (insbesondere damit man die
|
||||
Aufrufstelle findet)
|
||||
- Werden Werte an Aufrufparameter zurueckgegeben, heissen die
|
||||
f "pid"
|
||||
function f
|
||||
{
|
||||
local varname_pid
|
||||
...
|
||||
eval $varname_pid=pid
|
||||
}
|
||||
|
||||
|
||||
- Namenskonvention fuer lib - Funktionen und main (gemaess Moduln)
|
||||
- "includeable" conf Dateien (Beispiel
|
||||
./build_test_environment/resource/lib-resource.conf) evtl. via base-directory
|
||||
bis zu dem nach oben nach .conf - Dateien gesucht wird
|
||||
- Argumentzahlpruefung in allen Funktionen
|
||||
|
||||
- Problem, wenn man nur einen Test laufen lassen will => Subdirectory out und
|
||||
- out.conf gelinkt auf <test>.conf (Beispiel admin/apply_fetch/apply/out.conf)
|
||||
|
||||
|
||||
Meeting:
|
||||
passwort freier Zugang aller roots untereinander
|
||||
|
||||
|
||||
Robustheit:
|
||||
- sequence number vorne und hinten in die log Einträge
|
||||
- keine Nullbytes als Füllsel
|
||||
|
||||
|
||||
remote device:
|
||||
ln -s /tmp/fly@istore-test-bs7,remote-floppy /mars/resource-lv-2/_direct-002-istore-test-bap7
|
||||
|
||||
|
||||
|
||||
|
|
@ -19,34 +19,19 @@
|
|||
|
||||
#####################################################################
|
||||
|
||||
## the base directory of all .sh and .conf files
|
||||
main_base_directory=/home/fl/mars/mars-git-hub/test_suite
|
||||
## default for this directory: <git-repository>/test_suite.
|
||||
## Normally this should not be changed
|
||||
main_base_directory=$(dirname ${BASH_SOURCE[0]})
|
||||
|
||||
## hosts the tests are running on. The first host is initially always used as
|
||||
## primary host
|
||||
main_host_list=("istore-test-bs7" "istore-test-bap7")
|
||||
## flag to print debug messages
|
||||
main_verbose_script=1
|
||||
|
||||
## file to implement a primitive locking mechanism to avoid concurrent runs
|
||||
## of the test suite on the same host
|
||||
## of the test suite on the same host.
|
||||
## The use case specific file names must be provided in
|
||||
## global_user_dir/global.conf
|
||||
declare -g -A main_lock_file_list
|
||||
|
||||
## if main_host_list is changed in a *.conf file we need to recompute the values
|
||||
## of all dependent globals.
|
||||
function main_set_globals_depending_on_main_host_list
|
||||
{
|
||||
eval main_lock_file_list=($(for h in "${main_host_list[@]}";do printf "[$h]=/tmp/test-suite_on.$h ";done))
|
||||
|
||||
# see default-cluster.conf
|
||||
# we construct the expression cluster_mars_dir_lv_name_list=([host-1]=$cluster_mars_dir_lv [host-2]=$cluster_mars_dir_lv ...)
|
||||
if declare -p cluster_mars_dir_lv_name_list 2>/dev/null |\
|
||||
grep 'declare -A' >/dev/null
|
||||
then
|
||||
eval cluster_mars_dir_lv_name_list=($(x=(${main_host_list[@]/#/[}); y=(${x[@]/%/]=$cluster_mars_dir_lv}); echo ${y[@]}))
|
||||
fi
|
||||
}
|
||||
|
||||
main_set_globals_depending_on_main_host_list
|
||||
|
||||
## errorcode to prevent lib_exit from removing lock files
|
||||
## (main_lock_file_list)
|
||||
main_prevent_remove_lock_files_exit_code=42
|
||||
|
|
@ -57,32 +42,6 @@ main_prevent_remove_lock_files_exit_code=42
|
|||
## stdin, start_test.sh and 00_main.sh
|
||||
main_min_stack_level=3
|
||||
|
||||
## host indexed list of bootloaders. In case of lilo we check after installing
|
||||
## a kernel, that the label given in main_host_bootloader_label_list exists
|
||||
## in /etc/lilo.conf and call lilo -R <label>. For all other bootloaders we do
|
||||
## nothing. It's your responsibility to boot the correct kernel.
|
||||
declare -g -A main_host_bootloader_list
|
||||
main_host_bootloader_list=(\
|
||||
[istore-test-bs7]=grub \
|
||||
[istore-test-bap7]=lilo \
|
||||
[istore-test-bs4]=lilo \
|
||||
[istore-test-bap4]=lilo \
|
||||
[istore-test-bs2]=lilo \
|
||||
[ovzd-test-bs1]=grub \
|
||||
[ovzd-test-bap1]=grub \
|
||||
)
|
||||
|
||||
declare -g -A main_host_bootloader_label_list
|
||||
main_host_bootloader_label_list=(\
|
||||
[istore-test-bs7]=no_label \
|
||||
[istore-test-bap7]=mars \
|
||||
[istore-test-bs4]=mars \
|
||||
[istore-test-bap4]=mars \
|
||||
[istore-test-bs2]=mars \
|
||||
[ovzd-test-bs1]=no_label \
|
||||
[ovzd-test-bap1]=no_label \
|
||||
)
|
||||
|
||||
## associative array containing functions (as indexes) to be called in case of
|
||||
## unexpected errors or signals. See also lib_exit
|
||||
## The values of the array are the parameter lists for the function calls
|
||||
|
|
@ -94,29 +53,3 @@ declare -g -A main_error_recovery_functions
|
|||
|
||||
main_ssh_idfile_opt="-i $HOME/.ssh/id_dsa_no_pw"
|
||||
|
||||
|
||||
main_mars_directory="/mars"
|
||||
|
||||
## filesystem type of /mars
|
||||
main_mars_fs_type="ext4"
|
||||
|
||||
## egrep patterns to check for messsage categories
|
||||
main_mars_errmsg_prefix='(XIO|MARS)_error'
|
||||
main_mars_wrnmsg_prefix='(XIO|MARS)_warn'
|
||||
|
||||
## egrep pattern for error messages which are likely to appear only in
|
||||
## lib_err_total_log_file (5.total.log)
|
||||
main_errors_only_in_total_log_pattern='[A-Z]_error'
|
||||
|
||||
## to be able to kill all scripts started on a remote host these scripts
|
||||
## must have a fixed pattern in their name
|
||||
main_prefix_scripts='MARS-TEST-SCRIPT'
|
||||
|
||||
## results of checks of symlinks
|
||||
declare -g -A main_link_status
|
||||
main_link_status=(["link_ok"]=0 ["link_does_not_exist"]=1 \
|
||||
["link_has_wrong_value"]=2)
|
||||
|
||||
## function called in lib_exit to run general checks (e.g. for kernel stacks
|
||||
## caused by a test case)
|
||||
lib_general_checks_after_every_test=lib_general_mars_checks_after_every_test
|
||||
|
|
|
|||
|
|
@ -0,0 +1,21 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2010-2014 Frank Liepold / 1&1 Internet AG
|
||||
#
|
||||
# Email: frank.liepold@1und1.de
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
|
||||
#####################################################################
|
||||
colors_my_color="blue"
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2010-2014 Frank Liepold / 1&1 Internet AG
|
||||
#
|
||||
# Email: frank.liepold@1und1.de
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
|
||||
#####################################################################
|
||||
run_list="run_colors"
|
||||
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2010-2014 Frank Liepold / 1&1 Internet AG
|
||||
#
|
||||
# Email: frank.liepold@1und1.de
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
|
||||
#####################################################################
|
||||
## color used for some stupid tests
|
||||
colors_my_color="color not set"
|
||||
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2010-2014 Frank Liepold / 1&1 Internet AG
|
||||
#
|
||||
# Email: frank.liepold@1und1.de
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
|
||||
#####################################################################
|
||||
|
||||
# this file must exist, because the module lib_err.sh exists.
|
||||
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2010-2014 Frank Liepold / 1&1 Internet AG
|
||||
#
|
||||
# Email: frank.liepold@1und1.de
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
|
||||
#####################################################################
|
||||
|
||||
## path to your use case of the test suite.
|
||||
## You *must* set this variable.
|
||||
## This directory contains user specific sources and configurations and must
|
||||
## be a subdirectory of $main_base_directory.
|
||||
global_user_dir=$main_base_directory/example
|
||||
|
||||
## directory of your modules
|
||||
global_user_module_dir=$global_user_dir/modules
|
||||
|
||||
## function called in lib_exit to run general checks
|
||||
lib_general_checks_after_every_test_function=lib_err_general_checks_after_every_test_case
|
||||
|
||||
## in our example we want to see the complete stack
|
||||
main_min_stack_level=1
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2010-2014 Frank Liepold / 1&1 Internet AG
|
||||
#
|
||||
# Email: frank.liepold@1und1.de
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
|
||||
#####################################################################
|
||||
colors_my_color="green"
|
||||
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2010-2014 Frank Liepold / 1&1 Internet AG
|
||||
#
|
||||
# Email: frank.liepold@1und1.de
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
|
||||
#####################################################################
|
||||
|
||||
function run_colors
|
||||
{
|
||||
echo entering ${FUNCNAME[0]}
|
||||
# to show the use of main_error_recovery_functions (see also README)
|
||||
# in case of premature return we want to call colors_restore_something
|
||||
main_error_recovery_functions["colors_restore_something"]="param1 param2"
|
||||
echo my_color = $colors_my_color
|
||||
case "$colors_my_color" in # ((
|
||||
red|green|blue):;;
|
||||
*) lib_exit 1 "no valid color"
|
||||
esac
|
||||
main_error_recovery_functions["colors_restore_something"]=
|
||||
echo leaving ${FUNCNAME[0]}
|
||||
}
|
||||
|
||||
function colors_restore_something
|
||||
{
|
||||
echo entering ${FUNCNAME[0]}
|
||||
echo "parameter 1 = $1, parameter 2 = $2"
|
||||
echo leaving ${FUNCNAME[0]}
|
||||
}
|
||||
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2010-2014 Frank Liepold / 1&1 Internet AG
|
||||
#
|
||||
# Email: frank.liepold@1und1.de
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
|
||||
#####################################################################
|
||||
function lib_err_general_checks_after_every_test_case
|
||||
{
|
||||
echo entering ${FUNCNAME[0]}
|
||||
echo "doing some checks in ${FUNCNAME[0]}"
|
||||
echo leaving ${FUNCNAME[0]}
|
||||
}
|
||||
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2010-2014 Frank Liepold / 1&1 Internet AG
|
||||
#
|
||||
# Email: frank.liepold@1und1.de
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
|
||||
#####################################################################
|
||||
# we let colors_my_color intentionally empty
|
||||
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
---------------------------------------------------------------------------------------------------------------------
|
||||
category id testname description testcase and steps to check
|
||||
=====================================================================================================================
|
||||
hardcore H99 last_block write data file on H1.1 - pause-apply on secondary inspect last block
|
||||
data device of written data
|
||||
call halt
|
||||
=====================================================================================================================
|
||||
hardcore H98 gap_log_prim gap in logfile, higher logfiles already fetched
|
||||
force switch to primary must succeed
|
||||
|
|
@ -1,162 +0,0 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2010-2013 Frank Liepold / 1&1 Internet AG
|
||||
#
|
||||
# Email: frank.liepold@1und1.de
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
|
||||
#####################################################################
|
||||
|
||||
# This script is used to generate a given rate of write operations on a
|
||||
# given number of mars resources.
|
||||
# It is assumed that the data devices are named as follows (the strings
|
||||
# in <...> denotes shell variables defined below):
|
||||
# /dev/mars/lv-1-<dev_size_gb> ... /dev/mars/lv-<max_res_nr>-<dev_size_gb>
|
||||
#
|
||||
# The given write rate kb_per_sec is put into effect as follows:
|
||||
# 1.) at first kb_per_sec is divided in max_res_nr random defined
|
||||
# percentage write rates so that the sum of these
|
||||
# rates amount to kb_per_sec
|
||||
# 2.) for each of these percentage rates > 0 a write process is started
|
||||
# which realizes this rate on one resource.
|
||||
# 3.) after a random defined time between min_runtime_writer and
|
||||
# max_runtime_writer the processes are killed and we restart at 1.)
|
||||
|
||||
# number of resources
|
||||
max_res_nr=7
|
||||
|
||||
# min- and maxtime (seconds) the started write processes may write on the data
|
||||
# devices before they are killed
|
||||
max_runtime_writer=60
|
||||
min_runtime_writer=10
|
||||
|
||||
# max value of $RANDOM
|
||||
max_random=32767
|
||||
|
||||
# write rate in KB/s
|
||||
kb_per_sec=70000 # corresponds to 4G / 10min
|
||||
|
||||
# size of the data device in GB
|
||||
dev_size_gb=2
|
||||
|
||||
# size of the data device in KB
|
||||
device_size_kb=$(($dev_size_gb * 1024 * 1024))
|
||||
|
||||
# time (seconds) after which log-rotate and log-delete should be called
|
||||
log_rotate_delete_intervall=600
|
||||
|
||||
function calculate_kb_per_sec_per_resource
|
||||
{
|
||||
local i percentage_left=100
|
||||
for i in $(seq 1 1 $max_res_nr); do
|
||||
if [ $percentage_left -gt 0 ]; then
|
||||
percentage[$i]=$(( $RANDOM * $percentage_left / $max_random ))
|
||||
let percentage_left-=${percentage[$i]}
|
||||
else
|
||||
percentage[$i]=0
|
||||
fi
|
||||
kb_per_sec_res[$i]=$(( $kb_per_sec * ${percentage[$i]} / 100))
|
||||
done
|
||||
date && echo kb_per_sec_res="${kb_per_sec_res[@]}"
|
||||
}
|
||||
|
||||
function start_writer_on_resource
|
||||
{
|
||||
local res_nr=$1
|
||||
local kb_per_sec=${kb_per_sec_res[$i]}
|
||||
local output_dev=/dev/mars/lv-$res_nr-$dev_size_gb
|
||||
local loop_count=0
|
||||
date && echo "starting writer on $res_nr (kb_per_sec=$kb_per_sec)"
|
||||
if [ $kb_per_sec -eq 0 ]; then
|
||||
sleep 10 # because caller checks whether I'm running
|
||||
exit 0
|
||||
fi
|
||||
while true; do
|
||||
local maximal_offset_output=$(($device_size_kb - (2 * $kb_per_sec) ))
|
||||
local offset_output=$(( $RANDOM * $maximal_offset_output / $max_random ))
|
||||
local dd_cmd="dd of=$output_dev bs=1024 count=$kb_per_sec skip=$offset_output"
|
||||
date && echo "executing on res $res_nr(loop=$loop_count): $dd_cmd"
|
||||
yes hallo | $dd_cmd
|
||||
sleep 1
|
||||
let loop_count+=1
|
||||
if [ $loop_count -ge $max_runtime_writer ]; then
|
||||
exit 0
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
function check_process_running
|
||||
{
|
||||
local pid=$1 res_nr=$2 maxwait=5 waited=0
|
||||
date && echo "check process $pid for resource $res_nr"
|
||||
while true; do
|
||||
if ps -p $pid; then
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
let waited+=1
|
||||
date && echo "waited $waited for pid $pid"
|
||||
if [ $waited -eq $maxwait ]; then
|
||||
date && echo "maxwait $maxwait exceeded" >&2 && exit 1
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
function kill_process
|
||||
{
|
||||
local pid=$1 res_nr=$2
|
||||
local signal
|
||||
date && echo "killing process $pid for resource $res_nr"
|
||||
for signal in 1 9 "check"; do
|
||||
if ps -p $pid; then
|
||||
if [ "$signal" != "check" ]; then
|
||||
kill -$signal $pid
|
||||
sleep 1
|
||||
else
|
||||
echo "could not kill pid $pid" >&2 && exit 1
|
||||
fi
|
||||
else
|
||||
return
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
|
||||
sum_sleeptime=0
|
||||
|
||||
while true; do
|
||||
calculate_kb_per_sec_per_resource
|
||||
runtime_writer=$(( ($RANDOM * ($max_runtime_writer - $min_runtime_writer) / $max_random ) + $min_runtime_writer))
|
||||
date && echo runtime_writer=$runtime_writer
|
||||
for i in $(seq 1 1 $max_res_nr); do
|
||||
tmp_file=/tmp/dd.$$.$i
|
||||
rm -f $tmp_file || exit 1
|
||||
start_writer_on_resource $i >$tmp_file 2>&1 &
|
||||
pid[$i]=$!
|
||||
check_process_running ${pid[$i]} $i
|
||||
done
|
||||
date && echo sleeping $runtime_writer
|
||||
sleep $runtime_writer
|
||||
let sum_sleeptime+=$runtime_writer
|
||||
for i in $(seq 1 1 $max_res_nr); do
|
||||
kill_process ${pid[$i]} $i
|
||||
done
|
||||
if [ $sum_sleeptime -ge $log_rotate_delete_intervall ]; then
|
||||
marsadm log-rotate all
|
||||
marsadm log-delete all
|
||||
sum_sleeptime=0
|
||||
fi
|
||||
done
|
||||
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
#!/bin/bash
|
||||
|
||||
## installs the kernel, mars module and userpace tools on all test hosts
|
||||
## (main_host_list)
|
||||
## (global_host_list)
|
||||
## source and target directories must be specified in default-install_mars.conf
|
||||
|
||||
prepare_list="install_mars_prepare"
|
||||
|
|
@ -9,13 +9,13 @@
|
|||
#####################################################################
|
||||
|
||||
## host indexed list of logical volume names (see also lv_config_lv_name_list)
|
||||
## to be used for main_mars_directory.
|
||||
## The list is filled in main_set_globals_depending_on_main_host_list
|
||||
## to be used for global_mars_directory.
|
||||
## The list is filled in global_set_globals_depending_on_global_host_list
|
||||
## (default-main.conf).
|
||||
## As default we use the logical volume name cluster_mars_dir_lv for all hosts
|
||||
## To change this setting you have to edit default-main.conf
|
||||
|
||||
declare -g -A cluster_mars_dir_lv_name_list
|
||||
cluster_mars_dir_lv=lv-6-100
|
||||
main_set_globals_depending_on_main_host_list
|
||||
global_set_globals_depending_on_global_host_list
|
||||
|
||||
|
|
@ -20,9 +20,9 @@
|
|||
#####################################################################
|
||||
|
||||
## file containing all log messages
|
||||
lib_err_total_log_file="$main_mars_directory/5.total.log"
|
||||
lib_err_total_log_file="$global_mars_directory/5.total.log"
|
||||
|
||||
## file containing all error messages
|
||||
lib_err_total_err_file="$main_mars_directory/3.error.status"
|
||||
lib_err_total_err_file="$global_mars_directory/3.error.status"
|
||||
|
||||
|
||||
|
|
@ -16,7 +16,7 @@
|
|||
|
||||
lib_rw_file_to_write="file_to_write"
|
||||
|
||||
lib_rw_write_and_delete_script="/tmp/$main_prefix_scripts-write_and_delete.sh.$$"
|
||||
lib_rw_write_and_delete_script="/tmp/$global_prefix_scripts-write_and_delete.sh.$$"
|
||||
|
||||
## part of data device size, which is written in the dd statement in the
|
||||
## script wich writes the data device
|
||||
|
|
@ -39,4 +39,4 @@ logrotate_with_deletes=0
|
|||
## see logrotate_with_deletes
|
||||
logrotate_number_of_rotates_before_delete=3
|
||||
|
||||
logrotate_action_script_prefix="/tmp/$main_prefix_scripts-log_"
|
||||
logrotate_action_script_prefix="/tmp/$global_prefix_scripts-log_"
|
||||
|
|
@ -14,7 +14,7 @@
|
|||
|
||||
## the names of the resources used in the current test. Must be a subset of
|
||||
## lv_config_lv_name_list (default-lv_config.conf)
|
||||
## These resource are created on all hosts in main_host_list using the first
|
||||
## These resource are created on all hosts in global_host_list using the first
|
||||
## host as primary and the followings as secondaries
|
||||
resource_name_list=(lv-1-2)
|
||||
|
||||
|
|
@ -22,7 +22,7 @@ resource_name_list=(lv-1-2)
|
|||
## resource_set_globals_depending_on_resource_name_list fills the list
|
||||
declare -g -A resource_mount_point_list
|
||||
|
||||
## resource indexed list of resource directories in main_mars_directory
|
||||
## resource indexed list of resource directories in global_mars_directory
|
||||
## resource_set_globals_depending_on_resource_name_list fills the list
|
||||
declare -g -A resource_dir_list
|
||||
|
||||
|
|
@ -47,8 +47,8 @@ function resource_set_globals_depending_on_resource_name_list
|
|||
# maps resource name lv-<i>-<size> to /mnt/mars-test-<i>
|
||||
eval resource_mount_point_list=($(for i in ${!resource_name_list[*]};do printf "[${resource_name_list[$i]}]=/mnt/mars-test-$(($i + 1)) ";done))
|
||||
|
||||
# the resource dir of resource <res> is $main_mars_directory/resource-<res>
|
||||
eval resource_dir_list=($(for i in ${!resource_name_list[*]};do printf "[${resource_name_list[$i]}]=$main_mars_directory/resource-${resource_name_list[$i]} ";done))
|
||||
# the resource dir of resource <res> is $global_mars_directory/resource-<res>
|
||||
eval resource_dir_list=($(for i in ${!resource_name_list[*]};do printf "[${resource_name_list[$i]}]=$global_mars_directory/resource-${resource_name_list[$i]} ";done))
|
||||
|
||||
# use xfs for all data devices
|
||||
eval resource_fs_type_list=($(for i in ${!resource_name_list[*]};do printf "[${resource_name_list[$i]}]=xfs ";done))
|
||||
|
|
@ -121,7 +121,7 @@ resource_fs_on_data_device_necessary=1
|
|||
resource_use_data_dev_writes_to_fill_mars_dir=1
|
||||
|
||||
## file to use for filling /mars
|
||||
resource_big_file=$main_mars_directory/mars_test_bigfile
|
||||
resource_big_file=$global_mars_directory/mars_test_bigfile
|
||||
|
||||
## msg files
|
||||
declare -g -A resource_msgfile_list
|
||||
|
|
@ -130,7 +130,7 @@ resource_msgfile_list=([err]='3.error.status' [warn]='2.warn.status')
|
|||
## file used to reset after emergency mode
|
||||
resource_proc_sys_mars_reset_emergency_file="/proc/sys/mars/mars_reset_emergency"
|
||||
|
||||
resource_mars_dir_full_warn_pattern_list[0]='EMERGENCY: the space on '$main_mars_directory'/ is very low.'
|
||||
resource_mars_dir_full_warn_pattern_list[0]='EMERGENCY: the space on '$global_mars_directory'/ is very low.'
|
||||
|
||||
## flag to indicate whether the network connection should be cut while
|
||||
## synching (e.g. after join-resource or invalidate)
|
||||
|
|
@ -86,7 +86,7 @@ switch2primary_logrotate_new_primary=1
|
|||
|
||||
## prefix of the name of the script, which writes some data on
|
||||
## the data devices of the both primaries
|
||||
switch2primary_write_script_prefix="/tmp/$main_prefix_scripts-split_brain"
|
||||
switch2primary_write_script_prefix="/tmp/$global_prefix_scripts-split_brain"
|
||||
|
||||
## time for which the amount of data to sync must be constant
|
||||
## to declare the sync process as inactive
|
||||
|
|
@ -0,0 +1,103 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2010-2014 Frank Liepold / 1&1 Internet AG
|
||||
#
|
||||
# Email: frank.liepold@1und1.de
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or
|
||||
# modify it under the terms of the GNU General Public License
|
||||
# as published by the Free Software Foundation; either version 2
|
||||
# of the License, or (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
||||
|
||||
#####################################################################
|
||||
|
||||
## path to your use case of the test suite.
|
||||
## You *must* set this variable.
|
||||
## This directory contains user specific sources and configurations and must
|
||||
## be a subdirectory of $main_base_directory.
|
||||
global_user_dir=$main_base_directory/mars
|
||||
|
||||
## directory of your modules
|
||||
global_user_module_dir=$global_user_dir/modules
|
||||
|
||||
## hosts the tests are running on. The first host is initially always used as
|
||||
## primary host
|
||||
global_host_list=("istore-test-bs7" "istore-test-bap7")
|
||||
|
||||
## if global_host_list is changed in a *.conf file we need to recompute the
|
||||
## values of all dependent globals.
|
||||
function global_set_globals_depending_on_global_host_list
|
||||
{
|
||||
eval main_lock_file_list=($(for h in "${global_host_list[@]}";do printf "[$h]=/tmp/test-suite_on.$h ";done))
|
||||
|
||||
# see default-cluster.conf
|
||||
# we construct the expression cluster_mars_dir_lv_name_list=([host-1]=$cluster_mars_dir_lv [host-2]=$cluster_mars_dir_lv ...)
|
||||
if declare -p cluster_mars_dir_lv_name_list 2>/dev/null |\
|
||||
grep 'declare -A' >/dev/null
|
||||
then
|
||||
eval cluster_mars_dir_lv_name_list=($(x=(${global_host_list[@]/#/[}); y=(${x[@]/%/]=$cluster_mars_dir_lv}); echo ${y[@]}))
|
||||
fi
|
||||
}
|
||||
|
||||
global_set_globals_depending_on_global_host_list
|
||||
|
||||
## host indexed list of bootloaders. In case of lilo we check after installing
|
||||
## a kernel, that the label given in global_host_bootloader_label_list exists
|
||||
## in /etc/lilo.conf and call lilo -R <label>. For all other bootloaders we do
|
||||
## nothing. It's your responsibility to boot the correct kernel.
|
||||
declare -g -A global_host_bootloader_list
|
||||
global_host_bootloader_list=(\
|
||||
[istore-test-bs7]=grub \
|
||||
[istore-test-bap7]=lilo \
|
||||
[istore-test-bs4]=lilo \
|
||||
[istore-test-bap4]=lilo \
|
||||
[istore-test-bs2]=lilo \
|
||||
[ovzd-test-bs1]=grub \
|
||||
[ovzd-test-bap1]=grub \
|
||||
)
|
||||
|
||||
declare -g -A global_host_bootloader_label_list
|
||||
global_host_bootloader_label_list=(\
|
||||
[istore-test-bs7]=no_label \
|
||||
[istore-test-bap7]=mars \
|
||||
[istore-test-bs4]=mars \
|
||||
[istore-test-bap4]=mars \
|
||||
[istore-test-bs2]=mars \
|
||||
[ovzd-test-bs1]=no_label \
|
||||
[ovzd-test-bap1]=no_label \
|
||||
)
|
||||
|
||||
global_mars_directory="/mars"
|
||||
|
||||
## filesystem type of /mars
|
||||
global_mars_fs_type="ext4"
|
||||
|
||||
## egrep patterns to check for messsage categories
|
||||
global_mars_errmsg_prefix='(XIO|MARS)_error'
|
||||
global_mars_wrnmsg_prefix='(XIO|MARS)_warn'
|
||||
|
||||
## egrep pattern for error messages which are likely to appear only in
|
||||
## lib_err_total_log_file (5.total.log)
|
||||
global_errors_only_in_total_log_pattern='[A-Z]_error'
|
||||
|
||||
## to be able to kill all scripts started on a remote host these scripts
|
||||
## must have a fixed pattern in their name
|
||||
global_prefix_scripts='MARS-TEST-SCRIPT'
|
||||
|
||||
## results of checks of symlinks
|
||||
declare -g -A global_link_status
|
||||
global_link_status=(["link_ok"]=0 ["link_does_not_exist"]=1 \
|
||||
["link_has_wrong_value"]=2)
|
||||
|
||||
## function called in lib_exit to run general checks (e.g. for kernel stacks
|
||||
## caused by a test case)
|
||||
lib_general_checks_after_every_test_function=lib_general_mars_checks_after_every_test
|
||||
|
||||
|
|
@ -33,7 +33,7 @@
|
|||
##
|
||||
## directory-2 serves as parameter for option --config_root_dir of start_test.sh
|
||||
##
|
||||
# All directory paths are given relative to test_suite_dir
|
||||
# All directory paths are given relative to use_case_dir
|
||||
|
||||
function myexit
|
||||
{
|
||||
|
|
@ -53,22 +53,22 @@ function execute_tests
|
|||
local perf_msg="Performance-Failures:"$'\n'
|
||||
local errorfile_msg="Error-Files:"$'\n'
|
||||
local kernel_stack_msg="Kernel-Stacks:"$'\n'
|
||||
local perf_grep_cmd='grep PERFORMANCE-FAILURE '$tmp_file
|
||||
local errorfile_grep_cmd='grep ERROR-FILE '$tmp_file
|
||||
local kernel_stack_grep_cmd='grep KERNEL-STACK '$tmp_file
|
||||
local perf_grep_cmd="grep '^ *PERFORMANCE-FAILURE' "$tmp_file
|
||||
local errorfile_grep_cmd="grep 'ERROR-FILE' "$tmp_file
|
||||
local kernel_stack_grep_cmd="grep 'KERNEL-STACK' "$tmp_file
|
||||
|
||||
for t in "${tests_to_execute[@]}"; do
|
||||
local test_dir="${t%:*}" start_dirs s
|
||||
local config_root_dir=${t#*:}
|
||||
local config_root_dir_opt=${config_root_dir:+"--config_root_dir=$test_suite_dir/$config_root_dir"}
|
||||
local config_root_dir_opt=${config_root_dir:+"--config_root_dir=$use_case_dir/$config_root_dir"}
|
||||
case "$test_dir" in # ((
|
||||
*\*) test_dir=${test_dir%\*}
|
||||
start_dirs=($(find $test_suite_dir/$test_dir -type f \
|
||||
start_dirs=($(find $use_case_dir/$test_dir -type f \
|
||||
-name i_am_a_testdirectory -exec \
|
||||
dirname {} \;)
|
||||
)
|
||||
;;
|
||||
*) start_dirs=($test_suite_dir/$test_dir)
|
||||
*) start_dirs=($use_case_dir/$test_dir)
|
||||
;;
|
||||
esac
|
||||
for s in "${start_dirs[@]}"; do
|
||||
|
|
@ -83,16 +83,16 @@ function execute_tests
|
|||
fail_msg+="$s"$'\n'
|
||||
send_msg=1
|
||||
fi
|
||||
if $perf_grep_cmd >/dev/null; then
|
||||
perf_msg+="$s: $($perf_grep_cmd)"$'\n'
|
||||
if eval $perf_grep_cmd >/dev/null; then
|
||||
perf_msg+="$s: $(eval $perf_grep_cmd)"$'\n'
|
||||
send_msg=1
|
||||
fi
|
||||
if $errorfile_grep_cmd >/dev/null; then
|
||||
errorfile_msg+="$s: $($errorfile_grep_cmd)"$'\n'
|
||||
if eval $errorfile_grep_cmd >/dev/null; then
|
||||
errorfile_msg+="$s: $(eval $errorfile_grep_cmd)"$'\n'
|
||||
send_msg=1
|
||||
fi
|
||||
if $kernel_stack_grep_cmd >/dev/null; then
|
||||
kernel_stack_msg+="$s: $($kernel_stack_grep_cmd)"$'\n'
|
||||
if eval $kernel_stack_grep_cmd >/dev/null; then
|
||||
kernel_stack_msg+="$s: $(eval $kernel_stack_grep_cmd)"$'\n'
|
||||
send_msg=1
|
||||
fi
|
||||
if [ $rc -ne 0 -a $continue_after_failed_test -eq 0 ];then
|
||||
|
|
@ -122,7 +122,7 @@ function set_env
|
|||
|
||||
function usage
|
||||
{
|
||||
echo "usage: $my_name [-e] [-l] test_suite_dir" >&2
|
||||
echo "usage: $my_name [-e] [-l] test_suite_dir use_case_dir" >&2
|
||||
echo " -e: dont't continue if a test fails" >&2
|
||||
echo " -l: list all tests which will be executed but don't execute them" >&2
|
||||
exit 1
|
||||
|
|
@ -146,13 +146,23 @@ done
|
|||
|
||||
shift $(($OPTIND - 1))
|
||||
|
||||
[ $# -ne 1 ] && usage
|
||||
[ $# -ne 2 ] && usage
|
||||
|
||||
test_suite_dir=$1
|
||||
use_case_dir=$2
|
||||
|
||||
for n in test_suite use_case;do
|
||||
eval d='$'${n}_dir
|
||||
if [ ! -d $d ]; then
|
||||
echo "directory ${n}_dir = $d not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
|
||||
# main
|
||||
|
||||
echo Start $(basename $0) $test_suite_dir at $(date)
|
||||
echo Start $(basename $0) $test_suite_dir $use_case_dir at $(date)
|
||||
|
||||
mail_server=mxintern.schlund.de:587
|
||||
|
||||
|
|
@ -163,10 +173,7 @@ mail_to=("frank.liepold@1und1.de")
|
|||
start_script=$test_suite_dir/scripts/start_test.sh
|
||||
|
||||
# For documentation see header of this file
|
||||
tests_to_execute=(\
|
||||
"build_test_environment/checkout/checkout_mars_from_git:build_test_environment"
|
||||
"build_test_environment/make/make_mars/grub:build_test_environment"
|
||||
"build_test_environment/install_mars/install_via_rsync:build_test_environment"
|
||||
tests_to_execute=(
|
||||
"build_test_environment/lv_config/lv_recreate:build_test_environment"
|
||||
"build_test_environment/cluster/create_cluster:build_test_environment"
|
||||
"build_test_environment/resource/create_resource:build_test_environment"
|
||||
|
|
@ -24,8 +24,8 @@
|
|||
#####################################################################
|
||||
function apply_fetch_run
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host=${main_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host=${global_host_list[1]}
|
||||
local res=${resource_name_list[0]}
|
||||
local writer_pid writer_script write_count
|
||||
local logfile length_logfile time_waited net_throughput
|
||||
|
|
@ -4,7 +4,7 @@ function cluster_run
|
|||
{
|
||||
cluster_check_variables
|
||||
lib_check_access_to_remote_hosts "$main_ssh_idfile_opt" \
|
||||
"${main_host_list[@]}"
|
||||
"${global_host_list[@]}"
|
||||
cluster_check_devices_all
|
||||
cluster_umount_data_device_all
|
||||
cluster_rmmod_mars_all
|
||||
|
|
@ -19,7 +19,7 @@ function cluster_umount_data_device_all
|
|||
{
|
||||
local host
|
||||
local res=${resource_name_list[0]}
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
if mount_is_dir_mountpoint $host ${resource_mount_point_list[$res]}
|
||||
then
|
||||
mount_umount $host "device_does_not_matter" ${resource_mount_point_list[$res]}
|
||||
|
|
@ -31,7 +31,7 @@ function cluster_rmmod_mars_all
|
|||
{
|
||||
local host
|
||||
mount_umount_data_device_all
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
cluster_rmmod_mars $host
|
||||
done
|
||||
}
|
||||
|
|
@ -46,10 +46,10 @@ function cluster_rmmod_mars
|
|||
function cluster_clear_mars_dir_all
|
||||
{
|
||||
local host
|
||||
if [ -z "$main_mars_directory" ]; then
|
||||
lib_exit 1 " variable main_mars_directory empty"
|
||||
if [ -z "$global_mars_directory" ]; then
|
||||
lib_exit 1 " variable global_mars_directory empty"
|
||||
fi
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
cluster_clear_mars_dir $host
|
||||
done
|
||||
}
|
||||
|
|
@ -57,7 +57,7 @@ function cluster_clear_mars_dir_all
|
|||
function cluster_insert_mars_module_all
|
||||
{
|
||||
local host
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
cluster_insert_mars_module $host
|
||||
done
|
||||
}
|
||||
|
|
@ -74,20 +74,20 @@ function cluster_insert_mars_module
|
|||
function cluster_clear_mars_dir
|
||||
{
|
||||
local host=$1
|
||||
if [ -z "$main_mars_directory" ]; then
|
||||
lib_exit 1 "variable main_mars_directory empty"
|
||||
if [ -z "$global_mars_directory" ]; then
|
||||
lib_exit 1 "variable global_mars_directory empty"
|
||||
fi
|
||||
lib_vmsg " removing $host:$main_mars_directory/*"
|
||||
lib_remote_idfile $host "shopt -s dotglob && rm -rf $main_mars_directory/*" || lib_exit 1
|
||||
lib_vmsg " removing $host:$global_mars_directory/*"
|
||||
lib_remote_idfile $host "shopt -s dotglob && rm -rf $global_mars_directory/*" || lib_exit 1
|
||||
}
|
||||
|
||||
function cluster_clear_and_umount_mars_dir_all
|
||||
{
|
||||
local host
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
cluster_clear_mars_dir $host
|
||||
if mount_is_dir_mountpoint $host $main_mars_directory; then
|
||||
mount_umount $host "device_does_not_matter" $main_mars_directory || lib_exit 1
|
||||
if mount_is_dir_mountpoint $host $global_mars_directory; then
|
||||
mount_umount $host "device_does_not_matter" $global_mars_directory || lib_exit 1
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
|
@ -97,19 +97,19 @@ function cluster_mount_mars_dir
|
|||
local host=$1
|
||||
local dev="$(lv_config_get_lv_device ${cluster_mars_dir_lv_name_list[$host]})"
|
||||
local already_mounted_correctly=0
|
||||
if mount_is_dir_mountpoint $host $main_mars_directory; then
|
||||
if mount_is_dir_mountpoint $host $global_mars_directory; then
|
||||
local mount_point
|
||||
if mount_is_device_mounted $host $dev "mount_point" \
|
||||
&& [ "$mount_point" == "$main_mars_directory" ]
|
||||
&& [ "$mount_point" == "$global_mars_directory" ]
|
||||
then
|
||||
already_mounted_correctly=1
|
||||
else
|
||||
mount_umount $host "device_does_not_matter" $main_mars_directory
|
||||
mount_umount $host "device_does_not_matter" $global_mars_directory
|
||||
fi
|
||||
fi
|
||||
if [ $already_mounted_correctly -eq 0 ];then
|
||||
lib_rw_remote_check_device_fs $host $dev $main_mars_fs_type
|
||||
mount_mount $host $dev $main_mars_directory $main_mars_fs_type || lib_exit 1
|
||||
lib_rw_remote_check_device_fs $host $dev $global_mars_fs_type
|
||||
mount_mount $host $dev $global_mars_directory $global_mars_fs_type || lib_exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
|
|
@ -118,7 +118,7 @@ function cluster_clear_and_mount_mars_dir_all
|
|||
local host dev
|
||||
local primary_host_to_join
|
||||
local cluster_action="create-cluster"
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
cluster_mount_mars_dir $host
|
||||
cluster_clear_mars_dir $host
|
||||
marsadm_do_cmd $host "$cluster_action --force" $primary_host_to_join || lib_exit 1
|
||||
|
|
@ -131,7 +131,7 @@ function cluster_clear_and_mount_mars_dir_all
|
|||
function cluster_get_ip_linkname
|
||||
{
|
||||
local host=$1
|
||||
echo $main_mars_directory/ips/ip-$host
|
||||
echo $global_mars_directory/ips/ip-$host
|
||||
}
|
||||
|
||||
function cluster_get_ip_linkvalue_pattern
|
||||
|
|
@ -148,7 +148,7 @@ function cluster_exists
|
|||
|
||||
lib_linktree_check_link $host "$link" "$link_value_expected"
|
||||
link_status=$?
|
||||
if [ $link_status -eq ${main_link_status["link_ok"]} ]; then
|
||||
if [ $link_status -eq ${global_link_status["link_ok"]} ]; then
|
||||
lib_vmsg " cluster on $host already exists"
|
||||
return 0
|
||||
fi
|
||||
|
|
@ -157,7 +157,7 @@ function cluster_exists
|
|||
|
||||
function cluster_create
|
||||
{
|
||||
local host=${main_host_list[0]}
|
||||
local host=${global_host_list[0]}
|
||||
|
||||
if ! cluster_exists $host; then
|
||||
|
||||
|
|
@ -168,8 +168,8 @@ function cluster_create
|
|||
|
||||
function cluster_join
|
||||
{
|
||||
local host=${main_host_list[1]}
|
||||
local primary_host=${main_host_list[0]}
|
||||
local host=${global_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
|
||||
if ! cluster_exists $host; then
|
||||
|
||||
|
|
@ -208,7 +208,7 @@ function cluster_check_ok_link_list
|
|||
link_value_expected="${link_value_expected_list[$i]}"
|
||||
lib_linktree_check_link $host "$link" "$link_value_expected"
|
||||
link_status=$?
|
||||
if [ $link_status -ne ${main_link_status["link_ok"]} ]; then
|
||||
if [ $link_status -ne ${global_link_status["link_ok"]} ]; then
|
||||
local str=$(lib_linktree_status_to_string $link_status)
|
||||
lib_exit 1 "link $host:$link has state $str"
|
||||
fi
|
||||
|
|
@ -229,9 +229,9 @@ function cluster_check_links_after_create_cluster
|
|||
function cluster_check_devices_all
|
||||
{
|
||||
local dev host blkid_out rc
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
dev="$(lv_config_get_lv_device ${cluster_mars_dir_lv_name_list[$host]})"
|
||||
lib_rw_remote_check_device_fs $host $dev $main_mars_fs_type
|
||||
lib_rw_remote_check_device_fs $host $dev $global_mars_fs_type
|
||||
done
|
||||
}
|
||||
|
||||
|
|
@ -251,12 +251,12 @@ function cluster_remove_debugfiles
|
|||
|
||||
function cluster_check_variables
|
||||
{
|
||||
if [ ${#main_host_list[*]} -eq 0 ]; then
|
||||
if [ ${#global_host_list[*]} -eq 0 ]; then
|
||||
lib_exit 1 "no cluster hosts given"
|
||||
fi
|
||||
if [ ${#main_host_list[*]} -ne ${#cluster_mars_dir_lv_name_list[*]} ]
|
||||
if [ ${#global_host_list[*]} -ne ${#cluster_mars_dir_lv_name_list[*]} ]
|
||||
then
|
||||
lib_exit 1 "number of hosts = ${#main_host_list[*]} != ${#cluster_mars_dir_lv_name_list[*]} = number of devices"
|
||||
lib_exit 1 "number of hosts = ${#global_host_list[*]} != ${#cluster_mars_dir_lv_name_list[*]} = number of devices"
|
||||
fi
|
||||
local lv_name
|
||||
for lv_name in ${cluster_mars_dir_lv_name_list[@]}; do
|
||||
|
|
@ -269,9 +269,9 @@ function cluster_check_variables
|
|||
|
||||
function cluster_create_mars_dir_all
|
||||
{
|
||||
lib_vmsg " creating mars directory $main_mars_directory on ${main_host_list[*]}"
|
||||
lib_remote_all_idfile "${main_host_list[*]}" \
|
||||
"[ -d $main_mars_directory ] || mkdir $main_mars_directory" \
|
||||
lib_vmsg " creating mars directory $global_mars_directory on ${global_host_list[*]}"
|
||||
lib_remote_all_idfile "${global_host_list[*]}" \
|
||||
"[ -d $global_mars_directory ] || mkdir $global_mars_directory" \
|
||||
|| lib_exit 1
|
||||
}
|
||||
|
||||
|
|
@ -22,10 +22,10 @@
|
|||
|
||||
function crash_run
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host=${main_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host=${global_host_list[1]}
|
||||
local mars_dev=$(lv_config_get_lv_device ${cluster_mars_dir_lv_name_list[$primary_host]})
|
||||
local boot_label_name="${main_host_bootloader_label_list[$primary_host]}"
|
||||
local boot_label_name="${global_host_bootloader_label_list[$primary_host]}"
|
||||
local res=${resource_name_list[0]}
|
||||
local dev=$(lv_config_get_lv_device $res)
|
||||
local writer_pid writer_script logfile length_logfile time_waited
|
||||
|
|
@ -134,7 +134,7 @@ function crash_reboot
|
|||
then
|
||||
lib_exit 1 "to print symlink trees secondary_host must be given"
|
||||
fi
|
||||
if [ "${main_host_bootloader_list[$primary_host]}" = "lilo" ]; then
|
||||
if [ "${global_host_bootloader_list[$primary_host]}" = "lilo" ]; then
|
||||
install_mars_activate_kernel_to_boot_with_lilo $primary_host \
|
||||
"$boot_label_name"
|
||||
fi
|
||||
|
|
@ -21,8 +21,8 @@
|
|||
|
||||
function datadev_full_run
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host=${main_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host=${global_host_list[1]}
|
||||
local res=${resource_name_list[0]}
|
||||
local dev=$(lv_config_get_lv_device $res)
|
||||
local data_dev=$(resource_get_data_device $res)
|
||||
|
|
@ -22,8 +22,8 @@
|
|||
function file_destroy_run
|
||||
{
|
||||
|
||||
local primary_host="${main_host_list[0]}"
|
||||
local secondary_host="${main_host_list[1]}"
|
||||
local primary_host="${global_host_list[0]}"
|
||||
local secondary_host="${global_host_list[1]}"
|
||||
local res=${resource_name_list[0]}
|
||||
local logfile length_logfile writer_pid writer_script write_count
|
||||
local time_waited net_throughput
|
||||
|
|
@ -26,7 +26,7 @@ function install_mars_prepare
|
|||
install_mars_fill_userspace_target_dir_list
|
||||
|
||||
install_mars_check_access_to_remote_hosts \
|
||||
"${main_host_list[@]}" "$install_mars_source_host"
|
||||
"${global_host_list[@]}" "$install_mars_source_host"
|
||||
|
||||
# method specific actions
|
||||
case $install_mars_method in # ((
|
||||
|
|
@ -38,8 +38,8 @@ function install_mars_prepare
|
|||
|
||||
local dir
|
||||
for dir in "${install_mars_userspace_target_dir_list[@]}"; do
|
||||
lib_vmsg " checking userspace target directory $dir on hosts ${main_host_list[*]}"
|
||||
lib_remote_all_idfile "${main_host_list[*]}" "if ! test -d $dir; then mkdir -p $dir; fi" || lib_exit 1
|
||||
lib_vmsg " checking userspace target directory $dir on hosts ${global_host_list[*]}"
|
||||
lib_remote_all_idfile "${global_host_list[*]}" "if ! test -d $dir; then mkdir -p $dir; fi" || lib_exit 1
|
||||
done
|
||||
;;
|
||||
*) lib_exit 1 "undefined method $install_mars_method"
|
||||
|
|
@ -74,7 +74,7 @@ function install_mars_sync_mars_and_kernel
|
|||
done
|
||||
|
||||
local host
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
lib_vmsg " calling depmod $install_mars_kernel_release on $host"
|
||||
lib_remote_idfile $install_mars_source_host depmod $install_mars_kernel_release || lib_exit 1
|
||||
for dirname in "${!files_in_dir[@]}"; do
|
||||
|
|
@ -88,7 +88,7 @@ function install_mars_sync_mars_and_kernel
|
|||
function install_mars_sync_userspace
|
||||
{
|
||||
local host file target_dir
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
for file in ${!install_mars_userspace_target_dir_list[@]}; do
|
||||
target_dir=${install_mars_userspace_target_dir_list["$file"]}
|
||||
lib_vmsg " synching $install_mars_source_host:$file to $host:$target_dir"
|
||||
|
|
@ -109,10 +109,10 @@ function install_mars_update_bootloader_on_target_hosts
|
|||
{
|
||||
local host boot_loader
|
||||
local label_name
|
||||
for host in "${main_host_list[@]}"; do
|
||||
boot_loader="${main_host_bootloader_list[$host]}"
|
||||
for host in "${global_host_list[@]}"; do
|
||||
boot_loader="${global_host_bootloader_list[$host]}"
|
||||
case "$boot_loader" in # ((
|
||||
lilo) label_name="${main_host_bootloader_label_list[$host]}"
|
||||
lilo) label_name="${global_host_bootloader_label_list[$host]}"
|
||||
lib_vmsg "checking label $label_name on host $host"
|
||||
lib_remote_idfile $host lilo -I "$label_name" || lib_exit 1
|
||||
lib_vmsg " calling lilo on $host"
|
||||
|
|
@ -144,7 +144,7 @@ function install_mars_finish
|
|||
done
|
||||
|
||||
local host
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
install_mars_check_files_to_sync "$host" \
|
||||
"$install_mars_kernel_release" \
|
||||
"${install_mars_files_to_sync_list[@]}" \
|
||||
|
|
@ -166,7 +166,7 @@ function install_mars_check_access_to_remote_hosts
|
|||
kernel_and_modules_via_sync_from_host_to_host)
|
||||
# check access root@source_host -> root@target_hosts
|
||||
local host
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
lib_vmsg " testing access from root@$install_mars_source_host to root@$host"
|
||||
lib_remote_idfile $install_mars_source_host \
|
||||
ssh root@$host hostname || lib_exit 1
|
||||
|
|
@ -201,7 +201,7 @@ function install_mars_check_variables
|
|||
if ! [ "${install_mars_method_list["$install_mars_method"]}" -eq 1 ]; then
|
||||
lib_exit 1 "invalid method $install_mars_method"
|
||||
fi
|
||||
if [ ${#main_host_list[*]} -eq 0 ]; then
|
||||
if [ ${#global_host_list[*]} -eq 0 ]; then
|
||||
lib_exit 1 "no target hosts given"
|
||||
fi
|
||||
if [ -z "$install_mars_kernel_release" ];then
|
||||
|
|
@ -26,7 +26,7 @@ function lib_err_check_nonexistence_of_other_error_messages
|
|||
local rc
|
||||
lib_vmsg " checking non existence of $errmsg_pattern in $host@$msg_file"
|
||||
lib_remote_idfile $host \
|
||||
"egrep '$main_mars_errmsg_prefix' $msg_file | egrep -v '$errmsg_pattern'"
|
||||
"egrep '$global_mars_errmsg_prefix' $msg_file | egrep -v '$errmsg_pattern'"
|
||||
rc=$?
|
||||
if [ $rc -eq 0 ];then
|
||||
lib_exit 1 "other errors than $errmsg_pattern found in $host@$msg_file"
|
||||
|
|
@ -48,7 +48,7 @@ function lib_check_for_kernel_oops_after_start_time
|
|||
echo " cannot look for recent kernel oops" >&2
|
||||
return
|
||||
fi
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
last_stack_line="$(lib_remote_idfile $host "grep -w Stack $kern_log | tail -1")"
|
||||
if [ -n "$last_stack_line" ]; then
|
||||
lib_vmsg " last kernel stack on $host: $last_stack_line"
|
||||
|
|
@ -79,7 +79,7 @@ function lib_check_proc_sys_mars_variables
|
|||
{
|
||||
local host rc
|
||||
local file_pattern='/proc/sys/mars/mem*'
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
lib_vmsg " displaying $file_pattern on $host"
|
||||
lib_remote_idfile $host 'if ls -lrt '"$file_pattern"';then for f in '"$file_pattern"';do echo $f; cat $f;done;fi'
|
||||
done
|
||||
|
|
@ -96,7 +96,7 @@ function lib_err_check_and_copy_global_err_files_all
|
|||
local host rc
|
||||
local log_err=$lib_err_total_log_file.err
|
||||
local total_log_fetched=0
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
lib_vmsg " checking whether $host:$lib_err_total_err_file exists"
|
||||
lib_remote_idfile $host "ls -l $lib_err_total_err_file && test -s $lib_err_total_err_file"
|
||||
rc=$?
|
||||
|
|
@ -119,7 +119,7 @@ function lib_err_check_and_copy_global_err_files_all
|
|||
# may be that there are some errors only in the log file
|
||||
lib_vmsg " checking for errors in $host:$lib_err_total_log_file"
|
||||
lib_remote_idfile $host \
|
||||
"egrep '$main_errors_only_in_total_log_pattern' $lib_err_total_log_file >$log_err"
|
||||
"egrep '$global_errors_only_in_total_log_pattern' $lib_err_total_log_file >$log_err"
|
||||
rc=$?
|
||||
if [ $rc -eq 0 ];then
|
||||
echo "ERROR-IN-LOGFILE $host:$lib_err_total_log_file:" >&2
|
||||
|
|
@ -41,7 +41,7 @@ function lib_linktree_print_linktree
|
|||
lib_vmsg "lamport clock on $host:"
|
||||
lib_remote_idfile $host 'cat /proc/sys/mars/lamport_clock'
|
||||
lib_vmsg "printing link tree on $host"
|
||||
lib_remote_idfile $host 'ls -l --full-time $(find /'"$main_mars_directory"' \! -type d | sort)'
|
||||
lib_remote_idfile $host 'ls -l --full-time $(find /'"$global_mars_directory"' \! -type d | sort)'
|
||||
}
|
||||
|
||||
# the required link value may be specified with an unit (e.g. 3G)
|
||||
|
|
@ -103,7 +103,7 @@ function lib_linktree_check_link
|
|||
if [ $waited -ge $lib_linktree_maxtime_to_wait_for_link_value ]
|
||||
then
|
||||
lib_vmsg " max. wait time $lib_linktree_maxtime_to_wait_for_link_value exceeded"
|
||||
return ${main_link_status["link_has_wrong_value"]}
|
||||
return ${global_link_status["link_has_wrong_value"]}
|
||||
else
|
||||
sleep 1
|
||||
let waited+=1
|
||||
|
|
@ -111,10 +111,10 @@ function lib_linktree_check_link
|
|||
continue
|
||||
fi
|
||||
else
|
||||
return ${main_link_status["link_ok"]}
|
||||
return ${global_link_status["link_ok"]}
|
||||
fi
|
||||
else
|
||||
return ${main_link_status["link_does_not_exist"]}
|
||||
return ${global_link_status["link_does_not_exist"]}
|
||||
fi
|
||||
done
|
||||
lib_exit 1 "this code should not be reached"
|
||||
|
|
@ -123,8 +123,8 @@ function lib_linktree_check_link
|
|||
function lib_linktree_status_to_string
|
||||
{
|
||||
local link_status=$1 status
|
||||
for status in "${!main_link_status[@]}"; do
|
||||
if [ ${main_link_status["$status"]} = "$link_status" ]; then
|
||||
for status in "${!global_link_status[@]}"; do
|
||||
if [ ${global_link_status["$status"]} = "$link_status" ]; then
|
||||
echo $status
|
||||
return
|
||||
fi
|
||||
|
|
@ -146,7 +146,7 @@ function lib_rw_compare_checksums
|
|||
for role in "primary" "secondary"; do
|
||||
local dummy_file
|
||||
eval host='$'${role}_host
|
||||
dummy_file=$main_mars_directory/dummy-$host
|
||||
dummy_file=$global_mars_directory/dummy-$host
|
||||
marsadm_do_cmd $host "down" $res || lib_exit 1
|
||||
if [ $cmp_size -eq 0 ]; then
|
||||
local link_value
|
||||
|
|
@ -19,8 +19,8 @@
|
|||
|
||||
function logrotate_run
|
||||
{
|
||||
local primary_host="${main_host_list[0]}"
|
||||
local secondary_host="${main_host_list[1]}"
|
||||
local primary_host="${global_host_list[0]}"
|
||||
local secondary_host="${global_host_list[1]}"
|
||||
local res=${resource_name_list[0]}
|
||||
local dev=$(lv_config_get_lv_device $res)
|
||||
local writer_pid writer_script write_count
|
||||
|
|
@ -32,7 +32,7 @@ function lv_config_prepare
|
|||
function lv_config_delete_vg
|
||||
{
|
||||
local host out i rc
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
if [ ${lv_config_vg_recreatable_list[$host]} -eq 1 ]; then
|
||||
lib_vmsg " removing lvs in $lv_config_lvg_name on $host"
|
||||
out="$(lib_remote_idfile $host lvdisplay -c $lv_config_lvg_name)"
|
||||
|
|
@ -80,7 +80,7 @@ function lv_config_extract_int_from_lv_size_with_unit
|
|||
function lv_config_check_volume_group_existence_and_size
|
||||
{
|
||||
local host lvg_size_with_unit rc
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
lib_vmsg " checking volume group $lv_config_lvg_name on $host"
|
||||
lvg_size_with_unit=$(lib_remote_idfile $host vgs --noheadings \
|
||||
--units G -o vg_size $lv_config_lvg_name)
|
||||
|
|
@ -112,15 +112,15 @@ function lv_config_check_variables
|
|||
fi
|
||||
|
||||
local host
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
if [ -z "${lv_config_vg_recreatable_list[$host]}" ]; then
|
||||
lib_exit 1 "host $host is missing in array lv_config_vg_recreatable_list"
|
||||
fi
|
||||
done
|
||||
lib_check_access_to_remote_hosts "$main_ssh_idfile_opt" \
|
||||
"${main_host_list[@]}"
|
||||
"${global_host_list[@]}"
|
||||
|
||||
lv_config_check_volume_group_existence_and_size "${main_host_list[@]}"
|
||||
lv_config_check_volume_group_existence_and_size "${global_host_list[@]}"
|
||||
|
||||
}
|
||||
|
||||
|
|
@ -168,7 +168,7 @@ function lv_config_get_size_logical_volume
|
|||
function lv_config_recreate_logical_volumes
|
||||
{
|
||||
local create_volume_group_too=$1 host lv_name
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
if [ $create_volume_group_too -eq 1 ]; then
|
||||
if [ ${lv_config_vg_recreatable_list[$host]} -eq 0 ]; then
|
||||
lib_vmsg " skipping recreation of vg $lv_config_lvg_name on $host"
|
||||
|
|
@ -49,7 +49,7 @@ function marsadm_get_highest_to_delete_nr
|
|||
local ls_out_old="x" ls_out_act maxcount=10 count=0
|
||||
# wait until ls output is stable
|
||||
while true; do
|
||||
ls_out_act="$(lib_remote_idfile $host 'ls -1 $main_mars_directory/todo-global/delete-[0-9]*[0-9]')"
|
||||
ls_out_act="$(lib_remote_idfile $host 'ls -1 $global_mars_directory/todo-global/delete-[0-9]*[0-9]')"
|
||||
rc=$?
|
||||
if [ -z "$ls_out_act" -o $rc -ne 0 ]; then
|
||||
echo 0
|
||||
|
|
@ -69,7 +69,7 @@ function marsadm_get_highest_to_delete_nr
|
|||
done
|
||||
ret=$(echo "$ls_out_act" | sed 's/.*-//' | sort -n | tail -1)
|
||||
if ! expr "$ret" : '^[0-9][0-9]*$' >/dev/null; then
|
||||
lib_exit 1 "cannot determine number of $main_mars_directory/todo-global/delete-* files on $host (ret=$ret)"
|
||||
lib_exit 1 "cannot determine number of $global_mars_directory/todo-global/delete-* files on $host (ret=$ret)"
|
||||
fi
|
||||
echo $ret
|
||||
}
|
||||
|
|
@ -78,7 +78,7 @@ function marsadm_get_deleted_link_value
|
|||
{
|
||||
local host=$1
|
||||
local ret
|
||||
local link=$main_mars_directory/todo-global/deleted-$host
|
||||
local link=$global_mars_directory/todo-global/deleted-$host
|
||||
ret=$(lib_remote_idfile $host "readlink $link")
|
||||
if [ -z "$ret" ]; then
|
||||
ret=0
|
||||
|
|
@ -282,7 +282,7 @@ function marsadm_host_is_primary
|
|||
local link="$(lib_linktree_get_primary_linkname $host $res)"
|
||||
lib_linktree_check_link $host "$link" 1
|
||||
link_status=$?
|
||||
if [ $link_status -ne ${main_link_status["link_ok"]} ]; then
|
||||
if [ $link_status -ne ${global_link_status["link_ok"]} ]; then
|
||||
return 0
|
||||
fi
|
||||
return 1
|
||||
|
|
@ -295,7 +295,7 @@ function marsadm_host_is_designated_primary
|
|||
local link="$(lib_linktree_get_designated_primary_linkname $res)"
|
||||
lib_linktree_check_link $host "$link" $host
|
||||
link_status=$?
|
||||
if [ $link_status -ne ${main_link_status["link_ok"]} ]; then
|
||||
if [ $link_status -ne ${global_link_status["link_ok"]} ]; then
|
||||
return 0
|
||||
fi
|
||||
return 1
|
||||
|
|
@ -23,7 +23,7 @@
|
|||
|
||||
function memleak_run
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local res=${resource_name_list[0]}
|
||||
local data_dev=$(resource_get_data_device $res)
|
||||
local cmd='dd if=/dev/zero of='"$data_dev"' bs=4096 count=1000000 & sleep 1; kill -9 $(jobs -p); maxcount=20; count=0; while test $count -lt $maxcount && ! marsadm secondary '"$res"' ; do date; echo $count; sleep 1; let count+=1; done'
|
||||
|
|
@ -85,7 +85,7 @@ function mount_umount_data_device_all
|
|||
local res_no=${1:-0}
|
||||
local host
|
||||
local res=${resource_name_list[$res_no]}
|
||||
for host in ${main_host_list[@]}; do
|
||||
for host in ${global_host_list[@]}; do
|
||||
mount_umount_data_device $host $res
|
||||
done
|
||||
}
|
||||
|
|
@ -31,8 +31,8 @@ function multires_prepare
|
|||
|
||||
function multires_create_resources_all
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host=${main_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host=${global_host_list[1]}
|
||||
local res lv_dev count=0 maxwait=20
|
||||
for res in ${lv_config_lv_name_list[@]}; do
|
||||
local lv_dev=$(lv_config_get_lv_device $res)
|
||||
|
|
@ -21,8 +21,8 @@
|
|||
|
||||
function net_run
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host="${main_host_list[1]}"
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host="${global_host_list[1]}"
|
||||
local res=${resource_name_list[0]}
|
||||
local dev=$(lv_config_get_lv_device $res)
|
||||
local writer_pid writer_script write_count
|
||||
|
|
@ -129,7 +129,7 @@ function net_check_variables
|
|||
function net_clear_iptables_all
|
||||
{
|
||||
local host
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
lib_vmsg " flushing iptables on $host"
|
||||
lib_remote_idfile $host "iptables -F" || lib_exit 1
|
||||
done
|
||||
|
|
@ -30,8 +30,8 @@ function perftest_check_variables
|
|||
|
||||
function perftest_run
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host=${main_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host=${global_host_list[1]}
|
||||
local res=${resource_name_list[0]}
|
||||
|
||||
perftest_check_variables
|
||||
|
|
@ -57,8 +57,8 @@ function perftest_run
|
|||
|
||||
function perftest_finish
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host=${main_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host=${global_host_list[1]}
|
||||
perftest_stop_nttcp $secondary_host
|
||||
}
|
||||
|
||||
|
|
@ -20,8 +20,8 @@
|
|||
|
||||
function remote_dev_run
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host=${main_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host=${global_host_list[1]}
|
||||
local res=${resource_name_list[0]}
|
||||
local dev=$(lv_config_get_lv_device $res)
|
||||
local writer_pid writer_script write_count
|
||||
|
|
@ -65,7 +65,7 @@ function remote_dev_run
|
|||
lib_rw_stop_writing_data_device $primary_host $writer_script "write_count"
|
||||
main_error_recovery_functions["lib_rw_stop_scripts"]=
|
||||
|
||||
local boot_label_name="${main_host_bootloader_label_list[$secondary_host]}"
|
||||
local boot_label_name="${global_host_bootloader_label_list[$secondary_host]}"
|
||||
local mars_dev=$(lv_config_get_lv_device ${cluster_mars_dir_lv_name_list[$secondary_host]})
|
||||
crash_reboot $secondary_host "" $mars_dev $crash_maxtime_reboot \
|
||||
$crash_maxtime_to_become_unreachable \
|
||||
|
|
@ -83,8 +83,8 @@ function remote_dev_remove_magic_links
|
|||
mount_umount_data_device_all
|
||||
cluster_rmmod_mars_all
|
||||
|
||||
for host in "${main_host_list[@]}"; do
|
||||
local magic_link_pattern="$main_mars_directory/resource-*/$(remote_dev_get_magic_link_name $secondary_host)"
|
||||
for host in "${global_host_list[@]}"; do
|
||||
local magic_link_pattern="$global_mars_directory/resource-*/$(remote_dev_get_magic_link_name $secondary_host)"
|
||||
lib_vmsg " removing $magic_link_pattern on $host"
|
||||
lib_remote_idfile $host "rm $magic_link_pattern"
|
||||
done
|
||||
|
|
@ -19,8 +19,8 @@
|
|||
|
||||
function resize_prepare
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host="${main_host_list[1]}"
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host="${global_host_list[1]}"
|
||||
local res=${resource_name_list[0]}
|
||||
local data_lv_size_orig=$(lv_config_get_lv_size_from_name $res)
|
||||
local dev=$(lv_config_get_lv_device $res)
|
||||
|
|
@ -69,8 +69,8 @@ function resize_check_variables
|
|||
|
||||
function resize_run
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host=${main_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host=${global_host_list[1]}
|
||||
local res=${resource_name_list[0]}
|
||||
local data_lv_size_orig=$(lv_config_get_lv_size_from_name $res)
|
||||
local data_lv_size_new=$(($data_lv_size_orig + $resize_size_to_add))
|
||||
|
|
@ -37,8 +37,8 @@ function resource_prepare
|
|||
# - exactly one resource
|
||||
function resource_quick_prepare_first_resource
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host=${main_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host=${global_host_list[1]}
|
||||
local res=${resource_name_list[0]}
|
||||
local dev="$(lv_config_get_lv_device $res)"
|
||||
local data_dev=$(resource_get_data_device $res)
|
||||
|
|
@ -72,7 +72,7 @@ function resource_recreate_all
|
|||
resource_leave_all
|
||||
|
||||
local host primary_host_to_join res
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
for res in "${resource_name_list[@]}"; do
|
||||
if [ -z "$primary_host_to_join" ]; then
|
||||
resource_create $host $res
|
||||
|
|
@ -87,7 +87,7 @@ function resource_recreate_all
|
|||
function resource_leave_all
|
||||
{
|
||||
local host res
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
for res in "${resource_name_list[@]}"; do
|
||||
resource_leave $host $res
|
||||
done
|
||||
|
|
@ -158,8 +158,8 @@ function resource_do_after_leave_loops
|
|||
local link="${resource_dir_list[$res]}/actual-$host/open-count"
|
||||
lib_linktree_check_link $host "$link" "0"
|
||||
link_status=$?
|
||||
if [ $link_status -ne ${main_link_status["link_ok"]} \
|
||||
-a $link_status -ne ${main_link_status["link_does_not_exist"]} ]
|
||||
if [ $link_status -ne ${global_link_status["link_ok"]} \
|
||||
-a $link_status -ne ${global_link_status["link_does_not_exist"]} ]
|
||||
then
|
||||
let count+=1
|
||||
sleep 1
|
||||
|
|
@ -188,7 +188,7 @@ function resource_joined
|
|||
local link_value_expected=(".")
|
||||
lib_linktree_check_link $host "$link" "$link_value_expected"
|
||||
link_status=$?
|
||||
if [ $link_status -eq ${main_link_status["link_ok"]} ]; then
|
||||
if [ $link_status -eq ${global_link_status["link_ok"]} ]; then
|
||||
lib_vmsg " resource $res on $host exists"
|
||||
return 0
|
||||
else
|
||||
|
|
@ -214,12 +214,12 @@ function resource_run_all
|
|||
function resource_run
|
||||
{
|
||||
local res=$1 host i
|
||||
for i in ${!main_host_list[*]}; do
|
||||
host=${main_host_list[$i]}
|
||||
for i in ${!global_host_list[*]}; do
|
||||
host=${global_host_list[$i]}
|
||||
if [ $i -eq 0 ]; then
|
||||
resource_create $host $res
|
||||
else
|
||||
resource_join $host $res ${main_host_list[0]}
|
||||
resource_join $host $res ${global_host_list[0]}
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
|
@ -231,8 +231,8 @@ function resource_multi_res_run
|
|||
|
||||
function resource_fill_mars_dir
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host=${main_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host=${global_host_list[1]}
|
||||
local res=${resource_name_list[0]}
|
||||
local dev=$(lv_config_get_lv_device $res)
|
||||
local data_dev=$(resource_get_data_device $res)
|
||||
|
|
@ -245,7 +245,7 @@ function resource_fill_mars_dir
|
|||
|
||||
if [ $resource_use_data_dev_writes_to_fill_mars_dir -eq 1 ]; then
|
||||
resource_dd_until_mars_dir_full $primary_host $res \
|
||||
$main_mars_directory \
|
||||
$global_mars_directory \
|
||||
$data_dev $mars_dev_size \
|
||||
$data_dev_size "control_nr"
|
||||
resource_check_low_space_error $primary_host $res "sequence_hole"
|
||||
|
|
@ -253,7 +253,7 @@ function resource_fill_mars_dir
|
|||
lib_rw_start_writing_data_device $primary_host "writer_pid" \
|
||||
"writer_script" 0 2 $res
|
||||
resource_write_file_until_mars_dir_full $primary_host \
|
||||
$main_mars_directory \
|
||||
$global_mars_directory \
|
||||
$mars_dev_size \
|
||||
$resource_big_file
|
||||
fi
|
||||
|
|
@ -397,7 +397,7 @@ function resource_mount_mars_and_rm_resource_dir_all
|
|||
fi
|
||||
cluster_rmmod_mars_all
|
||||
|
||||
for host in "${main_host_list[@]}"; do
|
||||
for host in "${global_host_list[@]}"; do
|
||||
local mars_lv=${cluster_mars_dir_lv_name_list[$host]}
|
||||
local mars_dev=$(lv_config_get_lv_device $mars_lv)
|
||||
lib_vmsg " removing $host:$res_dir whether mounted or not"
|
||||
|
|
@ -539,8 +539,8 @@ function resource_check_mount_point_directories
|
|||
|
||||
function resource_write_and_check
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
eval local secondary_hosts=('"${main_host_list["{1..'${#main_host_list[*]}'}"]}"')
|
||||
local primary_host=${global_host_list[0]}
|
||||
eval local secondary_hosts=('"${global_host_list["{1..'${#global_host_list[*]}'}"]}"')
|
||||
local host
|
||||
local res=${resource_name_list[0]}
|
||||
local writer_pid writer_script write_count
|
||||
|
|
@ -626,7 +626,7 @@ function resource_check_links_after_join
|
|||
local link_value_expected="$primary_host"
|
||||
lib_linktree_check_link $host "$link" "$link_value_expected"
|
||||
link_status=$?
|
||||
if [ $link_status -ne ${main_link_status["link_ok"]} ]; then
|
||||
if [ $link_status -ne ${global_link_status["link_ok"]} ]; then
|
||||
lib_exit 1 "resource $res on $host has not been joined"
|
||||
fi
|
||||
}
|
||||
|
|
@ -639,7 +639,7 @@ function resource_check_links_after_create
|
|||
local link_value_expected="$host"
|
||||
lib_linktree_check_link $host "$link" "$link_value_expected"
|
||||
link_status=$?
|
||||
if [ $link_status -ne ${main_link_status["link_ok"]} ]; then
|
||||
if [ $link_status -ne ${global_link_status["link_ok"]} ]; then
|
||||
lib_exit 1 "resource $res on $host has not been created"
|
||||
fi
|
||||
}
|
||||
|
|
@ -662,9 +662,9 @@ function resource_clear_data_device
|
|||
function resource_kill_all_scripts
|
||||
{
|
||||
local host
|
||||
for host in "${main_host_list[@]}"; do
|
||||
lib_vmsg " killing all $main_prefix_scripts scripts on $host"
|
||||
lib_remote_idfile $host 'for p in $(pgrep -f '"$main_prefix_scripts"'); do if [ $p -ne $$ ] && ps -p $p >/dev/null; then echo killing:; ps -fp $p; kill -9 $p; fi; done'
|
||||
for host in "${global_host_list[@]}"; do
|
||||
lib_vmsg " killing all $global_prefix_scripts scripts on $host"
|
||||
lib_remote_idfile $host 'for p in $(pgrep -f '"$global_prefix_scripts"'); do if [ $p -ne $$ ] && ps -p $p >/dev/null; then echo killing:; ps -fp $p; kill -9 $p; fi; done'
|
||||
done
|
||||
}
|
||||
|
||||
|
|
@ -702,8 +702,8 @@ function resource_check_replication
|
|||
|
||||
function resource_leave_while_sync
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host=${main_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host=${global_host_list[1]}
|
||||
local res=${resource_name_list[0]}
|
||||
local dev="$(lv_config_get_lv_device $res)"
|
||||
local time_waited
|
||||
|
|
@ -767,8 +767,8 @@ function resource_check_sync
|
|||
|
||||
function resource_recreate_standalone
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host=${main_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host=${global_host_list[1]}
|
||||
local res=${resource_name_list[0]}
|
||||
local dev="$(lv_config_get_lv_device $res)"
|
||||
local time_waited
|
||||
|
|
@ -21,8 +21,8 @@
|
|||
|
||||
function switch2primary_run
|
||||
{
|
||||
local primary_host=${main_host_list[0]}
|
||||
local secondary_host=${main_host_list[1]}
|
||||
local primary_host=${global_host_list[0]}
|
||||
local secondary_host=${global_host_list[1]}
|
||||
local res=${resource_name_list[0]}
|
||||
local dev=$(lv_config_get_lv_device $res)
|
||||
local writer_pid writer_script write_count
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue