2018-04-23 13:25:32 +00:00
|
|
|
\begin{verbatim}
|
|
|
|
Usage:
|
|
|
|
./football.sh --help [--verbose]
|
|
|
|
Show help
|
|
|
|
./football.sh --variable=<value>
|
|
|
|
Override any shell variable
|
|
|
|
|
|
|
|
Actions for resource migration:
|
|
|
|
|
|
|
|
./football.sh migrate <resource> <target_primary> [<target_secondary>]
|
|
|
|
Run the sequence
|
|
|
|
migrate_prepare ; migrate_wait ; migrate_finish; migrate_cleanup.
|
|
|
|
|
2018-07-24 06:59:19 +00:00
|
|
|
Dto for testing of phases:
|
|
|
|
|
2018-04-23 13:25:32 +00:00
|
|
|
./football.sh migrate_prepare <resource> <target_primary> [<target_secondary>]
|
|
|
|
Allocate LVM space at the targets and start MARS replication.
|
|
|
|
|
|
|
|
./football.sh migrate_wait <resource> <target_primary> [<target_secondary>]
|
|
|
|
Wait until MARS replication reports UpToDate.
|
|
|
|
|
|
|
|
./football.sh migrate_finish <resource> <target_primary> [<target_secondary>]
|
|
|
|
Call hooks for handover to the targets.
|
|
|
|
|
|
|
|
./football.sh migrate_cleanup <resource>
|
|
|
|
Remove old / currently unused LV replicas from MARS and deallocate
|
|
|
|
from LVM.
|
|
|
|
|
2018-07-02 08:29:37 +00:00
|
|
|
Actions for inplace FS shrinking:
|
|
|
|
|
|
|
|
./football.sh shrink <resource> <percent>
|
|
|
|
Run the sequence shrink_prepare ; shrink_finish ; shrink_cleanup.
|
|
|
|
|
2018-07-24 06:59:19 +00:00
|
|
|
Dto for testing of phases:
|
|
|
|
|
2018-07-02 08:29:37 +00:00
|
|
|
./football.sh shrink_prepare <resource> [<percent>]
|
|
|
|
Allocate temporary LVM space (when possible) and create initial
|
|
|
|
raw FS copy.
|
|
|
|
Default percent value(when left out) is 85.
|
|
|
|
|
|
|
|
./football.sh shrink_finish <resource>
|
|
|
|
Incrementally update the FS copy, swap old <=> new copy with
|
|
|
|
small downtime.
|
|
|
|
|
|
|
|
./football.sh shrink_cleanup <resource>
|
|
|
|
Remove old FS copy from LVM.
|
|
|
|
|
|
|
|
Actions for inplace FS extension:
|
|
|
|
|
2018-07-10 12:02:58 +00:00
|
|
|
./football.sh expand <resource> <percent>
|
2018-07-02 08:29:37 +00:00
|
|
|
./football.sh extend <resource> <percent>
|
2018-07-10 12:02:58 +00:00
|
|
|
Increase mounted filesystem size during operations.
|
2018-07-02 08:29:37 +00:00
|
|
|
|
|
|
|
Combined actions:
|
|
|
|
|
|
|
|
./football.sh migrate+shrink <resource> <target_primary> [<target_secondary>] [<percent>]
|
|
|
|
Similar to migrate ; shrink but produces less network traffic.
|
|
|
|
Default percent value (when left out) is 85.
|
|
|
|
|
|
|
|
./football.sh migrate+shrink+back <resource> <tmp_primary> [<percent>]
|
|
|
|
Migrate temporarily to <tmp_primary>, then shrink there,
|
|
|
|
finally migrate back to old primary and secondaries.
|
|
|
|
Default percent value (when left out) is 85.
|
|
|
|
|
2018-04-23 13:25:32 +00:00
|
|
|
Actions for (manual) repair in emergency situations:
|
|
|
|
|
2018-07-24 06:59:19 +00:00
|
|
|
./football.sh manual_handover <resource> <target_primary>
|
|
|
|
This is useful in place of going to the machines and starting
|
|
|
|
handover on their command line. You dont need to log in.
|
|
|
|
All hooks (e.g. for downtime / reporting / etc) are automatically
|
|
|
|
called.
|
|
|
|
Notice: it will only work when there is already a replica
|
|
|
|
at <target_primary>, and when further constraints such as
|
|
|
|
clustermanager constraints will allow it.
|
|
|
|
For a full Football game between different clusters, use
|
|
|
|
"migrate" instead.
|
|
|
|
|
2018-04-23 13:25:32 +00:00
|
|
|
./football.sh manual_migrate_config <resource> <target_primary> [<target_secondary>]
|
|
|
|
Transfer only the cluster config, without changing the MARS replicas.
|
|
|
|
This does no resource stopping / restarting.
|
|
|
|
Useful for reverting a failed migration.
|
|
|
|
|
|
|
|
./football.sh manual_config_update <hostname>
|
|
|
|
Only update the cluster config, without changing anything else.
|
|
|
|
Useful for manual repair of failed migration.
|
|
|
|
|
|
|
|
./football.sh manual_merge_cluster <hostname1> <hostname2>
|
|
|
|
Run "marsadm merge-cluster" for the given hosts.
|
|
|
|
Hostnames must be from different (former) clusters.
|
|
|
|
|
|
|
|
./football.sh manual_split_cluster <hostname_list>
|
|
|
|
Run "marsadm split-cluster" at the given hosts.
|
|
|
|
Useful for fixing failed / asymmetric splits.
|
|
|
|
Hint: provide _all_ hostnames which have formerly participated
|
|
|
|
in the cluster.
|
|
|
|
|
|
|
|
./football.sh repair_vm <resource> <primary_candidate_list>
|
|
|
|
Try to restart the VM <resource> on one of the given machines.
|
|
|
|
Useful during unexpected customer downtime.
|
|
|
|
|
|
|
|
./football.sh repair_mars <resource> <primary_candidate_list>
|
|
|
|
Before restarting the VM like in repair_vm, try to find a local
|
|
|
|
LV where a stand-alone MARS resource can be found and built up.
|
|
|
|
Use this only when the MARS resources are gone, and when you are
|
|
|
|
desperate. Problem: this will likely create a MARS setup which is
|
|
|
|
not usable for production, and therefore must be corrected later
|
|
|
|
by hand. Use this only during an emergency situation in order to
|
|
|
|
get the customers online again, while buying the downsides of this
|
|
|
|
command.
|
|
|
|
|
2018-07-02 08:29:37 +00:00
|
|
|
./football.sh manual_lock <item> <host_list>
|
|
|
|
./football.sh manual_unlock <item> <host_list>
|
|
|
|
Manually lock or unlock an item at all of the given hosts, in
|
|
|
|
an atomic fashion. In most cases, use "ALL" for the item.
|
2018-04-23 13:25:32 +00:00
|
|
|
|
|
|
|
Global maintenance:
|
|
|
|
|
|
|
|
./football.sh lv_cleanup <resource>
|
|
|
|
|
|
|
|
General features:
|
|
|
|
|
|
|
|
- Instead of <percent>, an absolute amount of storage with suffix
|
|
|
|
'k' or 'm' or 'g' can be given.
|
|
|
|
|
|
|
|
- When <resource> is currently stopped, login to the container is
|
|
|
|
not possible, and in turn the hypervisor node and primary storage node
|
|
|
|
cannot be automatically determined. In such a case, the missing
|
|
|
|
nodes can be specified via the syntax
|
|
|
|
<resource>:<hypervisor>:<primary_storage>
|
|
|
|
|
|
|
|
- The following LV suffixes are used (naming convention):
|
|
|
|
-tmp = currently emerging version for shrinking
|
|
|
|
-preshrink = old version before shrinking took place
|
|
|
|
|
|
|
|
- By adding the option --screener, you can handover football execution
|
|
|
|
to ./screener.sh .
|
|
|
|
When some --enable_*_waiting is also added, then the critical
|
|
|
|
sections involving customer downtime are temporarily halted until
|
|
|
|
some sysadmins says "screener.sh continue $resource" or
|
|
|
|
attaches to the sessions and presses the RETURN key.
|
|
|
|
|
|
|
|
|
2018-07-02 08:29:37 +00:00
|
|
|
PLUGIN football-1and1config
|
|
|
|
|
|
|
|
1&1 specfic plugin for dealing with the cm3 clusters
|
2018-07-10 12:02:58 +00:00
|
|
|
and its concrete configuration.
|
|
|
|
|
2018-07-02 08:29:37 +00:00
|
|
|
|
2018-04-23 13:25:32 +00:00
|
|
|
PLUGIN football-cm3
|
|
|
|
|
|
|
|
1&1 specfic plugin for dealing with the cm3 cluster manager
|
|
|
|
and its concrete operating enviroment (singleton instance).
|
|
|
|
|
|
|
|
Current maximum cluster size limit:
|
|
|
|
|
|
|
|
Maximum #syncs running before migration can start:
|
|
|
|
|
|
|
|
Following marsadm --version must be installed:
|
|
|
|
|
|
|
|
Following mars kernel modules must be loaded:
|
|
|
|
|
2018-07-10 12:02:58 +00:00
|
|
|
Specific actions for plugin football-cm3:
|
|
|
|
|
|
|
|
./football.sh clustertool {GET|PUT} <url>
|
|
|
|
Call through to the clustertool via REST.
|
|
|
|
Useful for manual inspection and repair.
|
|
|
|
|
2018-07-24 06:59:19 +00:00
|
|
|
Specific features with plugin football-cm3:
|
|
|
|
|
|
|
|
- Parameter syntax "cluster123" instead of "icpu456 icpu457"
|
|
|
|
This is an alternate specification syntax, which is
|
|
|
|
automatically replaced with the real machine names.
|
|
|
|
It tries to minimize datacenter cross-traffic by
|
|
|
|
taking the new $target_primary at the same datacenter
|
|
|
|
location where the container is currenty running.
|
|
|
|
|
|
|
|
|
|
|
|
PLUGIN football-ticket
|
|
|
|
|
|
|
|
Generic plugin for creating and updating tickets,
|
|
|
|
e.g. Jira tickets.
|
|
|
|
|
|
|
|
You will need to hook in some external scripts which are
|
|
|
|
then creating / updating the tickets.
|
|
|
|
|
|
|
|
Comment texts may be provided with following conventions:
|
|
|
|
|
|
|
|
comment.$ticket_state.txt
|
|
|
|
comment.$ticket_phase.$ticket_state.txt
|
|
|
|
|
|
|
|
Directories where comments may reside:
|
|
|
|
|
|
|
|
football_creds=/usr/lib/mars/creds /etc/mars/creds /home/schoebel/mars/football-master.git/creds /home/schoebel/mars/football-master.git /home/schoebel/.mars/creds ./creds
|
|
|
|
football_confs=/usr/lib/mars/confs /etc/mars/confs /home/schoebel/mars/football-master.git/confs /home/schoebel/.mars/confs ./confs
|
|
|
|
football_includes=/usr/lib/mars/plugins /etc/mars/plugins /home/schoebel/mars/football-master.git/plugins /home/schoebel/.mars/plugins ./plugins
|
|
|
|
|
2018-04-23 13:25:32 +00:00
|
|
|
|
|
|
|
PLUGIN football-basic
|
|
|
|
|
|
|
|
Generic driver for systemd-controlled MARS pools.
|
|
|
|
The current version supports only a flat model:
|
|
|
|
(1) There is a single "big cluster" at metadata level.
|
|
|
|
All cluster members are joined via merge-cluster.
|
|
|
|
All occurring names need to be globally unique.
|
|
|
|
(2) The network uses BGP or other means, thus any hypervisor
|
|
|
|
can (potentially) start any VM at any time.
|
|
|
|
(3) iSCSI or remote devices are not supported for now
|
|
|
|
(LocalSharding model). This may be extended in a future
|
|
|
|
release.
|
|
|
|
This plugin is exclusive-or with cm3.
|
|
|
|
|
|
|
|
Plugin specific actions:
|
|
|
|
|
|
|
|
./football.sh basic_add_host <hostname>
|
|
|
|
Manually add another host to the hostname cache.
|
|
|
|
|
|
|
|
|
2018-07-10 12:02:58 +00:00
|
|
|
PLUGIN football-downtime
|
|
|
|
|
|
|
|
Generic plugin for communication of customer downtime.
|
|
|
|
|
|
|
|
|
2018-04-23 13:25:32 +00:00
|
|
|
PLUGIN football-motd
|
|
|
|
|
|
|
|
Generic plugin for motd. Communicate that Football is running
|
|
|
|
at login via motd.
|
|
|
|
|
|
|
|
|
|
|
|
PLUGIN football-report
|
|
|
|
|
|
|
|
Generic plugin for communication of reports.
|
|
|
|
|
|
|
|
|
|
|
|
PLUGIN football-waiting
|
|
|
|
|
|
|
|
Generic plugig, interfacing with screener: when this is used
|
|
|
|
by your script and enabled, then you will be able to wait for
|
|
|
|
"screener.sh continue" operations at certain points in your
|
|
|
|
script.
|
|
|
|
|
2018-07-10 12:02:58 +00:00
|
|
|
|
2018-04-23 13:25:32 +00:00
|
|
|
\end{verbatim}
|