mirror of
https://github.com/schoebel/mars
synced 2024-12-13 10:15:34 +00:00
194 lines
6.8 KiB
Plaintext
194 lines
6.8 KiB
Plaintext
\begin{verbatim}
|
|
./screener.sh: Run _unattended_ processes in screen sessions.
|
|
Useful for MASS automation, running hundreds of unattended
|
|
commands in parallel.
|
|
HINT: for running more than ~500 sessions in parallel, you might need
|
|
some system tuning (e.g. rlimits, kernel patches etc) for creating
|
|
a huge number of file descritor / sockets / etc.
|
|
ADVANTAGE: You may attach to individual screens, kill them, or continue
|
|
some waiting commands.
|
|
|
|
Synopsis:
|
|
./screener.sh --help [--verbose]
|
|
./screener.sh list-running
|
|
./screener.sh list-waiting
|
|
./screener.sh list-failed
|
|
./screener.sh list-critical
|
|
./screener.sh list-serious
|
|
./screener.sh list-done
|
|
./screener.sh list
|
|
./screener.sh list-screens
|
|
./screener.sh run <file.csv> [<condition_list>]
|
|
./screener.sh start <screen_id> <cmd> <args...>
|
|
./screener.sh [<options>] <operation> <screen_id>
|
|
|
|
Inquiry operations:
|
|
|
|
./screener.sh list-screens
|
|
Equivalent to screen -ls
|
|
|
|
./screener.sh list-<type>
|
|
Show a list of currently running, waiting (for continuation), failed,
|
|
and done/completed screen sessions.
|
|
|
|
./screener.sh list
|
|
First show a list of currently running screens, then
|
|
for each <type> a list of (old) failed / completed / sessions
|
|
(and so on).
|
|
|
|
./screener.sh status <screen_id>
|
|
Like list-*, but filter <sceen_id> and dont report timestamps.
|
|
|
|
./screener.sh show <screen_id>
|
|
Show the last logfile of <screen_id> at standard output.
|
|
|
|
./screener.sh less <screen_id>
|
|
Show the last logfile of <screen_id> using "less -r".
|
|
|
|
MASS starting of screen sessions:
|
|
|
|
./screener.sh run <file.csv> <condition_list>
|
|
Commands are launched in screen sessions via "./screener.sh start" commands,
|
|
unless the same <screen_id> is already running,
|
|
or is in some error state, or is already done (see below).
|
|
The commands are given by a column with CSV header name
|
|
containing "command", or by the first column.
|
|
The <screen_id> needs to be given by a column with CSV header
|
|
name matching "screen_id|resource".
|
|
The number and type of commands to launch can be reduced via
|
|
any combination of the following filter conditions:
|
|
|
|
--max=<number>
|
|
Limit the number of _new_ sessions additionally started this time.
|
|
|
|
--<column_name>==<value>
|
|
Only select lines where an arbitrary CSV column (given by its
|
|
CSV header name in C identifier syntax) has the given value.
|
|
|
|
--<column_name>!=<value>
|
|
Only select lines where the colum has _not_ the given value.
|
|
|
|
--<column_name>=~<bash_regex>
|
|
Only select lines where the bash regular expression matches
|
|
at the given column.
|
|
|
|
--max-per=<number>
|
|
Limit the number per _distinct_ value of the column denoted by
|
|
the _next_ filter condition.
|
|
Example: ./screener.sh run test.csv --dry-run --max-per=2 --dst_network=~.
|
|
would launch only 2 Football processes per destination network.
|
|
|
|
Hint: filter conditions can be easily checked by giving --dry-run.
|
|
|
|
Start / restart / kill / continue screen sessions:
|
|
|
|
./screener.sh start <screen_id> <cmd> <args...>
|
|
Start a new screen session, running arbitrary <cmd> and <args...>
|
|
inside.
|
|
|
|
./screener.sh restart <screen_id>
|
|
Works only when the last command for <screen_id> failed.
|
|
This will restart the old <cmd> and its <args...> as before.
|
|
Use only when you want to repeat the same command once again.
|
|
|
|
./screener.sh kill <screen_id>
|
|
Terminate the running screen session forcibly.
|
|
|
|
./screener.sh continue
|
|
./screener.sh continue <screen_id> [<screen_id_list>]
|
|
./screener.sh continue <number>
|
|
Useful for MASS automation of processes involving critical sections
|
|
such as customer downtime.
|
|
When giving a numerical <number> argument, up to that number
|
|
of sessions are resumed (ordered by age).
|
|
When no further arugment is given, _all_ currently waiting sessions
|
|
are continued.
|
|
When --auto-attach is given, it will sequentially resume the
|
|
sessions to be continued. By default, unless --force_attach is set,
|
|
it uses "screen -r" skipping those sessions which are already
|
|
attached to somebody else.
|
|
This feature works only with prepared scripts which are creating
|
|
an empty flagfile
|
|
/home/schoebel/mars/mars-migration.git/screener-logdir-testing/running/$screen_id.waiting
|
|
whenever they want to wait for manual intervention (for whatever reason).
|
|
Afterwards, the script must be polling this flagfile for removal.
|
|
This screener operation simply removes the flagfile, such that
|
|
the script will then continue afterwards.
|
|
Example: look into ./football.sh
|
|
and search for occurrences of substring "call_hook start_wait".
|
|
|
|
./screener.sh wakeup
|
|
./screener.sh wakeup <screen_id> [<screen_id_list>]
|
|
./screener.sh wakeup <number>
|
|
Similar to continue, but refers to delayed commands waiting for
|
|
a timeout. This can be used to individually shorten the timeout
|
|
period.
|
|
Example: Football cleanup operations may be artificially delayed
|
|
before doing "lvremove", to keep some sort of 'backup' for a
|
|
limited time. When your project is under time pressure, these
|
|
delays may be hindering.
|
|
Use this for premature ending of such artificial delays.
|
|
|
|
./screener.sh up <...>
|
|
Do both continue and wakeup.
|
|
|
|
./screener.sh auto <...>
|
|
Equivalent to ./screener.sh --auto-attach up <...>
|
|
Remember that only session without current attachment will be
|
|
attached to.
|
|
|
|
Attach to a running session:
|
|
|
|
./screener.sh attach <screen_id>
|
|
This is equivalent to screen -x $screen_id
|
|
|
|
./screener.sh resume <screen_id>
|
|
This is equivalent to screen -r $screen_id
|
|
|
|
Communication:
|
|
|
|
./screener.sh notify <screen_id> <txt>
|
|
May be called from external scripts to send emails etc.
|
|
|
|
Locking (only when supported by <cmd>):
|
|
|
|
./screener.sh lock
|
|
./screener.sh unlock
|
|
./screener.sh lock <screen_id>
|
|
./screener.sh unlock <screen_id>
|
|
|
|
Cleanup / bookkeeping:
|
|
|
|
./screener.sh clear-critical <screen_id>
|
|
./screener.sh clear-serious <screen_id>
|
|
./screener.sh clear-failed <screen_id>
|
|
Mark the status as "done" and move the logfile away.
|
|
|
|
./screener.sh purge [<days>]
|
|
This will remove all old logfiles which are older than
|
|
<days>. By default, the variable $screener_log_purge_period
|
|
will be used, which is currently set to '30'.
|
|
|
|
./screener.sh cron
|
|
You should call this regulary from a user cron job, in order
|
|
to purge old logfiles, or to detect hanging sessions, or to
|
|
automatically send pending emails, etc.
|
|
|
|
Options:
|
|
|
|
--variable
|
|
--variable=$value
|
|
These must come first, in order to prevent mixup with
|
|
options of <cmd> <args...>.
|
|
Allows overriding of any internal shell variable.
|
|
--help --verbose
|
|
Show all overridable shell variables, also for plugins.
|
|
|
|
|
|
PLUGIN screener-email
|
|
|
|
Generic plugin for sending emails (or SMS via gateways)
|
|
upon status changes, such as script failures.
|
|
|
|
\end{verbatim}
|