haproxy/doc/internals/initcalls.txt
Willy Tarreau 64a18534e3 DOC: internal: document the init calls
INITCALLs are used a lot in the code now and were not documented, resulting
in each user having to grep for functions in other files. This doc is not
perfect but aims at improving the situation. It documents what's been
available since 1.9 and may be backported there if it helps though it's
unlikely to be needed as it's mostly aimed at developers.
2019-11-20 16:52:56 +01:00

367 lines
19 KiB
Plaintext

Initialization stages aka how to get your code initialized at the right moment
1. Background
Originally all subsystems were initialized via a dedicated function call
from the huge main() function. Then some code started to become conditional
or a bit more modular and the #ifdef placed there became a mess, resulting
in init code being moved to function constructors in each subsystem's own
file. Then pools of various things were introduced, starting to make the
whole init sequence more complicated due to some forms of internal
dependencies. Later epoll was introduced, requiring a post-fork callback,
and finally threads arrived also requiring some post-thread init/deinit
and allocation, marking the old architecture's last breath. Finally the
whole thing resulted in lots of init code duplication and was simplified
in 1.9 with the introduction of initcalls and initialization stages.
2. New architecture
The new architecture relies on two layers :
- the registration functions
- the INITCALL macros and initialization stages
The first ones are mostly used to add a callback to a list. The second ones
are used to specify when to call a function. Both are totally independent,
however they are generally combined via another set consisting in the REGISTER
macros which make some registration functions be called at some specific points
during the init sequence.
3. Registration functions
Registration functions never fail. Or more precisely, if they fail it will only
be on out-of-memory condition, and they will cause the process to immediately
exit. As such they do not return any status and the caller doesn't have to care
about their success.
All available functions are described below in alphanumeric ordering. Please
make sure to respect this ordering when adding new ones.
- void hap_register_build_opts(const char *str, int must_free)
This appends the zero-terminated constant string <str> to the list of known
build options that will be reported on the output of "haproxy -vv". A line
feed character ('\n') will automatically be appended after the string when it
is displayed. The <must_free> argument must be zero, unless the string was
allocated by any malloc-compatible function such as malloc()/calloc()/
realloc()/strdup() or memprintf(), in which case it's better to pass a
non-null value so that the string is freed upon exit. Note that despite the
function's prototype taking a "const char *", the pointer will actually be
cast and freed. The const char* is here to leave more freedom to use consts
when making such options lists.
- void hap_register_per_thread_alloc(int (*fct)())
This adds a call to function <fct> to the list of functions to be called when
threads are started, at the beginning of the polling loop. This is also valid
for the main thread and will be called even if threads are disabled, so that
it is guaranteed that this function will be called in any circumstance. Each
thread will first call all these functions exactly once when it starts. Calls
are serialized by the init_mutex, so that locking is not necessary in these
functions. There is no relation between the thread numbers and the callback
ordering. The function is expected to return non-zero on success, or zero on
failure. A failure will make the process emit a succint error message and
immediately exit. See also hap_register_per_thread_free() for functions
called after these ones.
- void hap_register_per_thread_deinit(void (*fct)());
This adds a call to function <fct> to the list of functions to be called when
threads are gracefully stopped, at the end of the polling loop. This is also
valid for the main thread and will be called even if threads are disabled, so
that it is guaranteed that this function will be called in any circumstance
if the process experiences a soft stop. Each thread will call this function
exactly once when it stops. However contrary to _alloc() and _init(), the
calls are made without any protection, thus if any shared resource if touched
by the function, the function is responsible for protecting it. The reason
behind this is that such resources are very likely to be still in use in one
other thread and that most of the time the functions will in fact only touch
a refcount or deinitialize their private resources. See also
hap_register_per_thread_free() for functions called after these ones.
- void hap_register_per_thread_free(void (*fct)());
This adds a call to function <fct> to the list of functions to be called when
threads are gracefully stopped, at the end of the polling loop, after all calls
to _deinit() callbacks are done for this thread. This is also valid for the
main thread and will be called even if threads are disabled, so that it is
guaranteed that this function will be called in any circumstance if the
process experiences a soft stop. Each thread will call this function exactly
once when it stops. However contrary to _alloc() and _init(), the calls are
made without any protection, thus if any shared resource if touched by the
function, the function is responsible for protecting it. The reason behind
this is that such resources are very likely to be still in use in one other
thread and that most of the time the functions will in fact only touch a
refcount or deinitialize their private resources. See also
hap_register_per_thread_deinit() for functions called before these ones.
- void hap_register_per_thread_init(int (*fct)())
This adds a call to function <fct> to the list of functions to be called when
threads are started, at the beginning of the polling loop, right after the
list of _alloc() functions. This is also valid for the main thread and will
be called even if threads are disabled, so that it is guaranteed that this
function will be called in any circumstance. Each thread will call this
function exactly once when it starts, and calls are serialized by the
init_mutex which is held over all _alloc() and _init() calls, so that locking
is not necessary in these functions. In other words for all threads but the
current one, the sequence of _alloc() and _init() calls will be atomic. There
is no relation between the thread numbers and the callback ordering. The
function is expected to return non-zero on success, or zero on failure. A
failure will make the process emit a succint error message and immediately
exit. See also hap_register_per_thread_alloc() for functions called before
these ones.
- void hap_register_post_check(int (*fct)())
This adds a call to function <fct> to the list of functions to be called at
the end of the configuration validity checks, just at the point where the
program either forks or exits depending whether it's called with "-c" or not.
Such calls are suited for memory allocation or internal table pre-computation
that would preferably not be done on the fly to avoid inducing extra time to
a pure configuration check. Threads are not yet started so no protection is
required. The function is expected to return non-zero on success, or zero on
failure. A failure will make the process emit a succint error message and
immediately exit.
- void hap_register_post_deinit(void (*fct)())
This adds a call to function <fct> to the list of functions to be called when
freeing the global sections at the end of deinit(), after everything is
stopped. The process is single-threaded at this point, thus these functions
are suitable for releasing configuration elements provided that no other
_deinit() function uses them, i.e. only close/release what is strictly
private to the subsystem. Since such functions are mostly only called during
soft stops (reloads) or failed startups, they tend to experience much less
test coverage than others despite being more exposed, and as such a lot of
care must be taken to test them especially when facing partial subsystem
initializations followed by errors.
- void hap_register_post_proxy_check(int (*fct)(struct proxy *))
This adds a call to function <fct> to the list of functions to be called for
each proxy, after the calls to _post_server_check(). This can allow, for
example, to pre-configure default values for an option in a frontend based on
the "bind" lines or something in a backend based on the "server" lines. It's
worth being aware that such a function must be careful not to waste too much
time in order not to significantly slow down configurations with tens of
thousands of backends. The function is expected to return non-zero on
success, or zero on failure. A failure will make the process emit a succint
error message and immediately exit.
- void hap_register_post_server_check(int (*fct)(struct server *))
This adds a call to function <fct> to the list of functions to be called for
each server, after the call to check_config_validity(). This can allow, for
example, to preset a health state on a server or to allocate a protocol-
specific memory area. It's worth being aware that such a function must be
careful not to waste too much time in order not to significantly slow down
configurations with tens of thousands of servers. The function is expected
to return non-zero on success, or zero on failure. A failure will make the
process emit a succint error message and immediately exit.
- void hap_register_proxy_deinit(void (*fct)(struct proxy *))
This adds a call to function <fct> to the list of functions to be called when
freeing the resources during deinit(). These functions will be called as part
of the proxy's resource cleanup. Note that some of the proxy's fields will
already have been freed and others not, so such a function must not use any
information from the proxy that is subject to being released. In particular,
all servers have already been deleted. Since such functions are mostly only
called during soft stops (reloads) or failed startups, they tend to
experience much less test coverage than others despite being more exposed,
and as such a lot of care must be taken to test them especially when facing
partial subsystem initializations followed by errors. It's worth mentioning
that too slow functions could have a significant impact on the configuration
check or exit time especially on large configurations.
- void hap_register_server_deinit(void (*fct)(struct server *))
This adds a call to function <fct> to the list of functions to be called when
freeing the resources during deinit(). These functions will be called as part
of the server's resource cleanup. Note that some of the server's fields will
already have been freed and others not, so such a function must not use any
information from the server that is subject to being released. Since such
functions are mostly only called during soft stops (reloads) or failed
startups, they tend to experience much less test coverage than others despite
being more exposed, and as such a lot of care must be taken to test them
especially when facing partial subsystem initializations followed by errors.
It's worth mentioning that too slow functions could have a significant impact
on the configuration check or exit time especially on large configurations.
4. Initialization stages
In order to offer some guarantees, the startup of the program is split into
several stages. Some callbacks can be placed into each of these stages using
an INITCALL macro, with 0 to 3 arguments, respectively called INITCALL0 to
INITCALL3. These macros must be placed anywhere at the top level of a C file,
preferably at the end so that the referenced symbols have already been met,
but it may also be fine to place them right after the callbacks themselves.
Such callbacks are referenced into small structures containing a pointer to the
function and 3 arguments. NULL replaces unused arguments. The callbacks are
cast to (void (*)(void *, void *, void *)) and the arguments to (void *).
The first argument to the INITCALL macro is the initialization stage. The
second one is the callback function, and others if any are the arguments.
The init stage must be among the values of the "init_stage" enum, currently,
and in this execution order:
- STG_PREPARE : used to preset variables, pre-initialize lookup tables and
pre-initialize list heads
- STG_LOCK : used to pre-initialize locks
- STG_ALLOC : used to allocate the required structures
- STG_POOL : used to create pools
- STG_REGISTER : used to register static lists such as keywords
- STG_INIT : used to initialize subsystems
Each stage is guaranteed that previous stages have successfully completed. This
means that an INITCALL placed at stage STG_REGISTER is guaranteed that all
pools were already created and will be usable. Conversely, an INITCALL placed
at stage STG_PREPARE must not rely on any field that requires preliminary
allocation nor initialization. A callback cannot rely on other callbacks of the
same stage, as the execution order within a stage is undefined and essentially
depends on the linking order.
Example: register a very early call to init_log() with no argument, and another
call to cli_register_kw(&cli_kws) much later:
INITCALL0(STG_PREPARE, init_log);
INITCALL1(STG_REGISTER, cli_register_kw, &cli_kws);
Technically speaking, each call to such a macro adds a distinct local symbol
whose dynamic name involves the line number. These symbols are placed into a
separate section and the beginning and end section pointers are provided by the
linker. When too old a linker is used, a fallback is applied consisting in
placing them into a linked list which is built by a constructor function for
each initcall (this takes more room).
Due to the symbols internally using the line number, it is very important not
to place more than one INITCALL per line in the source file.
It is also strongly recommended that functions and referenced arguments are
static symbols local to the source file, unless they are global registration
functions like in the example above with cli_register_kw(), where only the
argument is a local keywords table.
INITCALLs do not expect the callback function to return anything and as such
do not perform any error check. As such, they are very similar to constructors
offered by the compiler except that they are segmented in stages. It is thus
the responsibility of the called functions to perform their own error checking
and to exit in case of error. This may change in the future.
5. REGISTER family of macros
The association of INITCALLs and registration functions allows to perform some
early dynamic registration of functions to be used anywhere, as well as values
to be added to existing lists without having to manipulate list elements. For
the sake of simplification, these combinations are available as a set of
REGISTER macros which register calls to certain functions at the appropriate
init stage. Such macros must be used at the top level in a file, just like
INITCALL macros. The following macros are currently supported. Please keep them
alphanumerically ordered:
- REGISTER_BUILD_OPTS(str)
Adds the constant string <str> to the list of build options. This is done by
registering a call to hap_register_build_opts(str, 0) at stage STG_REGISTER.
The string will not be freed.
- REGISTER_CONFIG_POSTPARSER(name, parser)
Adds a call to function <parser> at the end of the config parsing. The
function is called at the very end of check_config_validity() and may be used
to initialize a subsystem based on global settings for example. This is done
by registering a call to cfg_register_postparser(name, parser) at stage
STG_REGISTER.
- REGISTER_CONFIG_SECTION(name, parse, post)
Registers a new config section name <name> which will be parsed by function
<parse> (if not null), and with an optional call to function <post> at the
end of the section. Function <parse> must be of type (int (*parse)(const char
*file, int linenum, char **args, int inv)), and returns 0 on success or an
error code among the ERR_* set on failure. The <post> callback takes no
argument and returns a similar error code. This is achieved by registering a
call to cfg_register_section() with the three arguments at stage
STG_REGISTER.
- REGISTER_PER_THREAD_ALLOC(fct)
Registers a call to register_per_thread_alloc(fct) at stage STG_REGISTER.
- REGISTER_PER_THREAD_DEINIT(fct)
Registers a call to register_per_thread_deinit(fct) at stage STG_REGISTER.
- REGISTER_PER_THREAD_FREE(fct)
Registers a call to register_per_thread_free(fct) at stage STG_REGISTER.
- REGISTER_PER_THREAD_INIT(fct)
Registers a call to register_per_thread_init(fct) at stage STG_REGISTER.
- REGISTER_POOL(ptr, name, size)
Used internally to declare a new pool. This is made by calling function
create_pool_callback() with these arguments at stage STG_POOL. Do not use it
directly, use either DECLARE_POOL() or DECLARE_STATIC_POOL() instead (see
below).
- REGISTER_POST_CHECK(fct)
Registers a call to register_post_check(fct) at stage STG_REGISTER.
- REGISTER_POST_DEINIT(fct)
Registers a call to register_post_deinit(fct) at stage STG_REGISTER.
- REGISTER_POST_PROXY_CHECK(fct)
Registers a call to register_post_proxy_check(fct) at stage STG_REGISTER.
- REGISTER_POST_SERVER_CHECK(fct)
Registers a call to register_post_server_check(fct) at stage STG_REGISTER.
- REGISTER_PROXY_DEINIT(fct)
Registers a call to register_proxy_deinit(fct) at stage STG_REGISTER.
- REGISTER_SERVER_DEINIT(fct)
Registers a call to register_server_deinit(fct) at stage STG_REGISTER.
6. Other initialization macros
On top of the INITCALL family of macros, a few other convenient macros were
created in order to simplify declarations or allocations:
- DECLARE_POOL(ptr, name, size)
Placed at the top level of a file, this declares a global memory pool as
variable <ptr>, name <name> and size <size> bytes per element. This is made
via a call to REGISTER_POOL() and by assigning the resulting pointer to
variable <ptr>. <ptr> will be created of type "struct pool_head *". If the
pool needs to be visible outside of the function (which is likely), it will
also need to be declared somewhere as "extern struct pool_head *<ptr>;". It
is recommended to place such declarations very early in the source file so
that the variable is already known to all subsequent functions which may use
it.
- DECLARE_STATIC_POOL(ptr, name, size)
Placed at the top level of a file, this declares a static memory pool as
variable <ptr>, name <name> and size <size> bytes per element. This is made
via a call to REGISTER_POOL() and by assigning the resulting pointer to local
variable <ptr>. <ptr> will be created of type "static struct pool_head *". It
is recommended to place such declarations very early in the source file so
that the variable is already known to all subsequent functions which may use
it.