haproxy/doc/internals/mworker.md

Master Worker

2024-06-12

History

haproxy-systemd-wrapper

Back in 2013, distributions are discussing the adoption of systemd as the default init, this was controversial but fedora and archlinux already uses it. At this time HAProxy still had a multi-process model, and the way haproxy is working was incompatible with the daemon mode.

Systemd is compatible with traditionnal forking services, but somehow HAProxy is different. To work correctly, systemd needs a main PID, this is the PID of the process that systemd will supervises.

With nbproc 1 that could work, since systemd is able to guess the main PID, and even to read a PID file. But there is something uncommon that HAProxy is doing for a reload, which is not supported by systemd. Indeed the reload is in fact a new haproxy process, which will ask the old one to leave. This means the new main PID is supposed to change, but systemd is not supporting this, so it will just see the previous leaving and consider that the service broke and kill every other processes, meaning the new haproxy.

With nbproc > 1 this is worse, systemd is confused with all the processes, because they are independent, so there is not really a main process to supervise.

The systemd-wrapper appeared in HAProxy 1.5, it's a separated binary, which starts haproxy, so systemd can use the wrapper as the main PID, and the wrapper never change PID. Upon a reload, which is done with a SIGUSR2 signal the wrapper will launch a haproxy -sf. This was a non-intrusive work which a first step to deploy in systemd environments. Later contributions would add the support for upgrading the wrapper binary upon a reload.

However the wrapper suffered from several problems:

• It needed a intermediate haproxy process, it's basically a daemon mode, but instead of the first process leaving to daemonize, it is kept in foreground to waitpid() on all workers. Which means you need the wrapper + the -Ds + the haproxy workers, and each reload start a new -Ds.
• it was difficult to integrate new features since it wasn't in haproxy itself
• there were multiple issues with handling the failures during reload

mworker V1

HAProxy 1.8 got ride of the wrapper which was replaced by the master worker mode. This first version was basically a reintegration of the wrapper features within HAProxy. HAProxy is launched with the -W flag, read the configuration and then fork. In mworker mode, the master is usually launched as a root process, and will do chroot operations then setuid in the workers.

Like the wrapper, the master handle the SIGURS2 signal to reload, it is also able to forward the SIGUSR1 signal to the workers, to ask for a soft stop. The reload uses the same logic than the standard -sf method, but instead of starting a new process, it will exec() with -sf in the same PID. Which means that haproxy could upgrade its binary during the reload.

Once the SIGUSR2 signal is received, the master would block signals and unregister signals handler so no signals would halt haproxy reload, as it could kill the master to receive a USR2 if the signal is not register yet after the exec.

When doing the exec() upon a reload, a new argv array is constructed by copying the current argv and adding -sf and the list of PIDs in the children list, as well as the oldpids list.

When the workers are started, the master will first deinit the poller and clean the FDs that are not needed anymore (inherited fd need to be kept however), then the master will do a wait() loop instead of the haproxy polling loop, which will wait for its workers to leave, or for a signal.

When reloading haproxy, a non-working configuration could exits the master, which could end in killing all previous workers. This is a complex situation to handle, since all configuration parsing code was not written to let a process alive upon a failure. To handle this problem, an atexit() callback was used, so haproxy would reexec() upon a configuration loading failure, without any configuration, and without trying to fork new workers. This is called the master-worker "wait mode".

The master-worker mode also comes with a feature which automates the seamless reload (-x), meaning it would select the stats socket from the configuration to be added to the -x parameter for the next reload, so the FD of the bind could be retrieved automatically.

The master is supervising the workers, when a current worker (not a previous one from before the reload) is exiting without being asked for a reload, the master will emit an "exit-on-failure" error and will kill every workers with a SIGTERM and exits with the same error code than the failed master, this behavior can be changed by using the "no exit-on-failure" option in the global section.

While the master is supervising the workers using the wait() function, the workers is also surpervising the master. To achieve this, there is a pipe between the master and the workers. The FD of the worker side of the pipe is inserted in the poller so it can watch for a close. When the pipe is closed this means the master left, and this is not supposed to happen, so it could have crash. When it happens all workers are leaving. To survive the reloads of the master, the FD are saved in environment variables (HAPROXY_MWORKER_PIPE_{RD,WR})

The master-worker mode could be activated by using either "-W" or "master-worker" in the global section of the configuration, but it is prefered to use "-W".

The pidfile is usable in master-worker mode, instead of writing the PIDs of all workers, this will only write the PID of the master.

A systemd mode (-Ws) could also be used, it behaves the same way as -W, but will keep the master in foreground, and will send status messages to systemd using the sd_notify API.

mworker V2

HAProxy 1.9 go a little bit further with the master worker, instead of using the mworker_wait() fuction from V1, it uses the haproxy polling loop, so the signals will be handled directly by haproxy polling loop, removing the specific code.

Instead of using 1 pipe per haproxy instance, the V2 is using a socketpair per worker and the polling loop allows real network communication using these socketpairs. It needs to keep 1 FD per worker in the master, so they can be reused after a reload. The master keeps a linked list of processes, mworker_proc, containing socketpairs fd, PID, relative pid... This list is then serialized in the HAPROXY_PROCESSES environment variable to be unserialized upon a reload and the FD reinserted in the poller.

Since the FD are in the poller, there is a special flag in the listeners LI_O_WORKER, which specify that some FD mustn't be used in the worker, these FD are unbind once in the worker.

Meanwhile the thread support was implemented in haproxy, since mworker shares more code than before when using the polling loop, the nbthread configuration variable is not used for instancing the master, and the master will always remain with only 1 thread.

The HAPROXY_PROCESSES structures allow to store a lot more thing, the number of reload for each worker is kept, the PID etc...

The socketpairs are useful for bi-directional communication, so each socketpair are connected to a stats applet on the worker side, so the master could access to a stats socket for each worker.

The master implements a CLI proxy, which is an analyzer which is able to parse CLI input, which will split individual CLI commands and redirect them to the right worker. This is implemented like the HTTP pipelining with command being sent and responsed one after another. This proxy could be accessed by using the master CLI which is only bound using the -S option of the haproxy command. Special prefixed using @ syntax are used to select the right worker.

The master CLI implements its own commands set like show proc which shows the content of the HAPROXY_PROCESSES structure.

A 'reload' command was implemented so the reload could be asked from the master CLI without using the SIGUSR2 signal.

more features in mworker V2

HAProxy 2.0 implements a new configuration section called program this section allows to handle the start and stop of executables with the master-worker. One could launch the dataplane API from haproxy for example. The programs are shown in the show proc command. The programs will be added to the HAPROXY_PROCESSES structure. The option 'start-on-reload' allows to configure the behavior of a program during an haproxy reload, it can either start a new instance of the program or keep the previous one.

A mworker-max-reloads keyword was added in the global section, it allows to limit the number of reload a worker can endure. That helps limiting the number of remaining worker processes. This will send a SIGTERM to the worker once it reach this value, instead of a SIGUSR1, so any stuck worker is killed.

Version and starting time were added to HAPROXY_PROCESSES so they could be displayed in show proc.

HAProxy 2.1 added user/group to the program section so they could change their uid after the fork.

HAProxy 2.5 added the reexec of haproxy in wait mode after a successful loading, instead of doing it only after a configuration failure. It is useful to clear the memory of the master because charging the configuration from the master can take a lot of RAM, and there is no simple wait to free everything and decrease the memory space of the process.

In HAProxy 2.6, the seamless reload with the master-worker changed, instead of using a stats socket declared in the configuration, this uses the internal socketpair of the previous worker. The change is actually simple, instead of doing a -x /path/to/previous/socket it does a -x sockpair@FD using the FD number that can be found in HAPROXY_PROCESSES. With this change the stats socket in the configuration is less useful and everything can be done from the master CLI.

With 2.7, the reload mecanism of the master CLI evolved, with previous versions, this mecanism was asynchronous, so once the reload command was received, the master would reload, the active master CLI connection was closed, and there was no way to return a status as a response to the reload command. To achieve a synchronous reload, a dedicated sockpair is used, one side uses a master CLI applet and the other side wait to receive a socket. When the master CLI receives the reload command, it takes the FD of the active master CLI session, sends it in the socketpair and then does an exec. The FD is then stuck in the kernel during the reload, because the poller is disabled. Once haproxy reloaded and the poller active, the FD of the master CLI connection is received, so HAProxy can reply a success or failure status for the reload. When built with USE_SHM_OPEN=1, a shm is used to keep the warnings and errors when loading the configuration in a shared buffer so this could survive the rexec in wait mode and then be dumped as a response to the reload command after the status.

In 2.9 the master CLI command hard-reload was implemented, it works the same way as the reload command, but instead of exec() with -sf for a soft-stop, it starts with -st to achieve a hard stop on the previous worker.

Version 3.0 got rid of the libsystemd dependencies for sd_notify() after the events of xz/openssh, the function is now implemented directly in haproxy in src/systemd.c.