mirror of
http://git.haproxy.org/git/haproxy.git/
synced 2024-12-14 23:44:41 +00:00
haproxy public development tree
2bfefdbaef
Since threads were introduced, we've naturally had a number of bugs related to locking issues. In addition we've also got some issues with corrupted lists in certain rare cases not necessarily involving threads. Not only these events cause a lot of trouble to the production as it is very hard to detect that the process is stuck in a loop and doesn't deliver the service anymore, but it's often difficult (or too late) to collect more debugging information. The patch presented here implements a lockup detection mechanism, also known as "watchdog". The principle is that (on systems supporting it), each thread will have its own CPU timer which progresses as the thread consumes CPU cycles, and when a deadline is met, a signal is delivered (SIGALRM here since it doesn't interrupt gdb by default). The thread handling this signal (which is not necessarily the one which triggered the timer) figures the thread ID from the signal arguments and checks if it's really stuck by looking at the time spent since last exit from poll() and by checking that the thread's scheduler is still alive (so that even when dealing with configuration issues resulting in insane amount of tasks being called in turn, it is not possible to accidently trigger it). Checking the scheduler's activity will usually result in a second chance, thus doubling the detecting time. In order not to incorrectly flag a thread as being the cause of the lockup, the thread_harmless_mask is checked : a thread could very well be spinning on itself waiting for all other threads to join (typically what happens when issuing "show sess"). In this case, once all threads but one (or two) have joined, all the innocent ones are marked harmless and will not trigger the timer. Only the ones not reacting will. The deadline is set to one second, which already appears impossible to reach, especially since it's 1 second of CPU usage, not elapsed time with the CPU being preempted by other threads/processes/hypervisor. In practice due to the scheduler's health verification it takes up to two seconds to decide to panic. Once all conditions are met, the goal is to crash from the offending thread. So if it's the current one, we call ha_panic() otherwise the signal is bounced to the offending thread which deals with it. This will result in all threads being woken up in turn to dump their context, the whole state is emitted on stderr in hope that it can be logged, and the process aborts, leaving a chance for a core to be dumped and for a service manager to restart it. An alternative mechanism could be implemented for systems unable to wake up a thread once its CPU clock reaches a deadline (e.g. FreeBSD). Instead of waking the timer each and every deadline, it is possible to use a standard timer which is reset each time we leave poll(). Since the signal handler rechecks the CPU consumption this will also work. However a totally idle process may trigger it from time to time which may or may not confuse some debugging sessions. The same is true for alarm() which could be another option for systems not having such a broad choice of timers (but it seems that in this case they will not have per-thread CPU measurements available either). The feature is currently implemented only when threads are enabled in order to keep the code clean, since the main purpose is to detect and address inter-thread deadlocks. But if it proves useful for other situations this condition might be relaxed. |
||
|---|---|---|
| .github/ISSUE_TEMPLATE | ||
| contrib | ||
| doc | ||
| ebtree | ||
| examples | ||
| include | ||
| reg-tests | ||
| scripts | ||
| src | ||
| tests | ||
| .cirrus.yml | ||
| .gitignore | ||
| .travis.yml | ||
| CHANGELOG | ||
| CONTRIBUTING | ||
| INSTALL | ||
| LICENSE | ||
| MAINTAINERS | ||
| Makefile | ||
| README | ||
| ROADMAP | ||
| SUBVERS | ||
| VERDATE | ||
| VERSION | ||
The HAProxy documentation has been split into a number of different files for ease of use. Please refer to the following files depending on what you're looking for : - INSTALL for instructions on how to build and install HAProxy - LICENSE for the project's license - CONTRIBUTING for the process to follow to submit contributions The more detailed documentation is located into the doc/ directory : - doc/intro.txt for a quick introduction on HAProxy - doc/configuration.txt for the configuration's reference manual - doc/lua.txt for the Lua's reference manual - doc/SPOE.txt for how to use the SPOE engine - doc/network-namespaces.txt for how to use network namespaces under Linux - doc/management.txt for the management guide - doc/regression-testing.txt for how to use the regression testing suite - doc/peers.txt for the peers protocol reference - doc/coding-style.txt for how to adopt HAProxy's coding style - doc/internals for developer-specific documentation (not all up to date)