mirror of
http://git.haproxy.org/git/haproxy.git/
synced 2024-12-09 04:58:59 +00:00
cd8a6f28c6
Enable the 'slowstart' keyword for dynamic servers. The slowstart task is allocated in 'add server' handler if slowstart is used. As the server is created in disabled state, there is no need to start the task. The slowstart task will be automatically started on the first 'enable server' invocation.
3972 lines
202 KiB
Plaintext
3972 lines
202 KiB
Plaintext
------------------------
|
|
HAProxy Management Guide
|
|
------------------------
|
|
version 2.5
|
|
|
|
|
|
This document describes how to start, stop, manage, and troubleshoot HAProxy,
|
|
as well as some known limitations and traps to avoid. It does not describe how
|
|
to configure it (for this please read configuration.txt).
|
|
|
|
Note to documentation contributors :
|
|
This document is formatted with 80 columns per line, with even number of
|
|
spaces for indentation and without tabs. Please follow these rules strictly
|
|
so that it remains easily printable everywhere. If you add sections, please
|
|
update the summary below for easier searching.
|
|
|
|
|
|
Summary
|
|
-------
|
|
|
|
1. Prerequisites
|
|
2. Quick reminder about HAProxy's architecture
|
|
3. Starting HAProxy
|
|
4. Stopping and restarting HAProxy
|
|
5. File-descriptor limitations
|
|
6. Memory management
|
|
7. CPU usage
|
|
8. Logging
|
|
9. Statistics and monitoring
|
|
9.1. CSV format
|
|
9.2. Typed output format
|
|
9.3. Unix Socket commands
|
|
9.4. Master CLI
|
|
10. Tricks for easier configuration management
|
|
11. Well-known traps to avoid
|
|
12. Debugging and performance issues
|
|
13. Security considerations
|
|
|
|
|
|
1. Prerequisites
|
|
----------------
|
|
|
|
In this document it is assumed that the reader has sufficient administration
|
|
skills on a UNIX-like operating system, uses the shell on a daily basis and is
|
|
familiar with troubleshooting utilities such as strace and tcpdump.
|
|
|
|
|
|
2. Quick reminder about HAProxy's architecture
|
|
----------------------------------------------
|
|
|
|
HAProxy is a multi-threaded, event-driven, non-blocking daemon. This means is
|
|
uses event multiplexing to schedule all of its activities instead of relying on
|
|
the system to schedule between multiple activities. Most of the time it runs as
|
|
a single process, so the output of "ps aux" on a system will report only one
|
|
"haproxy" process, unless a soft reload is in progress and an older process is
|
|
finishing its job in parallel to the new one. It is thus always easy to trace
|
|
its activity using the strace utility. In order to scale with the number of
|
|
available processors, by default haproxy will start one worker thread per
|
|
processor it is allowed to run on. Unless explicitly configured differently,
|
|
the incoming traffic is spread over all these threads, all running the same
|
|
event loop. A great care is taken to limit inter-thread dependencies to the
|
|
strict minimum, so as to try to achieve near-linear scalability. This has some
|
|
impacts such as the fact that a given connection is served by a single thread.
|
|
Thus in order to use all available processing capacity, it is needed to have at
|
|
least as many connections as there are threads, which is almost always granted.
|
|
|
|
HAProxy is designed to isolate itself into a chroot jail during startup, where
|
|
it cannot perform any file-system access at all. This is also true for the
|
|
libraries it depends on (eg: libc, libssl, etc). The immediate effect is that
|
|
a running process will not be able to reload a configuration file to apply
|
|
changes, instead a new process will be started using the updated configuration
|
|
file. Some other less obvious effects are that some timezone files or resolver
|
|
files the libc might attempt to access at run time will not be found, though
|
|
this should generally not happen as they're not needed after startup. A nice
|
|
consequence of this principle is that the HAProxy process is totally stateless,
|
|
and no cleanup is needed after it's killed, so any killing method that works
|
|
will do the right thing.
|
|
|
|
HAProxy doesn't write log files, but it relies on the standard syslog protocol
|
|
to send logs to a remote server (which is often located on the same system).
|
|
|
|
HAProxy uses its internal clock to enforce timeouts, that is derived from the
|
|
system's time but where unexpected drift is corrected. This is done by limiting
|
|
the time spent waiting in poll() for an event, and measuring the time it really
|
|
took. In practice it never waits more than one second. This explains why, when
|
|
running strace over a completely idle process, periodic calls to poll() (or any
|
|
of its variants) surrounded by two gettimeofday() calls are noticed. They are
|
|
normal, completely harmless and so cheap that the load they imply is totally
|
|
undetectable at the system scale, so there's nothing abnormal there. Example :
|
|
|
|
16:35:40.002320 gettimeofday({1442759740, 2605}, NULL) = 0
|
|
16:35:40.002942 epoll_wait(0, {}, 200, 1000) = 0
|
|
16:35:41.007542 gettimeofday({1442759741, 7641}, NULL) = 0
|
|
16:35:41.007998 gettimeofday({1442759741, 8114}, NULL) = 0
|
|
16:35:41.008391 epoll_wait(0, {}, 200, 1000) = 0
|
|
16:35:42.011313 gettimeofday({1442759742, 11411}, NULL) = 0
|
|
|
|
HAProxy is a TCP proxy, not a router. It deals with established connections that
|
|
have been validated by the kernel, and not with packets of any form nor with
|
|
sockets in other states (eg: no SYN_RECV nor TIME_WAIT), though their existence
|
|
may prevent it from binding a port. It relies on the system to accept incoming
|
|
connections and to initiate outgoing connections. An immediate effect of this is
|
|
that there is no relation between packets observed on the two sides of a
|
|
forwarded connection, which can be of different size, numbers and even family.
|
|
Since a connection may only be accepted from a socket in LISTEN state, all the
|
|
sockets it is listening to are necessarily visible using the "netstat" utility
|
|
to show listening sockets. Example :
|
|
|
|
# netstat -ltnp
|
|
Active Internet connections (only servers)
|
|
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
|
|
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd
|
|
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy
|
|
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy
|
|
|
|
|
|
3. Starting HAProxy
|
|
-------------------
|
|
|
|
HAProxy is started by invoking the "haproxy" program with a number of arguments
|
|
passed on the command line. The actual syntax is :
|
|
|
|
$ haproxy [<options>]*
|
|
|
|
where [<options>]* is any number of options. An option always starts with '-'
|
|
followed by one of more letters, and possibly followed by one or multiple extra
|
|
arguments. Without any option, HAProxy displays the help page with a reminder
|
|
about supported options. Available options may vary slightly based on the
|
|
operating system. A fair number of these options overlap with an equivalent one
|
|
if the "global" section. In this case, the command line always has precedence
|
|
over the configuration file, so that the command line can be used to quickly
|
|
enforce some settings without touching the configuration files. The current
|
|
list of options is :
|
|
|
|
-- <cfgfile>* : all the arguments following "--" are paths to configuration
|
|
file/directory to be loaded and processed in the declaration order. It is
|
|
mostly useful when relying on the shell to load many files that are
|
|
numerically ordered. See also "-f". The difference between "--" and "-f" is
|
|
that one "-f" must be placed before each file name, while a single "--" is
|
|
needed before all file names. Both options can be used together, the
|
|
command line ordering still applies. When more than one file is specified,
|
|
each file must start on a section boundary, so the first keyword of each
|
|
file must be one of "global", "defaults", "peers", "listen", "frontend",
|
|
"backend", and so on. A file cannot contain just a server list for example.
|
|
|
|
-f <cfgfile|cfgdir> : adds <cfgfile> to the list of configuration files to be
|
|
loaded. If <cfgdir> is a directory, all the files (and only files) it
|
|
contains are added in lexical order (using LC_COLLATE=C) to the list of
|
|
configuration files to be loaded ; only files with ".cfg" extension are
|
|
added, only non hidden files (not prefixed with ".") are added.
|
|
Configuration files are loaded and processed in their declaration order.
|
|
This option may be specified multiple times to load multiple files. See
|
|
also "--". The difference between "--" and "-f" is that one "-f" must be
|
|
placed before each file name, while a single "--" is needed before all file
|
|
names. Both options can be used together, the command line ordering still
|
|
applies. When more than one file is specified, each file must start on a
|
|
section boundary, so the first keyword of each file must be one of
|
|
"global", "defaults", "peers", "listen", "frontend", "backend", and so on.
|
|
A file cannot contain just a server list for example.
|
|
|
|
-C <dir> : changes to directory <dir> before loading configuration
|
|
files. This is useful when using relative paths. Warning when using
|
|
wildcards after "--" which are in fact replaced by the shell before
|
|
starting haproxy.
|
|
|
|
-D : start as a daemon. The process detaches from the current terminal after
|
|
forking, and errors are not reported anymore in the terminal. It is
|
|
equivalent to the "daemon" keyword in the "global" section of the
|
|
configuration. It is recommended to always force it in any init script so
|
|
that a faulty configuration doesn't prevent the system from booting.
|
|
|
|
-L <name> : change the local peer name to <name>, which defaults to the local
|
|
hostname. This is used only with peers replication. You can use the
|
|
variable $HAPROXY_LOCALPEER in the configuration file to reference the
|
|
peer name.
|
|
|
|
-N <limit> : sets the default per-proxy maxconn to <limit> instead of the
|
|
builtin default value (usually 2000). Only useful for debugging.
|
|
|
|
-V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or
|
|
"quiet".
|
|
|
|
-W : master-worker mode. It is equivalent to the "master-worker" keyword in
|
|
the "global" section of the configuration. This mode will launch a "master"
|
|
which will monitor the "workers". Using this mode, you can reload HAProxy
|
|
directly by sending a SIGUSR2 signal to the master. The master-worker mode
|
|
is compatible either with the foreground or daemon mode. It is
|
|
recommended to use this mode with multiprocess and systemd.
|
|
|
|
-Ws : master-worker mode with support of `notify` type of systemd service.
|
|
This option is only available when HAProxy was built with `USE_SYSTEMD`
|
|
build option enabled.
|
|
|
|
-c : only performs a check of the configuration files and exits before trying
|
|
to bind. The exit status is zero if everything is OK, or non-zero if an
|
|
error is encountered. Presence of warnings will be reported if any.
|
|
|
|
-cc : evaluates a condition as used within a conditional block of the
|
|
configuration. The exit status is zero if the condition is true, 1 if the
|
|
condition is false or 2 if an error is encountered.
|
|
|
|
-d : enable debug mode. This disables daemon mode, forces the process to stay
|
|
in foreground and to show incoming and outgoing events. It must never be
|
|
used in an init script.
|
|
|
|
-dD : enable diagnostic mode. This mode will output extra warnings about
|
|
suspicious configuration statements. This will never prevent startup even in
|
|
"zero-warning" mode nor change the exit status code.
|
|
|
|
-dG : disable use of getaddrinfo() to resolve host names into addresses. It
|
|
can be used when suspecting that getaddrinfo() doesn't work as expected.
|
|
This option was made available because many bogus implementations of
|
|
getaddrinfo() exist on various systems and cause anomalies that are
|
|
difficult to troubleshoot.
|
|
|
|
-dM[<byte>] : forces memory poisoning, which means that each and every
|
|
memory region allocated with malloc() or pool_alloc() will be filled with
|
|
<byte> before being passed to the caller. When <byte> is not specified, it
|
|
defaults to 0x50 ('P'). While this slightly slows down operations, it is
|
|
useful to reliably trigger issues resulting from missing initializations in
|
|
the code that cause random crashes. Note that -dM0 has the effect of
|
|
turning any malloc() into a calloc(). In any case if a bug appears or
|
|
disappears when using this option it means there is a bug in haproxy, so
|
|
please report it.
|
|
|
|
-dS : disable use of the splice() system call. It is equivalent to the
|
|
"global" section's "nosplice" keyword. This may be used when splice() is
|
|
suspected to behave improperly or to cause performance issues, or when
|
|
using strace to see the forwarded data (which do not appear when using
|
|
splice()).
|
|
|
|
-dV : disable SSL verify on the server side. It is equivalent to having
|
|
"ssl-server-verify none" in the "global" section. This is useful when
|
|
trying to reproduce production issues out of the production
|
|
environment. Never use this in an init script as it degrades SSL security
|
|
to the servers.
|
|
|
|
-dW : if set, haproxy will refuse to start if any warning was emitted while
|
|
processing the configuration. This helps detect subtle mistakes and keep the
|
|
configuration clean and portable across versions. It is recommended to set
|
|
this option in service scripts when configurations are managed by humans,
|
|
but it is recommended not to use it with generated configurations, which
|
|
tend to emit more warnings. It may be combined with "-c" to cause warnings
|
|
in checked configurations to fail. This is equivalent to global option
|
|
"zero-warning".
|
|
|
|
-db : disable background mode and multi-process mode. The process remains in
|
|
foreground. It is mainly used during development or during small tests, as
|
|
Ctrl-C is enough to stop the process. Never use it in an init script.
|
|
|
|
-de : disable the use of the "epoll" poller. It is equivalent to the "global"
|
|
section's keyword "noepoll". It is mostly useful when suspecting a bug
|
|
related to this poller. On systems supporting epoll, the fallback will
|
|
generally be the "poll" poller.
|
|
|
|
-dk : disable the use of the "kqueue" poller. It is equivalent to the
|
|
"global" section's keyword "nokqueue". It is mostly useful when suspecting
|
|
a bug related to this poller. On systems supporting kqueue, the fallback
|
|
will generally be the "poll" poller.
|
|
|
|
-dp : disable the use of the "poll" poller. It is equivalent to the "global"
|
|
section's keyword "nopoll". It is mostly useful when suspecting a bug
|
|
related to this poller. On systems supporting poll, the fallback will
|
|
generally be the "select" poller, which cannot be disabled and is limited
|
|
to 1024 file descriptors.
|
|
|
|
-dr : ignore server address resolution failures. It is very common when
|
|
validating a configuration out of production not to have access to the same
|
|
resolvers and to fail on server address resolution, making it difficult to
|
|
test a configuration. This option simply appends the "none" method to the
|
|
list of address resolution methods for all servers, ensuring that even if
|
|
the libc fails to resolve an address, the startup sequence is not
|
|
interrupted.
|
|
|
|
-m <limit> : limit the total allocatable memory to <limit> megabytes across
|
|
all processes. This may cause some connection refusals or some slowdowns
|
|
depending on the amount of memory needed for normal operations. This is
|
|
mostly used to force the processes to work in a constrained resource usage
|
|
scenario. It is important to note that the memory is not shared between
|
|
processes, so in a multi-process scenario, this value is first divided by
|
|
global.nbproc before forking.
|
|
|
|
-n <limit> : limits the per-process connection limit to <limit>. This is
|
|
equivalent to the global section's keyword "maxconn". It has precedence
|
|
over this keyword. This may be used to quickly force lower limits to avoid
|
|
a service outage on systems where resource limits are too low.
|
|
|
|
-p <file> : write all processes' pids into <file> during startup. This is
|
|
equivalent to the "global" section's keyword "pidfile". The file is opened
|
|
before entering the chroot jail, and after doing the chdir() implied by
|
|
"-C". Each pid appears on its own line.
|
|
|
|
-q : set "quiet" mode. This disables some messages during the configuration
|
|
parsing and during startup. It can be used in combination with "-c" to
|
|
just check if a configuration file is valid or not.
|
|
|
|
-S <bind>[,bind_options...]: in master-worker mode, bind a master CLI, which
|
|
allows the access to every processes, running or leaving ones.
|
|
For security reasons, it is recommended to bind the master CLI to a local
|
|
UNIX socket. The bind options are the same as the keyword "bind" in
|
|
the configuration file with words separated by commas instead of spaces.
|
|
|
|
Note that this socket can't be used to retrieve the listening sockets from
|
|
an old process during a seamless reload.
|
|
|
|
-sf <pid>* : send the "finish" signal (SIGUSR1) to older processes after boot
|
|
completion to ask them to finish what they are doing and to leave. <pid>
|
|
is a list of pids to signal (one per argument). The list ends on any
|
|
option starting with a "-". It is not a problem if the list of pids is
|
|
empty, so that it can be built on the fly based on the result of a command
|
|
like "pidof" or "pgrep".
|
|
|
|
-st <pid>* : send the "terminate" signal (SIGTERM) to older processes after
|
|
boot completion to terminate them immediately without finishing what they
|
|
were doing. <pid> is a list of pids to signal (one per argument). The list
|
|
is ends on any option starting with a "-". It is not a problem if the list
|
|
of pids is empty, so that it can be built on the fly based on the result of
|
|
a command like "pidof" or "pgrep".
|
|
|
|
-v : report the version and build date.
|
|
|
|
-vv : display the version, build options, libraries versions and usable
|
|
pollers. This output is systematically requested when filing a bug report.
|
|
|
|
-x <unix_socket> : connect to the specified socket and try to retrieve any
|
|
listening sockets from the old process, and use them instead of trying to
|
|
bind new ones. This is useful to avoid missing any new connection when
|
|
reloading the configuration on Linux. The capability must be enable on the
|
|
stats socket using "expose-fd listeners" in your configuration.
|
|
|
|
A safe way to start HAProxy from an init file consists in forcing the daemon
|
|
mode, storing existing pids to a pid file and using this pid file to notify
|
|
older processes to finish before leaving :
|
|
|
|
haproxy -f /etc/haproxy.cfg \
|
|
-D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
|
|
|
|
When the configuration is split into a few specific files (eg: tcp vs http),
|
|
it is recommended to use the "-f" option :
|
|
|
|
haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
|
|
-f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
|
|
-f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
|
|
-D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
|
|
|
|
When an unknown number of files is expected, such as customer-specific files,
|
|
it is recommended to assign them a name starting with a fixed-size sequence
|
|
number and to use "--" to load them, possibly after loading some defaults :
|
|
|
|
haproxy -f /etc/haproxy/global.cfg -f /etc/haproxy/stats.cfg \
|
|
-f /etc/haproxy/default-tcp.cfg -f /etc/haproxy/tcp.cfg \
|
|
-f /etc/haproxy/default-http.cfg -f /etc/haproxy/http.cfg \
|
|
-D -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid) \
|
|
-f /etc/haproxy/default-customers.cfg -- /etc/haproxy/customers/*
|
|
|
|
Sometimes a failure to start may happen for whatever reason. Then it is
|
|
important to verify if the version of HAProxy you are invoking is the expected
|
|
version and if it supports the features you are expecting (eg: SSL, PCRE,
|
|
compression, Lua, etc). This can be verified using "haproxy -vv". Some
|
|
important information such as certain build options, the target system and
|
|
the versions of the libraries being used are reported there. It is also what
|
|
you will systematically be asked for when posting a bug report :
|
|
|
|
$ haproxy -vv
|
|
HAProxy version 1.6-dev7-a088d3-4 2015/10/08
|
|
Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>
|
|
|
|
Build options :
|
|
TARGET = linux2628
|
|
CPU = generic
|
|
CC = gcc
|
|
CFLAGS = -pg -O0 -g -fno-strict-aliasing -Wdeclaration-after-statement \
|
|
-DBUFSIZE=8030 -DMAXREWRITE=1030 -DSO_MARK=36 -DTCP_REPAIR=19
|
|
OPTIONS = USE_ZLIB=1 USE_DLMALLOC=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
|
|
|
|
Default settings :
|
|
maxconn = 2000, bufsize = 8030, maxrewrite = 1030, maxpollevents = 200
|
|
|
|
Encrypted password support via crypt(3): yes
|
|
Built with zlib version : 1.2.6
|
|
Compression algorithms supported : identity("identity"), deflate("deflate"), \
|
|
raw-deflate("deflate"), gzip("gzip")
|
|
Built with OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
|
|
Running on OpenSSL version : OpenSSL 1.0.1o 12 Jun 2015
|
|
OpenSSL library supports TLS extensions : yes
|
|
OpenSSL library supports SNI : yes
|
|
OpenSSL library supports prefer-server-ciphers : yes
|
|
Built with PCRE version : 8.12 2011-01-15
|
|
PCRE library supports JIT : no (USE_PCRE_JIT not set)
|
|
Built with Lua version : Lua 5.3.1
|
|
Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
|
|
|
|
Available polling systems :
|
|
epoll : pref=300, test result OK
|
|
poll : pref=200, test result OK
|
|
select : pref=150, test result OK
|
|
Total: 3 (3 usable), will use epoll.
|
|
|
|
The relevant information that many non-developer users can verify here are :
|
|
- the version : 1.6-dev7-a088d3-4 above means the code is currently at commit
|
|
ID "a088d3" which is the 4th one after after official version "1.6-dev7".
|
|
Version 1.6-dev7 would show as "1.6-dev7-8c1ad7". What matters here is in
|
|
fact "1.6-dev7". This is the 7th development version of what will become
|
|
version 1.6 in the future. A development version not suitable for use in
|
|
production (unless you know exactly what you are doing). A stable version
|
|
will show as a 3-numbers version, such as "1.5.14-16f863", indicating the
|
|
14th level of fix on top of version 1.5. This is a production-ready version.
|
|
|
|
- the release date : 2015/10/08. It is represented in the universal
|
|
year/month/day format. Here this means August 8th, 2015. Given that stable
|
|
releases are issued every few months (1-2 months at the beginning, sometimes
|
|
6 months once the product becomes very stable), if you're seeing an old date
|
|
here, it means you're probably affected by a number of bugs or security
|
|
issues that have since been fixed and that it might be worth checking on the
|
|
official site.
|
|
|
|
- build options : they are relevant to people who build their packages
|
|
themselves, they can explain why things are not behaving as expected. For
|
|
example the development version above was built for Linux 2.6.28 or later,
|
|
targeting a generic CPU (no CPU-specific optimizations), and lacks any
|
|
code optimization (-O0) so it will perform poorly in terms of performance.
|
|
|
|
- libraries versions : zlib version is reported as found in the library
|
|
itself. In general zlib is considered a very stable product and upgrades
|
|
are almost never needed. OpenSSL reports two versions, the version used at
|
|
build time and the one being used, as found on the system. These ones may
|
|
differ by the last letter but never by the numbers. The build date is also
|
|
reported because most OpenSSL bugs are security issues and need to be taken
|
|
seriously, so this library absolutely needs to be kept up to date. Seeing a
|
|
4-months old version here is highly suspicious and indeed an update was
|
|
missed. PCRE provides very fast regular expressions and is highly
|
|
recommended. Certain of its extensions such as JIT are not present in all
|
|
versions and still young so some people prefer not to build with them,
|
|
which is why the build status is reported as well. Regarding the Lua
|
|
scripting language, HAProxy expects version 5.3 which is very young since
|
|
it was released a little time before HAProxy 1.6. It is important to check
|
|
on the Lua web site if some fixes are proposed for this branch.
|
|
|
|
- Available polling systems will affect the process's scalability when
|
|
dealing with more than about one thousand of concurrent connections. These
|
|
ones are only available when the correct system was indicated in the TARGET
|
|
variable during the build. The "epoll" mechanism is highly recommended on
|
|
Linux, and the kqueue mechanism is highly recommended on BSD. Lacking them
|
|
will result in poll() or even select() being used, causing a high CPU usage
|
|
when dealing with a lot of connections.
|
|
|
|
|
|
4. Stopping and restarting HAProxy
|
|
----------------------------------
|
|
|
|
HAProxy supports a graceful and a hard stop. The hard stop is simple, when the
|
|
SIGTERM signal is sent to the haproxy process, it immediately quits and all
|
|
established connections are closed. The graceful stop is triggered when the
|
|
SIGUSR1 signal is sent to the haproxy process. It consists in only unbinding
|
|
from listening ports, but continue to process existing connections until they
|
|
close. Once the last connection is closed, the process leaves.
|
|
|
|
The hard stop method is used for the "stop" or "restart" actions of the service
|
|
management script. The graceful stop is used for the "reload" action which
|
|
tries to seamlessly reload a new configuration in a new process.
|
|
|
|
Both of these signals may be sent by the new haproxy process itself during a
|
|
reload or restart, so that they are sent at the latest possible moment and only
|
|
if absolutely required. This is what is performed by the "-st" (hard) and "-sf"
|
|
(graceful) options respectively.
|
|
|
|
In master-worker mode, it is not needed to start a new haproxy process in
|
|
order to reload the configuration. The master process reacts to the SIGUSR2
|
|
signal by reexecuting itself with the -sf parameter followed by the PIDs of
|
|
the workers. The master will then parse the configuration file and fork new
|
|
workers.
|
|
|
|
To understand better how these signals are used, it is important to understand
|
|
the whole restart mechanism.
|
|
|
|
First, an existing haproxy process is running. The administrator uses a system
|
|
specific command such as "/etc/init.d/haproxy reload" to indicate they want to
|
|
take the new configuration file into effect. What happens then is the following.
|
|
First, the service script (/etc/init.d/haproxy or equivalent) will verify that
|
|
the configuration file parses correctly using "haproxy -c". After that it will
|
|
try to start haproxy with this configuration file, using "-st" or "-sf".
|
|
|
|
Then HAProxy tries to bind to all listening ports. If some fatal errors happen
|
|
(eg: address not present on the system, permission denied), the process quits
|
|
with an error. If a socket binding fails because a port is already in use, then
|
|
the process will first send a SIGTTOU signal to all the pids specified in the
|
|
"-st" or "-sf" pid list. This is what is called the "pause" signal. It instructs
|
|
all existing haproxy processes to temporarily stop listening to their ports so
|
|
that the new process can try to bind again. During this time, the old process
|
|
continues to process existing connections. If the binding still fails (because
|
|
for example a port is shared with another daemon), then the new process sends a
|
|
SIGTTIN signal to the old processes to instruct them to resume operations just
|
|
as if nothing happened. The old processes will then restart listening to the
|
|
ports and continue to accept connections. Note that this mechanism is system
|
|
dependent and some operating systems may not support it in multi-process mode.
|
|
|
|
If the new process manages to bind correctly to all ports, then it sends either
|
|
the SIGTERM (hard stop in case of "-st") or the SIGUSR1 (graceful stop in case
|
|
of "-sf") to all processes to notify them that it is now in charge of operations
|
|
and that the old processes will have to leave, either immediately or once they
|
|
have finished their job.
|
|
|
|
It is important to note that during this timeframe, there are two small windows
|
|
of a few milliseconds each where it is possible that a few connection failures
|
|
will be noticed during high loads. Typically observed failure rates are around
|
|
1 failure during a reload operation every 10000 new connections per second,
|
|
which means that a heavily loaded site running at 30000 new connections per
|
|
second may see about 3 failed connection upon every reload. The two situations
|
|
where this happens are :
|
|
|
|
- if the new process fails to bind due to the presence of the old process,
|
|
it will first have to go through the SIGTTOU+SIGTTIN sequence, which
|
|
typically lasts about one millisecond for a few tens of frontends, and
|
|
during which some ports will not be bound to the old process and not yet
|
|
bound to the new one. HAProxy works around this on systems that support the
|
|
SO_REUSEPORT socket options, as it allows the new process to bind without
|
|
first asking the old one to unbind. Most BSD systems have been supporting
|
|
this almost forever. Linux has been supporting this in version 2.0 and
|
|
dropped it around 2.2, but some patches were floating around by then. It
|
|
was reintroduced in kernel 3.9, so if you are observing a connection
|
|
failure rate above the one mentioned above, please ensure that your kernel
|
|
is 3.9 or newer, or that relevant patches were backported to your kernel
|
|
(less likely).
|
|
|
|
- when the old processes close the listening ports, the kernel may not always
|
|
redistribute any pending connection that was remaining in the socket's
|
|
backlog. Under high loads, a SYN packet may happen just before the socket
|
|
is closed, and will lead to an RST packet being sent to the client. In some
|
|
critical environments where even one drop is not acceptable, these ones are
|
|
sometimes dealt with using firewall rules to block SYN packets during the
|
|
reload, forcing the client to retransmit. This is totally system-dependent,
|
|
as some systems might be able to visit other listening queues and avoid
|
|
this RST. A second case concerns the ACK from the client on a local socket
|
|
that was in SYN_RECV state just before the close. This ACK will lead to an
|
|
RST packet while the haproxy process is still not aware of it. This one is
|
|
harder to get rid of, though the firewall filtering rules mentioned above
|
|
will work well if applied one second or so before restarting the process.
|
|
|
|
For the vast majority of users, such drops will never ever happen since they
|
|
don't have enough load to trigger the race conditions. And for most high traffic
|
|
users, the failure rate is still fairly within the noise margin provided that at
|
|
least SO_REUSEPORT is properly supported on their systems.
|
|
|
|
|
|
5. File-descriptor limitations
|
|
------------------------------
|
|
|
|
In order to ensure that all incoming connections will successfully be served,
|
|
HAProxy computes at load time the total number of file descriptors that will be
|
|
needed during the process's life. A regular Unix process is generally granted
|
|
1024 file descriptors by default, and a privileged process can raise this limit
|
|
itself. This is one reason for starting HAProxy as root and letting it adjust
|
|
the limit. The default limit of 1024 file descriptors roughly allow about 500
|
|
concurrent connections to be processed. The computation is based on the global
|
|
maxconn parameter which limits the total number of connections per process, the
|
|
number of listeners, the number of servers which have a health check enabled,
|
|
the agent checks, the peers, the loggers and possibly a few other technical
|
|
requirements. A simple rough estimate of this number consists in simply
|
|
doubling the maxconn value and adding a few tens to get the approximate number
|
|
of file descriptors needed.
|
|
|
|
Originally HAProxy did not know how to compute this value, and it was necessary
|
|
to pass the value using the "ulimit-n" setting in the global section. This
|
|
explains why even today a lot of configurations are seen with this setting
|
|
present. Unfortunately it was often miscalculated resulting in connection
|
|
failures when approaching maxconn instead of throttling incoming connection
|
|
while waiting for the needed resources. For this reason it is important to
|
|
remove any vestigial "ulimit-n" setting that can remain from very old versions.
|
|
|
|
Raising the number of file descriptors to accept even moderate loads is
|
|
mandatory but comes with some OS-specific adjustments. First, the select()
|
|
polling system is limited to 1024 file descriptors. In fact on Linux it used
|
|
to be capable of handling more but since certain OS ship with excessively
|
|
restrictive SELinux policies forbidding the use of select() with more than
|
|
1024 file descriptors, HAProxy now refuses to start in this case in order to
|
|
avoid any issue at run time. On all supported operating systems, poll() is
|
|
available and will not suffer from this limitation. It is automatically picked
|
|
so there is nothing to do to get a working configuration. But poll's becomes
|
|
very slow when the number of file descriptors increases. While HAProxy does its
|
|
best to limit this performance impact (eg: via the use of the internal file
|
|
descriptor cache and batched processing), a good rule of thumb is that using
|
|
poll() with more than a thousand concurrent connections will use a lot of CPU.
|
|
|
|
For Linux systems base on kernels 2.6 and above, the epoll() system call will
|
|
be used. It's a much more scalable mechanism relying on callbacks in the kernel
|
|
that guarantee a constant wake up time regardless of the number of registered
|
|
monitored file descriptors. It is automatically used where detected, provided
|
|
that HAProxy had been built for one of the Linux flavors. Its presence and
|
|
support can be verified using "haproxy -vv".
|
|
|
|
For BSD systems which support it, kqueue() is available as an alternative. It
|
|
is much faster than poll() and even slightly faster than epoll() thanks to its
|
|
batched handling of changes. At least FreeBSD and OpenBSD support it. Just like
|
|
with Linux's epoll(), its support and availability are reported in the output
|
|
of "haproxy -vv".
|
|
|
|
Having a good poller is one thing, but it is mandatory that the process can
|
|
reach the limits. When HAProxy starts, it immediately sets the new process's
|
|
file descriptor limits and verifies if it succeeds. In case of failure, it
|
|
reports it before forking so that the administrator can see the problem. As
|
|
long as the process is started by as root, there should be no reason for this
|
|
setting to fail. However, it can fail if the process is started by an
|
|
unprivileged user. If there is a compelling reason for *not* starting haproxy
|
|
as root (eg: started by end users, or by a per-application account), then the
|
|
file descriptor limit can be raised by the system administrator for this
|
|
specific user. The effectiveness of the setting can be verified by issuing
|
|
"ulimit -n" from the user's command line. It should reflect the new limit.
|
|
|
|
Warning: when an unprivileged user's limits are changed in this user's account,
|
|
it is fairly common that these values are only considered when the user logs in
|
|
and not at all in some scripts run at system boot time nor in crontabs. This is
|
|
totally dependent on the operating system, keep in mind to check "ulimit -n"
|
|
before starting haproxy when running this way. The general advice is never to
|
|
start haproxy as an unprivileged user for production purposes. Another good
|
|
reason is that it prevents haproxy from enabling some security protections.
|
|
|
|
Once it is certain that the system will allow the haproxy process to use the
|
|
requested number of file descriptors, two new system-specific limits may be
|
|
encountered. The first one is the system-wide file descriptor limit, which is
|
|
the total number of file descriptors opened on the system, covering all
|
|
processes. When this limit is reached, accept() or socket() will typically
|
|
return ENFILE. The second one is the per-process hard limit on the number of
|
|
file descriptors, it prevents setrlimit() from being set higher. Both are very
|
|
dependent on the operating system. On Linux, the system limit is set at boot
|
|
based on the amount of memory. It can be changed with the "fs.file-max" sysctl.
|
|
And the per-process hard limit is set to 1048576 by default, but it can be
|
|
changed using the "fs.nr_open" sysctl.
|
|
|
|
File descriptor limitations may be observed on a running process when they are
|
|
set too low. The strace utility will report that accept() and socket() return
|
|
"-1 EMFILE" when the process's limits have been reached. In this case, simply
|
|
raising the "ulimit-n" value (or removing it) will solve the problem. If these
|
|
system calls return "-1 ENFILE" then it means that the kernel's limits have
|
|
been reached and that something must be done on a system-wide parameter. These
|
|
trouble must absolutely be addressed, as they result in high CPU usage (when
|
|
accept() fails) and failed connections that are generally visible to the user.
|
|
One solution also consists in lowering the global maxconn value to enforce
|
|
serialization, and possibly to disable HTTP keep-alive to force connections
|
|
to be released and reused faster.
|
|
|
|
|
|
6. Memory management
|
|
--------------------
|
|
|
|
HAProxy uses a simple and fast pool-based memory management. Since it relies on
|
|
a small number of different object types, it's much more efficient to pick new
|
|
objects from a pool which already contains objects of the appropriate size than
|
|
to call malloc() for each different size. The pools are organized as a stack or
|
|
LIFO, so that newly allocated objects are taken from recently released objects
|
|
still hot in the CPU caches. Pools of similar sizes are merged together, in
|
|
order to limit memory fragmentation.
|
|
|
|
By default, since the focus is set on performance, each released object is put
|
|
back into the pool it came from, and allocated objects are never freed since
|
|
they are expected to be reused very soon.
|
|
|
|
On the CLI, it is possible to check how memory is being used in pools thanks to
|
|
the "show pools" command :
|
|
|
|
> show pools
|
|
Dumping pools usage. Use SIGQUIT to flush them.
|
|
- Pool cache_st (16 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccc40=03 [SHARED]
|
|
- Pool pipe (32 bytes) : 5 allocated (160 bytes), 5 used, 0 failures, 2 users, @0x9ccac0=00 [SHARED]
|
|
- Pool comp_state (48 bytes) : 3 allocated (144 bytes), 3 used, 0 failures, 5 users, @0x9cccc0=04 [SHARED]
|
|
- Pool filter (64 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 3 users, @0x9ccbc0=02 [SHARED]
|
|
- Pool vars (80 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccb40=01 [SHARED]
|
|
- Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9cd240=15 [SHARED]
|
|
- Pool task (144 bytes) : 55 allocated (7920 bytes), 55 used, 0 failures, 1 users, @0x9cd040=11 [SHARED]
|
|
- Pool session (160 bytes) : 1 allocated (160 bytes), 1 used, 0 failures, 1 users, @0x9cd140=13 [SHARED]
|
|
- Pool h2s (208 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccec0=08 [SHARED]
|
|
- Pool h2c (288 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cce40=07 [SHARED]
|
|
- Pool spoe_ctx (304 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 2 users, @0x9ccf40=09 [SHARED]
|
|
- Pool connection (400 bytes) : 2 allocated (800 bytes), 2 used, 0 failures, 1 users, @0x9cd1c0=14 [SHARED]
|
|
- Pool hdr_idx (416 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd340=17 [SHARED]
|
|
- Pool dns_resolut (480 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccdc0=06 [SHARED]
|
|
- Pool dns_answer_ (576 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9ccd40=05 [SHARED]
|
|
- Pool stream (960 bytes) : 1 allocated (960 bytes), 1 used, 0 failures, 1 users, @0x9cd0c0=12 [SHARED]
|
|
- Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 0 failures, 1 users, @0x9cd2c0=16 [SHARED]
|
|
- Pool buffer (8030 bytes) : 3 allocated (24090 bytes), 2 used, 0 failures, 1 users, @0x9cd3c0=18 [SHARED]
|
|
- Pool trash (8062 bytes) : 1 allocated (8062 bytes), 1 used, 0 failures, 1 users, @0x9cd440=19
|
|
Total: 19 pools, 42296 bytes allocated, 34266 used.
|
|
|
|
The pool name is only indicative, it's the name of the first object type using
|
|
this pool. The size in parenthesis is the object size for objects in this pool.
|
|
Object sizes are always rounded up to the closest multiple of 16 bytes. The
|
|
number of objects currently allocated and the equivalent number of bytes is
|
|
reported so that it is easy to know which pool is responsible for the highest
|
|
memory usage. The number of objects currently in use is reported as well in the
|
|
"used" field. The difference between "allocated" and "used" corresponds to the
|
|
objects that have been freed and are available for immediate use. The address
|
|
at the end of the line is the pool's address, and the following number is the
|
|
pool index when it exists, or is reported as -1 if no index was assigned.
|
|
|
|
It is possible to limit the amount of memory allocated per process using the
|
|
"-m" command line option, followed by a number of megabytes. It covers all of
|
|
the process's addressable space, so that includes memory used by some libraries
|
|
as well as the stack, but it is a reliable limit when building a resource
|
|
constrained system. It works the same way as "ulimit -v" on systems which have
|
|
it, or "ulimit -d" for the other ones.
|
|
|
|
If a memory allocation fails due to the memory limit being reached or because
|
|
the system doesn't have any enough memory, then haproxy will first start to
|
|
free all available objects from all pools before attempting to allocate memory
|
|
again. This mechanism of releasing unused memory can be triggered by sending
|
|
the signal SIGQUIT to the haproxy process. When doing so, the pools state prior
|
|
to the flush will also be reported to stderr when the process runs in
|
|
foreground.
|
|
|
|
During a reload operation, the process switched to the graceful stop state also
|
|
automatically performs some flushes after releasing any connection so that all
|
|
possible memory is released to save it for the new process.
|
|
|
|
|
|
7. CPU usage
|
|
------------
|
|
|
|
HAProxy normally spends most of its time in the system and a smaller part in
|
|
userland. A finely tuned 3.5 GHz CPU can sustain a rate about 80000 end-to-end
|
|
connection setups and closes per second at 100% CPU on a single core. When one
|
|
core is saturated, typical figures are :
|
|
- 95% system, 5% user for long TCP connections or large HTTP objects
|
|
- 85% system and 15% user for short TCP connections or small HTTP objects in
|
|
close mode
|
|
- 70% system and 30% user for small HTTP objects in keep-alive mode
|
|
|
|
The amount of rules processing and regular expressions will increase the user
|
|
land part. The presence of firewall rules, connection tracking, complex routing
|
|
tables in the system will instead increase the system part.
|
|
|
|
On most systems, the CPU time observed during network transfers can be cut in 4
|
|
parts :
|
|
- the interrupt part, which concerns all the processing performed upon I/O
|
|
receipt, before the target process is even known. Typically Rx packets are
|
|
accounted for in interrupt. On some systems such as Linux where interrupt
|
|
processing may be deferred to a dedicated thread, it can appear as softirq,
|
|
and the thread is called ksoftirqd/0 (for CPU 0). The CPU taking care of
|
|
this load is generally defined by the hardware settings, though in the case
|
|
of softirq it is often possible to remap the processing to another CPU.
|
|
This interrupt part will often be perceived as parasitic since it's not
|
|
associated with any process, but it actually is some processing being done
|
|
to prepare the work for the process.
|
|
|
|
- the system part, which concerns all the processing done using kernel code
|
|
called from userland. System calls are accounted as system for example. All
|
|
synchronously delivered Tx packets will be accounted for as system time. If
|
|
some packets have to be deferred due to queues filling up, they may then be
|
|
processed in interrupt context later (eg: upon receipt of an ACK opening a
|
|
TCP window).
|
|
|
|
- the user part, which exclusively runs application code in userland. HAProxy
|
|
runs exclusively in this part, though it makes heavy use of system calls.
|
|
Rules processing, regular expressions, compression, encryption all add to
|
|
the user portion of CPU consumption.
|
|
|
|
- the idle part, which is what the CPU does when there is nothing to do. For
|
|
example HAProxy waits for an incoming connection, or waits for some data to
|
|
leave, meaning the system is waiting for an ACK from the client to push
|
|
these data.
|
|
|
|
In practice regarding HAProxy's activity, it is in general reasonably accurate
|
|
(but totally inexact) to consider that interrupt/softirq are caused by Rx
|
|
processing in kernel drivers, that user-land is caused by layer 7 processing
|
|
in HAProxy, and that system time is caused by network processing on the Tx
|
|
path.
|
|
|
|
Since HAProxy runs around an event loop, it waits for new events using poll()
|
|
(or any alternative) and processes all these events as fast as possible before
|
|
going back to poll() waiting for new events. It measures the time spent waiting
|
|
in poll() compared to the time spent doing processing events. The ratio of
|
|
polling time vs total time is called the "idle" time, it's the amount of time
|
|
spent waiting for something to happen. This ratio is reported in the stats page
|
|
on the "idle" line, or "Idle_pct" on the CLI. When it's close to 100%, it means
|
|
the load is extremely low. When it's close to 0%, it means that there is
|
|
constantly some activity. While it cannot be very accurate on an overloaded
|
|
system due to other processes possibly preempting the CPU from the haproxy
|
|
process, it still provides a good estimate about how HAProxy considers it is
|
|
working : if the load is low and the idle ratio is low as well, it may indicate
|
|
that HAProxy has a lot of work to do, possibly due to very expensive rules that
|
|
have to be processed. Conversely, if HAProxy indicates the idle is close to
|
|
100% while things are slow, it means that it cannot do anything to speed things
|
|
up because it is already waiting for incoming data to process. In the example
|
|
below, haproxy is completely idle :
|
|
|
|
$ echo "show info" | socat - /var/run/haproxy.sock | grep ^Idle
|
|
Idle_pct: 100
|
|
|
|
When the idle ratio starts to become very low, it is important to tune the
|
|
system and place processes and interrupts correctly to save the most possible
|
|
CPU resources for all tasks. If a firewall is present, it may be worth trying
|
|
to disable it or to tune it to ensure it is not responsible for a large part
|
|
of the performance limitation. It's worth noting that unloading a stateful
|
|
firewall generally reduces both the amount of interrupt/softirq and of system
|
|
usage since such firewalls act both on the Rx and the Tx paths. On Linux,
|
|
unloading the nf_conntrack and ip_conntrack modules will show whether there is
|
|
anything to gain. If so, then the module runs with default settings and you'll
|
|
have to figure how to tune it for better performance. In general this consists
|
|
in considerably increasing the hash table size. On FreeBSD, "pfctl -d" will
|
|
disable the "pf" firewall and its stateful engine at the same time.
|
|
|
|
If it is observed that a lot of time is spent in interrupt/softirq, it is
|
|
important to ensure that they don't run on the same CPU. Most systems tend to
|
|
pin the tasks on the CPU where they receive the network traffic because for
|
|
certain workloads it improves things. But with heavily network-bound workloads
|
|
it is the opposite as the haproxy process will have to fight against its kernel
|
|
counterpart. Pinning haproxy to one CPU core and the interrupts to another one,
|
|
all sharing the same L3 cache tends to sensibly increase network performance
|
|
because in practice the amount of work for haproxy and the network stack are
|
|
quite close, so they can almost fill an entire CPU each. On Linux this is done
|
|
using taskset (for haproxy) or using cpu-map (from the haproxy config), and the
|
|
interrupts are assigned under /proc/irq. Many network interfaces support
|
|
multiple queues and multiple interrupts. In general it helps to spread them
|
|
across a small number of CPU cores provided they all share the same L3 cache.
|
|
Please always stop irq_balance which always does the worst possible thing on
|
|
such workloads.
|
|
|
|
For CPU-bound workloads consisting in a lot of SSL traffic or a lot of
|
|
compression, it may be worth using multiple processes dedicated to certain
|
|
tasks, though there is no universal rule here and experimentation will have to
|
|
be performed.
|
|
|
|
In order to increase the CPU capacity, it is possible to make HAProxy run as
|
|
several processes, using the "nbproc" directive in the global section. There
|
|
are some limitations though :
|
|
- health checks are run per process, so the target servers will get as many
|
|
checks as there are running processes ;
|
|
- maxconn values and queues are per-process so the correct value must be set
|
|
to avoid overloading the servers ;
|
|
- outgoing connections should avoid using port ranges to avoid conflicts
|
|
- stick-tables are per process and are not shared between processes ;
|
|
- each peers section may only run on a single process at a time ;
|
|
- the CLI operations will only act on a single process at a time.
|
|
|
|
With this in mind, it appears that the easiest setup often consists in having
|
|
one first layer running on multiple processes and in charge for the heavy
|
|
processing, passing the traffic to a second layer running in a single process.
|
|
This mechanism is suited to SSL and compression which are the two CPU-heavy
|
|
features. Instances can easily be chained over UNIX sockets (which are cheaper
|
|
than TCP sockets and which do not waste ports), and the proxy protocol which is
|
|
useful to pass client information to the next stage. When doing so, it is
|
|
generally a good idea to bind all the single-process tasks to process number 1
|
|
and extra tasks to next processes, as this will make it easier to generate
|
|
similar configurations for different machines.
|
|
|
|
On Linux versions 3.9 and above, running HAProxy in multi-process mode is much
|
|
more efficient when each process uses a distinct listening socket on the same
|
|
IP:port ; this will make the kernel evenly distribute the load across all
|
|
processes instead of waking them all up. Please check the "process" option of
|
|
the "bind" keyword lines in the configuration manual for more information.
|
|
|
|
|
|
8. Logging
|
|
----------
|
|
|
|
For logging, HAProxy always relies on a syslog server since it does not perform
|
|
any file-system access. The standard way of using it is to send logs over UDP
|
|
to the log server (by default on port 514). Very commonly this is configured to
|
|
127.0.0.1 where the local syslog daemon is running, but it's also used over the
|
|
network to log to a central server. The central server provides additional
|
|
benefits especially in active-active scenarios where it is desirable to keep
|
|
the logs merged in arrival order. HAProxy may also make use of a UNIX socket to
|
|
send its logs to the local syslog daemon, but it is not recommended at all,
|
|
because if the syslog server is restarted while haproxy runs, the socket will
|
|
be replaced and new logs will be lost. Since HAProxy will be isolated inside a
|
|
chroot jail, it will not have the ability to reconnect to the new socket. It
|
|
has also been observed in field that the log buffers in use on UNIX sockets are
|
|
very small and lead to lost messages even at very light loads. But this can be
|
|
fine for testing however.
|
|
|
|
It is recommended to add the following directive to the "global" section to
|
|
make HAProxy log to the local daemon using facility "local0" :
|
|
|
|
log 127.0.0.1:514 local0
|
|
|
|
and then to add the following one to each "defaults" section or to each frontend
|
|
and backend section :
|
|
|
|
log global
|
|
|
|
This way, all logs will be centralized through the global definition of where
|
|
the log server is.
|
|
|
|
Some syslog daemons do not listen to UDP traffic by default, so depending on
|
|
the daemon being used, the syntax to enable this will vary :
|
|
|
|
- on sysklogd, you need to pass argument "-r" on the daemon's command line
|
|
so that it listens to a UDP socket for "remote" logs ; note that there is
|
|
no way to limit it to address 127.0.0.1 so it will also receive logs from
|
|
remote systems ;
|
|
|
|
- on rsyslogd, the following lines must be added to the configuration file :
|
|
|
|
$ModLoad imudp
|
|
$UDPServerAddress *
|
|
$UDPServerRun 514
|
|
|
|
- on syslog-ng, a new source can be created the following way, it then needs
|
|
to be added as a valid source in one of the "log" directives :
|
|
|
|
source s_udp {
|
|
udp(ip(127.0.0.1) port(514));
|
|
};
|
|
|
|
Please consult your syslog daemon's manual for more information. If no logs are
|
|
seen in the system's log files, please consider the following tests :
|
|
|
|
- restart haproxy. Each frontend and backend logs one line indicating it's
|
|
starting. If these logs are received, it means logs are working.
|
|
|
|
- run "strace -tt -s100 -etrace=sendmsg -p <haproxy's pid>" and perform some
|
|
activity that you expect to be logged. You should see the log messages
|
|
being sent using sendmsg() there. If they don't appear, restart using
|
|
strace on top of haproxy. If you still see no logs, it definitely means
|
|
that something is wrong in your configuration.
|
|
|
|
- run tcpdump to watch for port 514, for example on the loopback interface if
|
|
the traffic is being sent locally : "tcpdump -As0 -ni lo port 514". If the
|
|
packets are seen there, it's the proof they're sent then the syslogd daemon
|
|
needs to be troubleshooted.
|
|
|
|
While traffic logs are sent from the frontends (where the incoming connections
|
|
are accepted), backends also need to be able to send logs in order to report a
|
|
server state change consecutive to a health check. Please consult HAProxy's
|
|
configuration manual for more information regarding all possible log settings.
|
|
|
|
It is convenient to chose a facility that is not used by other daemons. HAProxy
|
|
examples often suggest "local0" for traffic logs and "local1" for admin logs
|
|
because they're never seen in field. A single facility would be enough as well.
|
|
Having separate logs is convenient for log analysis, but it's also important to
|
|
remember that logs may sometimes convey confidential information, and as such
|
|
they must not be mixed with other logs that may accidentally be handed out to
|
|
unauthorized people.
|
|
|
|
For in-field troubleshooting without impacting the server's capacity too much,
|
|
it is recommended to make use of the "halog" utility provided with HAProxy.
|
|
This is sort of a grep-like utility designed to process HAProxy log files at
|
|
a very fast data rate. Typical figures range between 1 and 2 GB of logs per
|
|
second. It is capable of extracting only certain logs (eg: search for some
|
|
classes of HTTP status codes, connection termination status, search by response
|
|
time ranges, look for errors only), count lines, limit the output to a number
|
|
of lines, and perform some more advanced statistics such as sorting servers
|
|
by response time or error counts, sorting URLs by time or count, sorting client
|
|
addresses by access count, and so on. It is pretty convenient to quickly spot
|
|
anomalies such as a bot looping on the site, and block them.
|
|
|
|
|
|
9. Statistics and monitoring
|
|
----------------------------
|
|
|
|
It is possible to query HAProxy about its status. The most commonly used
|
|
mechanism is the HTTP statistics page. This page also exposes an alternative
|
|
CSV output format for monitoring tools. The same format is provided on the
|
|
Unix socket.
|
|
|
|
Statistics are regroup in categories labelled as domains, corresponding to the
|
|
multiple components of HAProxy. There are two domains available: proxy and dns.
|
|
If not specified, the proxy domain is selected. Note that only the proxy
|
|
statistics are printed on the HTTP page.
|
|
|
|
9.1. CSV format
|
|
---------------
|
|
|
|
The statistics may be consulted either from the unix socket or from the HTTP
|
|
page. Both means provide a CSV format whose fields follow. The first line
|
|
begins with a sharp ('#') and has one word per comma-delimited field which
|
|
represents the title of the column. All other lines starting at the second one
|
|
use a classical CSV format using a comma as the delimiter, and the double quote
|
|
('"') as an optional text delimiter, but only if the enclosed text is ambiguous
|
|
(if it contains a quote or a comma). The double-quote character ('"') in the
|
|
text is doubled ('""'), which is the format that most tools recognize. Please
|
|
do not insert any column before these ones in order not to break tools which
|
|
use hard-coded column positions.
|
|
|
|
For proxy statistics, after each field name, the types which may have a value
|
|
for that field are specified in brackets. The types are L (Listeners), F
|
|
(Frontends), B (Backends), and S (Servers). There is a fixed set of static
|
|
fields that are always available in the same order. A column containing the
|
|
character '-' delimits the end of the static fields, after which presence or
|
|
order of the fields are not guaranteed.
|
|
|
|
Here is the list of static fields using the proxy statistics domain:
|
|
0. pxname [LFBS]: proxy name
|
|
1. svname [LFBS]: service name (FRONTEND for frontend, BACKEND for backend,
|
|
any name for server/listener)
|
|
2. qcur [..BS]: current queued requests. For the backend this reports the
|
|
number queued without a server assigned.
|
|
3. qmax [..BS]: max value of qcur
|
|
4. scur [LFBS]: current sessions
|
|
5. smax [LFBS]: max sessions
|
|
6. slim [LFBS]: configured session limit
|
|
7. stot [LFBS]: cumulative number of sessions
|
|
8. bin [LFBS]: bytes in
|
|
9. bout [LFBS]: bytes out
|
|
10. dreq [LFB.]: requests denied because of security concerns.
|
|
- For tcp this is because of a matched tcp-request content rule.
|
|
- For http this is because of a matched http-request or tarpit rule.
|
|
11. dresp [LFBS]: responses denied because of security concerns.
|
|
- For http this is because of a matched http-request rule, or
|
|
"option checkcache".
|
|
12. ereq [LF..]: request errors. Some of the possible causes are:
|
|
- early termination from the client, before the request has been sent.
|
|
- read error from the client
|
|
- client timeout
|
|
- client closed connection
|
|
- various bad requests from the client.
|
|
- request was tarpitted.
|
|
13. econ [..BS]: number of requests that encountered an error trying to
|
|
connect to a backend server. The backend stat is the sum of the stat
|
|
for all servers of that backend, plus any connection errors not
|
|
associated with a particular server (such as the backend having no
|
|
active servers).
|
|
14. eresp [..BS]: response errors. srv_abrt will be counted here also.
|
|
Some other errors are:
|
|
- write error on the client socket (won't be counted for the server stat)
|
|
- failure applying filters to the response.
|
|
15. wretr [..BS]: number of times a connection to a server was retried.
|
|
16. wredis [..BS]: number of times a request was redispatched to another
|
|
server. The server value counts the number of times that server was
|
|
switched away from.
|
|
17. status [LFBS]: status (UP/DOWN/NOLB/MAINT/MAINT(via)/MAINT(resolution)...)
|
|
18. weight [..BS]: total effective weight (backend), effective weight (server)
|
|
19. act [..BS]: number of active servers (backend), server is active (server)
|
|
20. bck [..BS]: number of backup servers (backend), server is backup (server)
|
|
21. chkfail [...S]: number of failed checks. (Only counts checks failed when
|
|
the server is up.)
|
|
22. chkdown [..BS]: number of UP->DOWN transitions. The backend counter counts
|
|
transitions to the whole backend being down, rather than the sum of the
|
|
counters for each server.
|
|
23. lastchg [..BS]: number of seconds since the last UP<->DOWN transition
|
|
24. downtime [..BS]: total downtime (in seconds). The value for the backend
|
|
is the downtime for the whole backend, not the sum of the server downtime.
|
|
25. qlimit [...S]: configured maxqueue for the server, or nothing in the
|
|
value is 0 (default, meaning no limit)
|
|
26. pid [LFBS]: process id (0 for first instance, 1 for second, ...)
|
|
27. iid [LFBS]: unique proxy id
|
|
28. sid [L..S]: server id (unique inside a proxy)
|
|
29. throttle [...S]: current throttle percentage for the server, when
|
|
slowstart is active, or no value if not in slowstart.
|
|
30. lbtot [..BS]: total number of times a server was selected, either for new
|
|
sessions, or when re-dispatching. The server counter is the number
|
|
of times that server was selected.
|
|
31. tracked [...S]: id of proxy/server if tracking is enabled.
|
|
32. type [LFBS]: (0=frontend, 1=backend, 2=server, 3=socket/listener)
|
|
33. rate [.FBS]: number of sessions per second over last elapsed second
|
|
34. rate_lim [.F..]: configured limit on new sessions per second
|
|
35. rate_max [.FBS]: max number of new sessions per second
|
|
36. check_status [...S]: status of last health check, one of:
|
|
UNK -> unknown
|
|
INI -> initializing
|
|
SOCKERR -> socket error
|
|
L4OK -> check passed on layer 4, no upper layers testing enabled
|
|
L4TOUT -> layer 1-4 timeout
|
|
L4CON -> layer 1-4 connection problem, for example
|
|
"Connection refused" (tcp rst) or "No route to host" (icmp)
|
|
L6OK -> check passed on layer 6
|
|
L6TOUT -> layer 6 (SSL) timeout
|
|
L6RSP -> layer 6 invalid response - protocol error
|
|
L7OK -> check passed on layer 7
|
|
L7OKC -> check conditionally passed on layer 7, for example 404 with
|
|
disable-on-404
|
|
L7TOUT -> layer 7 (HTTP/SMTP) timeout
|
|
L7RSP -> layer 7 invalid response - protocol error
|
|
L7STS -> layer 7 response error, for example HTTP 5xx
|
|
Notice: If a check is currently running, the last known status will be
|
|
reported, prefixed with "* ". e. g. "* L7OK".
|
|
37. check_code [...S]: layer5-7 code, if available
|
|
38. check_duration [...S]: time in ms took to finish last health check
|
|
39. hrsp_1xx [.FBS]: http responses with 1xx code
|
|
40. hrsp_2xx [.FBS]: http responses with 2xx code
|
|
41. hrsp_3xx [.FBS]: http responses with 3xx code
|
|
42. hrsp_4xx [.FBS]: http responses with 4xx code
|
|
43. hrsp_5xx [.FBS]: http responses with 5xx code
|
|
44. hrsp_other [.FBS]: http responses with other codes (protocol error)
|
|
45. hanafail [...S]: failed health checks details
|
|
46. req_rate [.F..]: HTTP requests per second over last elapsed second
|
|
47. req_rate_max [.F..]: max number of HTTP requests per second observed
|
|
48. req_tot [.FB.]: total number of HTTP requests received
|
|
49. cli_abrt [..BS]: number of data transfers aborted by the client
|
|
50. srv_abrt [..BS]: number of data transfers aborted by the server
|
|
(inc. in eresp)
|
|
51. comp_in [.FB.]: number of HTTP response bytes fed to the compressor
|
|
52. comp_out [.FB.]: number of HTTP response bytes emitted by the compressor
|
|
53. comp_byp [.FB.]: number of bytes that bypassed the HTTP compressor
|
|
(CPU/BW limit)
|
|
54. comp_rsp [.FB.]: number of HTTP responses that were compressed
|
|
55. lastsess [..BS]: number of seconds since last session assigned to
|
|
server/backend
|
|
56. last_chk [...S]: last health check contents or textual error
|
|
57. last_agt [...S]: last agent check contents or textual error
|
|
58. qtime [..BS]: the average queue time in ms over the 1024 last requests
|
|
59. ctime [..BS]: the average connect time in ms over the 1024 last requests
|
|
60. rtime [..BS]: the average response time in ms over the 1024 last requests
|
|
(0 for TCP)
|
|
61. ttime [..BS]: the average total session time in ms over the 1024 last
|
|
requests
|
|
62. agent_status [...S]: status of last agent check, one of:
|
|
UNK -> unknown
|
|
INI -> initializing
|
|
SOCKERR -> socket error
|
|
L4OK -> check passed on layer 4, no upper layers testing enabled
|
|
L4TOUT -> layer 1-4 timeout
|
|
L4CON -> layer 1-4 connection problem, for example
|
|
"Connection refused" (tcp rst) or "No route to host" (icmp)
|
|
L7OK -> agent reported "up"
|
|
L7STS -> agent reported "fail", "stop", or "down"
|
|
63. agent_code [...S]: numeric code reported by agent if any (unused for now)
|
|
64. agent_duration [...S]: time in ms taken to finish last check
|
|
65. check_desc [...S]: short human-readable description of check_status
|
|
66. agent_desc [...S]: short human-readable description of agent_status
|
|
67. check_rise [...S]: server's "rise" parameter used by checks
|
|
68. check_fall [...S]: server's "fall" parameter used by checks
|
|
69. check_health [...S]: server's health check value between 0 and rise+fall-1
|
|
70. agent_rise [...S]: agent's "rise" parameter, normally 1
|
|
71. agent_fall [...S]: agent's "fall" parameter, normally 1
|
|
72. agent_health [...S]: agent's health parameter, between 0 and rise+fall-1
|
|
73. addr [L..S]: address:port or "unix". IPv6 has brackets around the address.
|
|
74: cookie [..BS]: server's cookie value or backend's cookie name
|
|
75: mode [LFBS]: proxy mode (tcp, http, health, unknown)
|
|
76: algo [..B.]: load balancing algorithm
|
|
77: conn_rate [.F..]: number of connections over the last elapsed second
|
|
78: conn_rate_max [.F..]: highest known conn_rate
|
|
79: conn_tot [.F..]: cumulative number of connections
|
|
80: intercepted [.FB.]: cum. number of intercepted requests (monitor, stats)
|
|
81: dcon [LF..]: requests denied by "tcp-request connection" rules
|
|
82: dses [LF..]: requests denied by "tcp-request session" rules
|
|
83: wrew [LFBS]: cumulative number of failed header rewriting warnings
|
|
84: connect [..BS]: cumulative number of connection establishment attempts
|
|
85: reuse [..BS]: cumulative number of connection reuses
|
|
86: cache_lookups [.FB.]: cumulative number of cache lookups
|
|
87: cache_hits [.FB.]: cumulative number of cache hits
|
|
88: srv_icur [...S]: current number of idle connections available for reuse
|
|
89: src_ilim [...S]: limit on the number of available idle connections
|
|
90. qtime_max [..BS]: the maximum observed queue time in ms
|
|
91. ctime_max [..BS]: the maximum observed connect time in ms
|
|
92. rtime_max [..BS]: the maximum observed response time in ms (0 for TCP)
|
|
93. ttime_max [..BS]: the maximum observed total session time in ms
|
|
94. eint [LFBS]: cumulative number of internal errors
|
|
95. idle_conn_cur [...S]: current number of unsafe idle connections
|
|
96. safe_conn_cur [...S]: current number of safe idle connections
|
|
97. used_conn_cur [...S]: current number of connections in use
|
|
98. need_conn_est [...S]: estimated needed number of connections
|
|
99. uweight [..BS]: total user weight (backend), server user weight (server)
|
|
|
|
For all other statistics domains, the presence or the order of the fields are
|
|
not guaranteed. In this case, the header line should always be used to parse
|
|
the CSV data.
|
|
|
|
9.2. Typed output format
|
|
------------------------
|
|
|
|
Both "show info" and "show stat" support a mode where each output value comes
|
|
with its type and sufficient information to know how the value is supposed to
|
|
be aggregated between processes and how it evolves.
|
|
|
|
In all cases, the output consists in having a single value per line with all
|
|
the information split into fields delimited by colons (':').
|
|
|
|
The first column designates the object or metric being dumped. Its format is
|
|
specific to the command producing this output and will not be described in this
|
|
section. Usually it will consist in a series of identifiers and field names.
|
|
|
|
The second column contains 3 characters respectively indicating the origin, the
|
|
nature and the scope of the value being reported. The first character (the
|
|
origin) indicates where the value was extracted from. Possible characters are :
|
|
|
|
M The value is a metric. It is valid at one instant any may change depending
|
|
on its nature .
|
|
|
|
S The value is a status. It represents a discrete value which by definition
|
|
cannot be aggregated. It may be the status of a server ("UP" or "DOWN"),
|
|
the PID of the process, etc.
|
|
|
|
K The value is a sorting key. It represents an identifier which may be used
|
|
to group some values together because it is unique among its class. All
|
|
internal identifiers are keys. Some names can be listed as keys if they
|
|
are unique (eg: a frontend name is unique). In general keys come from the
|
|
configuration, even though some of them may automatically be assigned. For
|
|
most purposes keys may be considered as equivalent to configuration.
|
|
|
|
C The value comes from the configuration. Certain configuration values make
|
|
sense on the output, for example a concurrent connection limit or a cookie
|
|
name. By definition these values are the same in all processes started
|
|
from the same configuration file.
|
|
|
|
P The value comes from the product itself. There are very few such values,
|
|
most common use is to report the product name, version and release date.
|
|
These elements are also the same between all processes.
|
|
|
|
The second character (the nature) indicates the nature of the information
|
|
carried by the field in order to let an aggregator decide on what operation to
|
|
use to aggregate multiple values. Possible characters are :
|
|
|
|
A The value represents an age since a last event. This is a bit different
|
|
from the duration in that an age is automatically computed based on the
|
|
current date. A typical example is how long ago did the last session
|
|
happen on a server. Ages are generally aggregated by taking the minimum
|
|
value and do not need to be stored.
|
|
|
|
a The value represents an already averaged value. The average response times
|
|
and server weights are of this nature. Averages can typically be averaged
|
|
between processes.
|
|
|
|
C The value represents a cumulative counter. Such measures perpetually
|
|
increase until they wrap around. Some monitoring protocols need to tell
|
|
the difference between a counter and a gauge to report a different type.
|
|
In general counters may simply be summed since they represent events or
|
|
volumes. Examples of metrics of this nature are connection counts or byte
|
|
counts.
|
|
|
|
D The value represents a duration for a status. There are a few usages of
|
|
this, most of them include the time taken by the last health check and
|
|
the time a server has spent down. Durations are generally not summed,
|
|
most of the time the maximum will be retained to compute an SLA.
|
|
|
|
G The value represents a gauge. It's a measure at one instant. The memory
|
|
usage or the current number of active connections are of this nature.
|
|
Metrics of this type are typically summed during aggregation.
|
|
|
|
L The value represents a limit (generally a configured one). By nature,
|
|
limits are harder to aggregate since they are specific to the point where
|
|
they were retrieved. In certain situations they may be summed or be kept
|
|
separate.
|
|
|
|
M The value represents a maximum. In general it will apply to a gauge and
|
|
keep the highest known value. An example of such a metric could be the
|
|
maximum amount of concurrent connections that was encountered in the
|
|
product's life time. To correctly aggregate maxima, you are supposed to
|
|
output a range going from the maximum of all maxima and the sum of all
|
|
of them. There is indeed no way to know if they were encountered
|
|
simultaneously or not.
|
|
|
|
m The value represents a minimum. In general it will apply to a gauge and
|
|
keep the lowest known value. An example of such a metric could be the
|
|
minimum amount of free memory pools that was encountered in the product's
|
|
life time. To correctly aggregate minima, you are supposed to output a
|
|
range going from the minimum of all minima and the sum of all of them.
|
|
There is indeed no way to know if they were encountered simultaneously
|
|
or not.
|
|
|
|
N The value represents a name, so it is a string. It is used to report
|
|
proxy names, server names and cookie names. Names have configuration or
|
|
keys as their origin and are supposed to be the same among all processes.
|
|
|
|
O The value represents a free text output. Outputs from various commands,
|
|
returns from health checks, node descriptions are of such nature.
|
|
|
|
R The value represents an event rate. It's a measure at one instant. It is
|
|
quite similar to a gauge except that the recipient knows that this measure
|
|
moves slowly and may decide not to keep all values. An example of such a
|
|
metric is the measured amount of connections per second. Metrics of this
|
|
type are typically summed during aggregation.
|
|
|
|
T The value represents a date or time. A field emitting the current date
|
|
would be of this type. The method to aggregate such information is left
|
|
as an implementation choice. For now no field uses this type.
|
|
|
|
The third character (the scope) indicates what extent the value reflects. Some
|
|
elements may be per process while others may be per configuration or per system.
|
|
The distinction is important to know whether or not a single value should be
|
|
kept during aggregation or if values have to be aggregated. The following
|
|
characters are currently supported :
|
|
|
|
C The value is valid for a whole cluster of nodes, which is the set of nodes
|
|
communicating over the peers protocol. An example could be the amount of
|
|
entries present in a stick table that is replicated with other peers. At
|
|
the moment no metric use this scope.
|
|
|
|
P The value is valid only for the process reporting it. Most metrics use
|
|
this scope.
|
|
|
|
S The value is valid for the whole service, which is the set of processes
|
|
started together from the same configuration file. All metrics originating
|
|
from the configuration use this scope. Some other metrics may use it as
|
|
well for some shared resources (eg: shared SSL cache statistics).
|
|
|
|
s The value is valid for the whole system, such as the system's hostname,
|
|
current date or resource usage. At the moment this scope is not used by
|
|
any metric.
|
|
|
|
Consumers of these information will generally have enough of these 3 characters
|
|
to determine how to accurately report aggregated information across multiple
|
|
processes.
|
|
|
|
After this column, the third column indicates the type of the field, among "s32"
|
|
(signed 32-bit integer), "s64" (signed 64-bit integer), "u32" (unsigned 32-bit
|
|
integer), "u64" (unsigned 64-bit integer), "str" (string). It is important to
|
|
know the type before parsing the value in order to properly read it. For example
|
|
a string containing only digits is still a string an not an integer (eg: an
|
|
error code extracted by a check).
|
|
|
|
Then the fourth column is the value itself, encoded according to its type.
|
|
Strings are dumped as-is immediately after the colon without any leading space.
|
|
If a string contains a colon, it will appear normally. This means that the
|
|
output should not be exclusively split around colons or some check outputs
|
|
or server addresses might be truncated.
|
|
|
|
|
|
9.3. Unix Socket commands
|
|
-------------------------
|
|
|
|
The stats socket is not enabled by default. In order to enable it, it is
|
|
necessary to add one line in the global section of the haproxy configuration.
|
|
A second line is recommended to set a larger timeout, always appreciated when
|
|
issuing commands by hand :
|
|
|
|
global
|
|
stats socket /var/run/haproxy.sock mode 600 level admin
|
|
stats timeout 2m
|
|
|
|
It is also possible to add multiple instances of the stats socket by repeating
|
|
the line, and make them listen to a TCP port instead of a UNIX socket. This is
|
|
never done by default because this is dangerous, but can be handy in some
|
|
situations :
|
|
|
|
global
|
|
stats socket /var/run/haproxy.sock mode 600 level admin
|
|
stats socket ipv4@192.168.0.1:9999 level admin
|
|
stats timeout 2m
|
|
|
|
To access the socket, an external utility such as "socat" is required. Socat is
|
|
a swiss-army knife to connect anything to anything. We use it to connect
|
|
terminals to the socket, or a couple of stdin/stdout pipes to it for scripts.
|
|
The two main syntaxes we'll use are the following :
|
|
|
|
# socat /var/run/haproxy.sock stdio
|
|
# socat /var/run/haproxy.sock readline
|
|
|
|
The first one is used with scripts. It is possible to send the output of a
|
|
script to haproxy, and pass haproxy's output to another script. That's useful
|
|
for retrieving counters or attack traces for example.
|
|
|
|
The second one is only useful for issuing commands by hand. It has the benefit
|
|
that the terminal is handled by the readline library which supports line
|
|
editing and history, which is very convenient when issuing repeated commands
|
|
(eg: watch a counter).
|
|
|
|
The socket supports two operation modes :
|
|
- interactive
|
|
- non-interactive
|
|
|
|
The non-interactive mode is the default when socat connects to the socket. In
|
|
this mode, a single line may be sent. It is processed as a whole, responses are
|
|
sent back, and the connection closes after the end of the response. This is the
|
|
mode that scripts and monitoring tools use. It is possible to send multiple
|
|
commands in this mode, they need to be delimited by a semi-colon (';'). For
|
|
example :
|
|
|
|
# echo "show info;show stat;show table" | socat /var/run/haproxy stdio
|
|
|
|
If a command needs to use a semi-colon or a backslash (eg: in a value), it
|
|
must be preceded by a backslash ('\').
|
|
|
|
The interactive mode displays a prompt ('>') and waits for commands to be
|
|
entered on the line, then processes them, and displays the prompt again to wait
|
|
for a new command. This mode is entered via the "prompt" command which must be
|
|
sent on the first line in non-interactive mode. The mode is a flip switch, if
|
|
"prompt" is sent in interactive mode, it is disabled and the connection closes
|
|
after processing the last command of the same line.
|
|
|
|
For this reason, when debugging by hand, it's quite common to start with the
|
|
"prompt" command :
|
|
|
|
# socat /var/run/haproxy readline
|
|
prompt
|
|
> show info
|
|
...
|
|
>
|
|
|
|
Since multiple commands may be issued at once, haproxy uses the empty line as a
|
|
delimiter to mark an end of output for each command, and takes care of ensuring
|
|
that no command can emit an empty line on output. A script can thus easily
|
|
parse the output even when multiple commands were pipelined on a single line.
|
|
|
|
Some commands may take an optional payload. To add one to a command, the first
|
|
line needs to end with the "<<\n" pattern. The next lines will be treated as
|
|
the payload and can contain as many lines as needed. To validate a command with
|
|
a payload, it needs to end with an empty line.
|
|
|
|
Limitations do exist: the length of the whole buffer passed to the CLI must
|
|
not be greater than tune.bfsize and the pattern "<<" must not be glued to the
|
|
last word of the line.
|
|
|
|
When entering a paylod while in interactive mode, the prompt will change from
|
|
"> " to "+ ".
|
|
|
|
It is important to understand that when multiple haproxy processes are started
|
|
on the same sockets, any process may pick up the request and will output its
|
|
own stats.
|
|
|
|
The list of commands currently supported on the stats socket is provided below.
|
|
If an unknown command is sent, haproxy displays the usage message which reminds
|
|
all supported commands. Some commands support a more complex syntax, generally
|
|
it will explain what part of the command is invalid when this happens.
|
|
|
|
Some commands require a higher level of privilege to work. If you do not have
|
|
enough privilege, you will get an error "Permission denied". Please check
|
|
the "level" option of the "bind" keyword lines in the configuration manual
|
|
for more information.
|
|
|
|
abort ssl ca-file <cafile>
|
|
Abort and destroy a temporary CA file update transaction.
|
|
|
|
See also "set ssl ca-file" and "commit ssl ca-file".
|
|
|
|
abort ssl cert <filename>
|
|
Abort and destroy a temporary SSL certificate update transaction.
|
|
|
|
See also "set ssl cert" and "commit ssl cert".
|
|
|
|
abort ssl crl-file <crlfile>
|
|
Abort and destroy a temporary CRL file update transaction.
|
|
|
|
See also "set ssl crl-file" and "commit ssl crl-file".
|
|
|
|
add acl [@<ver>] <acl> <pattern>
|
|
Add an entry into the acl <acl>. <acl> is the #<id> or the <file> returned by
|
|
"show acl". This command does not verify if the entry already exists. Entries
|
|
are added to the current version of the ACL, unless a specific version is
|
|
specified with "@<ver>". This version number must have preliminary been
|
|
allocated by "prepare acl", and it will be comprised between the versions
|
|
reported in "curr_ver" and "next_ver" on the output of "show acl". Entries
|
|
added with a specific version number will not match until a "commit acl"
|
|
operation is performed on them. They may however be consulted using the
|
|
"show acl @<ver>" command, and cleared using a "clear acl @<ver>" command.
|
|
This command cannot be used if the reference <acl> is a file also used with
|
|
a map. In this case, the "add map" command must be used instead.
|
|
|
|
add map [@<ver>] <map> <key> <value>
|
|
add map [@<ver>] <map> <payload>
|
|
Add an entry into the map <map> to associate the value <value> to the key
|
|
<key>. This command does not verify if the entry already exists. It is
|
|
mainly used to fill a map after a "clear" or "prepare" operation. Entries
|
|
are added to the current version of the ACL, unless a specific version is
|
|
specified with "@<ver>". This version number must have preliminary been
|
|
allocated by "prepare acl", and it will be comprised between the versions
|
|
reported in "curr_ver" and "next_ver" on the output of "show acl". Entries
|
|
added with a specific version number will not match until a "commit map"
|
|
operation is performed on them. They may however be consulted using the
|
|
"show map @<ver>" command, and cleared using a "clear acl @<ver>" command.
|
|
If the designated map is also used as an ACL, the ACL will only match the
|
|
<key> part and will ignore the <value> part. Using the payload syntax it is
|
|
possible to add multiple key/value pairs by entering them on separate lines.
|
|
On each new line, the first word is the key and the rest of the line is
|
|
considered to be the value which can even contains spaces.
|
|
|
|
Example:
|
|
|
|
# socat /tmp/sock1 -
|
|
prompt
|
|
|
|
> add map #-1 <<
|
|
+ key1 value1
|
|
+ key2 value2 with spaces
|
|
+ key3 value3 also with spaces
|
|
+ key4 value4
|
|
|
|
>
|
|
|
|
add server <backend>/<server> [args]*
|
|
Instantiate a new server attached to the backend <backend>. Only supported on
|
|
a CLI connection running in experimental mode (see "experimental-mode on").
|
|
This method is still in development and may change in the future.
|
|
|
|
The <server> name must not be already used in the backend. A special
|
|
restriction is put on the backend which must used a dynamic load-balancing
|
|
algorithm. A subset of keywords from the server config file statement can be
|
|
used to configure the server behavior. Also note that no settings will be
|
|
reused from an hypothetical 'default-server' statement in the same backend.
|
|
|
|
Currently a dynamic server is statically initialized with the "none"
|
|
init-addr method. This means that no resolution will be undertaken if a FQDN
|
|
is specified as an address, even if the server creation will be validated.
|
|
|
|
To support the reload operations, it is expected that the server created via
|
|
the CLI is also manually inserted in the relevant haproxy configuration file.
|
|
A dynamic server not present in the configuration won't be restored after a
|
|
reload operation.
|
|
|
|
A dynamic server may use the "track" keyword to follow the check status of
|
|
another server from the configuration. However, it is not possible to track
|
|
another dynamic server. This is to ensure that the tracking chain is kept
|
|
consistent even in the case of dynamic servers deletion.
|
|
|
|
Use the "check" keyword to enable health-check support. Note that the
|
|
health-check is disabled by default and must be enabled independently from
|
|
the server using the "enable health" command. For agent checks, use the
|
|
"agent-check" keyword and the "enable agent" command. Note that in this case
|
|
the server may be activated via the agent depending on the status reported,
|
|
without an explicit "enable server" command. This also means that extra care
|
|
is required when removing a dynamic server with agent check. The agent should
|
|
be first deactivated via "disable agent" to be able to put the server in the
|
|
required maintenance mode before removal.
|
|
|
|
It may be possible to reach the fd limit when using a large number of dynamic
|
|
servers. Please refer to the "u-limit" global keyword documentation in this
|
|
case.
|
|
|
|
Here is the list of the currently supported keywords :
|
|
|
|
- agent-addr
|
|
- agent-check
|
|
- agent-inter
|
|
- agent-port
|
|
- agent-send
|
|
- allow-0rtt
|
|
- alpn
|
|
- addr
|
|
- backup
|
|
- ca-file
|
|
- check
|
|
- check-alpn
|
|
- check-proto
|
|
- check-send-proxy
|
|
- check-sni
|
|
- check-ssl
|
|
- check-via-socks4
|
|
- ciphers
|
|
- ciphersuites
|
|
- crl-file
|
|
- crt
|
|
- disabled
|
|
- downinter
|
|
- enabled
|
|
- error-limit
|
|
- fall
|
|
- fastinter
|
|
- force-sslv3/tlsv10/tlsv11/tlsv12/tlsv13
|
|
- id
|
|
- inter
|
|
- maxconn
|
|
- maxqueue
|
|
- minconn
|
|
- no-ssl-reuse
|
|
- no-sslv3/tlsv10/tlsv11/tlsv12/tlsv13
|
|
- no-tls-tickets
|
|
- npn
|
|
- observe
|
|
- on-error
|
|
- on-marked-down
|
|
- on-marked-up
|
|
- pool-low-conn
|
|
- pool-max-conn
|
|
- pool-purge-delay
|
|
- port
|
|
- proto
|
|
- proxy-v2-options
|
|
- rise
|
|
- send-proxy
|
|
- send-proxy-v2
|
|
- send-proxy-v2-ssl
|
|
- send-proxy-v2-ssl-cn
|
|
- slowstart
|
|
- sni
|
|
- source
|
|
- ssl
|
|
- ssl-max-ver
|
|
- ssl-min-ver
|
|
- tfo
|
|
- tls-tickets
|
|
- track
|
|
- usesrc
|
|
- verify
|
|
- verifyhost
|
|
- weight
|
|
|
|
Their syntax is similar to the server line from the configuration file,
|
|
please refer to their individual documentation for details.
|
|
|
|
add ssl crt-list <crtlist> <certificate>
|
|
add ssl crt-list <crtlist> <payload>
|
|
Add an certificate in a crt-list. It can also be used for directories since
|
|
directories are now loaded the same way as the crt-lists. This command allow
|
|
you to use a certificate name in parameter, to use SSL options or filters a
|
|
crt-list line must sent as a payload instead. Only one crt-list line is
|
|
supported in the payload. This command will load the certificate for every
|
|
bind lines using the crt-list. To push a new certificate to HAProxy the
|
|
commands "new ssl cert" and "set ssl cert" must be used.
|
|
|
|
Example:
|
|
$ echo "new ssl cert foobar.pem" | socat /tmp/sock1 -
|
|
$ echo -e "set ssl cert foobar.pem <<\n$(cat foobar.pem)\n" | socat
|
|
/tmp/sock1 -
|
|
$ echo "commit ssl cert foobar.pem" | socat /tmp/sock1 -
|
|
$ echo "add ssl crt-list certlist1 foobar.pem" | socat /tmp/sock1 -
|
|
|
|
$ echo -e 'add ssl crt-list certlist1 <<\nfoobar.pem [allow-0rtt] foo.bar.com
|
|
!test1.com\n' | socat /tmp/sock1 -
|
|
|
|
clear counters
|
|
Clear the max values of the statistics counters in each proxy (frontend &
|
|
backend) and in each server. The accumulated counters are not affected. The
|
|
internal activity counters reported by "show activity" are also reset. This
|
|
can be used to get clean counters after an incident, without having to
|
|
restart nor to clear traffic counters. This command is restricted and can
|
|
only be issued on sockets configured for levels "operator" or "admin".
|
|
|
|
clear counters all
|
|
Clear all statistics counters in each proxy (frontend & backend) and in each
|
|
server. This has the same effect as restarting. This command is restricted
|
|
and can only be issued on sockets configured for level "admin".
|
|
|
|
clear acl [@<ver>] <acl>
|
|
Remove all entries from the acl <acl>. <acl> is the #<id> or the <file>
|
|
returned by "show acl". Note that if the reference <acl> is a file and is
|
|
shared with a map, this map will be also cleared. By default only the current
|
|
version of the ACL is cleared (the one being matched against). However it is
|
|
possible to specify another version using '@' followed by this version.
|
|
|
|
clear map [@<ver>] <map>
|
|
Remove all entries from the map <map>. <map> is the #<id> or the <file>
|
|
returned by "show map". Note that if the reference <map> is a file and is
|
|
shared with a acl, this acl will be also cleared. By default only the current
|
|
version of the map is cleared (the one being matched against). However it is
|
|
possible to specify another version using '@' followed by this version.
|
|
|
|
clear table <table> [ data.<type> <operator> <value> ] | [ key <key> ]
|
|
Remove entries from the stick-table <table>.
|
|
|
|
This is typically used to unblock some users complaining they have been
|
|
abusively denied access to a service, but this can also be used to clear some
|
|
stickiness entries matching a server that is going to be replaced (see "show
|
|
table" below for details). Note that sometimes, removal of an entry will be
|
|
refused because it is currently tracked by a session. Retrying a few seconds
|
|
later after the session ends is usual enough.
|
|
|
|
In the case where no options arguments are given all entries will be removed.
|
|
|
|
When the "data." form is used entries matching a filter applied using the
|
|
stored data (see "stick-table" in section 4.2) are removed. A stored data
|
|
type must be specified in <type>, and this data type must be stored in the
|
|
table otherwise an error is reported. The data is compared according to
|
|
<operator> with the 64-bit integer <value>. Operators are the same as with
|
|
the ACLs :
|
|
|
|
- eq : match entries whose data is equal to this value
|
|
- ne : match entries whose data is not equal to this value
|
|
- le : match entries whose data is less than or equal to this value
|
|
- ge : match entries whose data is greater than or equal to this value
|
|
- lt : match entries whose data is less than this value
|
|
- gt : match entries whose data is greater than this value
|
|
|
|
When the key form is used the entry <key> is removed. The key must be of the
|
|
same type as the table, which currently is limited to IPv4, IPv6, integer and
|
|
string.
|
|
|
|
Example :
|
|
$ echo "show table http_proxy" | socat stdio /tmp/sock1
|
|
>>> # table: http_proxy, type: ip, size:204800, used:2
|
|
>>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
|
|
bytes_out_rate(60000)=187
|
|
>>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
|
|
bytes_out_rate(60000)=191
|
|
|
|
$ echo "clear table http_proxy key 127.0.0.1" | socat stdio /tmp/sock1
|
|
|
|
$ echo "show table http_proxy" | socat stdio /tmp/sock1
|
|
>>> # table: http_proxy, type: ip, size:204800, used:1
|
|
>>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
|
|
bytes_out_rate(60000)=191
|
|
$ echo "clear table http_proxy data.gpc0 eq 1" | socat stdio /tmp/sock1
|
|
$ echo "show table http_proxy" | socat stdio /tmp/sock1
|
|
>>> # table: http_proxy, type: ip, size:204800, used:1
|
|
|
|
commit acl @<ver> <acl>
|
|
Commit all changes made to version <ver> of ACL <acl>, and deletes all past
|
|
versions. <acl> is the #<id> or the <file> returned by "show acl". The
|
|
version number must be between "curr_ver"+1 and "next_ver" as reported in
|
|
"show acl". The contents to be committed to the ACL can be consulted with
|
|
"show acl @<ver> <acl>" if desired. The specified version number has normally
|
|
been created with the "prepare acl" command. The replacement is atomic. It
|
|
consists in atomically updating the current version to the specified version,
|
|
which will instantly cause all entries in other versions to become invisible,
|
|
and all entries in the new version to become visible. It is also possible to
|
|
use this command to perform an atomic removal of all visible entries of an
|
|
ACL by calling "prepare acl" first then committing without adding any
|
|
entries. This command cannot be used if the reference <acl> is a file also
|
|
used as a map. In this case, the "commit map" command must be used instead.
|
|
|
|
commit map @<ver> <map>
|
|
Commit all changes made to version <ver> of map <map>, and deletes all past
|
|
versions. <map> is the #<id> or the <file> returned by "show map". The
|
|
version number must be between "curr_ver"+1 and "next_ver" as reported in
|
|
"show map". The contents to be committed to the map can be consulted with
|
|
"show map @<ver> <map>" if desired. The specified version number has normally
|
|
been created with the "prepare map" command. The replacement is atomic. It
|
|
consists in atomically updating the current version to the specified version,
|
|
which will instantly cause all entries in other versions to become invisible,
|
|
and all entries in the new version to become visible. It is also possible to
|
|
use this command to perform an atomic removal of all visible entries of an
|
|
map by calling "prepare map" first then committing without adding any
|
|
entries.
|
|
|
|
commit ssl ca-file <cafile>
|
|
Commit a temporary SSL CA file update transaction.
|
|
|
|
In the case of an existing CA file (in a "Used" state in "show ssl ca-file"),
|
|
the new CA file tree entry is inserted in the CA file tree and every instance
|
|
that used the CA file entry is rebuilt, along with the SSL contexts it needs.
|
|
All the contexts previously used by the rebuilt instances are removed.
|
|
Upon success, the previous CA file entry is removed from the tree.
|
|
Upon failure, nothing is removed or deleted, and all the original SSL
|
|
contexts are kept and used.
|
|
Once the temporary transaction is committed, it is destroyed.
|
|
|
|
In the case of a new CA file (after a "new ssl ca-file" and in a "Unused"
|
|
state in "show ssl ca-file"), the CA file will be inserted in the CA file
|
|
tree but it won't be used anywhere in HAProxy. To use it and generate SSL
|
|
contexts that use it, you will need to add it to a crt-list with "add ssl
|
|
crt-list".
|
|
|
|
See also "new ssl ca-file", "set ssl ca-file", "abort ssl ca-file" and
|
|
"add ssl crt-list".
|
|
|
|
commit ssl cert <filename>
|
|
Commit a temporary SSL certificate update transaction.
|
|
|
|
In the case of an existing certificate (in a "Used" state in "show ssl
|
|
cert"), generate every SSL contextes and SNIs it need, insert them, and
|
|
remove the previous ones. Replace in memory the previous SSL certificates
|
|
everywhere the <filename> was used in the configuration. Upon failure it
|
|
doesn't remove or insert anything. Once the temporary transaction is
|
|
committed, it is destroyed.
|
|
|
|
In the case of a new certificate (after a "new ssl cert" and in a "Unused"
|
|
state in "show ssl cert"), the certificate will be committed in a certificate
|
|
storage, but it won't be used anywhere in haproxy. To use it and generate
|
|
its SNIs you will need to add it to a crt-list or a directory with "add ssl
|
|
crt-list".
|
|
|
|
See also "new ssl cert", "set ssl cert", "abort ssl cert" and
|
|
"add ssl crt-list".
|
|
|
|
commit ssl crl-file <crlfile>
|
|
Commit a temporary SSL CRL file update transaction.
|
|
|
|
In the case of an existing CRL file (in a "Used" state in "show ssl
|
|
crl-file"), the new CRL file entry is inserted in the CA file tree (which
|
|
holds both the CA files and the CRL files) and every instance that used the
|
|
CRL file entry is rebuilt, along with the SSL contexts it needs.
|
|
All the contexts previously used by the rebuilt instances are removed.
|
|
Upon success, the previous CRL file entry is removed from the tree.
|
|
Upon failure, nothing is removed or deleted, and all the original SSL
|
|
contexts are kept and used.
|
|
Once the temporary transaction is committed, it is destroyed.
|
|
|
|
In the case of a new CRL file (after a "new ssl crl-file" and in a "Unused"
|
|
state in "show ssl crl-file"), the CRL file will be inserted in the CRL file
|
|
tree but it won't be used anywhere in HAProxy. To use it and generate SSL
|
|
contexts that use it, you will need to add it to a crt-list with "add ssl
|
|
crt-list".
|
|
|
|
See also "new ssl crl-file", "set ssl crl-file", "abort ssl crl-file" and
|
|
"add ssl crt-list".
|
|
|
|
debug dev <command> [args]*
|
|
Call a developer-specific command. Only supported on a CLI connection running
|
|
in expert mode (see "expert-mode on"). Such commands are extremely dangerous
|
|
and not forgiving, any misuse may result in a crash of the process. They are
|
|
intended for experts only, and must really not be used unless told to do so.
|
|
Some of them are only available when haproxy is built with DEBUG_DEV defined
|
|
because they may have security implications. All of these commands require
|
|
admin privileges, and are purposely not documented to avoid encouraging their
|
|
use by people who are not at ease with the source code.
|
|
|
|
del acl <acl> [<key>|#<ref>]
|
|
Delete all the acl entries from the acl <acl> corresponding to the key <key>.
|
|
<acl> is the #<id> or the <file> returned by "show acl". If the <ref> is used,
|
|
this command delete only the listed reference. The reference can be found with
|
|
listing the content of the acl. Note that if the reference <acl> is a file and
|
|
is shared with a map, the entry will be also deleted in the map.
|
|
|
|
del map <map> [<key>|#<ref>]
|
|
Delete all the map entries from the map <map> corresponding to the key <key>.
|
|
<map> is the #<id> or the <file> returned by "show map". If the <ref> is used,
|
|
this command delete only the listed reference. The reference can be found with
|
|
listing the content of the map. Note that if the reference <map> is a file and
|
|
is shared with a acl, the entry will be also deleted in the map.
|
|
|
|
del ssl ca-file <cafile>
|
|
Delete a CA file tree entry from HAProxy. The CA file must be unused and
|
|
removed from any crt-list. "show ssl ca-file" displays the status of the CA
|
|
files. The deletion doesn't work with a certificate referenced directly with
|
|
the "ca-file" or "ca-verify-file" directives in the configuration.
|
|
|
|
del ssl cert <certfile>
|
|
Delete a certificate store from HAProxy. The certificate must be unused and
|
|
removed from any crt-list or directory. "show ssl cert" displays the status
|
|
of the certificate. The deletion doesn't work with a certificate referenced
|
|
directly with the "crt" directive in the configuration.
|
|
|
|
del ssl crl-file <crlfile>
|
|
Delete a CRL file tree entry from HAProxy. The CRL file must be unused and
|
|
removed from any crt-list. "show ssl crl-file" displays the status of the CRL
|
|
files. The deletion doesn't work with a certificate referenced directly with
|
|
the "crl-file" directive in the configuration.
|
|
|
|
del ssl crt-list <filename> <certfile[:line]>
|
|
Delete an entry in a crt-list. This will delete every SNIs used for this
|
|
entry in the frontends. If a certificate is used several time in a crt-list,
|
|
you will need to provide which line you want to delete. To display the line
|
|
numbers, use "show ssl crt-list -n <crtlist>".
|
|
|
|
del server <backend>/<server>
|
|
Remove a server attached to the backend <backend>. All servers are eligible,
|
|
except servers which are referenced by other configuration elements. The
|
|
server must be put in maintenance mode prior to its deletion. The operation
|
|
is cancelled if the serveur still has active or idle connection or its
|
|
connection queue is not empty.
|
|
|
|
disable agent <backend>/<server>
|
|
Mark the auxiliary agent check as temporarily stopped.
|
|
|
|
In the case where an agent check is being run as a auxiliary check, due
|
|
to the agent-check parameter of a server directive, new checks are only
|
|
initialized when the agent is in the enabled. Thus, disable agent will
|
|
prevent any new agent checks from begin initiated until the agent
|
|
re-enabled using enable agent.
|
|
|
|
When an agent is disabled the processing of an auxiliary agent check that
|
|
was initiated while the agent was set as enabled is as follows: All
|
|
results that would alter the weight, specifically "drain" or a weight
|
|
returned by the agent, are ignored. The processing of agent check is
|
|
otherwise unchanged.
|
|
|
|
The motivation for this feature is to allow the weight changing effects
|
|
of the agent checks to be paused to allow the weight of a server to be
|
|
configured using set weight without being overridden by the agent.
|
|
|
|
This command is restricted and can only be issued on sockets configured for
|
|
level "admin".
|
|
|
|
disable dynamic-cookie backend <backend>
|
|
Disable the generation of dynamic cookies for the backend <backend>
|
|
|
|
disable frontend <frontend>
|
|
Mark the frontend as temporarily stopped. This corresponds to the mode which
|
|
is used during a soft restart : the frontend releases the port but can be
|
|
enabled again if needed. This should be used with care as some non-Linux OSes
|
|
are unable to enable it back. This is intended to be used in environments
|
|
where stopping a proxy is not even imaginable but a misconfigured proxy must
|
|
be fixed. That way it's possible to release the port and bind it into another
|
|
process to restore operations. The frontend will appear with status "STOP"
|
|
on the stats page.
|
|
|
|
The frontend may be specified either by its name or by its numeric ID,
|
|
prefixed with a sharp ('#').
|
|
|
|
This command is restricted and can only be issued on sockets configured for
|
|
level "admin".
|
|
|
|
disable health <backend>/<server>
|
|
Mark the primary health check as temporarily stopped. This will disable
|
|
sending of health checks, and the last health check result will be ignored.
|
|
The server will be in unchecked state and considered UP unless an auxiliary
|
|
agent check forces it down.
|
|
|
|
This command is restricted and can only be issued on sockets configured for
|
|
level "admin".
|
|
|
|
disable server <backend>/<server>
|
|
Mark the server DOWN for maintenance. In this mode, no more checks will be
|
|
performed on the server until it leaves maintenance.
|
|
If the server is tracked by other servers, those servers will be set to DOWN
|
|
during the maintenance.
|
|
|
|
In the statistics page, a server DOWN for maintenance will appear with a
|
|
"MAINT" status, its tracking servers with the "MAINT(via)" one.
|
|
|
|
Both the backend and the server may be specified either by their name or by
|
|
their numeric ID, prefixed with a sharp ('#').
|
|
|
|
This command is restricted and can only be issued on sockets configured for
|
|
level "admin".
|
|
|
|
enable agent <backend>/<server>
|
|
Resume auxiliary agent check that was temporarily stopped.
|
|
|
|
See "disable agent" for details of the effect of temporarily starting
|
|
and stopping an auxiliary agent.
|
|
|
|
This command is restricted and can only be issued on sockets configured for
|
|
level "admin".
|
|
|
|
enable dynamic-cookie backend <backend>
|
|
Enable the generation of dynamic cookies for the backend <backend>.
|
|
A secret key must also be provided.
|
|
|
|
enable frontend <frontend>
|
|
Resume a frontend which was temporarily stopped. It is possible that some of
|
|
the listening ports won't be able to bind anymore (eg: if another process
|
|
took them since the 'disable frontend' operation). If this happens, an error
|
|
is displayed. Some operating systems might not be able to resume a frontend
|
|
which was disabled.
|
|
|
|
The frontend may be specified either by its name or by its numeric ID,
|
|
prefixed with a sharp ('#').
|
|
|
|
This command is restricted and can only be issued on sockets configured for
|
|
level "admin".
|
|
|
|
enable health <backend>/<server>
|
|
Resume a primary health check that was temporarily stopped. This will enable
|
|
sending of health checks again. Please see "disable health" for details.
|
|
|
|
This command is restricted and can only be issued on sockets configured for
|
|
level "admin".
|
|
|
|
enable server <backend>/<server>
|
|
If the server was previously marked as DOWN for maintenance, this marks the
|
|
server UP and checks are re-enabled.
|
|
|
|
Both the backend and the server may be specified either by their name or by
|
|
their numeric ID, prefixed with a sharp ('#').
|
|
|
|
This command is restricted and can only be issued on sockets configured for
|
|
level "admin".
|
|
|
|
experimental-mode [on|off]
|
|
Without options, this indicates whether the experimental mode is enabled or
|
|
disabled on the current connection. When passed "on", it turns the
|
|
experimental mode on for the current CLI connection only. With "off" it turns
|
|
it off.
|
|
|
|
The experimental mode is used to access to extra features still in
|
|
development. These features are currently not stable and should be used with
|
|
care. They may be subject to breaking changes across versions.
|
|
|
|
expert-mode [on|off]
|
|
This command is similar to experimental-mode but is used to toggle the
|
|
expert mode.
|
|
|
|
The expert mode enables displaying of expert commands that can be extremely
|
|
dangerous for the process and which may occasionally help developers collect
|
|
important information about complex bugs. Any misuse of these features will
|
|
likely lead to a process crash. Do not use this option without being invited
|
|
to do so. Note that this command is purposely not listed in the help message.
|
|
This command is only accessible in admin level. Changing to another level
|
|
automatically resets the expert mode.
|
|
|
|
get map <map> <value>
|
|
get acl <acl> <value>
|
|
Lookup the value <value> in the map <map> or in the ACL <acl>. <map> or <acl>
|
|
are the #<id> or the <file> returned by "show map" or "show acl". This command
|
|
returns all the matching patterns associated with this map. This is useful for
|
|
debugging maps and ACLs. The output format is composed by one line par
|
|
matching type. Each line is composed by space-delimited series of words.
|
|
|
|
The first two words are:
|
|
|
|
<match method>: The match method applied. It can be "found", "bool",
|
|
"int", "ip", "bin", "len", "str", "beg", "sub", "dir",
|
|
"dom", "end" or "reg".
|
|
|
|
<match result>: The result. Can be "match" or "no-match".
|
|
|
|
The following words are returned only if the pattern matches an entry.
|
|
|
|
<index type>: "tree" or "list". The internal lookup algorithm.
|
|
|
|
<case>: "case-insensitive" or "case-sensitive". The
|
|
interpretation of the case.
|
|
|
|
<entry matched>: match="<entry>". Return the matched pattern. It is
|
|
useful with regular expressions.
|
|
|
|
The two last word are used to show the returned value and its type. With the
|
|
"acl" case, the pattern doesn't exist.
|
|
|
|
return=nothing: No return because there are no "map".
|
|
return="<value>": The value returned in the string format.
|
|
return=cannot-display: The value cannot be converted as string.
|
|
|
|
type="<type>": The type of the returned sample.
|
|
|
|
get var <name>
|
|
Show the existence, type and contents of the process-wide variable 'name'.
|
|
Only process-wide variables are readable, so the name must begin with
|
|
'proc.' otherwise no variable will be found. This command requires levels
|
|
"operator" or "admin".
|
|
|
|
get weight <backend>/<server>
|
|
Report the current weight and the initial weight of server <server> in
|
|
backend <backend> or an error if either doesn't exist. The initial weight is
|
|
the one that appears in the configuration file. Both are normally equal
|
|
unless the current weight has been changed. Both the backend and the server
|
|
may be specified either by their name or by their numeric ID, prefixed with a
|
|
sharp ('#').
|
|
|
|
help [<command>]
|
|
Print the list of known keywords and their basic usage, or commands matching
|
|
the requested one. The same help screen is also displayed for unknown
|
|
commands.
|
|
|
|
new ssl ca-file <cafile>
|
|
Create a new empty CA file tree entry to be filled with a set of CA
|
|
certificates and added to a crt-list. This command should be used in
|
|
combination with "set ssl ca-file" and "add ssl crt-list".
|
|
|
|
new ssl cert <filename>
|
|
Create a new empty SSL certificate store to be filled with a certificate and
|
|
added to a directory or a crt-list. This command should be used in
|
|
combination with "set ssl cert" and "add ssl crt-list".
|
|
|
|
new ssl crl-file <crlfile>
|
|
Create a new empty CRL file tree entry to be filled with a set of CRLs
|
|
and added to a crt-list. This command should be used in combination with "set
|
|
ssl crl-file" and "add ssl crt-list".
|
|
|
|
prepare acl <acl>
|
|
Allocate a new version number in ACL <acl> for atomic replacement. <acl> is
|
|
the #<id> or the <file> returned by "show acl". The new version number is
|
|
shown in response after "New version created:". This number will then be
|
|
usable to prepare additions of new entries into the ACL which will then
|
|
atomically replace the current ones once committed. It is reported as
|
|
"next_ver" in "show acl". There is no impact of allocating new versions, as
|
|
unused versions will automatically be removed once a more recent version is
|
|
committed. Version numbers are unsigned 32-bit values which wrap at the end,
|
|
so care must be taken when comparing them in an external program. This
|
|
command cannot be used if the reference <acl> is a file also used as a map.
|
|
In this case, the "prepare map" command must be used instead.
|
|
|
|
prepare map <map>
|
|
Allocate a new version number in map <map> for atomic replacement. <map> is
|
|
the #<id> or the <file> returned by "show map". The new version number is
|
|
shown in response after "New version created:". This number will then be
|
|
usable to prepare additions of new entries into the map which will then
|
|
atomically replace the current ones once committed. It is reported as
|
|
"next_ver" in "show map". There is no impact of allocating new versions, as
|
|
unused versions will automatically be removed once a more recent version is
|
|
committed. Version numbers are unsigned 32-bit values which wrap at the end,
|
|
so care must be taken when comparing them in an external program.
|
|
|
|
prompt
|
|
Toggle the prompt at the beginning of the line and enter or leave interactive
|
|
mode. In interactive mode, the connection is not closed after a command
|
|
completes. Instead, the prompt will appear again, indicating the user that
|
|
the interpreter is waiting for a new command. The prompt consists in a right
|
|
angle bracket followed by a space "> ". This mode is particularly convenient
|
|
when one wants to periodically check information such as stats or errors.
|
|
It is also a good idea to enter interactive mode before issuing a "help"
|
|
command.
|
|
|
|
quit
|
|
Close the connection when in interactive mode.
|
|
|
|
set dynamic-cookie-key backend <backend> <value>
|
|
Modify the secret key used to generate the dynamic persistent cookies.
|
|
This will break the existing sessions.
|
|
|
|
set map <map> [<key>|#<ref>] <value>
|
|
Modify the value corresponding to each key <key> in a map <map>. <map> is the
|
|
#<id> or <file> returned by "show map". If the <ref> is used in place of
|
|
<key>, only the entry pointed by <ref> is changed. The new value is <value>.
|
|
|
|
set maxconn frontend <frontend> <value>
|
|
Dynamically change the specified frontend's maxconn setting. Any positive
|
|
value is allowed including zero, but setting values larger than the global
|
|
maxconn does not make much sense. If the limit is increased and connections
|
|
were pending, they will immediately be accepted. If it is lowered to a value
|
|
below the current number of connections, new connections acceptation will be
|
|
delayed until the threshold is reached. The frontend might be specified by
|
|
either its name or its numeric ID prefixed with a sharp ('#').
|
|
|
|
set maxconn server <backend/server> <value>
|
|
Dynamically change the specified server's maxconn setting. Any positive
|
|
value is allowed including zero, but setting values larger than the global
|
|
maxconn does not make much sense.
|
|
|
|
set maxconn global <maxconn>
|
|
Dynamically change the global maxconn setting within the range defined by the
|
|
initial global maxconn setting. If it is increased and connections were
|
|
pending, they will immediately be accepted. If it is lowered to a value below
|
|
the current number of connections, new connections acceptation will be
|
|
delayed until the threshold is reached. A value of zero restores the initial
|
|
setting.
|
|
|
|
set profiling { tasks | memory } { auto | on | off }
|
|
Enables or disables CPU or memory profiling for the indicated subsystem. This
|
|
is equivalent to setting or clearing the "profiling" settings in the "global"
|
|
section of the configuration file. Please also see "show profiling". Note
|
|
that manually setting the tasks profiling to "on" automatically resets the
|
|
scheduler statistics, thus allows to check activity over a given interval.
|
|
The memory profiling is limited to certain operating systems (known to work
|
|
on the linux-glibc target), and requires USE_MEMORY_PROFILING to be set at
|
|
compile time.
|
|
|
|
set rate-limit connections global <value>
|
|
Change the process-wide connection rate limit, which is set by the global
|
|
'maxconnrate' setting. A value of zero disables the limitation. This limit
|
|
applies to all frontends and the change has an immediate effect. The value
|
|
is passed in number of connections per second.
|
|
|
|
set rate-limit http-compression global <value>
|
|
Change the maximum input compression rate, which is set by the global
|
|
'maxcomprate' setting. A value of zero disables the limitation. The value is
|
|
passed in number of kilobytes per second. The value is available in the "show
|
|
info" on the line "CompressBpsRateLim" in bytes.
|
|
|
|
set rate-limit sessions global <value>
|
|
Change the process-wide session rate limit, which is set by the global
|
|
'maxsessrate' setting. A value of zero disables the limitation. This limit
|
|
applies to all frontends and the change has an immediate effect. The value
|
|
is passed in number of sessions per second.
|
|
|
|
set rate-limit ssl-sessions global <value>
|
|
Change the process-wide SSL session rate limit, which is set by the global
|
|
'maxsslrate' setting. A value of zero disables the limitation. This limit
|
|
applies to all frontends and the change has an immediate effect. The value
|
|
is passed in number of sessions per second sent to the SSL stack. It applies
|
|
before the handshake in order to protect the stack against handshake abuses.
|
|
|
|
set server <backend>/<server> addr <ip4 or ip6 address> [port <port>]
|
|
Replace the current IP address of a server by the one provided.
|
|
Optionally, the port can be changed using the 'port' parameter.
|
|
Note that changing the port also support switching from/to port mapping
|
|
(notation with +X or -Y), only if a port is configured for the health check.
|
|
|
|
set server <backend>/<server> agent [ up | down ]
|
|
Force a server's agent to a new state. This can be useful to immediately
|
|
switch a server's state regardless of some slow agent checks for example.
|
|
Note that the change is propagated to tracking servers if any.
|
|
|
|
set server <backend>/<server> agent-addr <addr> [port <port>]
|
|
Change addr for servers agent checks. Allows to migrate agent-checks to
|
|
another address at runtime. You can specify both IP and hostname, it will be
|
|
resolved.
|
|
Optionally, change the port agent.
|
|
|
|
set server <backend>/<server> agent-port <port>
|
|
Change the port used for agent checks.
|
|
|
|
set server <backend>/<server> agent-send <value>
|
|
Change agent string sent to agent check target. Allows to update string while
|
|
changing server address to keep those two matching.
|
|
|
|
set server <backend>/<server> health [ up | stopping | down ]
|
|
Force a server's health to a new state. This can be useful to immediately
|
|
switch a server's state regardless of some slow health checks for example.
|
|
Note that the change is propagated to tracking servers if any.
|
|
|
|
set server <backend>/<server> check-addr <ip4 | ip6> [port <port>]
|
|
Change the IP address used for server health checks.
|
|
Optionally, change the port used for server health checks.
|
|
|
|
set server <backend>/<server> check-port <port>
|
|
Change the port used for health checking to <port>
|
|
|
|
set server <backend>/<server> state [ ready | drain | maint ]
|
|
Force a server's administrative state to a new state. This can be useful to
|
|
disable load balancing and/or any traffic to a server. Setting the state to
|
|
"ready" puts the server in normal mode, and the command is the equivalent of
|
|
the "enable server" command. Setting the state to "maint" disables any traffic
|
|
to the server as well as any health checks. This is the equivalent of the
|
|
"disable server" command. Setting the mode to "drain" only removes the server
|
|
from load balancing but still allows it to be checked and to accept new
|
|
persistent connections. Changes are propagated to tracking servers if any.
|
|
|
|
set server <backend>/<server> weight <weight>[%]
|
|
Change a server's weight to the value passed in argument. This is the exact
|
|
equivalent of the "set weight" command below.
|
|
|
|
set server <backend>/<server> fqdn <FQDN>
|
|
Change a server's FQDN to the value passed in argument. This requires the
|
|
internal run-time DNS resolver to be configured and enabled for this server.
|
|
|
|
set server <backend>/<server> ssl [ on | off ]
|
|
This option configures SSL ciphering on outgoing connections to the server.
|
|
|
|
set severity-output [ none | number | string ]
|
|
Change the severity output format of the stats socket connected to for the
|
|
duration of the current session.
|
|
|
|
set ssl ca-file <cafile> <payload>
|
|
This command is part of a transaction system, the "commit ssl ca-file" and
|
|
"abort ssl ca-file" commands could be required.
|
|
If there is no on-going transaction, it will create a CA file tree entry into
|
|
which the certificates contained in the payload will be stored. The CA file
|
|
entry will not be stored in the CA file tree and will only be kept in a
|
|
temporary transaction. If a transaction with the same filename already exists,
|
|
the previous CA file entry will be deleted and replaced by the new one.
|
|
Once the modifications are done, you have to commit the transaction through
|
|
a "commit ssl ca-file" call.
|
|
|
|
Example:
|
|
echo -e "set ssl ca-file cafile.pem <<\n$(cat rootCA.crt)\n" | \
|
|
socat /var/run/haproxy.stat -
|
|
echo "commit ssl ca-file cafile.pem" | socat /var/run/haproxy.stat -
|
|
|
|
set ssl cert <filename> <payload>
|
|
This command is part of a transaction system, the "commit ssl cert" and
|
|
"abort ssl cert" commands could be required.
|
|
This whole transaction system works on any certificate displayed by the
|
|
"show ssl cert" command, so on any frontend or backend certificate.
|
|
If there is no on-going transaction, it will duplicate the certificate
|
|
<filename> in memory to a temporary transaction, then update this
|
|
transaction with the PEM file in the payload. If a transaction exists with
|
|
the same filename, it will update this transaction. It's also possible to
|
|
update the files linked to a certificate (.issuer, .sctl, .oscp etc.)
|
|
Once the modification are done, you have to "commit ssl cert" the
|
|
transaction.
|
|
|
|
Injection of files over the CLI must be done with caution since an empty line
|
|
is used to notify the end of the payload. It is recommended to inject a PEM
|
|
file which has been sanitized. A simple method would be to remove every empty
|
|
line and only leave what are in the PEM sections. It could be achieved with a
|
|
sed command.
|
|
|
|
Example:
|
|
|
|
# With some simple sanitizing
|
|
echo -e "set ssl cert localhost.pem <<\n$(sed -n '/^$/d;/-BEGIN/,/-END/p' 127.0.0.1.pem)\n" | \
|
|
socat /var/run/haproxy.stat -
|
|
|
|
# Complete example with commit
|
|
echo -e "set ssl cert localhost.pem <<\n$(cat 127.0.0.1.pem)\n" | \
|
|
socat /var/run/haproxy.stat -
|
|
echo -e \
|
|
"set ssl cert localhost.pem.issuer <<\n $(cat 127.0.0.1.pem.issuer)\n" | \
|
|
socat /var/run/haproxy.stat -
|
|
echo -e \
|
|
"set ssl cert localhost.pem.ocsp <<\n$(base64 -w 1000 127.0.0.1.pem.ocsp)\n" | \
|
|
socat /var/run/haproxy.stat -
|
|
echo "commit ssl cert localhost.pem" | socat /var/run/haproxy.stat -
|
|
|
|
set ssl crl-file <crlfile> <payload>
|
|
This command is part of a transaction system, the "commit ssl crl-file" and
|
|
"abort ssl crl-file" commands could be required.
|
|
If there is no on-going transaction, it will create a CRL file tree entry into
|
|
which the Revocation Lists contained in the payload will be stored. The CRL
|
|
file entry will not be stored in the CRL file tree and will only be kept in a
|
|
temporary transaction. If a transaction with the same filename already exists,
|
|
the previous CRL file entry will be deleted and replaced by the new one.
|
|
Once the modifications are done, you have to commit the transaction through
|
|
a "commit ssl crl-file" call.
|
|
|
|
Example:
|
|
echo -e "set ssl crl-file crlfile.pem <<\n$(cat rootCRL.pem)\n" | \
|
|
socat /var/run/haproxy.stat -
|
|
echo "commit ssl crl-file crlfile.pem" | socat /var/run/haproxy.stat -
|
|
|
|
set ssl ocsp-response <response | payload>
|
|
This command is used to update an OCSP Response for a certificate (see "crt"
|
|
on "bind" lines). Same controls are performed as during the initial loading of
|
|
the response. The <response> must be passed as a base64 encoded string of the
|
|
DER encoded response from the OCSP server. This command is not supported with
|
|
BoringSSL.
|
|
|
|
Example:
|
|
openssl ocsp -issuer issuer.pem -cert server.pem \
|
|
-host ocsp.issuer.com:80 -respout resp.der
|
|
echo "set ssl ocsp-response $(base64 -w 10000 resp.der)" | \
|
|
socat stdio /var/run/haproxy.stat
|
|
|
|
using the payload syntax:
|
|
echo -e "set ssl ocsp-response <<\n$(base64 resp.der)\n" | \
|
|
socat stdio /var/run/haproxy.stat
|
|
|
|
set ssl tls-key <id> <tlskey>
|
|
Set the next TLS key for the <id> listener to <tlskey>. This key becomes the
|
|
ultimate key, while the penultimate one is used for encryption (others just
|
|
decrypt). The oldest TLS key present is overwritten. <id> is either a numeric
|
|
#<id> or <file> returned by "show tls-keys". <tlskey> is a base64 encoded 48
|
|
or 80 bits TLS ticket key (ex. openssl rand 80 | openssl base64 -A).
|
|
|
|
set table <table> key <key> [data.<data_type> <value>]*
|
|
Create or update a stick-table entry in the table. If the key is not present,
|
|
an entry is inserted. See stick-table in section 4.2 to find all possible
|
|
values for <data_type>. The most likely use consists in dynamically entering
|
|
entries for source IP addresses, with a flag in gpc0 to dynamically block an
|
|
IP address or affect its quality of service. It is possible to pass multiple
|
|
data_types in a single call.
|
|
|
|
set timeout cli <delay>
|
|
Change the CLI interface timeout for current connection. This can be useful
|
|
during long debugging sessions where the user needs to constantly inspect
|
|
some indicators without being disconnected. The delay is passed in seconds.
|
|
|
|
set var <name> <expression>
|
|
set var <name> expr <expression>
|
|
set var <name> fmt <format>
|
|
Allows to set or overwrite the process-wide variable 'name' with the result
|
|
of expression <expression> or format string <format>. Only process-wide
|
|
variables may be used, so the name must begin with 'proc.' otherwise no
|
|
variable will be set. The <expression> and <format> may only involve
|
|
"internal" sample fetch keywords and converters even though the most likely
|
|
useful ones will be str('something'), int(), simple strings or references to
|
|
other variables. Note that the command line parser doesn't know about quotes,
|
|
so any space in the expression must be preceded by a backslash. This command
|
|
requires levels "operator" or "admin". This command is only supported on a
|
|
CLI connection running in experimental mode (see "experimental-mode on").
|
|
|
|
set weight <backend>/<server> <weight>[%]
|
|
Change a server's weight to the value passed in argument. If the value ends
|
|
with the '%' sign, then the new weight will be relative to the initially
|
|
configured weight. Absolute weights are permitted between 0 and 256.
|
|
Relative weights must be positive with the resulting absolute weight is
|
|
capped at 256. Servers which are part of a farm running a static
|
|
load-balancing algorithm have stricter limitations because the weight
|
|
cannot change once set. Thus for these servers, the only accepted values
|
|
are 0 and 100% (or 0 and the initial weight). Changes take effect
|
|
immediately, though certain LB algorithms require a certain amount of
|
|
requests to consider changes. A typical usage of this command is to
|
|
disable a server during an update by setting its weight to zero, then to
|
|
enable it again after the update by setting it back to 100%. This command
|
|
is restricted and can only be issued on sockets configured for level
|
|
"admin". Both the backend and the server may be specified either by their
|
|
name or by their numeric ID, prefixed with a sharp ('#').
|
|
|
|
show acl [[@<ver>] <acl>]
|
|
Dump info about acl converters. Without argument, the list of all available
|
|
acls is returned. If a <acl> is specified, its contents are dumped. <acl> is
|
|
the #<id> or <file>. By default the current version of the ACL is shown (the
|
|
version currently being matched against and reported as 'curr_ver' in the ACL
|
|
list). It is possible to instead dump other versions by prepending '@<ver>'
|
|
before the ACL's identifier. The version works as a filter and non-existing
|
|
versions will simply report no result. The dump format is the same as for the
|
|
maps even for the sample values. The data returned are not a list of
|
|
available ACL, but are the list of all patterns composing any ACL. Many of
|
|
these patterns can be shared with maps. The 'entry_cnt' value represents the
|
|
count of all the ACL entries, not just the active ones, which means that it
|
|
also includes entries currently being added.
|
|
|
|
show backend
|
|
Dump the list of backends available in the running process
|
|
|
|
show cli level
|
|
Display the CLI level of the current CLI session. The result could be
|
|
'admin', 'operator' or 'user'. See also the 'operator' and 'user' commands.
|
|
|
|
Example :
|
|
|
|
$ socat /tmp/sock1 readline
|
|
prompt
|
|
> operator
|
|
> show cli level
|
|
operator
|
|
> user
|
|
> show cli level
|
|
user
|
|
> operator
|
|
Permission denied
|
|
|
|
operator
|
|
Decrease the CLI level of the current CLI session to operator. It can't be
|
|
increased. It also drops expert and experimental mode. See also "show cli
|
|
level".
|
|
|
|
user
|
|
Decrease the CLI level of the current CLI session to user. It can't be
|
|
increased. It also drops expert and experimental mode. See also "show cli
|
|
level".
|
|
|
|
show activity
|
|
Reports some counters about internal events that will help developers and
|
|
more generally people who know haproxy well enough to narrow down the causes
|
|
of reports of abnormal behaviours. A typical example would be a properly
|
|
running process never sleeping and eating 100% of the CPU. The output fields
|
|
will be made of one line per metric, and per-thread counters on the same
|
|
line. These counters are 32-bit and will wrap during the process's life, which
|
|
is not a problem since calls to this command will typically be performed
|
|
twice. The fields are purposely not documented so that their exact meaning is
|
|
verified in the code where the counters are fed. These values are also reset
|
|
by the "clear counters" command.
|
|
|
|
show cli sockets
|
|
List CLI sockets. The output format is composed of 3 fields separated by
|
|
spaces. The first field is the socket address, it can be a unix socket, a
|
|
ipv4 address:port couple or a ipv6 one. Socket of other types won't be dump.
|
|
The second field describe the level of the socket: 'admin', 'user' or
|
|
'operator'. The last field list the processes on which the socket is bound,
|
|
separated by commas, it can be numbers or 'all'.
|
|
|
|
Example :
|
|
|
|
$ echo 'show cli sockets' | socat stdio /tmp/sock1
|
|
# socket lvl processes
|
|
/tmp/sock1 admin all
|
|
127.0.0.1:9999 user 2,3,4
|
|
127.0.0.2:9969 user 2
|
|
[::1]:9999 operator 2
|
|
|
|
show cache
|
|
List the configured caches and the objects stored in each cache tree.
|
|
|
|
$ echo 'show cache' | socat stdio /tmp/sock1
|
|
0x7f6ac6c5b03a: foobar (shctx:0x7f6ac6c5b000, available blocks:3918)
|
|
1 2 3 4
|
|
|
|
1. pointer to the cache structure
|
|
2. cache name
|
|
3. pointer to the mmap area (shctx)
|
|
4. number of blocks available for reuse in the shctx
|
|
|
|
0x7f6ac6c5b4cc hash:286881868 vary:0x0011223344556677 size:39114 (39 blocks), refcount:9, expire:237
|
|
1 2 3 4 5 6 7
|
|
|
|
1. pointer to the cache entry
|
|
2. first 32 bits of the hash
|
|
3. secondary hash of the entry in case of vary
|
|
4. size of the object in bytes
|
|
5. number of blocks used for the object
|
|
6. number of transactions using the entry
|
|
7. expiration time, can be negative if already expired
|
|
|
|
show env [<name>]
|
|
Dump one or all environment variables known by the process. Without any
|
|
argument, all variables are dumped. With an argument, only the specified
|
|
variable is dumped if it exists. Otherwise "Variable not found" is emitted.
|
|
Variables are dumped in the same format as they are stored or returned by the
|
|
"env" utility, that is, "<name>=<value>". This can be handy when debugging
|
|
certain configuration files making heavy use of environment variables to
|
|
ensure that they contain the expected values. This command is restricted and
|
|
can only be issued on sockets configured for levels "operator" or "admin".
|
|
|
|
show errors [<iid>|<proxy>] [request|response]
|
|
Dump last known request and response errors collected by frontends and
|
|
backends. If <iid> is specified, the limit the dump to errors concerning
|
|
either frontend or backend whose ID is <iid>. Proxy ID "-1" will cause
|
|
all instances to be dumped. If a proxy name is specified instead, its ID
|
|
will be used as the filter. If "request" or "response" is added after the
|
|
proxy name or ID, only request or response errors will be dumped. This
|
|
command is restricted and can only be issued on sockets configured for
|
|
levels "operator" or "admin".
|
|
|
|
The errors which may be collected are the last request and response errors
|
|
caused by protocol violations, often due to invalid characters in header
|
|
names. The report precisely indicates what exact character violated the
|
|
protocol. Other important information such as the exact date the error was
|
|
detected, frontend and backend names, the server name (when known), the
|
|
internal session ID and the source address which has initiated the session
|
|
are reported too.
|
|
|
|
All characters are returned, and non-printable characters are encoded. The
|
|
most common ones (\t = 9, \n = 10, \r = 13 and \e = 27) are encoded as one
|
|
letter following a backslash. The backslash itself is encoded as '\\' to
|
|
avoid confusion. Other non-printable characters are encoded '\xNN' where
|
|
NN is the two-digits hexadecimal representation of the character's ASCII
|
|
code.
|
|
|
|
Lines are prefixed with the position of their first character, starting at 0
|
|
for the beginning of the buffer. At most one input line is printed per line,
|
|
and large lines will be broken into multiple consecutive output lines so that
|
|
the output never goes beyond 79 characters wide. It is easy to detect if a
|
|
line was broken, because it will not end with '\n' and the next line's offset
|
|
will be followed by a '+' sign, indicating it is a continuation of previous
|
|
line.
|
|
|
|
Example :
|
|
$ echo "show errors -1 response" | socat stdio /tmp/sock1
|
|
>>> [04/Mar/2009:15:46:56.081] backend http-in (#2) : invalid response
|
|
src 127.0.0.1, session #54, frontend fe-eth0 (#1), server s2 (#1)
|
|
response length 213 bytes, error at position 23:
|
|
|
|
00000 HTTP/1.0 200 OK\r\n
|
|
00017 header/bizarre:blah\r\n
|
|
00038 Location: blah\r\n
|
|
00054 Long-line: this is a very long line which should b
|
|
00104+ e broken into multiple lines on the output buffer,
|
|
00154+ otherwise it would be too large to print in a ter
|
|
00204+ minal\r\n
|
|
00211 \r\n
|
|
|
|
In the example above, we see that the backend "http-in" which has internal
|
|
ID 2 has blocked an invalid response from its server s2 which has internal
|
|
ID 1. The request was on session 54 initiated by source 127.0.0.1 and
|
|
received by frontend fe-eth0 whose ID is 1. The total response length was
|
|
213 bytes when the error was detected, and the error was at byte 23. This
|
|
is the slash ('/') in header name "header/bizarre", which is not a valid
|
|
HTTP character for a header name.
|
|
|
|
show events [<sink>] [-w] [-n]
|
|
With no option, this lists all known event sinks and their types. With an
|
|
option, it will dump all available events in the designated sink if it is of
|
|
type buffer. If option "-w" is passed after the sink name, then once the end
|
|
of the buffer is reached, the command will wait for new events and display
|
|
them. It is possible to stop the operation by entering any input (which will
|
|
be discarded) or by closing the session. Finally, option "-n" is used to
|
|
directly seek to the end of the buffer, which is often convenient when
|
|
combined with "-w" to only report new events. For convenience, "-wn" or "-nw"
|
|
may be used to enable both options at once.
|
|
|
|
show fd [<fd>]
|
|
Dump the list of either all open file descriptors or just the one number <fd>
|
|
if specified. This is only aimed at developers who need to observe internal
|
|
states in order to debug complex issues such as abnormal CPU usages. One fd
|
|
is reported per lines, and for each of them, its state in the poller using
|
|
upper case letters for enabled flags and lower case for disabled flags, using
|
|
"P" for "polled", "R" for "ready", "A" for "active", the events status using
|
|
"H" for "hangup", "E" for "error", "O" for "output", "P" for "priority" and
|
|
"I" for "input", a few other flags like "N" for "new" (just added into the fd
|
|
cache), "U" for "updated" (received an update in the fd cache), "L" for
|
|
"linger_risk", "C" for "cloned", then the cached entry position, the pointer
|
|
to the internal owner, the pointer to the I/O callback and its name when
|
|
known. When the owner is a connection, the connection flags, and the target
|
|
are reported (frontend, proxy or server). When the owner is a listener, the
|
|
listener's state and its frontend are reported. There is no point in using
|
|
this command without a good knowledge of the internals. It's worth noting
|
|
that the output format may evolve over time so this output must not be parsed
|
|
by tools designed to be durable. Some internal structure states may look
|
|
suspicious to the function listing them, in this case the output line will be
|
|
suffixed with an exclamation mark ('!'). This may help find a starting point
|
|
when trying to diagnose an incident.
|
|
|
|
show info [typed|json] [desc] [float]
|
|
Dump info about haproxy status on current process. If "typed" is passed as an
|
|
optional argument, field numbers, names and types are emitted as well so that
|
|
external monitoring products can easily retrieve, possibly aggregate, then
|
|
report information found in fields they don't know. Each field is dumped on
|
|
its own line. If "json" is passed as an optional argument then
|
|
information provided by "typed" output is provided in JSON format as a
|
|
list of JSON objects. By default, the format contains only two columns
|
|
delimited by a colon (':'). The left one is the field name and the right
|
|
one is the value. It is very important to note that in typed output
|
|
format, the dump for a single object is contiguous so that there is no
|
|
need for a consumer to store everything at once. If "float" is passed as an
|
|
optional argument, some fields usually emitted as integers may switch to
|
|
floats for higher accuracy. It is purposely unspecified which ones are
|
|
concerned as this might evolve over time. Using this option implies that the
|
|
consumer is able to process floats. The output format used is sprintf("%f").
|
|
|
|
When using the typed output format, each line is made of 4 columns delimited
|
|
by colons (':'). The first column is a dot-delimited series of 3 elements. The
|
|
first element is the numeric position of the field in the list (starting at
|
|
zero). This position shall not change over time, but holes are to be expected,
|
|
depending on build options or if some fields are deleted in the future. The
|
|
second element is the field name as it appears in the default "show info"
|
|
output. The third element is the relative process number starting at 1.
|
|
|
|
The rest of the line starting after the first colon follows the "typed output
|
|
format" described in the section above. In short, the second column (after the
|
|
first ':') indicates the origin, nature and scope of the variable. The third
|
|
column indicates the type of the field, among "s32", "s64", "u32", "u64" and
|
|
"str". Then the fourth column is the value itself, which the consumer knows
|
|
how to parse thanks to column 3 and how to process thanks to column 2.
|
|
|
|
Thus the overall line format in typed mode is :
|
|
|
|
<field_pos>.<field_name>.<process_num>:<tags>:<type>:<value>
|
|
|
|
When "desc" is appended to the command, one extra colon followed by a quoted
|
|
string is appended with a description for the metric. At the time of writing,
|
|
this is only supported for the "typed" and default output formats.
|
|
|
|
Example :
|
|
|
|
> show info
|
|
Name: HAProxy
|
|
Version: 1.7-dev1-de52ea-146
|
|
Release_date: 2016/03/11
|
|
Nbproc: 1
|
|
Process_num: 1
|
|
Pid: 28105
|
|
Uptime: 0d 0h00m04s
|
|
Uptime_sec: 4
|
|
Memmax_MB: 0
|
|
PoolAlloc_MB: 0
|
|
PoolUsed_MB: 0
|
|
PoolFailed: 0
|
|
(...)
|
|
|
|
> show info typed
|
|
0.Name.1:POS:str:HAProxy
|
|
1.Version.1:POS:str:1.7-dev1-de52ea-146
|
|
2.Release_date.1:POS:str:2016/03/11
|
|
3.Nbproc.1:CGS:u32:1
|
|
4.Process_num.1:KGP:u32:1
|
|
5.Pid.1:SGP:u32:28105
|
|
6.Uptime.1:MDP:str:0d 0h00m08s
|
|
7.Uptime_sec.1:MDP:u32:8
|
|
8.Memmax_MB.1:CLP:u32:0
|
|
9.PoolAlloc_MB.1:MGP:u32:0
|
|
10.PoolUsed_MB.1:MGP:u32:0
|
|
11.PoolFailed.1:MCP:u32:0
|
|
(...)
|
|
|
|
In the typed format, the presence of the process ID at the end of the
|
|
first column makes it very easy to visually aggregate outputs from
|
|
multiple processes.
|
|
Example :
|
|
|
|
$ ( echo show info typed | socat /var/run/haproxy.sock1 ; \
|
|
echo show info typed | socat /var/run/haproxy.sock2 ) | \
|
|
sort -t . -k 1,1n -k 2,2 -k 3,3n
|
|
0.Name.1:POS:str:HAProxy
|
|
0.Name.2:POS:str:HAProxy
|
|
1.Version.1:POS:str:1.7-dev1-868ab3-148
|
|
1.Version.2:POS:str:1.7-dev1-868ab3-148
|
|
2.Release_date.1:POS:str:2016/03/11
|
|
2.Release_date.2:POS:str:2016/03/11
|
|
3.Nbproc.1:CGS:u32:2
|
|
3.Nbproc.2:CGS:u32:2
|
|
4.Process_num.1:KGP:u32:1
|
|
4.Process_num.2:KGP:u32:2
|
|
5.Pid.1:SGP:u32:30120
|
|
5.Pid.2:SGP:u32:30121
|
|
6.Uptime.1:MDP:str:0d 0h01m28s
|
|
6.Uptime.2:MDP:str:0d 0h01m28s
|
|
(...)
|
|
|
|
The format of JSON output is described in a schema which may be output
|
|
using "show schema json".
|
|
|
|
The JSON output contains no extra whitespace in order to reduce the
|
|
volume of output. For human consumption passing the output through a
|
|
pretty printer may be helpful. Example :
|
|
|
|
$ echo "show info json" | socat /var/run/haproxy.sock stdio | \
|
|
python -m json.tool
|
|
|
|
The JSON output contains no extra whitespace in order to reduce the
|
|
volume of output. For human consumption passing the output through a
|
|
pretty printer may be helpful. Example :
|
|
|
|
$ echo "show info json" | socat /var/run/haproxy.sock stdio | \
|
|
python -m json.tool
|
|
|
|
show map [[@<ver>] <map>]
|
|
Dump info about map converters. Without argument, the list of all available
|
|
maps is returned. If a <map> is specified, its contents are dumped. <map> is
|
|
the #<id> or <file>. By default the current version of the map is shown (the
|
|
version currently being matched against and reported as 'curr_ver' in the map
|
|
list). It is possible to instead dump other versions by prepending '@<ver>'
|
|
before the map's identifier. The version works as a filter and non-existing
|
|
versions will simply report no result. The 'entry_cnt' value represents the
|
|
count of all the map entries, not just the active ones, which means that it
|
|
also includes entries currently being added.
|
|
|
|
In the output, the first column is a unique entry identifier, which is usable
|
|
as a reference for operations "del map" and "set map". The second column is
|
|
the pattern and the third column is the sample if available. The data returned
|
|
are not directly a list of available maps, but are the list of all patterns
|
|
composing any map. Many of these patterns can be shared with ACL.
|
|
|
|
show peers [dict|-] [<peers section>]
|
|
Dump info about the peers configured in "peers" sections. Without argument,
|
|
the list of the peers belonging to all the "peers" sections are listed. If
|
|
<peers section> is specified, only the information about the peers belonging
|
|
to this "peers" section are dumped. When "dict" is specified before the peers
|
|
section name, the entire Tx/Rx dictionary caches will also be dumped (very
|
|
large). Passing "-" may be required to dump a peers section called "dict".
|
|
|
|
Here are two examples of outputs where hostA, hostB and hostC peers belong to
|
|
"sharedlb" peers sections. Only hostA and hostB are connected. Only hostA has
|
|
sent data to hostB.
|
|
|
|
$ echo "show peers" | socat - /tmp/hostA
|
|
0x55deb0224320: [15/Apr/2019:11:28:01] id=sharedlb state=0 flags=0x3 \
|
|
resync_timeout=<PAST> task_calls=45122
|
|
0x55deb022b540: id=hostC(remote) addr=127.0.0.12:10002 status=CONN \
|
|
reconnect=4s confirm=0
|
|
flags=0x0
|
|
0x55deb022a440: id=hostA(local) addr=127.0.0.10:10000 status=NONE \
|
|
reconnect=<NEVER> confirm=0
|
|
flags=0x0
|
|
0x55deb0227d70: id=hostB(remote) addr=127.0.0.11:10001 status=ESTA
|
|
reconnect=2s confirm=0
|
|
flags=0x20000200 appctx:0x55deb028fba0 st0=7 st1=0 task_calls=14456 \
|
|
state=EST
|
|
xprt=RAW src=127.0.0.1:37257 addr=127.0.0.10:10000
|
|
remote_table:0x55deb0224a10 id=stkt local_id=1 remote_id=1
|
|
last_local_table:0x55deb0224a10 id=stkt local_id=1 remote_id=1
|
|
shared tables:
|
|
0x55deb0224a10 local_id=1 remote_id=1 flags=0x0 remote_data=0x65
|
|
last_acked=0 last_pushed=3 last_get=0 teaching_origin=0 update=3
|
|
table:0x55deb022d6a0 id=stkt update=3 localupdate=3 \
|
|
commitupdate=3 syncing=0
|
|
|
|
$ echo "show peers" | socat - /tmp/hostB
|
|
0x55871b5ab320: [15/Apr/2019:11:28:03] id=sharedlb state=0 flags=0x3 \
|
|
resync_timeout=<PAST> task_calls=3
|
|
0x55871b5b2540: id=hostC(remote) addr=127.0.0.12:10002 status=CONN \
|
|
reconnect=3s confirm=0
|
|
flags=0x0
|
|
0x55871b5b1440: id=hostB(local) addr=127.0.0.11:10001 status=NONE \
|
|
reconnect=<NEVER> confirm=0
|
|
flags=0x0
|
|
0x55871b5aed70: id=hostA(remote) addr=127.0.0.10:10000 status=ESTA \
|
|
reconnect=2s confirm=0
|
|
flags=0x20000200 appctx:0x7fa46800ee00 st0=7 st1=0 task_calls=62356 \
|
|
state=EST
|
|
remote_table:0x55871b5ab960 id=stkt local_id=1 remote_id=1
|
|
last_local_table:0x55871b5ab960 id=stkt local_id=1 remote_id=1
|
|
shared tables:
|
|
0x55871b5ab960 local_id=1 remote_id=1 flags=0x0 remote_data=0x65
|
|
last_acked=3 last_pushed=0 last_get=3 teaching_origin=0 update=0
|
|
table:0x55871b5b46a0 id=stkt update=1 localupdate=0 \
|
|
commitupdate=0 syncing=0
|
|
|
|
show pools
|
|
Dump the status of internal memory pools. This is useful to track memory
|
|
usage when suspecting a memory leak for example. It does exactly the same
|
|
as the SIGQUIT when running in foreground except that it does not flush
|
|
the pools.
|
|
|
|
show profiling [{all | status | tasks | memory}] [byaddr] [<max_lines>]
|
|
Dumps the current profiling settings, one per line, as well as the command
|
|
needed to change them. When tasks profiling is enabled, some per-function
|
|
statistics collected by the scheduler will also be emitted, with a summary
|
|
covering the number of calls, total/avg CPU time and total/avg latency. When
|
|
memory profiling is enabled, some information such as the number of
|
|
allocations/releases and their sizes will be reported. It is possible to
|
|
limit the dump to only the profiling status, the tasks, or the memory
|
|
profiling by specifying the respective keywords; by default all profiling
|
|
information are dumped. It is also possible to limit the number of lines
|
|
of output of each category by specifying a numeric limit. If is possible to
|
|
request that the output is sorted by address instead of usage, e.g. to ease
|
|
comparisons between subsequent calls. Please note that profiling is
|
|
essentially aimed at developers since it gives hints about where CPU cycles
|
|
or memory are wasted in the code. There is nothing useful to monitor there.
|
|
|
|
show resolvers [<resolvers section id>]
|
|
Dump statistics for the given resolvers section, or all resolvers sections
|
|
if no section is supplied.
|
|
|
|
For each name server, the following counters are reported:
|
|
sent: number of DNS requests sent to this server
|
|
valid: number of DNS valid responses received from this server
|
|
update: number of DNS responses used to update the server's IP address
|
|
cname: number of CNAME responses
|
|
cname_error: CNAME errors encountered with this server
|
|
any_err: number of empty response (IE: server does not support ANY type)
|
|
nx: non existent domain response received from this server
|
|
timeout: how many time this server did not answer in time
|
|
refused: number of requests refused by this server
|
|
other: any other DNS errors
|
|
invalid: invalid DNS response (from a protocol point of view)
|
|
too_big: too big response
|
|
outdated: number of response arrived too late (after an other name server)
|
|
|
|
show servers conn [<backend>]
|
|
Dump the current and idle connections state of the servers belonging to the
|
|
designated backend (or all backends if none specified). A backend name or
|
|
identifier may be used.
|
|
|
|
The output consists in a header line showing the fields titles, then one
|
|
server per line with for each, the backend name and ID, server name and ID,
|
|
the address, port and a series or values. The number of fields varies
|
|
depending on thread count.
|
|
|
|
Given the threaded nature of idle connections, it's important to understand
|
|
that some values may change once read, and that as such, consistency within a
|
|
line isn't granted. This output is mostly provided as a debugging tool and is
|
|
not relevant to be routinely monitored nor graphed.
|
|
|
|
show servers state [<backend>]
|
|
Dump the state of the servers found in the running configuration. A backend
|
|
name or identifier may be provided to limit the output to this backend only.
|
|
|
|
The dump has the following format:
|
|
- first line contains the format version (1 in this specification);
|
|
- second line contains the column headers, prefixed by a sharp ('#');
|
|
- third line and next ones contain data;
|
|
- each line starting by a sharp ('#') is considered as a comment.
|
|
|
|
Since multiple versions of the output may co-exist, below is the list of
|
|
fields and their order per file format version :
|
|
1:
|
|
be_id: Backend unique id.
|
|
be_name: Backend label.
|
|
srv_id: Server unique id (in the backend).
|
|
srv_name: Server label.
|
|
srv_addr: Server IP address.
|
|
srv_op_state: Server operational state (UP/DOWN/...).
|
|
0 = SRV_ST_STOPPED
|
|
The server is down.
|
|
1 = SRV_ST_STARTING
|
|
The server is warming up (up but
|
|
throttled).
|
|
2 = SRV_ST_RUNNING
|
|
The server is fully up.
|
|
3 = SRV_ST_STOPPING
|
|
The server is up but soft-stopping
|
|
(eg: 404).
|
|
srv_admin_state: Server administrative state (MAINT/DRAIN/...).
|
|
The state is actually a mask of values :
|
|
0x01 = SRV_ADMF_FMAINT
|
|
The server was explicitly forced into
|
|
maintenance.
|
|
0x02 = SRV_ADMF_IMAINT
|
|
The server has inherited the maintenance
|
|
status from a tracked server.
|
|
0x04 = SRV_ADMF_CMAINT
|
|
The server is in maintenance because of
|
|
the configuration.
|
|
0x08 = SRV_ADMF_FDRAIN
|
|
The server was explicitly forced into
|
|
drain state.
|
|
0x10 = SRV_ADMF_IDRAIN
|
|
The server has inherited the drain status
|
|
from a tracked server.
|
|
0x20 = SRV_ADMF_RMAINT
|
|
The server is in maintenance because of an
|
|
IP address resolution failure.
|
|
0x40 = SRV_ADMF_HMAINT
|
|
The server FQDN was set from stats socket.
|
|
|
|
srv_uweight: User visible server's weight.
|
|
srv_iweight: Server's initial weight.
|
|
srv_time_since_last_change: Time since last operational change.
|
|
srv_check_status: Last health check status.
|
|
srv_check_result: Last check result (FAILED/PASSED/...).
|
|
0 = CHK_RES_UNKNOWN
|
|
Initialized to this by default.
|
|
1 = CHK_RES_NEUTRAL
|
|
Valid check but no status information.
|
|
2 = CHK_RES_FAILED
|
|
Check failed.
|
|
3 = CHK_RES_PASSED
|
|
Check succeeded and server is fully up
|
|
again.
|
|
4 = CHK_RES_CONDPASS
|
|
Check reports the server doesn't want new
|
|
sessions.
|
|
srv_check_health: Checks rise / fall current counter.
|
|
srv_check_state: State of the check (ENABLED/PAUSED/...).
|
|
The state is actually a mask of values :
|
|
0x01 = CHK_ST_INPROGRESS
|
|
A check is currently running.
|
|
0x02 = CHK_ST_CONFIGURED
|
|
This check is configured and may be
|
|
enabled.
|
|
0x04 = CHK_ST_ENABLED
|
|
This check is currently administratively
|
|
enabled.
|
|
0x08 = CHK_ST_PAUSED
|
|
Checks are paused because of maintenance
|
|
(health only).
|
|
srv_agent_state: State of the agent check (ENABLED/PAUSED/...).
|
|
This state uses the same mask values as
|
|
"srv_check_state", adding this specific one :
|
|
0x10 = CHK_ST_AGENT
|
|
Check is an agent check (otherwise it's a
|
|
health check).
|
|
bk_f_forced_id: Flag to know if the backend ID is forced by
|
|
configuration.
|
|
srv_f_forced_id: Flag to know if the server's ID is forced by
|
|
configuration.
|
|
srv_fqdn: Server FQDN.
|
|
srv_port: Server port.
|
|
srvrecord: DNS SRV record associated to this SRV.
|
|
srv_use_ssl: use ssl for server connections.
|
|
srv_check_port: Server health check port.
|
|
srv_check_addr: Server health check address.
|
|
srv_agent_addr: Server health agent address.
|
|
srv_agent_port: Server health agent port.
|
|
|
|
show sess
|
|
Dump all known sessions. Avoid doing this on slow connections as this can
|
|
be huge. This command is restricted and can only be issued on sockets
|
|
configured for levels "operator" or "admin". Note that on machines with
|
|
quickly recycled connections, it is possible that this output reports less
|
|
entries than really exist because it will dump all existing sessions up to
|
|
the last one that was created before the command was entered; those which
|
|
die in the mean time will not appear.
|
|
|
|
show sess <id>
|
|
Display a lot of internal information about the specified session identifier.
|
|
This identifier is the first field at the beginning of the lines in the dumps
|
|
of "show sess" (it corresponds to the session pointer). Those information are
|
|
useless to most users but may be used by haproxy developers to troubleshoot a
|
|
complex bug. The output format is intentionally not documented so that it can
|
|
freely evolve depending on demands. You may find a description of all fields
|
|
returned in src/dumpstats.c
|
|
|
|
The special id "all" dumps the states of all sessions, which must be avoided
|
|
as much as possible as it is highly CPU intensive and can take a lot of time.
|
|
|
|
show stat [domain <dns|proxy>] [{<iid>|<proxy>} <type> <sid>] [typed|json] \
|
|
[desc] [up|no-maint]
|
|
Dump statistics. The domain is used to select which statistics to print; dns
|
|
and proxy are available for now. By default, the CSV format is used; you can
|
|
activate the extended typed output format described in the section above if
|
|
"typed" is passed after the other arguments; or in JSON if "json" is passed
|
|
after the other arguments. By passing <id>, <type> and <sid>, it is possible
|
|
to dump only selected items :
|
|
- <iid> is a proxy ID, -1 to dump everything. Alternatively, a proxy name
|
|
<proxy> may be specified. In this case, this proxy's ID will be used as
|
|
the ID selector.
|
|
- <type> selects the type of dumpable objects : 1 for frontends, 2 for
|
|
backends, 4 for servers, -1 for everything. These values can be ORed,
|
|
for example:
|
|
1 + 2 = 3 -> frontend + backend.
|
|
1 + 2 + 4 = 7 -> frontend + backend + server.
|
|
- <sid> is a server ID, -1 to dump everything from the selected proxy.
|
|
|
|
Example :
|
|
$ echo "show info;show stat" | socat stdio unix-connect:/tmp/sock1
|
|
>>> Name: HAProxy
|
|
Version: 1.4-dev2-49
|
|
Release_date: 2009/09/23
|
|
Nbproc: 1
|
|
Process_num: 1
|
|
(...)
|
|
|
|
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq, (...)
|
|
stats,FRONTEND,,,0,0,1000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0, (...)
|
|
stats,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,250,(...)
|
|
(...)
|
|
www1,BACKEND,0,0,0,0,1000,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,250, (...)
|
|
|
|
$
|
|
|
|
In this example, two commands have been issued at once. That way it's easy to
|
|
find which process the stats apply to in multi-process mode. This is not
|
|
needed in the typed output format as the process number is reported on each
|
|
line. Notice the empty line after the information output which marks the end
|
|
of the first block. A similar empty line appears at the end of the second
|
|
block (stats) so that the reader knows the output has not been truncated.
|
|
|
|
When "typed" is specified, the output format is more suitable to monitoring
|
|
tools because it provides numeric positions and indicates the type of each
|
|
output field. Each value stands on its own line with process number, element
|
|
number, nature, origin and scope. This same format is available via the HTTP
|
|
stats by passing ";typed" after the URI. It is very important to note that in
|
|
typed output format, the dump for a single object is contiguous so that there
|
|
is no need for a consumer to store everything at once.
|
|
|
|
The "up" modifier will result in listing only servers which reportedly up or
|
|
not checked. Those down, unresolved, or in maintenance will not be listed.
|
|
This is analogous to the ";up" option on the HTTP stats. Similarly, the
|
|
"no-maint" modifier will act like the ";no-maint" HTTP modifier and will
|
|
result in disabled servers not to be listed. The difference is that those
|
|
which are enabled but down will not be evicted.
|
|
|
|
When using the typed output format, each line is made of 4 columns delimited
|
|
by colons (':'). The first column is a dot-delimited series of 5 elements. The
|
|
first element is a letter indicating the type of the object being described.
|
|
At the moment the following object types are known : 'F' for a frontend, 'B'
|
|
for a backend, 'L' for a listener, and 'S' for a server. The second element
|
|
The second element is a positive integer representing the unique identifier of
|
|
the proxy the object belongs to. It is equivalent to the "iid" column of the
|
|
CSV output and matches the value in front of the optional "id" directive found
|
|
in the frontend or backend section. The third element is a positive integer
|
|
containing the unique object identifier inside the proxy, and corresponds to
|
|
the "sid" column of the CSV output. ID 0 is reported when dumping a frontend
|
|
or a backend. For a listener or a server, this corresponds to their respective
|
|
ID inside the proxy. The fourth element is the numeric position of the field
|
|
in the list (starting at zero). This position shall not change over time, but
|
|
holes are to be expected, depending on build options or if some fields are
|
|
deleted in the future. The fifth element is the field name as it appears in
|
|
the CSV output. The sixth element is a positive integer and is the relative
|
|
process number starting at 1.
|
|
|
|
The rest of the line starting after the first colon follows the "typed output
|
|
format" described in the section above. In short, the second column (after the
|
|
first ':') indicates the origin, nature and scope of the variable. The third
|
|
column indicates the field type, among "s32", "s64", "u32", "u64", "flt' and
|
|
"str". Then the fourth column is the value itself, which the consumer knows
|
|
how to parse thanks to column 3 and how to process thanks to column 2.
|
|
|
|
When "desc" is appended to the command, one extra colon followed by a quoted
|
|
string is appended with a description for the metric. At the time of writing,
|
|
this is only supported for the "typed" output format.
|
|
|
|
Thus the overall line format in typed mode is :
|
|
|
|
<obj>.<px_id>.<id>.<fpos>.<fname>.<process_num>:<tags>:<type>:<value>
|
|
|
|
Here's an example of typed output format :
|
|
|
|
$ echo "show stat typed" | socat stdio unix-connect:/tmp/sock1
|
|
F.2.0.0.pxname.1:MGP:str:private-frontend
|
|
F.2.0.1.svname.1:MGP:str:FRONTEND
|
|
F.2.0.8.bin.1:MGP:u64:0
|
|
F.2.0.9.bout.1:MGP:u64:0
|
|
F.2.0.40.hrsp_2xx.1:MGP:u64:0
|
|
L.2.1.0.pxname.1:MGP:str:private-frontend
|
|
L.2.1.1.svname.1:MGP:str:sock-1
|
|
L.2.1.17.status.1:MGP:str:OPEN
|
|
L.2.1.73.addr.1:MGP:str:0.0.0.0:8001
|
|
S.3.13.60.rtime.1:MCP:u32:0
|
|
S.3.13.61.ttime.1:MCP:u32:0
|
|
S.3.13.62.agent_status.1:MGP:str:L4TOUT
|
|
S.3.13.64.agent_duration.1:MGP:u64:2001
|
|
S.3.13.65.check_desc.1:MCP:str:Layer4 timeout
|
|
S.3.13.66.agent_desc.1:MCP:str:Layer4 timeout
|
|
S.3.13.67.check_rise.1:MCP:u32:2
|
|
S.3.13.68.check_fall.1:MCP:u32:3
|
|
S.3.13.69.check_health.1:SGP:u32:0
|
|
S.3.13.70.agent_rise.1:MaP:u32:1
|
|
S.3.13.71.agent_fall.1:SGP:u32:1
|
|
S.3.13.72.agent_health.1:SGP:u32:1
|
|
S.3.13.73.addr.1:MCP:str:1.255.255.255:8888
|
|
S.3.13.75.mode.1:MAP:str:http
|
|
B.3.0.0.pxname.1:MGP:str:private-backend
|
|
B.3.0.1.svname.1:MGP:str:BACKEND
|
|
B.3.0.2.qcur.1:MGP:u32:0
|
|
B.3.0.3.qmax.1:MGP:u32:0
|
|
B.3.0.4.scur.1:MGP:u32:0
|
|
B.3.0.5.smax.1:MGP:u32:0
|
|
B.3.0.6.slim.1:MGP:u32:1000
|
|
B.3.0.55.lastsess.1:MMP:s32:-1
|
|
(...)
|
|
|
|
In the typed format, the presence of the process ID at the end of the
|
|
first column makes it very easy to visually aggregate outputs from
|
|
multiple processes, as show in the example below where each line appears
|
|
for each process :
|
|
|
|
$ ( echo show stat typed | socat /var/run/haproxy.sock1 - ; \
|
|
echo show stat typed | socat /var/run/haproxy.sock2 - ) | \
|
|
sort -t . -k 1,1 -k 2,2n -k 3,3n -k 4,4n -k 5,5 -k 6,6n
|
|
B.3.0.0.pxname.1:MGP:str:private-backend
|
|
B.3.0.0.pxname.2:MGP:str:private-backend
|
|
B.3.0.1.svname.1:MGP:str:BACKEND
|
|
B.3.0.1.svname.2:MGP:str:BACKEND
|
|
B.3.0.2.qcur.1:MGP:u32:0
|
|
B.3.0.2.qcur.2:MGP:u32:0
|
|
B.3.0.3.qmax.1:MGP:u32:0
|
|
B.3.0.3.qmax.2:MGP:u32:0
|
|
B.3.0.4.scur.1:MGP:u32:0
|
|
B.3.0.4.scur.2:MGP:u32:0
|
|
B.3.0.5.smax.1:MGP:u32:0
|
|
B.3.0.5.smax.2:MGP:u32:0
|
|
B.3.0.6.slim.1:MGP:u32:1000
|
|
B.3.0.6.slim.2:MGP:u32:1000
|
|
(...)
|
|
|
|
The format of JSON output is described in a schema which may be output
|
|
using "show schema json".
|
|
|
|
The JSON output contains no extra whitespace in order to reduce the
|
|
volume of output. For human consumption passing the output through a
|
|
pretty printer may be helpful. Example :
|
|
|
|
$ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
|
|
python -m json.tool
|
|
|
|
The JSON output contains no extra whitespace in order to reduce the
|
|
volume of output. For human consumption passing the output through a
|
|
pretty printer may be helpful. Example :
|
|
|
|
$ echo "show stat json" | socat /var/run/haproxy.sock stdio | \
|
|
python -m json.tool
|
|
|
|
show ssl ca-file [<cafile>[:<index>]]
|
|
Display the list of CA files used by HAProxy and their respective certificate
|
|
counts. If a filename is prefixed by an asterisk, it is a transaction which
|
|
is not committed yet. If a <cafile> is specified without <index>, it will show
|
|
the status of the CA file ("Used"/"Unused") followed by details about all the
|
|
certificates contained in the CA file. The details displayed for every
|
|
certificate are the same as the ones displayed by a "show ssl cert" command.
|
|
If a <cafile> is specified followed by an <index>, it will only display the
|
|
details of the certificate having the specified index. Indexes start from 1.
|
|
If the index is invalid (too big for instance), nothing will be displayed.
|
|
This command can be useful to check if a CA file was properly updated.
|
|
You can also display the details of an ongoing transaction by prefixing the
|
|
filename by an asterisk.
|
|
|
|
Example :
|
|
|
|
$ echo "show ssl ca-file" | socat /var/run/haproxy.master -
|
|
# transaction
|
|
*cafile.crt - 2 certificate(s)
|
|
# filename
|
|
cafile.crt - 1 certificate(s)
|
|
|
|
$ echo "show ssl ca-file cafile.crt" | socat /var/run/haproxy.master -
|
|
Filename: /home/tricot/work/haproxy/reg-tests/ssl/set_cafile_ca2.crt
|
|
Status: Used
|
|
|
|
Certificate #1:
|
|
Serial: 11A4D2200DC84376E7D233CAFF39DF44BF8D1211
|
|
notBefore: Apr 1 07:40:53 2021 GMT
|
|
notAfter: Aug 17 07:40:53 2048 GMT
|
|
Subject Alternative Name:
|
|
Algorithm: RSA4096
|
|
SHA1 FingerPrint: A111EF0FEFCDE11D47FE3F33ADCA8435EBEA4864
|
|
Subject: /C=FR/ST=Some-State/O=HAProxy Technologies/CN=HAProxy Technologies CA
|
|
Issuer: /C=FR/ST=Some-State/O=HAProxy Technologies/CN=HAProxy Technologies CA
|
|
|
|
$ echo "show ssl ca-file *cafile.crt:2" | socat /var/run/haproxy.master -
|
|
Filename: */home/tricot/work/haproxy/reg-tests/ssl/set_cafile_ca2.crt
|
|
Status: Unused
|
|
|
|
Certificate #2:
|
|
Serial: 587A1CE5ED855040A0C82BF255FF300ADB7C8136
|
|
[...]
|
|
|
|
show ssl cert [<filename>]
|
|
Display the list of certificates used on frontends and backends.
|
|
If a filename is prefixed by an asterisk, it is a transaction which is not
|
|
committed yet. If a filename is specified, it will show details about the
|
|
certificate. This command can be useful to check if a certificate was well
|
|
updated. You can also display details on a transaction by prefixing the
|
|
filename by an asterisk.
|
|
This command can also be used to display the details of a certificate's OCSP
|
|
response by suffixing the filename with a ".ocsp" extension. It works for
|
|
committed certificates as well as for ongoing transactions. On a committed
|
|
certificate, this command is equivalent to calling "show ssl ocsp-response"
|
|
with the certificate's corresponding OCSP response ID.
|
|
|
|
Example :
|
|
|
|
$ echo "@1 show ssl cert" | socat /var/run/haproxy.master -
|
|
# transaction
|
|
*test.local.pem
|
|
# filename
|
|
test.local.pem
|
|
|
|
$ echo "@1 show ssl cert test.local.pem" | socat /var/run/haproxy.master -
|
|
Filename: test.local.pem
|
|
Serial: 03ECC19BA54B25E85ABA46EE561B9A10D26F
|
|
notBefore: Sep 13 21:20:24 2019 GMT
|
|
notAfter: Dec 12 21:20:24 2019 GMT
|
|
Issuer: /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
|
|
Subject: /CN=test.local
|
|
Subject Alternative Name: DNS:test.local, DNS:imap.test.local
|
|
Algorithm: RSA2048
|
|
SHA1 FingerPrint: 417A11CAE25F607B24F638B4A8AEE51D1E211477
|
|
|
|
$ echo "@1 show ssl cert *test.local.pem" | socat /var/run/haproxy.master -
|
|
Filename: *test.local.pem
|
|
[...]
|
|
|
|
show ssl crl-file [<crlfile>[:<index>]]
|
|
Display the list of CRL files used by HAProxy.
|
|
If a filename is prefixed by an asterisk, it is a transaction which is not
|
|
committed yet. If a <crlfile> is specified without <index>, it will show the
|
|
status of the CRL file ("Used"/"Unused") followed by details about all the
|
|
Revocation Lists contained in the CRL file. The details displayed for every
|
|
list are based on the output of "openssl crl -text -noout -in <file>".
|
|
If a <crlfile> is specified followed by an <index>, it will only display the
|
|
details of the list having the specified index. Indexes start from 1.
|
|
If the index is invalid (too big for instance), nothing will be displayed.
|
|
This command can be useful to check if a CRL file was properly updated.
|
|
You can also display the details of an ongoing transaction by prefixing the
|
|
filename by an asterisk.
|
|
|
|
Example :
|
|
|
|
$ echo "show ssl crl-file" | socat /var/run/haproxy.master -
|
|
# transaction
|
|
*crlfile.pem
|
|
# filename
|
|
crlfile.pem
|
|
|
|
$ echo "show ssl crl-file crlfile.pem" | socat /var/run/haproxy.master -
|
|
Filename: /home/tricot/work/haproxy/reg-tests/ssl/crlfile.pem
|
|
Status: Used
|
|
|
|
Certificate Revocation List #1:
|
|
Version 1
|
|
Signature Algorithm: sha256WithRSAEncryption
|
|
Issuer: /C=FR/O=HAProxy Technologies/CN=Intermediate CA2
|
|
Last Update: Apr 23 14:45:39 2021 GMT
|
|
Next Update: Sep 8 14:45:39 2048 GMT
|
|
Revoked Certificates:
|
|
Serial Number: 1008
|
|
Revocation Date: Apr 23 14:45:36 2021 GMT
|
|
|
|
Certificate Revocation List #2:
|
|
Version 1
|
|
Signature Algorithm: sha256WithRSAEncryption
|
|
Issuer: /C=FR/O=HAProxy Technologies/CN=Root CA
|
|
Last Update: Apr 23 14:30:44 2021 GMT
|
|
Next Update: Sep 8 14:30:44 2048 GMT
|
|
No Revoked Certificates.
|
|
|
|
show ssl crt-list [-n] [<filename>]
|
|
Display the list of crt-list and directories used in the HAProxy
|
|
configuration. If a filename is specified, dump the content of a crt-list or
|
|
a directory. Once dumped the output can be used as a crt-list file.
|
|
The '-n' option can be used to display the line number, which is useful when
|
|
combined with the 'del ssl crt-list' option when a entry is duplicated. The
|
|
output with the '-n' option is not compatible with the crt-list format and
|
|
not loadable by haproxy.
|
|
|
|
Example:
|
|
echo "show ssl crt-list -n localhost.crt-list" | socat /tmp/sock1 -
|
|
# localhost.crt-list
|
|
common.pem:1 !not.test1.com *.test1.com !localhost
|
|
common.pem:2
|
|
ecdsa.pem:3 [verify none allow-0rtt ssl-min-ver TLSv1.0 ssl-max-ver TLSv1.3] localhost !www.test1.com
|
|
ecdsa.pem:4 [verify none allow-0rtt ssl-min-ver TLSv1.0 ssl-max-ver TLSv1.3]
|
|
|
|
show ssl ocsp-response [<id>]
|
|
Display the IDs of the OCSP tree entries corresponding to all the OCSP
|
|
responses used in HAProxy, as well as the issuer's name and key hash and the
|
|
serial number of the certificate for which the OCSP response was built.
|
|
If a valid <id> is provided, display the contents of the corresponding OCSP
|
|
response. The information displayed is the same as in an "openssl ocsp -respin
|
|
<ocsp-response> -text" call.
|
|
|
|
Example :
|
|
|
|
$ echo "show ssl ocsp-response" | socat /var/run/haproxy.master -
|
|
# Certificate IDs
|
|
Certificate ID key : 303b300906052b0e03021a050004148a83e0060faff709ca7e9b95522a2e81635fda0a0414f652b0e435d5ea923851508f0adbe92d85de007a0202100a
|
|
Certificate ID:
|
|
|
|
Issuer Name Hash: 8A83E0060FAFF709CA7E9B95522A2E81635FDA0A
|
|
Issuer Key Hash: F652B0E435D5EA923851508F0ADBE92D85DE007A
|
|
Serial Number: 100A
|
|
|
|
$ echo "show ssl ocsp-response 303b300906052b0e03021a050004148a83e0060faff709ca7e9b95522a2e81635fda0a0414f652b0e435d5ea923851508f0adbe92d85de007a0202100a" | socat /var/run/haproxy.master -
|
|
OCSP Response Data:
|
|
OCSP Response Status: successful (0x0)
|
|
Response Type: Basic OCSP Response
|
|
Version: 1 (0x0)
|
|
Responder Id: C = FR, O = HAProxy Technologies, CN = ocsp.haproxy.com
|
|
Produced At: May 27 15:43:38 2021 GMT
|
|
Responses:
|
|
Certificate ID:
|
|
Hash Algorithm: sha1
|
|
Issuer Name Hash: 8A83E0060FAFF709CA7E9B95522A2E81635FDA0A
|
|
Issuer Key Hash: F652B0E435D5EA923851508F0ADBE92D85DE007A
|
|
Serial Number: 100A
|
|
Cert Status: good
|
|
This Update: May 27 15:43:38 2021 GMT
|
|
Next Update: Oct 12 15:43:38 2048 GMT
|
|
[...]
|
|
|
|
show table
|
|
Dump general information on all known stick-tables. Their name is returned
|
|
(the name of the proxy which holds them), their type (currently zero, always
|
|
IP), their size in maximum possible number of entries, and the number of
|
|
entries currently in use.
|
|
|
|
Example :
|
|
$ echo "show table" | socat stdio /tmp/sock1
|
|
>>> # table: front_pub, type: ip, size:204800, used:171454
|
|
>>> # table: back_rdp, type: ip, size:204800, used:0
|
|
|
|
show table <name> [ data.<type> <operator> <value> [data.<type> ...]] | [ key <key> ]
|
|
Dump contents of stick-table <name>. In this mode, a first line of generic
|
|
information about the table is reported as with "show table", then all
|
|
entries are dumped. Since this can be quite heavy, it is possible to specify
|
|
a filter in order to specify what entries to display.
|
|
|
|
When the "data." form is used the filter applies to the stored data (see
|
|
"stick-table" in section 4.2). A stored data type must be specified
|
|
in <type>, and this data type must be stored in the table otherwise an
|
|
error is reported. The data is compared according to <operator> with the
|
|
64-bit integer <value>. Operators are the same as with the ACLs :
|
|
|
|
- eq : match entries whose data is equal to this value
|
|
- ne : match entries whose data is not equal to this value
|
|
- le : match entries whose data is less than or equal to this value
|
|
- ge : match entries whose data is greater than or equal to this value
|
|
- lt : match entries whose data is less than this value
|
|
- gt : match entries whose data is greater than this value
|
|
|
|
In this form, you can use multiple data filter entries, up to a maximum
|
|
defined during build time (4 by default).
|
|
|
|
When the key form is used the entry <key> is shown. The key must be of the
|
|
same type as the table, which currently is limited to IPv4, IPv6, integer,
|
|
and string.
|
|
|
|
Example :
|
|
$ echo "show table http_proxy" | socat stdio /tmp/sock1
|
|
>>> # table: http_proxy, type: ip, size:204800, used:2
|
|
>>> 0x80e6a4c: key=127.0.0.1 use=0 exp=3594729 gpc0=0 conn_rate(30000)=1 \
|
|
bytes_out_rate(60000)=187
|
|
>>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
|
|
bytes_out_rate(60000)=191
|
|
|
|
$ echo "show table http_proxy data.gpc0 gt 0" | socat stdio /tmp/sock1
|
|
>>> # table: http_proxy, type: ip, size:204800, used:2
|
|
>>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
|
|
bytes_out_rate(60000)=191
|
|
|
|
$ echo "show table http_proxy data.conn_rate gt 5" | \
|
|
socat stdio /tmp/sock1
|
|
>>> # table: http_proxy, type: ip, size:204800, used:2
|
|
>>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
|
|
bytes_out_rate(60000)=191
|
|
|
|
$ echo "show table http_proxy key 127.0.0.2" | \
|
|
socat stdio /tmp/sock1
|
|
>>> # table: http_proxy, type: ip, size:204800, used:2
|
|
>>> 0x80e6a80: key=127.0.0.2 use=0 exp=3594740 gpc0=1 conn_rate(30000)=10 \
|
|
bytes_out_rate(60000)=191
|
|
|
|
When the data criterion applies to a dynamic value dependent on time such as
|
|
a bytes rate, the value is dynamically computed during the evaluation of the
|
|
entry in order to decide whether it has to be dumped or not. This means that
|
|
such a filter could match for some time then not match anymore because as
|
|
time goes, the average event rate drops.
|
|
|
|
It is possible to use this to extract lists of IP addresses abusing the
|
|
service, in order to monitor them or even blacklist them in a firewall.
|
|
Example :
|
|
$ echo "show table http_proxy data.gpc0 gt 0" \
|
|
| socat stdio /tmp/sock1 \
|
|
| fgrep 'key=' | cut -d' ' -f2 | cut -d= -f2 > abusers-ip.txt
|
|
( or | awk '/key/{ print a[split($2,a,"=")]; }' )
|
|
|
|
show tasks
|
|
Dumps the number of tasks currently in the run queue, with the number of
|
|
occurrences for each function, and their average latency when it's known
|
|
(for pure tasks with task profiling enabled). The dump is a snapshot of the
|
|
instant it's done, and there may be variations depending on what tasks are
|
|
left in the queue at the moment it happens, especially in mono-thread mode
|
|
as there's less chance that I/Os can refill the queue (unless the queue is
|
|
full). This command takes exclusive access to the process and can cause
|
|
minor but measurable latencies when issued on a highly loaded process, so
|
|
it must not be abused by monitoring bots.
|
|
|
|
show threads
|
|
Dumps some internal states and structures for each thread, that may be useful
|
|
to help developers understand a problem. The output tries to be readable by
|
|
showing one block per thread. When haproxy is built with USE_THREAD_DUMP=1,
|
|
an advanced dump mechanism involving thread signals is used so that each
|
|
thread can dump its own state in turn. Without this option, the thread
|
|
processing the command shows all its details but the other ones are less
|
|
detailed. A star ('*') is displayed in front of the thread handling the
|
|
command. A right angle bracket ('>') may also be displayed in front of
|
|
threads which didn't make any progress since last invocation of this command,
|
|
indicating a bug in the code which must absolutely be reported. When this
|
|
happens between two threads it usually indicates a deadlock. If a thread is
|
|
alone, it's a different bug like a corrupted list. In all cases the process
|
|
needs is not fully functional anymore and needs to be restarted.
|
|
|
|
The output format is purposely not documented so that it can easily evolve as
|
|
new needs are identified, without having to maintain any form of backwards
|
|
compatibility, and just like with "show activity", the values are meaningless
|
|
without the code at hand.
|
|
|
|
show tls-keys [id|*]
|
|
Dump all loaded TLS ticket keys references. The TLS ticket key reference ID
|
|
and the file from which the keys have been loaded is shown. Both of those
|
|
can be used to update the TLS keys using "set ssl tls-key". If an ID is
|
|
specified as parameter, it will dump the tickets, using * it will dump every
|
|
keys from every references.
|
|
|
|
show schema json
|
|
Dump the schema used for the output of "show info json" and "show stat json".
|
|
|
|
The contains no extra whitespace in order to reduce the volume of output.
|
|
For human consumption passing the output through a pretty printer may be
|
|
helpful. Example :
|
|
|
|
$ echo "show schema json" | socat /var/run/haproxy.sock stdio | \
|
|
python -m json.tool
|
|
|
|
The schema follows "JSON Schema" (json-schema.org) and accordingly
|
|
verifiers may be used to verify the output of "show info json" and "show
|
|
stat json" against the schema.
|
|
|
|
show trace [<source>]
|
|
Show the current trace status. For each source a line is displayed with a
|
|
single-character status indicating if the trace is stopped, waiting, or
|
|
running. The output sink used by the trace is indicated (or "none" if none
|
|
was set), as well as the number of dropped events in this sink, followed by a
|
|
brief description of the source. If a source name is specified, a detailed
|
|
list of all events supported by the source, and their status for each action
|
|
(report, start, pause, stop), indicated by a "+" if they are enabled, or a
|
|
"-" otherwise. All these events are independent and an event might trigger
|
|
a start without being reported and conversely.
|
|
|
|
shutdown frontend <frontend>
|
|
Completely delete the specified frontend. All the ports it was bound to will
|
|
be released. It will not be possible to enable the frontend anymore after
|
|
this operation. This is intended to be used in environments where stopping a
|
|
proxy is not even imaginable but a misconfigured proxy must be fixed. That
|
|
way it's possible to release the port and bind it into another process to
|
|
restore operations. The frontend will not appear at all on the stats page
|
|
once it is terminated.
|
|
|
|
The frontend may be specified either by its name or by its numeric ID,
|
|
prefixed with a sharp ('#').
|
|
|
|
This command is restricted and can only be issued on sockets configured for
|
|
level "admin".
|
|
|
|
shutdown session <id>
|
|
Immediately terminate the session matching the specified session identifier.
|
|
This identifier is the first field at the beginning of the lines in the dumps
|
|
of "show sess" (it corresponds to the session pointer). This can be used to
|
|
terminate a long-running session without waiting for a timeout or when an
|
|
endless transfer is ongoing. Such terminated sessions are reported with a 'K'
|
|
flag in the logs.
|
|
|
|
shutdown sessions server <backend>/<server>
|
|
Immediately terminate all the sessions attached to the specified server. This
|
|
can be used to terminate long-running sessions after a server is put into
|
|
maintenance mode, for instance. Such terminated sessions are reported with a
|
|
'K' flag in the logs.
|
|
|
|
trace
|
|
The "trace" command alone lists the trace sources, their current status, and
|
|
their brief descriptions. It is only meant as a menu to enter next levels,
|
|
see other "trace" commands below.
|
|
|
|
trace 0
|
|
Immediately stops all traces. This is made to be used as a quick solution
|
|
to terminate a debugging session or as an emergency action to be used in case
|
|
complex traces were enabled on multiple sources and impact the service.
|
|
|
|
trace <source> event [ [+|-|!]<name> ]
|
|
Without argument, this will list all the events supported by the designated
|
|
source. They are prefixed with a "-" if they are not enabled, or a "+" if
|
|
they are enabled. It is important to note that a single trace may be labelled
|
|
with multiple events, and as long as any of the enabled events matches one of
|
|
the events labelled on the trace, the event will be passed to the trace
|
|
subsystem. For example, receiving an HTTP/2 frame of type HEADERS may trigger
|
|
a frame event and a stream event since the frame creates a new stream. If
|
|
either the frame event or the stream event are enabled for this source, the
|
|
frame will be passed to the trace framework.
|
|
|
|
With an argument, it is possible to toggle the state of each event and
|
|
individually enable or disable them. Two special keywords are supported,
|
|
"none", which matches no event, and is used to disable all events at once,
|
|
and "any" which matches all events, and is used to enable all events at
|
|
once. Other events are specific to the event source. It is possible to
|
|
enable one event by specifying its name, optionally prefixed with '+' for
|
|
better readability. It is possible to disable one event by specifying its
|
|
name prefixed by a '-' or a '!'.
|
|
|
|
One way to completely disable a trace source is to pass "event none", and
|
|
this source will instantly be totally ignored.
|
|
|
|
trace <source> level [<level>]
|
|
Without argument, this will list all trace levels for this source, and the
|
|
current one will be indicated by a star ('*') prepended in front of it. With
|
|
an argument, this will change the trace level to the specified level. Detail
|
|
levels are a form of filters that are applied before reporting the events.
|
|
These filters are used to selectively include or exclude events depending on
|
|
their level of importance. For example a developer might need to know
|
|
precisely where in the code an HTTP header was considered invalid while the
|
|
end user may not even care about this header's validity at all. There are
|
|
currently 5 distinct levels for a trace :
|
|
|
|
user this will report information that are suitable for use by a
|
|
regular haproxy user who wants to observe his traffic.
|
|
Typically some HTTP requests and responses will be reported
|
|
without much detail. Most sources will set this as the
|
|
default level to ease operations.
|
|
|
|
proto in addition to what is reported at the "user" level, it also
|
|
displays protocol-level updates. This can for example be the
|
|
frame types or HTTP headers after decoding.
|
|
|
|
state in addition to what is reported at the "proto" level, it
|
|
will also display state transitions (or failed transitions)
|
|
which happen in parsers, so this will show attempts to
|
|
perform an operation while the "proto" level only shows
|
|
the final operation.
|
|
|
|
data in addition to what is reported at the "state" level, it
|
|
will also include data transfers between the various layers.
|
|
|
|
developer it reports everything available, which can include advanced
|
|
information such as "breaking out of this loop" that are
|
|
only relevant to a developer trying to understand a bug that
|
|
only happens once in a while in field. Function names are
|
|
only reported at this level.
|
|
|
|
It is highly recommended to always use the "user" level only and switch to
|
|
other levels only if instructed to do so by a developer. Also it is a good
|
|
idea to first configure the events before switching to higher levels, as it
|
|
may save from dumping many lines if no filter is applied.
|
|
|
|
trace <source> lock [criterion]
|
|
Without argument, this will list all the criteria supported by this source
|
|
for lock-on processing, and display the current choice by a star ('*') in
|
|
front of it. Lock-on means that the source will focus on the first matching
|
|
event and only stick to the criterion which triggered this event, and ignore
|
|
all other ones until the trace stops. This allows for example to take a trace
|
|
on a single connection or on a single stream. The following criteria are
|
|
supported by some traces, though not necessarily all, since some of them
|
|
might not be available to the source :
|
|
|
|
backend lock on the backend that started the trace
|
|
connection lock on the connection that started the trace
|
|
frontend lock on the frontend that started the trace
|
|
listener lock on the listener that started the trace
|
|
nothing do not lock on anything
|
|
server lock on the server that started the trace
|
|
session lock on the session that started the trace
|
|
thread lock on the thread that started the trace
|
|
|
|
In addition to this, each source may provide up to 4 specific criteria such
|
|
as internal states or connection IDs. For example in HTTP/2 it is possible
|
|
to lock on the H2 stream and ignore other streams once a strace starts.
|
|
|
|
When a criterion is passed in argument, this one is used instead of the
|
|
other ones and any existing tracking is immediately terminated so that it can
|
|
restart with the new criterion. The special keyword "nothing" is supported by
|
|
all sources to permanently disable tracking.
|
|
|
|
trace <source> { pause | start | stop } [ [+|-|!]event]
|
|
Without argument, this will list the events enabled to automatically pause,
|
|
start, or stop a trace for this source. These events are specific to each
|
|
trace source. With an argument, this will either enable the event for the
|
|
specified action (if optionally prefixed by a '+') or disable it (if
|
|
prefixed by a '-' or '!'). The special keyword "now" is not an event and
|
|
requests to take the action immediately. The keywords "none" and "any" are
|
|
supported just like in "trace event".
|
|
|
|
The 3 supported actions are respectively "pause", "start" and "stop". The
|
|
"pause" action enumerates events which will cause a running trace to stop and
|
|
wait for a new start event to restart it. The "start" action enumerates the
|
|
events which switch the trace into the waiting mode until one of the start
|
|
events appears. And the "stop" action enumerates the events which definitely
|
|
stop the trace until it is manually enabled again. In practice it makes sense
|
|
to manually start a trace using "start now" without caring about events, and
|
|
to stop it using "stop now". In order to capture more subtle event sequences,
|
|
setting "start" to a normal event (like receiving an HTTP request) and "stop"
|
|
to a very rare event like emitting a certain error, will ensure that the last
|
|
captured events will match the desired criteria. And the pause event is
|
|
useful to detect the end of a sequence, disable the lock-on and wait for
|
|
another opportunity to take a capture. In this case it can make sense to
|
|
enable lock-on to spot only one specific criterion (e.g. a stream), and have
|
|
"start" set to anything that starts this criterion (e.g. all events which
|
|
create a stream), "stop" set to the expected anomaly, and "pause" to anything
|
|
that ends that criterion (e.g. any end of stream event). In this case the
|
|
trace log will contain complete sequences of perfectly clean series affecting
|
|
a single object, until the last sequence containing everything from the
|
|
beginning to the anomaly.
|
|
|
|
trace <source> sink [<sink>]
|
|
Without argument, this will list all event sinks available for this source,
|
|
and the currently configured one will have a star ('*') prepended in front
|
|
of it. Sink "none" is always available and means that all events are simply
|
|
dropped, though their processing is not ignored (e.g. lock-on does occur).
|
|
Other sinks are available depending on configuration and build options, but
|
|
typically "stdout" and "stderr" will be usable in debug mode, and in-memory
|
|
ring buffers should be available as well. When a name is specified, the sink
|
|
instantly changes for the specified source. Events are not changed during a
|
|
sink change. In the worst case some may be lost if an invalid sink is used
|
|
(or "none"), but operations do continue to a different destination.
|
|
|
|
trace <source> verbosity [<level>]
|
|
Without argument, this will list all verbosity levels for this source, and the
|
|
current one will be indicated by a star ('*') prepended in front of it. With
|
|
an argument, this will change the verbosity level to the specified one.
|
|
|
|
Verbosity levels indicate how far the trace decoder should go to provide
|
|
detailed information. It depends on the trace source, since some sources will
|
|
not even provide a specific decoder. Level "quiet" is always available and
|
|
disables any decoding. It can be useful when trying to figure what's
|
|
happening before trying to understand the details, since it will have a very
|
|
low impact on performance and trace size. When no verbosity levels are
|
|
declared by a source, level "default" is available and will cause a decoder
|
|
to be called when specified in the traces. It is an opportunistic decoding.
|
|
When the source declares some verbosity levels, these ones are listed with
|
|
a description of what they correspond to. In this case the trace decoder
|
|
provided by the source will be as accurate as possible based on the
|
|
information available at the trace point. The first level above "quiet" is
|
|
set by default.
|
|
|
|
|
|
9.4. Master CLI
|
|
---------------
|
|
|
|
The master CLI is a socket bound to the master process in master-worker mode.
|
|
This CLI gives access to the unix socket commands in every running or leaving
|
|
processes and allows a basic supervision of those processes.
|
|
|
|
The master CLI is configurable only from the haproxy program arguments with
|
|
the -S option. This option also takes bind options separated by commas.
|
|
|
|
Example:
|
|
|
|
# haproxy -W -S 127.0.0.1:1234 -f test1.cfg
|
|
# haproxy -Ws -S /tmp/master-socket,uid,1000,gid,1000,mode,600 -f test1.cfg
|
|
# haproxy -W -S /tmp/master-socket,level,user -f test1.cfg
|
|
|
|
The master CLI introduces a new 'show proc' command to surpervise the
|
|
processes:
|
|
|
|
Example:
|
|
|
|
$ echo 'show proc' | socat /var/run/haproxy-master.sock -
|
|
#<PID> <type> <relative PID> <reloads> <uptime> <version>
|
|
1162 master 0 5 0d00h02m07s 2.0-dev7-0124c9-7
|
|
# workers
|
|
1271 worker 1 0 0d00h00m00s 2.0-dev7-0124c9-7
|
|
1272 worker 2 0 0d00h00m00s 2.0-dev7-0124c9-7
|
|
# old workers
|
|
1233 worker [was: 1] 3 0d00h00m43s 2.0-dev3-6019f6-289
|
|
|
|
|
|
In this example, the master has been reloaded 5 times but one of the old
|
|
worker is still running and survived 3 reloads. You could access the CLI of
|
|
this worker to understand what's going on.
|
|
|
|
When the prompt is enabled (via the "prompt" command), the context the CLI is
|
|
working on is displayed in the prompt. The master is identified by the "master"
|
|
string, and other processes are identified with their PID. In case the last
|
|
reload failed, the master prompt will be changed to "master[ReloadFailed]>" so
|
|
that it becomes visible that the process is still running on the previous
|
|
configuration and that the new configuration is not operational.
|
|
|
|
The master CLI uses a special prefix notation to access the multiple
|
|
processes. This notation is easily identifiable as it begins by a @.
|
|
|
|
A @ prefix can be followed by a relative process number or by an exclamation
|
|
point and a PID. (e.g. @1 or @!1271). A @ alone could be use to specify the
|
|
master. Leaving processes are only accessible with the PID as relative process
|
|
number are only usable with the current processes.
|
|
|
|
Examples:
|
|
|
|
$ socat /var/run/haproxy-master.sock readline
|
|
prompt
|
|
master> @1 show info; @2 show info
|
|
[...]
|
|
Process_num: 1
|
|
Pid: 1271
|
|
[...]
|
|
Process_num: 2
|
|
Pid: 1272
|
|
[...]
|
|
master>
|
|
|
|
$ echo '@!1271 show info; @!1272 show info' | socat /var/run/haproxy-master.sock -
|
|
[...]
|
|
|
|
A prefix could be use as a command, which will send every next commands to
|
|
the specified process.
|
|
|
|
Examples:
|
|
|
|
$ socat /var/run/haproxy-master.sock readline
|
|
prompt
|
|
master> @1
|
|
1271> show info
|
|
[...]
|
|
1271> show stat
|
|
[...]
|
|
1271> @
|
|
master>
|
|
|
|
$ echo '@1; show info; show stat; @2; show info; show stat' | socat /var/run/haproxy-master.sock -
|
|
[...]
|
|
|
|
You can also reload the HAProxy master process with the "reload" command which
|
|
does the same as a `kill -USR2` on the master process, provided that the user
|
|
has at least "operator" or "admin" privileges.
|
|
|
|
Example:
|
|
|
|
$ echo "reload" | socat /var/run/haproxy-master.sock stdin
|
|
|
|
Note that a reload will close the connection to the master CLI.
|
|
|
|
|
|
10. Tricks for easier configuration management
|
|
----------------------------------------------
|
|
|
|
It is very common that two HAProxy nodes constituting a cluster share exactly
|
|
the same configuration modulo a few addresses. Instead of having to maintain a
|
|
duplicate configuration for each node, which will inevitably diverge, it is
|
|
possible to include environment variables in the configuration. Thus multiple
|
|
configuration may share the exact same file with only a few different system
|
|
wide environment variables. This started in version 1.5 where only addresses
|
|
were allowed to include environment variables, and 1.6 goes further by
|
|
supporting environment variables everywhere. The syntax is the same as in the
|
|
UNIX shell, a variable starts with a dollar sign ('$'), followed by an opening
|
|
curly brace ('{'), then the variable name followed by the closing brace ('}').
|
|
Except for addresses, environment variables are only interpreted in arguments
|
|
surrounded with double quotes (this was necessary not to break existing setups
|
|
using regular expressions involving the dollar symbol).
|
|
|
|
Environment variables also make it convenient to write configurations which are
|
|
expected to work on various sites where only the address changes. It can also
|
|
permit to remove passwords from some configs. Example below where the the file
|
|
"site1.env" file is sourced by the init script upon startup :
|
|
|
|
$ cat site1.env
|
|
LISTEN=192.168.1.1
|
|
CACHE_PFX=192.168.11
|
|
SERVER_PFX=192.168.22
|
|
LOGGER=192.168.33.1
|
|
STATSLP=admin:pa$$w0rd
|
|
ABUSERS=/etc/haproxy/abuse.lst
|
|
TIMEOUT=10s
|
|
|
|
$ cat haproxy.cfg
|
|
global
|
|
log "${LOGGER}:514" local0
|
|
|
|
defaults
|
|
mode http
|
|
timeout client "${TIMEOUT}"
|
|
timeout server "${TIMEOUT}"
|
|
timeout connect 5s
|
|
|
|
frontend public
|
|
bind "${LISTEN}:80"
|
|
http-request reject if { src -f "${ABUSERS}" }
|
|
stats uri /stats
|
|
stats auth "${STATSLP}"
|
|
use_backend cache if { path_end .jpg .css .ico }
|
|
default_backend server
|
|
|
|
backend cache
|
|
server cache1 "${CACHE_PFX}.1:18080" check
|
|
server cache2 "${CACHE_PFX}.2:18080" check
|
|
|
|
backend server
|
|
server cache1 "${SERVER_PFX}.1:8080" check
|
|
server cache2 "${SERVER_PFX}.2:8080" check
|
|
|
|
|
|
11. Well-known traps to avoid
|
|
-----------------------------
|
|
|
|
Once in a while, someone reports that after a system reboot, the haproxy
|
|
service wasn't started, and that once they start it by hand it works. Most
|
|
often, these people are running a clustered IP address mechanism such as
|
|
keepalived, to assign the service IP address to the master node only, and while
|
|
it used to work when they used to bind haproxy to address 0.0.0.0, it stopped
|
|
working after they bound it to the virtual IP address. What happens here is
|
|
that when the service starts, the virtual IP address is not yet owned by the
|
|
local node, so when HAProxy wants to bind to it, the system rejects this
|
|
because it is not a local IP address. The fix doesn't consist in delaying the
|
|
haproxy service startup (since it wouldn't stand a restart), but instead to
|
|
properly configure the system to allow binding to non-local addresses. This is
|
|
easily done on Linux by setting the net.ipv4.ip_nonlocal_bind sysctl to 1. This
|
|
is also needed in order to transparently intercept the IP traffic that passes
|
|
through HAProxy for a specific target address.
|
|
|
|
Multi-process configurations involving source port ranges may apparently seem
|
|
to work but they will cause some random failures under high loads because more
|
|
than one process may try to use the same source port to connect to the same
|
|
server, which is not possible. The system will report an error and a retry will
|
|
happen, picking another port. A high value in the "retries" parameter may hide
|
|
the effect to a certain extent but this also comes with increased CPU usage and
|
|
processing time. Logs will also report a certain number of retries. For this
|
|
reason, port ranges should be avoided in multi-process configurations.
|
|
|
|
Since HAProxy uses SO_REUSEPORT and supports having multiple independent
|
|
processes bound to the same IP:port, during troubleshooting it can happen that
|
|
an old process was not stopped before a new one was started. This provides
|
|
absurd test results which tend to indicate that any change to the configuration
|
|
is ignored. The reason is that in fact even the new process is restarted with a
|
|
new configuration, the old one also gets some incoming connections and
|
|
processes them, returning unexpected results. When in doubt, just stop the new
|
|
process and try again. If it still works, it very likely means that an old
|
|
process remains alive and has to be stopped. Linux's "netstat -lntp" is of good
|
|
help here.
|
|
|
|
When adding entries to an ACL from the command line (eg: when blacklisting a
|
|
source address), it is important to keep in mind that these entries are not
|
|
synchronized to the file and that if someone reloads the configuration, these
|
|
updates will be lost. While this is often the desired effect (for blacklisting)
|
|
it may not necessarily match expectations when the change was made as a fix for
|
|
a problem. See the "add acl" action of the CLI interface.
|
|
|
|
|
|
12. Debugging and performance issues
|
|
------------------------------------
|
|
|
|
When HAProxy is started with the "-d" option, it will stay in the foreground
|
|
and will print one line per event, such as an incoming connection, the end of a
|
|
connection, and for each request or response header line seen. This debug
|
|
output is emitted before the contents are processed, so they don't consider the
|
|
local modifications. The main use is to show the request and response without
|
|
having to run a network sniffer. The output is less readable when multiple
|
|
connections are handled in parallel, though the "debug2ansi" and "debug2html"
|
|
scripts found in the examples/ directory definitely help here by coloring the
|
|
output.
|
|
|
|
If a request or response is rejected because HAProxy finds it is malformed, the
|
|
best thing to do is to connect to the CLI and issue "show errors", which will
|
|
report the last captured faulty request and response for each frontend and
|
|
backend, with all the necessary information to indicate precisely the first
|
|
character of the input stream that was rejected. This is sometimes needed to
|
|
prove to customers or to developers that a bug is present in their code. In
|
|
this case it is often possible to relax the checks (but still keep the
|
|
captures) using "option accept-invalid-http-request" or its equivalent for
|
|
responses coming from the server "option accept-invalid-http-response". Please
|
|
see the configuration manual for more details.
|
|
|
|
Example :
|
|
|
|
> show errors
|
|
Total events captured on [13/Oct/2015:13:43:47.169] : 1
|
|
|
|
[13/Oct/2015:13:43:40.918] frontend HAProxyLocalStats (#2): invalid request
|
|
backend <NONE> (#-1), server <NONE> (#-1), event #0
|
|
src 127.0.0.1:51981, session #0, session flags 0x00000080
|
|
HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
|
|
HTTP chunk len 0 bytes, HTTP body len 0 bytes
|
|
buffer flags 0x00808002, out 0 bytes, total 31 bytes
|
|
pending 31 bytes, wrapping at 8040, error at position 13:
|
|
|
|
00000 GET /invalid request HTTP/1.1\r\n
|
|
|
|
|
|
The output of "show info" on the CLI provides a number of useful information
|
|
regarding the maximum connection rate ever reached, maximum SSL key rate ever
|
|
reached, and in general all information which can help to explain temporary
|
|
issues regarding CPU or memory usage. Example :
|
|
|
|
> show info
|
|
Name: HAProxy
|
|
Version: 1.6-dev7-e32d18-17
|
|
Release_date: 2015/10/12
|
|
Nbproc: 1
|
|
Process_num: 1
|
|
Pid: 7949
|
|
Uptime: 0d 0h02m39s
|
|
Uptime_sec: 159
|
|
Memmax_MB: 0
|
|
Ulimit-n: 120032
|
|
Maxsock: 120032
|
|
Maxconn: 60000
|
|
Hard_maxconn: 60000
|
|
CurrConns: 0
|
|
CumConns: 3
|
|
CumReq: 3
|
|
MaxSslConns: 0
|
|
CurrSslConns: 0
|
|
CumSslConns: 0
|
|
Maxpipes: 0
|
|
PipesUsed: 0
|
|
PipesFree: 0
|
|
ConnRate: 0
|
|
ConnRateLimit: 0
|
|
MaxConnRate: 1
|
|
SessRate: 0
|
|
SessRateLimit: 0
|
|
MaxSessRate: 1
|
|
SslRate: 0
|
|
SslRateLimit: 0
|
|
MaxSslRate: 0
|
|
SslFrontendKeyRate: 0
|
|
SslFrontendMaxKeyRate: 0
|
|
SslFrontendSessionReuse_pct: 0
|
|
SslBackendKeyRate: 0
|
|
SslBackendMaxKeyRate: 0
|
|
SslCacheLookups: 0
|
|
SslCacheMisses: 0
|
|
CompressBpsIn: 0
|
|
CompressBpsOut: 0
|
|
CompressBpsRateLim: 0
|
|
ZlibMemUsage: 0
|
|
MaxZlibMemUsage: 0
|
|
Tasks: 5
|
|
Run_queue: 1
|
|
Idle_pct: 100
|
|
node: wtap
|
|
description:
|
|
|
|
When an issue seems to randomly appear on a new version of HAProxy (eg: every
|
|
second request is aborted, occasional crash, etc), it is worth trying to enable
|
|
memory poisoning so that each call to malloc() is immediately followed by the
|
|
filling of the memory area with a configurable byte. By default this byte is
|
|
0x50 (ASCII for 'P'), but any other byte can be used, including zero (which
|
|
will have the same effect as a calloc() and which may make issues disappear).
|
|
Memory poisoning is enabled on the command line using the "-dM" option. It
|
|
slightly hurts performance and is not recommended for use in production. If
|
|
an issue happens all the time with it or never happens when poisoning uses
|
|
byte zero, it clearly means you've found a bug and you definitely need to
|
|
report it. Otherwise if there's no clear change, the problem it is not related.
|
|
|
|
When debugging some latency issues, it is important to use both strace and
|
|
tcpdump on the local machine, and another tcpdump on the remote system. The
|
|
reason for this is that there are delays everywhere in the processing chain and
|
|
it is important to know which one is causing latency to know where to act. In
|
|
practice, the local tcpdump will indicate when the input data come in. Strace
|
|
will indicate when haproxy receives these data (using recv/recvfrom). Warning,
|
|
openssl uses read()/write() syscalls instead of recv()/send(). Strace will also
|
|
show when haproxy sends the data, and tcpdump will show when the system sends
|
|
these data to the interface. Then the external tcpdump will show when the data
|
|
sent are really received (since the local one only shows when the packets are
|
|
queued). The benefit of sniffing on the local system is that strace and tcpdump
|
|
will use the same reference clock. Strace should be used with "-tts200" to get
|
|
complete timestamps and report large enough chunks of data to read them.
|
|
Tcpdump should be used with "-nvvttSs0" to report full packets, real sequence
|
|
numbers and complete timestamps.
|
|
|
|
In practice, received data are almost always immediately received by haproxy
|
|
(unless the machine has a saturated CPU or these data are invalid and not
|
|
delivered). If these data are received but not sent, it generally is because
|
|
the output buffer is saturated (ie: recipient doesn't consume the data fast
|
|
enough). This can be confirmed by seeing that the polling doesn't notify of
|
|
the ability to write on the output file descriptor for some time (it's often
|
|
easier to spot in the strace output when the data finally leave and then roll
|
|
back to see when the write event was notified). It generally matches an ACK
|
|
received from the recipient, and detected by tcpdump. Once the data are sent,
|
|
they may spend some time in the system doing nothing. Here again, the TCP
|
|
congestion window may be limited and not allow these data to leave, waiting for
|
|
an ACK to open the window. If the traffic is idle and the data take 40 ms or
|
|
200 ms to leave, it's a different issue (which is not an issue), it's the fact
|
|
that the Nagle algorithm prevents empty packets from leaving immediately, in
|
|
hope that they will be merged with subsequent data. HAProxy automatically
|
|
disables Nagle in pure TCP mode and in tunnels. However it definitely remains
|
|
enabled when forwarding an HTTP body (and this contributes to the performance
|
|
improvement there by reducing the number of packets). Some HTTP non-compliant
|
|
applications may be sensitive to the latency when delivering incomplete HTTP
|
|
response messages. In this case you will have to enable "option http-no-delay"
|
|
to disable Nagle in order to work around their design, keeping in mind that any
|
|
other proxy in the chain may similarly be impacted. If tcpdump reports that data
|
|
leave immediately but the other end doesn't see them quickly, it can mean there
|
|
is a congested WAN link, a congested LAN with flow control enabled and
|
|
preventing the data from leaving, or more commonly that HAProxy is in fact
|
|
running in a virtual machine and that for whatever reason the hypervisor has
|
|
decided that the data didn't need to be sent immediately. In virtualized
|
|
environments, latency issues are almost always caused by the virtualization
|
|
layer, so in order to save time, it's worth first comparing tcpdump in the VM
|
|
and on the external components. Any difference has to be credited to the
|
|
hypervisor and its accompanying drivers.
|
|
|
|
When some TCP SACK segments are seen in tcpdump traces (using -vv), it always
|
|
means that the side sending them has got the proof of a lost packet. While not
|
|
seeing them doesn't mean there are no losses, seeing them definitely means the
|
|
network is lossy. Losses are normal on a network, but at a rate where SACKs are
|
|
not noticeable at the naked eye. If they appear a lot in the traces, it is
|
|
worth investigating exactly what happens and where the packets are lost. HTTP
|
|
doesn't cope well with TCP losses, which introduce huge latencies.
|
|
|
|
The "netstat -i" command will report statistics per interface. An interface
|
|
where the Rx-Ovr counter grows indicates that the system doesn't have enough
|
|
resources to receive all incoming packets and that they're lost before being
|
|
processed by the network driver. Rx-Drp indicates that some received packets
|
|
were lost in the network stack because the application doesn't process them
|
|
fast enough. This can happen during some attacks as well. Tx-Drp means that
|
|
the output queues were full and packets had to be dropped. When using TCP it
|
|
should be very rare, but will possibly indicate a saturated outgoing link.
|
|
|
|
|
|
13. Security considerations
|
|
---------------------------
|
|
|
|
HAProxy is designed to run with very limited privileges. The standard way to
|
|
use it is to isolate it into a chroot jail and to drop its privileges to a
|
|
non-root user without any permissions inside this jail so that if any future
|
|
vulnerability were to be discovered, its compromise would not affect the rest
|
|
of the system.
|
|
|
|
In order to perform a chroot, it first needs to be started as a root user. It is
|
|
pointless to build hand-made chroots to start the process there, these ones are
|
|
painful to build, are never properly maintained and always contain way more
|
|
bugs than the main file-system. And in case of compromise, the intruder can use
|
|
the purposely built file-system. Unfortunately many administrators confuse
|
|
"start as root" and "run as root", resulting in the uid change to be done prior
|
|
to starting haproxy, and reducing the effective security restrictions.
|
|
|
|
HAProxy will need to be started as root in order to :
|
|
- adjust the file descriptor limits
|
|
- bind to privileged port numbers
|
|
- bind to a specific network interface
|
|
- transparently listen to a foreign address
|
|
- isolate itself inside the chroot jail
|
|
- drop to another non-privileged UID
|
|
|
|
HAProxy may require to be run as root in order to :
|
|
- bind to an interface for outgoing connections
|
|
- bind to privileged source ports for outgoing connections
|
|
- transparently bind to a foreign address for outgoing connections
|
|
|
|
Most users will never need the "run as root" case. But the "start as root"
|
|
covers most usages.
|
|
|
|
A safe configuration will have :
|
|
|
|
- a chroot statement pointing to an empty location without any access
|
|
permissions. This can be prepared this way on the UNIX command line :
|
|
|
|
# mkdir /var/empty && chmod 0 /var/empty || echo "Failed"
|
|
|
|
and referenced like this in the HAProxy configuration's global section :
|
|
|
|
chroot /var/empty
|
|
|
|
- both a uid/user and gid/group statements in the global section :
|
|
|
|
user haproxy
|
|
group haproxy
|
|
|
|
- a stats socket whose mode, uid and gid are set to match the user and/or
|
|
group allowed to access the CLI so that nobody may access it :
|
|
|
|
stats socket /var/run/haproxy.stat uid hatop gid hatop mode 600
|
|
|