Sometimes it would be convenient to have a log counter so that from a log
server we know whether some logs were lost or not. The frontend's log counter
serves exactly this purpose. It's incremented each time a traffic log is
produced. If a log is disabled using "http-request set-log-level silent",
the counter will not be incremented. However, admin logs are not accounted
for. Also, if logs are filtered out before being sent to the server because
of a minimum level set on the log line, the counter will be increased anyway.
The counter is 32-bit, so it will wrap, but that's not an issue considering
that 4 billion logs are rarely in the same file, let alone close to each
other.
Add support for http-request track-sc, similar to what is done in
tcp-request for backends. A new act_prm field was added to HTTP
request rules to store the track params (table, counter). Just
like for TCP rules, the table is resolved while checking for
config validity. The code was mostly copied from the TCP code
with the exception that here we also count the HTTP request count
and rate by hand. Probably that something could be factored out in
the future.
It seems like tracking flags should be improved to mark each hook
which tracks a key so that we can have some check points where to
increase counters of the past if not done yet, a bit like is done
for TRACK_BACKEND.
From time to time it's useful to hash input data (scramble input, or
reduce the space needed in a stick table). This patch provides 3 simple
converters allowing use of the available hash functions to hash input
data. The output is an unsigned integer which can be passed into a header,
a log or used as an index for a stick table. One nice usage is to scramble
source IP addresses before logging when there are requirements to hide them.
Konstantin Romanenko reported a typo in the HTML documentation. The typo is
already present in the raw text version : the "shutdown sessions" command
should be "shutdown sessions server".
This one is not inherited from defaults into frontends nor backends
because it would create a confusion situation where it would be hard
to disable it (since both frontend and backend would enable it).
Daniel Dubovik reported an interesting bug showing that the request body
processing was still not 100% fixed. If a POST request contained short
enough data to be forwarded at once before trying to establish the
connection to the server, we had no way to correctly rewind the body.
The first visible case is that balancing on a header does not always work
on such POST requests since the header cannot be found. But there are even
nastier implications which are that http-send-name-header would apply to
the wrong location and possibly even affect part of the request's body
due to an incorrect rewinding.
There are two options to fix the problem :
- first one is to force the HTTP_MSG_F_WAIT_CONN flag on all hash-based
balancing algorithms and http-send-name-header, but there's always a
risk that any new algorithm forgets to set it ;
- the second option is to account for the amount of skipped data before
the connection establishes so that we always know the position of the
request's body relative to the buffer's origin.
The second option is much more reliable and fits very well in the spirit
of the past changes to fix forwarding. Indeed, at the moment we have
msg->sov which points to the start of the body before headers are forwarded
and which equals zero afterwards (so it still points to the start of the
body before forwarding data). A minor change consists in always making it
point to the start of the body even after data have been forwarded. It means
that it can get a negative value (so we need to change its type to signed)..
In order to avoid wrapping, we only do this as long as the other side of
the buffer is not connected yet.
Doing this definitely fixes the issues above for the requests. Since the
response cannot be rewound we don't need to perform any change there.
This bug was introduced/remained unfixed in 1.5-dev23 so the fix must be
backported to 1.5.
This patch adds two converters :
ltime(<format>[,<offset>])
utime(<format>[,<offset>])
Both use strftime() to emit the output string from an input date. ltime()
provides local time, while utime() provides the UTC time.
These new converters make it possible to look up any sample expression
in a table, and check whether an equivalent key exists or not, and if it
exists, to retrieve the associated data (eg: gpc0, request rate, etc...).
Till now it was only possible using tracking, but sometimes tracking is
not suited to only retrieving such counters, either because it's done too
early or because too many items need to be checked without necessarily
being tracked.
These converters all take a string on input, and then convert it again to
the table's type. This means that if an input sample is of type IPv4 and
the table is of type IP, it will first be converted to a string, then back
to an IP address. This is a limitation of the current design which does not
allow converters to declare that "any" type is supported on input. Since
strings are the only types which can be cast to any other one, this method
always works.
The following converters were added :
in_table, table_bytes_in_rate, table_bytes_out_rate, table_conn_cnt,
table_conn_cur, table_conn_rate, table_gpc0, table_gpc0_rate,
table_http_err_cnt, table_http_err_rate, table_http_req_cnt,
table_http_req_rate, table_kbytes_in, table_kbytes_out,
table_server_id, table_sess_cnt, table_sess_rate, table_trackers.
Listening to an abstract namespace socket is quite convenient but
comes with some drawbacks that must be clearly understood when the
socket is being listened to by multiple processes. The trouble is
that the socket cannot be rebound if a new process attempts a soft
restart and fails, so only one of the initially bound processes
will still be bound to it, the other ones will fail to rebind. For
most situations it's not an issue but it needs to be indicated.
With all the goodies supported by logformat, people find that the limit
of 1024 chars for log lines is too short. Some servers do not support
larger lines and can simply drop them, so changing the default value is
not always the best choice.
This patch takes a different approach. Log line length is specified per
log server on the "log" line, with a value between 80 and 65535. That
way it's possibly to satisfy all needs, even with some fat local servers
and small remote ones.
This new branch is based on 1.5.0, which 1.6-dev0 is 100% equivalent to.
The README has been updated to mention that it is a development branch.
Released version 1.6-dev0 with the following main changes :
- exact copy of 1.5.0
Released version 1.5.0 with the following main changes :
- MEDIUM: ssl: ignored file names ending as '.issuer' or '.ocsp'.
- MEDIUM: ssl: basic OCSP stapling support.
- MINOR: ssl/cli: Fix unapropriate comment in code on 'set ssl ocsp-response'
- MEDIUM: ssl: add 300s supported time skew on OCSP response update.
- MINOR: checks: mysql-check: Add support for v4.1+ authentication
- MEDIUM: ssl: Add the option to use standardized DH parameters >= 1024 bits
- MEDIUM: ssl: fix detection of ephemeral diffie-hellman key exchange by using the cipher description.
- MEDIUM: http: add actions "replace-header" and "replace-values" in http-req/resp
- MEDIUM: Break out check establishment into connect_chk()
- MEDIUM: Add port_to_str helper
- BUG/MEDIUM: fix ignored values for half-closed timeouts (client-fin and server-fin) in defaults section.
- BUG/MEDIUM: Fix unhandled connections problem with systemd daemon mode and SO_REUSEPORT.
- MINOR: regex: fix a little configuration memory leak.
- MINOR: regex: Create JIT compatible function that return match strings
- MEDIUM: regex: replace all standard regex function by own functions
- MEDIUM: regex: Remove null terminated strings.
- MINOR: regex: Use native PCRE API.
- MINOR: missing regex.h include
- DOC: Add Exim as Proxy Protocol implementer.
- BUILD: don't use type "uint" which is not portable
- BUILD: stats: workaround stupid and bogus -Werror=format-security behaviour
- BUG/MEDIUM: http: clear CF_READ_NOEXP when preparing a new transaction
- CLEANUP: http: don't clear CF_READ_NOEXP twice
- DOC: fix proxy protocol v2 decoder example
- DOC: fix remaining occurrences of "pattern extraction"
- MINOR: log: allow the HTTP status code to be logged even in TCP frontends
- MINOR: logs: don't limit HTTP header captures to HTTP frontends
- MINOR: sample: improve sample_fetch_string() to report partial contents
- MINOR: capture: extend the captures to support non-header keys
- MINOR: tcp: prepare support for the "capture" action
- MEDIUM: tcp: add a new tcp-request capture directive
- MEDIUM: session: allow shorter retry delay if timeout connect is small
- MEDIUM: session: don't apply the retry delay when redispatching
- MEDIUM: session: redispatch earlier when possible
- MINOR: config: warn when tcp-check rules are used without option tcp-check
- BUG/MINOR: connection: make proxy protocol v1 support the UNKNOWN protocol
- DOC: proxy protocol example parser was still wrong
- DOC: minor updates to the proxy protocol doc
- CLEANUP: connection: merge proxy proto v2 header and address block
- MEDIUM: connection: add support for proxy protocol v2 in accept-proxy
- MINOR: tools: add new functions to quote-encode strings
- DOC: clarify the CSV format
- MEDIUM: stats: report the last check and last agent's output on the CSV status
- MINOR: freq_ctr: introduce a new averaging method
- MEDIUM: session: maintain per-backend and per-server time statistics
- MEDIUM: stats: report per-backend and per-server time stats in HTML and CSV outputs
- BUG/MINOR: http: fix typos in previous patch
- DOC: remove the ultra-obsolete TODO file
- DOC: update roadmap
- DOC: minor updates to the README
- DOC: mention the maxconn limitations with the select poller
- DOC: commit a few old design thoughts files
These ones were design notes and ideas collected during the 1.5
development phase lying on my development machine. There might still
be some value in keeping them for future reference since they mention
certain corner cases.
Select()'s safe area is limited to 1024 FDs, and anything higher
than this will report "select: FAILED" on startup in debug mode,
so better document it.
The support is all based on static responses. This doesn't add any
request / response logic to HAProxy, but allows a way to update
information through the socket interface.
Currently certificates specified using "crt" or "crt-list" on "bind" lines
are loaded as PEM files.
For each PEM file, haproxy checks for the presence of file at the same path
suffixed by ".ocsp". If such file is found, support for the TLS Certificate
Status Request extension (also known as "OCSP stapling") is automatically
enabled. The content of this file is optional. If not empty, it must contain
a valid OCSP Response in DER format. In order to be valid an OCSP Response
must comply with the following rules: it has to indicate a good status,
it has to be a single response for the certificate of the PEM file, and it
has to be valid at the moment of addition. If these rules are not respected
the OCSP Response is ignored and a warning is emitted. In order to identify
which certificate an OCSP Response applies to, the issuer's certificate is
necessary. If the issuer's certificate is not found in the PEM file, it will
be loaded from a file at the same path as the PEM file suffixed by ".issuer"
if it exists otherwise it will fail with an error.
It is possible to update an OCSP Response from the unix socket using:
set ssl ocsp-response <response>
This command is used to update an OCSP Response for a certificate (see "crt"
on "bind" lines). Same controls are performed as during the initial loading of
the response. The <response> must be passed as a base64 encoded string of the
DER encoded response from the OCSP server.
Example:
openssl ocsp -issuer issuer.pem -cert server.pem \
-host ocsp.issuer.com:80 -respout resp.der
echo "set ssl ocsp-response $(base64 -w 10000 resp.der)" | \
socat stdio /var/run/haproxy.stat
This feature is automatically enabled on openssl 0.9.8h and above.
This work was performed jointly by Dirkjan Bussink of GitHub and
Emeric Brun of HAProxy Technologies.
This patch adds two new actions to http-request and http-response rulesets :
- replace-header : replace a whole header line, suited for headers
which might contain commas
- replace-value : replace a single header value, suited for headers
defined as lists.
The match consists in a regex, and the replacement string takes a log-format
and supports back-references.
The time statistics computed by previous patches are now reported in the
HTML stats in the tips related to the total sessions for backend and servers,
and as separate columns for the CSV stats.
Now that we can quote unsafe string, it becomes possible to dump the health
check responses on the CSV page as well. The two new fields are "last_chk"
and "last_agt".
Indicate that the text cells in the CSV format may contain quotes to
escape ambiguous texts. We don't have this case right now since we limit
the output, but it may happen in the future.
The "accept-proxy" statement of bind lines was still limited to version
1 of the protocol, while send-proxy-v2 is now available on the server
lines. This patch adds support for parsing v2 of the protocol on incoming
connections. The v2 header is automatically recognized so there is no
need for a new option.
This new directive captures the specified fetch expression, converts
it to text and puts it into the next capture slot. The capture slots
are shared with header captures so that it is possible to dump all
captures at once or selectively in logs and header processing.
The purpose is to permit logs to contain whatever payload is found in
a request, for example bytes at a fixed location or the SNI of forwarded
SSL traffic.
Similar to previous patches, HTTP header captures are performed when
a TCP frontend switches to an HTTP backend, but are not possible to
report. So let's relax the check to explicitly allow them to be present
in TCP frontends.
Log format is defined in the frontend, and some frontends may be chained to
an HTTP backend. Sometimes it's very convenient to be able to log the HTTP
status code of these HTTP backends. This status is definitely present in
the internal structures, it's just that we used to limit it to be used in
HTTP frontends. So let's simply relax the check to allow it to be used in
TCP frontends as well.
When no static DH parameters are specified, this patch makes haproxy
use standardized (rfc 2409 / rfc 3526) DH parameters with prime lenghts
of 1024, 2048, 4096 or 8192 bits for DHE key exchange. The size of the
temporary/ephemeral DH key is computed as the minimum of the RSA/DSA server
key size and the value of a new option named tune.ssl.default-dh-param.
Richard Russo reported that the example code in the PP spec is wrong
now that we slightly changed the format to merge <ver> and <cmd>. Also
rename the field <ver_cmd> to avoid any ambiguity on the usage.
MySQL will in stop supporting pre-4.1 authentication packets in the future
and is already giving us a hard time regarding non-silencable warnings
which are logged on each health check. Warnings look like the following:
"[Warning] Client failed to provide its character set. 'latin1' will be used
as client character set."
This patch adds basic support for post-4.1 authentication by sending the proper
authentication packet with the character set, along with the QUIT command.
Released version 1.5-dev26 with the following main changes :
- BUG/MEDIUM: polling: fix possible CPU hogging of worker processes after receiving SIGUSR1.
- BUG/MINOR: stats: fix a typo on a closing tag for a server tracking another one
- OPTIM: stats: avoid the calculation of a useless link on tracking servers in maintenance
- MINOR: fix a few memory usage errors
- CONTRIB: halog: Filter input lines by date and time through timestamp
- MINOR: ssl: SSL_CTX_set_options() and SSL_CTX_set_mode() take a long, not an int
- BUG/MEDIUM: regex: fix risk of buffer overrun in exp_replace()
- MINOR: acl: set "str" as default match for strings
- DOC: Add some precisions about acl default matching method
- MEDIUM: acl: strenghten the option parser to report invalid options
- BUG/MEDIUM: config: a stats-less config crashes in 1.5-dev25
- BUG/MINOR: checks: tcp-check must not stop on '\0' for binary checks
- MINOR: stats: improve alignment of color codes to save one line of header
- MINOR: checks: simplify and improve reporting of state changes when using log-health-checks
- MINOR: server: remove the SRV_DRAIN flag which can always be deduced
- MINOR: server: use functions to detect state changes and to update them
- MINOR: server: create srv_was_usable() from srv_is_usable() and use a pointer
- BUG/MINOR: stats: do not report "100%" in the thottle column when server is draining
- BUG/MAJOR: config: don't free valid regex memory
- BUG/MEDIUM: session: don't clear CF_READ_NOEXP if analysers are not called
- BUG/MINOR: stats: tracking servers may incorrectly report an inherited DRAIN status
- MEDIUM: proxy: make timeout parser a bit stricter
- REORG/MEDIUM: server: split server state and flags in two different variables
- REORG/MEDIUM: server: move the maintenance bits out of the server state
- MAJOR: server: use states instead of flags to store the server state
- REORG: checks: put the functions in the appropriate files !
- MEDIUM: server: properly support and propagate the maintenance status
- MEDIUM: server: allow multi-level server tracking
- CLEANUP: checks: rename the server_status_printf function
- MEDIUM: checks: simplify server up/down/nolb transitions
- MAJOR: checks: move health checks changes to set_server_check_status()
- MINOR: server: make the status reporting function support a reason
- MINOR: checks: simplify health check reporting functions
- MINOR: server: implement srv_set_stopped()
- MINOR: server: implement srv_set_running()
- MINOR: server: implement srv_set_stopping()
- MEDIUM: checks: simplify failure notification using srv_set_stopped()
- MEDIUM: checks: simplify success notification using srv_set_running()
- MEDIUM: checks: simplify stopping mode notification using srv_set_stopping()
- MEDIUM: stats: report a server's own state instead of the tracked one's
- MINOR: server: make use of srv_is_usable() instead of checking eweight
- MAJOR: checks: add support for a new "drain" administrative mode
- MINOR: stats: use the admin flags for soft enable/disable/stop/start on the web page
- MEDIUM: stats: introduce new actions to simplify admin status management
- MINOR: cli: introduce a new "set server" command
- MINOR: stats: report a distinct output for DOWN caused by agent
- MINOR: checks: support specific check reporting for the agent
- MINOR: checks: support a neutral check result
- BUG/MINOR: cli: "agent" was missing from the "enable"/"disable" help message
- MEDIUM: cli: add support for enabling/disabling health checks.
- MEDIUM: stats: report down caused by agent prior to reporting up
- MAJOR: agent: rework the response processing and support additional actions
- MINOR: stats: improve the stats web page to support more actions
- CONTRIB: halog: avoid calling time/localtime/mktime for each line
- DOC: document the workarouds for Google Chrome's bogus pre-connect
- MINOR: stats: report SSL key computations per second
- MINOR: stats: add counters for SSL cache lookups and misses
More and more people are complaining about the bugs experienced by
Chrome users due to the pre-connect feature and the fact that Chrome
does not monitor its connections and happily displays the error page
instead of re-opening a new connection. Since we can work around this
bug, let's document how to do it.
We now retrieve a lot of information from a single line of response, which
can be made up of various words delimited by spaces/tabs/commas. We try to
arrange all this and report whatever unusual we detect. The agent now supports :
- "up", "down", "stopped", "fail" for the operational states
- "ready", "drain", "maint" for the administrative states
- any "%" number for the weight
- an optional reason after a "#" that can be reported on the stats page
The line parser and processor should move to its own function so that
we can reuse the exact same one for http-based agent checks later.
This command supports "agent", "health", "state" and "weight" to adjust
various server attributes as well as changing server health check statuses
on the fly or setting the drain mode.
Now that it is possible to know whether a server is in forced maintenance
or inherits its maintenance status from another one, it is possible to
allow server tracking at more than one level. We still provide a loop
detection however.
Note that for the stats it's a bit trickier since we have to report the
check state which corresponds to the state of the server at the end of
the chain.
Function set_server_check_status() is very weird. It is called at the
end of a check to update the server's state before the new state is even
calculated, and possibly to log status changes, only if the proxy has
"option log-health-checks" set.
In order to do so, it employs an exhaustive list of the combinations
which can lead to a state change, while in practice almost all of
them may simply be deduced from the change of check status. Better,
some changes of check status are currently not detected while they
can be very valuable (eg: changes between L4/L6/TOUT/HTTP 500 for
example).
The doc was updated to reflect this.
Also, a minor change was made to consider s->uweight and not s->eweight
as meaning "DRAIN" since eweight can be null without the DRAIN mode (eg:
throttle, NOLB, ...).
Released version 1.5-dev25 with the following main changes :
- MEDIUM: connection: Implement and extented PROXY Protocol V2
- MINOR: ssl: clean unused ACLs declarations
- MINOR: ssl: adds fetchs and ACLs for ssl back connection.
- MINOR: ssl: merge client's and frontend's certificate functions.
- MINOR: ssl: adds ssl_f_sha1 fetch to return frontend's certificate fingerprint
- MINOR: ssl: adds sample converter base64 for binary type.
- MINOR: ssl: convert to binary ssl_fc_unique_id and ssl_bc_unique_id.
- BUG/MAJOR: ssl: Fallback to private session cache if current lock mode is not supported.
- MAJOR: ssl: Change default locks on ssl session cache.
- BUG/MINOR: chunk: Fix function chunk_strcmp and chunk_strcasecmp match a substring.
- MINOR: ssl: add global statement tune.ssl.force-private-cache.
- MINOR: ssl: remove fallback to SSL session private cache if lock init fails.
- BUG/MEDIUM: patterns: last fix was still not enough
- MINOR: http: export the smp_fetch_cookie function
- MINOR: http: generic pointer to rule argument
- BUG/MEDIUM: pattern: a typo breaks automatic acl/map numbering
- BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns
- BUG/MINOR: proxy: unsafe initialization of HTTP transaction when switching from TCP frontend
- BUG/MINOR: http: log 407 in case of proxy auth
- MINOR: http: rely on the message body parser to send 100-continue
- MEDIUM: http: move reqadd after execution of http_request redirect
- MEDIUM: http: jump to dedicated labels after http-request processing
- BUG/MINOR: http: block rules forgot to increment the denied_req counter
- BUG/MINOR: http: block rules forgot to increment the session's request counter
- MEDIUM: http: move Connection header processing earlier
- MEDIUM: http: remove even more of the spaghetti in the request path
- MINOR: http: silently support the "block" action for http-request
- CLEANUP: proxy: rename "block_cond" to "block_rules"
- MEDIUM: http: emulate "block" rules using "http-request" rules
- MINOR: http: remove the now unused loop over "block" rules
- MEDIUM: http: factorize the "auth" action of http-request and stats
- MEDIUM: http: make http-request rules processing return a verdict instead of a rule
- MINOR: config: add minimum support for emitting warnings only once
- MEDIUM: config: inform the user about the deprecatedness of "block" rules
- MEDIUM: config: inform the user that "reqsetbe" is deprecated
- MEDIUM: config: inform the user only once that "redispatch" is deprecated
- MEDIUM: config: warn that '{cli,con,srv}timeout' are deprecated
- BUG/MINOR: auth: fix wrong return type in pat_match_auth()
- BUILD: config: remove a warning with clang
- BUG/MAJOR: http: connection setup may stall on balance url_param
- BUG/MEDIUM: http/session: disable client-side expiration only after body
- BUG/MEDIUM: http: correctly report request body timeouts
- BUG/MEDIUM: http: disable server-side expiration until client has sent the body
- MEDIUM: listener: make the accept function more robust against pauses
- BUILD: syscalls: remove improper inline statement in front of syscalls
- BUILD: ssl: SSL_CTX_set_msg_callback() needs openssl >= 0.9.7
- BUG/MAJOR: session: recover the correct connection pointer in half-initialized sessions
- DOC: add some explanation on the shared cache build options in the readme.
- MEDIUM: proxy: only adjust the backend's bind-process when already set
- MEDIUM: config: limit nbproc to the machine's word size
- MEDIUM: config: check the bind-process settings according to nbproc
- MEDIUM: listener: parse the new "process" bind keyword
- MEDIUM: listener: inherit the process mask from the proxy
- MAJOR: listener: only start listeners bound to the same processes
- MINOR: config: only report a warning when stats sockets are bound to more than 1 process
- CLEANUP: config: set the maxaccept value for peers listeners earlier
- BUG/MINOR: backend: only match IPv4 addresses with RDP cookies
- BUG/MINOR: checks: correctly configure the address family and protocol
- MINOR: tools: split is_addr() and is_inet_addr()
- MINOR: protocols: use is_inet_addr() when only INET addresses are desired
- MEDIUM: unix: add preliminary support for connecting to servers over UNIX sockets
- MEDIUM: checks: only complain about the missing port when the check uses TCP
- MEDIUM: unix: implement support for Linux abstract namespace sockets
- DOC: map_beg was missing from the table of map_* converters
- DOC: ebtree: indicate that prefix insertion/lookup may be used with strings
- MEDIUM: pattern: use ebtree's longest match to index/lookup string beginning
- BUILD: remove the obsolete BSD and OSX makefiles
- MEDIUM: unix: avoid a double connect probe when no data are sent
- DOC: stop referencing the slow git repository in the README
- BUILD: only build the systemd wrapper on Linux 2.6 and above
- DOC: update roadmap with completed tasks
- MEDIUM: session: implement half-closed timeouts (client-fin and server-fin)
Long-lived sessions are often subject to half-closed sessions resulting in
a lot of sessions appearing in FIN_WAIT state in the system tables, and no
way for haproxy to get rid of them. This typically happens because clients
suddenly disconnect without sending any packet (eg: FIN or RST was lost in
the path), and while the server detects this using an applicative heart
beat, haproxy does not close the connection.
This patch adds two new timeouts : "timeout client-fin" and
"timeout server-fin". The former allows one to override the client-facing
timeout when a FIN has been received or sent. The latter does the same for
server-facing connections, which is less useful.
These sockets are the same as Unix sockets except that there's no need
for any filesystem access. The address may be whatever string both sides
agree upon. This can be really convenient for inter-process communications
as well as for chaining backends to frontends.
These addresses are forced by prepending their address with "abns@" for
"abstract namespace".
