Commit Graph

12053 Commits

Author SHA1 Message Date
William Lallemand
2f44a59c7f MEDIUM: ssl: use TLSv1.2 as the minimum default on bind lines
Since HAProxy 1.8, the TLS default minimum version was set to TLSv1.0 to
avoid using the deprecated SSLv3.0. Since then, the standard changed and
the recommended TLS version is now TLSv1.2.

This patch changes the minimum default version to TLSv1.2 on bind lines.
If you need to use prior TLS version, this is still possible by
using the ssl-min-ver keyword.
2020-05-29 09:05:45 +02:00
Christopher Faulet
56192cc60b BUG/MEDIUM: checks: Don't add a tcpcheck ruleset twice in the shared tree
When a tcpcheck ruleset is created, it is automatically inserted in a global
tree. Unfortunately for applicative health checks (redis, mysql...), the created
ruleset is inserted a second time during the directive parsing. The leads to a
infinite loop when haproxy is stopped when we try to scan the tree to release
all tcpcheck rulesets.

Now, only the function responsible to create the tcpcheck ruleset insert it into
the tree.

No backport needed.
2020-05-29 08:20:17 +02:00
Willy Tarreau
78675252fb BUG/MINOR: nameservers: fix error handling in parsing of resolv.conf
In issue #657, Coverity found a bug in the "nameserver" parser for the
resolv.conf when "parse-resolv-conf" is set. What happens is that if an
unparsable address appears on a "nameserver" line, it will destroy the
previously allocated pointer before reporting the warning, then the next
"nameserver" line will dereference it again and wlil cause a crash. If
the faulty nameserver is the last one, it will only be a memory leak.
Let's just make sure we preserve the pointer when handling the error.
The patch also fixes a typo in the warning.

The bug was introduced in 1.9 with commit 44e609bfa ("MINOR: dns:
Implement `parse-resolv-conf` directive") so the fix needs to be backported
up to 1.9 or 2.0.
2020-05-28 18:12:22 +02:00
Ilya Shipitsin
143f4ce051 CI: cirrus-ci: skip reg-tests/connection/proxy_protocol_send_unique_id_alpn.vtc on CentOS 6
as reg-tests/connection/proxy_protocol_send_unique_id_alpn.vtc requires ALPN, which is not
supported on CentOS 6, let us skip that test
2020-05-28 16:51:36 +02:00
Emeric Brun
fa9d780119 BUG/MEDIUM: logs: fix trailing zeros on log message.
This patch removes all trailing LFs and Zeros from
log messages. Previously only the last LF was removed.

It's a regression from e8ea0ae6f6 "BUG/MINOR: logs:
prevent double line returns in some events."

This should fix github issue #654
2020-05-28 15:30:51 +02:00
Nathan Neulinger
31a841c323 BUG/MINOR: lua: Add missing string length for lua sticktable lookup
In hlua_stktable_lookup(), the key length is never set so all
stktable:lookup("key") calls return nil from lua.

This patch must be backported as far as 1.9.

[Cf: I slightly updated the patch to use lua_tolstring() instead of
     luaL_checkstring() + strlen()]
2020-05-28 15:15:52 +02:00
Christopher Faulet
3d5e121842 MINOR: checks: I/O callback function only rely on the data layer wake callback
Most of code in event_srv_chk_io() function is inherited from the checks before
the recent refactoring. Now, it is enough to only call wake_srv_chk(). Since the
refactoring, the removed code is dead and never called. wake_srv_chk() may only
return 0 if tcpcheck_main() returns 0 and the check status is unknown
(CHK_RES_UNKNOWN). When this happens, nothing is performed in event_srv_chk_io().
2020-05-28 15:07:20 +02:00
Christopher Faulet
030ed4b738 BUG/MEDIUM: checks: Don't blindly subscribe for receive if waiting for connect
When an health check is waiting for a connection establishment, it subscribe for
receive or send events, depending on the next rule to be evaluated. For
subscription for send events, there is no problem. It works as expected. For
subscription for receive events, It only works for HTTP checks because the
underlying multiplexer takes care to do a receive before subscribing again,
updating the fd state. For TCP checks, the PT multiplexer only forwards
subscriptions at the transport layer. So the check itself is woken up. This
leads to a subscribe/notify loop waiting the connection establishment or a
timeout, uselessly eating CPU.

Thus, when a check is waiting for a connect, instead of blindly resubscribe for
receive events when it is woken up, we now try to receive data.

This patch should fix the issue #635. No backport needed.
2020-05-28 15:07:20 +02:00
Christopher Faulet
0bac4cdf1a CLEANUP: http: Remove unused HTTP message templates
HTTP_1XX, HTTP_3XX and HTTP_4XX message templates are no longer used. Only
HTTP_302 and HTTP_303 are used during configuration parsing by "errorloc" family
directives. So these templates are removed from the generic http code. And
HTTP_302 and HTTP_303 templates are moved as static strings in the function
parsing "errorloc" directives.
2020-05-28 15:07:20 +02:00
Christopher Faulet
b304883754 MINOR: http-rules: Use an action function to eval http-request auth rules
Now http-request auth rules are evaluated in a dedicated function and no longer
handled "in place" during the HTTP rules evaluation. Thus the action name
ACT_HTTP_REQ_AUTH is removed. In additionn, http_reply_40x_unauthorized() is
also removed. This part is now handled in the new action_ptr callback function.
2020-05-28 15:07:20 +02:00
Christopher Faulet
612f2eafe9 MINOR: http-ana: Use proxy's error replies to emit 401/407 responses
There is no reason to not use proxy's error replies to emit 401/407
responses. The function http_reply_40x_unauthorized(), responsible to emit those
responses, is not really complex. It only adds a
WWW-Authenticate/Proxy-Authenticate header to a generic message.

So now, error replies can be defined for 401 and 407 status codes, using
errorfile or http-error directives. When an http-request auth rule is evaluated,
the corresponding error reply is used. For 401 responses, all occurrences of the
WWW-Authenticate header are removed and replaced by a new one with a basic
authentication challenge for the configured realm. For 407 responses, the same
is done on the Proxy-Authenticate header. If the error reply must not be
altered, "http-request return" rule must be used instead.
2020-05-28 15:07:20 +02:00
Christopher Faulet
ae43b6c446 MINOR: http-ana: Make the function http_reply_to_htx() public
This function may be used from anywhere to convert an HTTP reply to an HTX
message.
2020-05-28 15:07:20 +02:00
Tim Duesterhus
9947f77a59 REGTEST: Add connection/proxy_protocol_send_unique_id_alpn
This reg-test checks that sending unique IDs via PPv2 works for servers
with the `alpn` option specified (issue #640). As a side effect it also
checks that PPv2 works with ALPN (issue #651).

It has been verified that the test fails without the following commits
applied and succeeds with them applied.

   1f9a4ecea BUG/MEDIUM: backend: set the connection owner to the session when using alpn.
   083fd42d5 BUG/MEDIUM: connection: Ignore PP2 unique ID for stream-less connections
   eb9ba3cb2 BUG/MINOR: connection: Always get the stream when available to send PP2 line

Without the first two commits HAProxy crashes during execution of the
test. Without the last commit the test will fail, because no unique ID
is received.
2020-05-27 13:29:56 +02:00
Willy Tarreau
63a8738724 MEDIUM: pools: directly free objects when pools are too much crowded
During pool_free(), when the ->allocated value is 125% of needed_avg or
more, instead of putting the object back into the pool, it's immediately
freed using free(). By doing this we manage to significantly reduce the
amount of memory pinned in pools after transient traffic spikes.

During a test involving a constant load of 100 concurrent connections
each delivering 100 requests per second, the memory usage was a steady
21 MB RSS. Adding a 1 minute parallel load of 40k connections all looping
on 100kB objects made the memory usage climb to 938 MB before this patch.
With the patch it was only 660 MB. But when this parasit load stopped,
before the patch the RSS would remain at 938 MB while with the patch,
it went down to 480 then 180 MB after a few seconds, to stabilize around
69 MB after about 20 seconds.

This can be particularly important to improve reloads where the memory
has to be shared between the old and new process.

Another improvement would be welcome, we ought to have a periodic task
to check pools usage and continue to free up unused objects regardless
of any call to pool_free(), because the needed_avg value depends on the
past and will not cover recently refilled objects.
2020-05-27 08:32:42 +02:00
Willy Tarreau
a1e4f8c27c MINOR: pools: compute an estimate of each pool's average needed objects
This adds a sliding estimate of the pools' usage. The goal is to be able
to use this to start to more aggressively free memory instead of keeping
lots of unused objects in pools. The average is calculated as a sliding
average over the last 1024 consecutive measures of ->used during calls to
pool_free(), and is bumped up for 1/4 of its history from ->allocated when
allocation from the pool fails and results in a call to malloc().

The result is a floating value between ->used and ->allocated, that tries
to react fast to under-estimates that result in expensive malloc() but
still maintains itself well in case of stable usage, and progressively
goes down if usage shrinks over time.

This new metric is reported as "needed_avg" in "show pools".

Sadly due to yet another include dependency hell, we couldn't reuse the
functions from freq_ctr.h so they were temporarily duplicated into memory.h.
2020-05-27 08:32:42 +02:00
Olivier Houchard
68ad53cb37 BUG/MEDIUM: backend: set the connection owner to the session when using alpn.
In connect_server(), if we can't create the mux immediately because we have
to wait until the alpn is negociated, store the session as the connection's
owner. conn_create_mux() expects it to be set, and provides it to the mux
init() method. Failure to do so will result to crashes later if the
connection is private, and even if we didn't do so it would prevent connection
reuse for private connections.
This should fix github issue #651.
2020-05-27 01:34:33 +02:00
Christopher Faulet
d82056c319 BUG/MINOR: connection: Always get the stream when available to send PP2 line
When a PROXY protocol line must be sent, it is important to always get the
stream if it exists. It is mandatory to send an unique ID when the unique-id
option is enabled. In conn_si_send_proxy(), to get the stream, we first retrieve
the conn-stream attached to the backend connection. Then if the conn-stream data
callback is si_conn_cb, it is possible to get the stream. But for now, it only
works for connections with a multiplexer. Thus, for mux-less connections, the
unique ID is never sent. This happens for all SSL connections relying on the
alpn to choose the good multiplexer. But it is possible to use the context of
such connections to get the conn-stream.

The bug was introduced by the commit cf6e0c8a8 ("MEDIUM: proxy_protocol: Support
sending unique IDs using PPv2"). Thus, this patch must be backported to the same
versions as the commit above.
2020-05-26 17:39:39 +02:00
Christopher Faulet
3ab504f5ff BUG/MEDIUM: connection: Ignore PP2 unique ID for stream-less connections
It is possible to send a unique ID when the PROXY protocol v2 is used. It relies
on the stream to do so. So we must be sure to have a stream. Locally initiated
connections may not be linked to a stream. For instance, outgoing connections
created by health checks have no stream. Moreover, the stream is not retrieved
for mux-less connections (this bug will be fixed in another commit).

Unfortunately, in make_proxy_line_v2() function, the stream is not tested before
generating the unique-id.  This bug leads to a segfault when a health check is
performed for a server with the PROXY protocol v2 and the unique-id option
enabled. It also crashes for servers using SSL connections with alpn. The bug
was introduced by the commit cf6e0c8a8 ("MEDIUM: proxy_protocol: Support sending
unique IDs using PPv2")

This patch should fix the issue #640. It must be backported to the same versions
as the commit above.
2020-05-26 17:36:01 +02:00
Bertrand Jacquin
54f7823ed5 BUG/MEDIUM: contrib/spoa: do not register python3.8 if --embed fail
spoa server fails to build when python3.8 is not available. If
python3-config --embed fails, the output of the command is registered in
check_python_config.  However when it's later used to define
PYTHON_DEFAULT_INC and PYTHON_DEFAULT_LIB it's content does not match
and fallback to python2.7

Content of check_python_config when building with python3.6:

  Usage: bin/python3-config --prefix|--exec-prefix|--includes|--libs|--cflags|--ldflags|--extension-suffix|--help|--abiflags|--configdir python3

As we are only looking for return code, this commit ensure we always
ignore the output of python3-config or hash commands.
2020-05-26 16:16:19 +02:00
Christopher Faulet
b0b8b26e16 MINOR: checks: Remove useless tests on the connection and conn-stream
During an health check execution, the conn-stream and the conncetion may only be
NULL before the evaluation of the first rule, which is always a connect, or if
the first conn-stream allocation failed. Thus, in tcpcheck_main(), useless tests
on the conn-stream or on the connection have been removed. A comment has been
added to make it clear.

No backport needed.
2020-05-26 13:36:30 +02:00
Christopher Faulet
3cbdd22df5 BUG/MEDIUM: checks: Refresh the conn-stream and the connection after a connect
When a connect rule is evaluated, the conn-stream and the connection must be
refreshed in tcpcheck_main(). Otherwise, in case of synchronous connect, these
variables point on outdated values (NULL for the first connect or released
otherwise).

No backport needed.
2020-05-26 13:36:30 +02:00
Christopher Faulet
80f59e5db1 REGTESTS: Require the version 2.2 to execute lua/set_var
This script depends on LUA changes introduced in HAProxy 2.2 and not backported.
2020-05-26 13:36:30 +02:00
Christopher Faulet
185a64309c REGTESTS: Add missing OPENSSL to REQUIRE_OPTIONS for compression/lua_validation
The test uses the `ssl` keyword, add `OPENSSL` as a requirement.

Should be backported to all branches with that test.
2020-05-26 13:36:30 +02:00
Emeric Brun
99c453df9d MEDIUM: ring: new section ring to declare custom ring buffers.
It is possible to globally declare ring-buffers, to be used as target for log
servers or traces.

ring <ringname>
  Creates a new ring-buffer with name <ringname>.

description <text>
  The descritpition is an optional description string of the ring. It will
  appear on CLI. By default, <name> is reused to fill this field.

format <format>
  Format used to store events into the ring buffer.

  Arguments:
    <format> is the log format used when generating syslog messages. It may be
             one of the following :

      iso     A message containing only the ISO date, followed by the text.
              The PID, process name and system name are omitted. This is
              designed to be used with a local log server.

      raw     A message containing only the text. The level, PID, date, time,
              process name and system name are omitted. This is designed to be
              used in containers or during development, where the severity
              only depends on the file descriptor used (stdout/stderr). This
              is the default.

      rfc3164 The RFC3164 syslog message format. This is the default.
              (https://tools.ietf.org/html/rfc3164)

      rfc5424 The RFC5424 syslog message format.
              (https://tools.ietf.org/html/rfc5424)

      short   A message containing only a level between angle brackets such as
              '<3>', followed by the text. The PID, date, time, process name
              and system name are omitted. This is designed to be used with a
              local log server. This format is compatible with what the systemd
              logger consumes.

      timed   A message containing only a level between angle brackets such as
              '<3>', followed by ISO date and by the text. The PID, process
              name and system name are omitted. This is designed to be
              used with a local log server.

maxlen <length>
  The maximum length of an event message stored into the ring,
  including formatted header. If an event message is longer than
  <length>, it will be truncated to this length.

size <size>
  This is the optional size in bytes for the ring-buffer. Default value is
  set to BUFSIZE.

  Example:
    global
        log ring@myring local7

    ring myring
        description "My local buffer"
        format rfc3164
        maxlen 1200

Note: ring names are resolved during post configuration processing.
2020-05-26 08:03:15 +02:00
Tim Duesterhus
4e172c93f9 MEDIUM: lua: Add ifexist parameter to set_var
As discussed in GitHub issue #624 Lua scripts should not use
variables that are never going to be read, because the memory
for variable names is never going to be freed.

Add an optional `ifexist` parameter to the `set_var` function
that allows a Lua developer to set variables that are going to
be ignored if the variable name was not used elsewhere before.

Usually this mean that there is no `var()` sample fetch for the
variable in question within the configuration.
2020-05-25 08:12:35 +02:00
Tim Duesterhus
84ebc136a1 MINOR: lua: Make set_var() and unset_var() return success
This patch makes `set_var()` and `unset_var()` return a boolean indicating
success.
2020-05-25 08:12:31 +02:00
Tim Duesterhus
b4fac1eb3c MINOR: vars: Make vars_(un|)set_by_name(_ifexist|) return a success value
Change the return type from `void` to `int` and return whether setting
the variable was successful.
2020-05-25 08:12:27 +02:00
Tim Duesterhus
7329327333 CLEANUP: vars: Remove void vars_unset_by_name(const char*, size_t, struct sample*)
With "MINOR: lua: Use vars_unset_by_name_ifexist()" the last user was
removed and as outlined in that commit there is no good reason for this
function to exist.

May be backported together with the commit mentioned above.
2020-05-25 08:12:23 +02:00
Tim Duesterhus
94fa973f3e MINOR: lua: Use vars_unset_by_name_ifexist()
There is no good reason to register a variable name, just to unset
that value that could not even be set without the variable existing.

This change should be safe, may be backported if desired.
2020-05-25 08:12:17 +02:00
Tim Duesterhus
de5021a133 REGTESTS: Add missing OPENSSL to REQUIRE_OPTIONS for lua/txn_get_priv
The test uses the `ssl` keyword, add `OPENSSL` as a requirement.

Should be backported to all branches with that test.
2020-05-25 08:12:10 +02:00
Christopher Faulet
ed48657e02 REGTESTS: checks: Fix tls_health_checks when IPv6 addresses are used
In tls_health_checks.vtc, when IPv6 addresses are used, A config error is
reported because of the "addr" server parameter. Because there is no specified
port, the IPv6 address must be enclosed into brackets to be properly parsed. It
also works with IPv4 addresses. But instead, a dummy port is added to the addr
parameter. This way, we also check the port parameter, when specified, is used
in priority over the port found in the addr parameter.

This patch should fix the issue #646.
2020-05-25 08:06:30 +02:00
Christopher Faulet
99ff105bd1 MINOR: checks: Remove dead code from process_chk_conn()
With the checks refactoring, all connections are performed into a tcp-check
ruleset. So, in process_chk_conn(), it is no longer required to check
synchronous error on the connect when an health check is started. This part is
now handled in tcpcheck_main(). And because it is impossible to start a health
check with a connection, all tests on the connection during the health check
startup can be safely removed.

This patch should fix the issue #636. No backport needed.
2020-05-25 07:32:28 +02:00
Christopher Faulet
d6e3123176 BUG/MINOR: http-htx: Fix a leak on error path during http reply parsing
When "hdr" arguments of an http reply are parsed, the allocated header may leak
on error path. Adding it to the header list earlier fixes the issue.

This patch should partly fix the issue #645.

No backport needed.
2020-05-25 07:32:03 +02:00
Christopher Faulet
63d4824f64 BUG/MINOR: http-htx: Don't forget to release the http reply in release function
The http reply must be released in the function responsible to release it. This
leak was introduced when the http return was refactored to use http reply.

This patch should partly fix the issue #645.

No backport needed.
2020-05-25 07:32:03 +02:00
Willy Tarreau
73b943b964 [RELEASE] Released version 2.2-dev8
Released version 2.2-dev8 with the following main changes :
    - MINOR: checks: Improve report of unexpected errors for expect rules
    - MEDIUM: checks: Add matching on log-format string for expect rules
    - DOC: Fix req.body and co documentation to be accurate
    - MEDIUM: checks: Remove dedicated sample fetches and use response ones instead
    - CLEANUP: checks: sort and rename tcpcheck_expect_type types
    - MINOR: checks: Use dedicated actions to send log-format strings in send rules
    - MINOR: checks: Simplify matching on HTTP headers in HTTP expect rules
    - MINOR: checks/sample: Remove unnecessary tests on the sample session
    - REGTEST: checks: Adapt SSL error message reported when connection is rejected
    - MINOR: mworker: replace ha_alert by ha_warning when exiting successfuly
    - MINOR: checks: Support log-format string to set the URI for HTTP send rules
    - MINOR: checks: Support log-format string to set the body for HTTP send rules
    - DOC: Be more explicit about configurable check ok/error/timeout status
    - MINOR: checks: Make matching on HTTP headers for expect rules less obscure
    - BUG/MEDIUM: lua: Fix dumping of stick table entries for STD_T_DICT
    - BUG/MINOR: config: Make use_backend and use-server post-parsing less obscur
    - REGTESTS: make the http-check-send test require version 2.2
    - BUG/MINOR: http-ana: fix NTLM response parsing again
    - BUG/MEDIUM: http_ana: make the detection of NTLM variants safer
    - BUG/MINOR: cfgparse: Abort parsing the current line if an invalid \x sequence is encountered
    - MINOR: cfgparse: Improve error message for invalid \x sequences
    - CI: travis-ci: enable arm64 builds again
    - MEDIUM: ssl: increase default-dh-param to 2048
    - CI: travis-ci: skip pcre2 on arm64 build
    - CI: travis-ci: extend the build time for SSL to 60 minutes
    - CLEANUP: config: drop unused setting CONFIG_HAP_MEM_OPTIM
    - CLEANUP: config: drop unused setting CONFIG_HAP_INLINE_FD_SET
    - CLENAUP: config: move CONFIG_HAP_LOCKLESS_POOLS out of config.h
    - CLEANUP: remove THREAD_LOCAL from config.h
    - CI: travis-ci: upgrade LibreSSL versions
    - DOC: assorted typo fixes in the documentation
    - CI: extend spellchecker whitelist
    - CLEANUP: assorted typo fixes in the code and comments
    - MAJOR: contrib: porting spoa_server to support python3
    - BUG/MEDIUM: checks: Subscribe to I/O events on an unfinished connect
    - BUG/MINOR: checks: Don't subscribe to I/O events if it is already done
    - BUG/MINOR: checks: Rely on next I/O oriented rule when waiting for a connection
    - MINOR: checks: Don't try to send outgoing data if waiting to be able to send
    - MINOR: sample: Move aes_gcm_dec implementation into sample.c
    - MINOR: sample: Add digest and hmac converters
    - BUG/MEDIUM: checks: Subscribe to I/O events only if a mux was installed
    - BUG/MINOR: sample/ssl: Fix digest converter for openssl < 1.1.0
    - BUG/MINOR: pools: use %u not %d to report pool stats in "show pools"
    - BUG/MINOR: pollers: remove uneeded free in global init
    - CLEANUP: select: enhance readability in init
    - BUG/MINOR: soft-stop: always wake up waiting threads on stopping
    - MINOR: soft-stop: let the first stopper only signal other threads
    - BUILD: select: only declare existing local labels to appease clang
    - BUG/MEDIUM: streams: Remove SF_ADDR_SET if we're retrying due to L7 retry.
    - BUG/MEDIUM: stream: Only allow L7 retries when using HTTP.
    - DOC: retry-on can only be used with mode http
    - MEDIUM: ssl: allow to register callbacks for SSL/TLS protocol messages
    - MEDIUM: ssl: split ssl_sock_msgcbk() and use a new callback mechanism
    - MINOR: ssl: add a new function ssl_sock_get_ssl_object()
    - MEDIUM: ssl: use ssl_sock_get_ssl_object() in fetchers where appropriate
    - REORG: ssl: move macros and structure definitions to ssl_sock.h
    - CLEANUP: ssl: remove the shsess_* macros
    - REORG: move the crt-list structures in their own .h
    - REORG: ssl: move the ckch structures to types/ssl_ckch.h
    - CLEANUP: ssl: add ckch prototypes in proto/ssl_ckch.h
    - REORG: ssl: move crtlist functions to src/ssl_crtlist.c
    - CLEANUP: ssl: avoid circular dependencies in ssl_crtlist.h
    - REORG: ssl: move the ckch_store related functions to src/ssl_ckch.c
    - REORG: ssl: move ckch_inst functions to src/ssl_ckch.c
    - REORG: ssl: move the crt-list CLI functions in src/ssl_crtlist.c
    - REORG: ssl: move the CLI 'cert' functions to src/ssl_ckch.c
    - REORG: ssl: move ssl configuration to cfgparse-ssl.c
    - MINOR: ssl: remove static keyword in some SSL utility functions
    - REORG: ssl: move ssl_sock_ctx and fix cross-dependencies issues
    - REORG: ssl: move sample fetches to src/ssl_sample.c
    - REORG: ssl: move utility functions to src/ssl_utils.c
    - DOC: ssl: update MAINTAINERS file
    - CI: travis-ci: switch arm64 builds to use openssl from distro
    - MINOR: stats: Prepare for more accurate moving averages
    - MINOR: stats: Expose native cum_req metric for a server
    - MEDIUM: stats: Enable more accurate moving average calculation for stats
    - BUILD: ssl: include buffer common headers for ssl_sock_ctx
    - BUILD: ssl: include errno.h in ssl_crtlist.c
    - CLEANUP: acl: remove unused assignment
    - DOC/MINOR: halog: Add long help info for ic flag
    - BUILD: ssl: fix build without OPENSSL_NO_ENGINE
    - DOC: SPOE is no longer experimental
    - BUG/MINOR: cache: Don't needlessly test "cache" keyword in parse_cache_flt()
    - MINOR: config: Don't dump keywords if argument is NULL
    - MEDIUM: checks: Make post-41 the default mode for mysql checks
    - BUG/MINOR: logs: prevent double line returns in some events.
    - MEDIUM: sink: build header in sink_write for log formats
    - MEDIUM: logs: buffer targets now rely on new sink_write
    - MEDIUM: sink: add global statement to create a new ring (sink buffer)
    - MEDIUM: hpack: use a pool for the hpack table
    - BUG/MAJOR: mux-fcgi: Stop sending loop if FCGI stream is blocked for any reason
    - BUG/MEDIUM: ring: write-lock the ring while attaching/detaching
    - MINOR: applet: adopt the wait list entry from the CLI
    - MINOR: ring: make the applet code not depend on the CLI
    - Revert "MEDIUM: sink: add global statement to create a new ring (sink buffer)"
    - CI: travis-ci: fix libslz download URL
    - MINOR: ssl: split config and runtime variable for ssl-{min,max}-ver
    - CLEANUP: http_ana: Remove unused TXN flags
    - BUG/MINOR: http-rules: Mark http return rules as final
    - MINOR: http-htx: Add http_reply type based on what is used for http return rules
    - CLEANUP: http-htx: Rename http_error structure into http_error_msg
    - MINOR: http-rules: Use http_reply structure for http return rules
    - MINOR: http-htx: Use a dedicated function to release http_reply objects
    - MINOR: http-htx: Use a dedicated function to parse http reply arguments
    - MINOR: http-htx: Use a dedicated function to check http reply validity
    - MINOR: http-ana: Use a dedicated function to send a response from an http reply
    - MEDIUM: http-rules: Rely on http reply for http deny/tarpit rules
    - MINOR: http-htx: Store default error messages in a global http reply array
    - MINOR: http-htx: Store messages of an http-errors section in a http reply array
    - MINOR: http-htx: Store errorloc/errorfile messages in http replies
    - MINOR: proxy: Add references on http replies for proxy error messages
    - MINOR: http-htx: Use http reply from the http-errors section
    - MINOR: http-ana: Use a TXN flag to prevent after-response ruleset evaluation
    - MEDIUM: http-ana: Use http replies for HTTP error messages
    - CLEANUP: http-htx: Remove unused storage of error messages in buffers
    - MINOR: htx: Add a function to copy a buffer in an HTX message
    - CLEANUP: channel: Remove channel_htx_copy_msg() function
    - MINOR: http-ana: Add a function to write an http reply in an HTX message
    - MINOR: http-htx/proxy: Add http-error directive using http return syntax
    - DOC: Fix "errorfile" description in the configuration manual
    - BUG/MINOR: checks: Respect check-ssl param when a port or an addr is specified
    - BUILD: hpack: make sure the hpack table can still be built standalone
    - CONTRIB: hpack: make use of the simplified standalone HPACK API
    - MINOR: connection: add pp2-never-send-local to support old PP2 behavior
2020-05-22 16:19:04 +02:00
Willy Tarreau
119e50e0cc MINOR: connection: add pp2-never-send-local to support old PP2 behavior
A bug in the PROXY protocol v2 implementation was present in HAProxy up to
version 2.1, causing it to emit a PROXY command instead of a LOCAL command
for health checks. This is particularly minor but confuses some servers'
logs. Sadly, the bug was discovered very late and revealed that some servers
which possibly only tested their PROXY protocol implementation against
HAProxy fail to properly handle the LOCAL command, and permanently remain in
the "down" state when HAProxy checks them. When this happens, it is possible
to enable this global option to revert to the older (bogus) behavior for the
time it takes to contact the affected components' vendors and get them fixed.
This option is disabled by default and acts on all servers having the
"send-proxy-v2" statement.

Older versions were reverted to the old behavior and should not attempt to
be fixed by default again. However a variant of this patch could possibly
be implemented to ask to explicitly send LOCAL if needed by some servers.

More context here:
   https://www.mail-archive.com/haproxy@formilux.org/msg36890.html
   https://www.mail-archive.com/haproxy@formilux.org/msg37218.html
2020-05-22 13:55:32 +02:00
Willy Tarreau
318adf4364 CONTRIB: hpack: make use of the simplified standalone HPACK API
As reported in oss-fuzz issue 22388, the contrib/hpack code doesn't
build anymore for now. Thanks to previous patch we can now just define
HPACK_STANDALONE in the few files which include hpack-tbl.h and we can
declare a dummy pool to hold the DHT size. This is enough to make the
code work again outside of the haproxy process. It was successfully
tested using:

  $ echo 82 84 87 90 | ./decode
2020-05-22 12:13:43 +02:00
Willy Tarreau
0ff9b3d64f BUILD: hpack: make sure the hpack table can still be built standalone
Recent commit 2bdcc70fa7 ("MEDIUM: hpack: use a pool for the hpack table")
made the hpack code finally use a pool with very unintrusive code that was
assumed to be trivial enough to adjust if the code needed to be reused
outside of haproxy. Unfortunately the code in contrib/hpack already uses
it and broke the oss-fuzz tests as it doesn't build anymore.

This patch adds an HPACK_STANDALONE macro to decide if we should use the
pools or malloc+free. The resulting macros are called hpack_alloc() and
hpack_free() respectively, and the size must be passed into the pool
itself.
2020-05-22 12:13:43 +02:00
Christopher Faulet
66163ec616 BUG/MINOR: checks: Respect check-ssl param when a port or an addr is specified
When a check port or a check address is specified, the check transport layer is
ignored. So it is impossible to do a SSL check in this case. This bug was
introduced by the commit 8892e5d30 ("BUG/MEDIUM: server/checks: Init server
check during config validity check").

This patch should fix the issue #643. It must be backported to all branches
where the above commit was backported.
2020-05-20 22:55:05 +02:00
Christopher Faulet
7017067d68 DOC: Fix "errorfile" description in the configuration manual
"errorfile" description is outdated. Now, error messages are parsed at startup
and converted to HTX messages. So they must be valid according to the HTTP
standards. In addition, there is now a comment about the reserved buffer space
that should remain available to allow header rewrites.
2020-05-20 18:27:14 +02:00
Christopher Faulet
3b967c1210 MINOR: http-htx/proxy: Add http-error directive using http return syntax
The http-error directive can now be used instead of errorfile to define an error
message in a proxy section (including default sections). This directive uses the
same syntax that http return rules. The only real difference is the limitation
on status code that may be specified. Only status codes supported by errorfile
directives are supported for this new directive. Parsing of errorfile directive
remains independent from http-error parsing. But functionally, it may be
expressed in terms of http-errors :

  errorfile <status> <file> ==> http-errror status <status> errorfile <file>
2020-05-20 18:27:14 +02:00
Christopher Faulet
97e466c9e3 MINOR: http-ana: Add a function to write an http reply in an HTX message
When an error response is sent to a client, the write of the http reply in the
channel buffer and its sending are performed in different functions. The
http_reply_to_htx() function is used to write an http reply in HTX message. This
way, it could be possible to use the http replies in a different context.
2020-05-20 18:27:14 +02:00
Christopher Faulet
963ce5bc06 CLEANUP: channel: Remove channel_htx_copy_msg() function
This function is now unused. So it is removed.
2020-05-20 18:27:14 +02:00
Christopher Faulet
2056736453 MINOR: htx: Add a function to copy a buffer in an HTX message
The htx_copy_msg() function can now be used to copy the HTX message stored in a
buffer in an existing HTX message. It takes care to not overwrite existing
data. If the destination message is empty, a raw copy is performed. All the
message is copied or nothing.

This function is used instead of channel_htx_copy_msg().
2020-05-20 18:27:14 +02:00
Christopher Faulet
f1fedc3cce CLEANUP: http-htx: Remove unused storage of error messages in buffers
Now, error messages are all stored in http replies. So the storage as a buffer
can safely be removed.
2020-05-20 18:27:14 +02:00
Christopher Faulet
8dfeccf6d3 MEDIUM: http-ana: Use http replies for HTTP error messages
When HAProxy returns an http error message, the corresponding http reply is now
used instead of the buffer containing the corresponding HTX message. So,
http_error_message() function now returns the http reply to use for a given
stream. And the http_reply_and_close() function now relies on
http_reply_message() to send the response to the client.
2020-05-20 18:27:14 +02:00
Christopher Faulet
507479b096 MINOR: http-ana: Use a TXN flag to prevent after-response ruleset evaluation
The txn flag TX_CONST_REPLY may now be used to prevent after-response ruleset
evaluation. It is used if this ruleset evaluation failed on an internal error
response. Before, it was done incrementing the parameter <final>. But it is not
really convenient if an intermediary function is used to produce the
response. Using a txn flag could also be a good way to prevent after-response
ruleset evaluation in a different context.
2020-05-20 18:27:13 +02:00
Christopher Faulet
e29a97e51a MINOR: http-htx: Use http reply from the http-errors section
When an http reply is configured to use an error message from an http-errors
section, instead of referencing the error message, the http reply is used. To do
so the new http reply type HTTP_REPLY_INDIRECT has been added.
2020-05-20 18:27:13 +02:00
Christopher Faulet
40e8569676 MINOR: proxy: Add references on http replies for proxy error messages
Error messages defined in proxy section or inherited from a default section are
now also referenced using an array of http replies. This is done during the
configuration validity check.
2020-05-20 18:27:13 +02:00
Christopher Faulet
5809e10b48 MINOR: http-htx: Store errorloc/errorfile messages in http replies
During configuration parsing, error messages resulting of parsing of errorloc
and errorfile directives are now also stored as an http reply. So, for now,
these messages are stored as a buffer and as an http reply. To be able to
release all these http replies when haproxy is stopped, a global list is
used. We must do that because the same http reply may be referenced several
times by different proxies if it is defined in a default section.
2020-05-20 18:27:13 +02:00