Commit Graph

1740 Commits

Author SHA1 Message Date
Tim Duesterhus
f38175cf6e MINOR: sample: Add secure_memcmp converter
secure_memcmp compares two binary strings in constant time. It's only
available when haproxy is compiled with USE_OPENSSL.
2020-06-09 22:04:13 +02:00
William Lallemand
7c26ed73f7 DOC: add a line about comments in crt-list
Add a line about comments in crt-list.

Fix issue #514.
2020-06-03 17:36:39 +02:00
Christopher Faulet
89aed32bff MINOR: mux-h1/proxy: Add a proxy option to disable clear h2 upgrade
By default, HAProxy is able to implicitly upgrade an H1 client connection to an
H2 connection if the first request it receives from a given HTTP connection
matches the HTTP/2 connection preface. This way, it is possible to support H1
and H2 clients on a non-SSL connections. It could be a problem if for any
reason, the H2 upgrade is not acceptable. "option disable-h2-upgrade" may now be
used to disable it, per proxy. The main puprose of this option is to let an
admin to totally disable the H2 support for security reasons. Recently, a
critical issue in the HPACK decoder was fixed, forcing everyone to upgrade their
HAProxy version to fix the bug. It is possible to disable H2 for SSL
connections, but not on clear ones. This option would have been a viable
workaround.
2020-06-03 10:23:39 +02:00
Emeric Brun
530ba38a70 BUG/MINOR: peers: fix internal/network key type mapping.
Network types were directly and mistakenly mapped on sample types:

This patch fix the doc with values effectively used to keep backward
compatiblitiy on existing implementations.

In addition it adds an internal/network mapping for key types to avoid
further mistakes adding or modifying internals types.

This patch should be backported on all maintained branches,
particularly until v1.8 included for documentation part.
2020-06-02 16:25:19 +02:00
William Lallemand
50df1cb1e5 MINOR: ssl: set ssl-min-ver in ambiguous configurations
Using ssl-max-ver without ssl-min-ver is ambiguous.

When the ssl-min-ver is not configured, and ssl-max-ver is set to a
value lower than the default ssl-min-ver (which is TLSv1.2 currently),
set the ssl-min-ver to the value of ssl-max-ver, and emit a warning.
2020-06-02 11:13:12 +02:00
Emeric Brun
975564784f MEDIUM: ring: add new srv statement to support octet counting forward
log-proto <logproto>
  The "log-proto" specifies the protocol used to forward event messages to
  a server configured in a ring section. Possible values are "legacy"
  and "octet-count" corresponding respectively to "Non-transparent-framing"
  and "Octet counting" in rfc6587. "legacy" is the default.

Notes: a separated io_handler was created to avoid per messages test
and to prepare code to set different log protocols such as
request- response based ones.
2020-05-31 10:49:43 +02:00
Emeric Brun
494c505703 MEDIUM: ring: add server statement to forward messages from a ring
This patch adds new statement "server" into ring section, and the
related "timeout connect" and "timeout server".

server <name> <address> [param*]
  Used to configure a syslog tcp server to forward messages from ring buffer.
  This supports for all "server" parameters found in 5.2 paragraph.
  Some of these parameters are irrelevant for "ring" sections.

timeout connect <timeout>
  Set the maximum time to wait for a connection attempt to a server to succeed.

  Arguments :
    <timeout> is the timeout value specified in milliseconds by default, but
              can be in any other unit if the number is suffixed by the unit,
              as explained at the top of this document.

timeout server <timeout>
  Set the maximum time for pending data staying into output buffer.

  Arguments :
    <timeout> is the timeout value specified in milliseconds by default, but
              can be in any other unit if the number is suffixed by the unit,
              as explained at the top of this document.

  Example:
    global
        log ring@myring local7

    ring myring
        description "My local buffer"
        format rfc3164
        maxlen 1200
        size 32764
        timeout connect 5s
        timeout server 10s
        server mysyslogsrv 127.0.0.1:6514
2020-05-31 10:46:13 +02:00
Christopher Faulet
612f2eafe9 MINOR: http-ana: Use proxy's error replies to emit 401/407 responses
There is no reason to not use proxy's error replies to emit 401/407
responses. The function http_reply_40x_unauthorized(), responsible to emit those
responses, is not really complex. It only adds a
WWW-Authenticate/Proxy-Authenticate header to a generic message.

So now, error replies can be defined for 401 and 407 status codes, using
errorfile or http-error directives. When an http-request auth rule is evaluated,
the corresponding error reply is used. For 401 responses, all occurrences of the
WWW-Authenticate header are removed and replaced by a new one with a basic
authentication challenge for the configured realm. For 407 responses, the same
is done on the Proxy-Authenticate header. If the error reply must not be
altered, "http-request return" rule must be used instead.
2020-05-28 15:07:20 +02:00
Emeric Brun
99c453df9d MEDIUM: ring: new section ring to declare custom ring buffers.
It is possible to globally declare ring-buffers, to be used as target for log
servers or traces.

ring <ringname>
  Creates a new ring-buffer with name <ringname>.

description <text>
  The descritpition is an optional description string of the ring. It will
  appear on CLI. By default, <name> is reused to fill this field.

format <format>
  Format used to store events into the ring buffer.

  Arguments:
    <format> is the log format used when generating syslog messages. It may be
             one of the following :

      iso     A message containing only the ISO date, followed by the text.
              The PID, process name and system name are omitted. This is
              designed to be used with a local log server.

      raw     A message containing only the text. The level, PID, date, time,
              process name and system name are omitted. This is designed to be
              used in containers or during development, where the severity
              only depends on the file descriptor used (stdout/stderr). This
              is the default.

      rfc3164 The RFC3164 syslog message format. This is the default.
              (https://tools.ietf.org/html/rfc3164)

      rfc5424 The RFC5424 syslog message format.
              (https://tools.ietf.org/html/rfc5424)

      short   A message containing only a level between angle brackets such as
              '<3>', followed by the text. The PID, date, time, process name
              and system name are omitted. This is designed to be used with a
              local log server. This format is compatible with what the systemd
              logger consumes.

      timed   A message containing only a level between angle brackets such as
              '<3>', followed by ISO date and by the text. The PID, process
              name and system name are omitted. This is designed to be
              used with a local log server.

maxlen <length>
  The maximum length of an event message stored into the ring,
  including formatted header. If an event message is longer than
  <length>, it will be truncated to this length.

size <size>
  This is the optional size in bytes for the ring-buffer. Default value is
  set to BUFSIZE.

  Example:
    global
        log ring@myring local7

    ring myring
        description "My local buffer"
        format rfc3164
        maxlen 1200

Note: ring names are resolved during post configuration processing.
2020-05-26 08:03:15 +02:00
Tim Duesterhus
4e172c93f9 MEDIUM: lua: Add ifexist parameter to set_var
As discussed in GitHub issue #624 Lua scripts should not use
variables that are never going to be read, because the memory
for variable names is never going to be freed.

Add an optional `ifexist` parameter to the `set_var` function
that allows a Lua developer to set variables that are going to
be ignored if the variable name was not used elsewhere before.

Usually this mean that there is no `var()` sample fetch for the
variable in question within the configuration.
2020-05-25 08:12:35 +02:00
Willy Tarreau
73b943b964 [RELEASE] Released version 2.2-dev8
Released version 2.2-dev8 with the following main changes :
    - MINOR: checks: Improve report of unexpected errors for expect rules
    - MEDIUM: checks: Add matching on log-format string for expect rules
    - DOC: Fix req.body and co documentation to be accurate
    - MEDIUM: checks: Remove dedicated sample fetches and use response ones instead
    - CLEANUP: checks: sort and rename tcpcheck_expect_type types
    - MINOR: checks: Use dedicated actions to send log-format strings in send rules
    - MINOR: checks: Simplify matching on HTTP headers in HTTP expect rules
    - MINOR: checks/sample: Remove unnecessary tests on the sample session
    - REGTEST: checks: Adapt SSL error message reported when connection is rejected
    - MINOR: mworker: replace ha_alert by ha_warning when exiting successfuly
    - MINOR: checks: Support log-format string to set the URI for HTTP send rules
    - MINOR: checks: Support log-format string to set the body for HTTP send rules
    - DOC: Be more explicit about configurable check ok/error/timeout status
    - MINOR: checks: Make matching on HTTP headers for expect rules less obscure
    - BUG/MEDIUM: lua: Fix dumping of stick table entries for STD_T_DICT
    - BUG/MINOR: config: Make use_backend and use-server post-parsing less obscur
    - REGTESTS: make the http-check-send test require version 2.2
    - BUG/MINOR: http-ana: fix NTLM response parsing again
    - BUG/MEDIUM: http_ana: make the detection of NTLM variants safer
    - BUG/MINOR: cfgparse: Abort parsing the current line if an invalid \x sequence is encountered
    - MINOR: cfgparse: Improve error message for invalid \x sequences
    - CI: travis-ci: enable arm64 builds again
    - MEDIUM: ssl: increase default-dh-param to 2048
    - CI: travis-ci: skip pcre2 on arm64 build
    - CI: travis-ci: extend the build time for SSL to 60 minutes
    - CLEANUP: config: drop unused setting CONFIG_HAP_MEM_OPTIM
    - CLEANUP: config: drop unused setting CONFIG_HAP_INLINE_FD_SET
    - CLENAUP: config: move CONFIG_HAP_LOCKLESS_POOLS out of config.h
    - CLEANUP: remove THREAD_LOCAL from config.h
    - CI: travis-ci: upgrade LibreSSL versions
    - DOC: assorted typo fixes in the documentation
    - CI: extend spellchecker whitelist
    - CLEANUP: assorted typo fixes in the code and comments
    - MAJOR: contrib: porting spoa_server to support python3
    - BUG/MEDIUM: checks: Subscribe to I/O events on an unfinished connect
    - BUG/MINOR: checks: Don't subscribe to I/O events if it is already done
    - BUG/MINOR: checks: Rely on next I/O oriented rule when waiting for a connection
    - MINOR: checks: Don't try to send outgoing data if waiting to be able to send
    - MINOR: sample: Move aes_gcm_dec implementation into sample.c
    - MINOR: sample: Add digest and hmac converters
    - BUG/MEDIUM: checks: Subscribe to I/O events only if a mux was installed
    - BUG/MINOR: sample/ssl: Fix digest converter for openssl < 1.1.0
    - BUG/MINOR: pools: use %u not %d to report pool stats in "show pools"
    - BUG/MINOR: pollers: remove uneeded free in global init
    - CLEANUP: select: enhance readability in init
    - BUG/MINOR: soft-stop: always wake up waiting threads on stopping
    - MINOR: soft-stop: let the first stopper only signal other threads
    - BUILD: select: only declare existing local labels to appease clang
    - BUG/MEDIUM: streams: Remove SF_ADDR_SET if we're retrying due to L7 retry.
    - BUG/MEDIUM: stream: Only allow L7 retries when using HTTP.
    - DOC: retry-on can only be used with mode http
    - MEDIUM: ssl: allow to register callbacks for SSL/TLS protocol messages
    - MEDIUM: ssl: split ssl_sock_msgcbk() and use a new callback mechanism
    - MINOR: ssl: add a new function ssl_sock_get_ssl_object()
    - MEDIUM: ssl: use ssl_sock_get_ssl_object() in fetchers where appropriate
    - REORG: ssl: move macros and structure definitions to ssl_sock.h
    - CLEANUP: ssl: remove the shsess_* macros
    - REORG: move the crt-list structures in their own .h
    - REORG: ssl: move the ckch structures to types/ssl_ckch.h
    - CLEANUP: ssl: add ckch prototypes in proto/ssl_ckch.h
    - REORG: ssl: move crtlist functions to src/ssl_crtlist.c
    - CLEANUP: ssl: avoid circular dependencies in ssl_crtlist.h
    - REORG: ssl: move the ckch_store related functions to src/ssl_ckch.c
    - REORG: ssl: move ckch_inst functions to src/ssl_ckch.c
    - REORG: ssl: move the crt-list CLI functions in src/ssl_crtlist.c
    - REORG: ssl: move the CLI 'cert' functions to src/ssl_ckch.c
    - REORG: ssl: move ssl configuration to cfgparse-ssl.c
    - MINOR: ssl: remove static keyword in some SSL utility functions
    - REORG: ssl: move ssl_sock_ctx and fix cross-dependencies issues
    - REORG: ssl: move sample fetches to src/ssl_sample.c
    - REORG: ssl: move utility functions to src/ssl_utils.c
    - DOC: ssl: update MAINTAINERS file
    - CI: travis-ci: switch arm64 builds to use openssl from distro
    - MINOR: stats: Prepare for more accurate moving averages
    - MINOR: stats: Expose native cum_req metric for a server
    - MEDIUM: stats: Enable more accurate moving average calculation for stats
    - BUILD: ssl: include buffer common headers for ssl_sock_ctx
    - BUILD: ssl: include errno.h in ssl_crtlist.c
    - CLEANUP: acl: remove unused assignment
    - DOC/MINOR: halog: Add long help info for ic flag
    - BUILD: ssl: fix build without OPENSSL_NO_ENGINE
    - DOC: SPOE is no longer experimental
    - BUG/MINOR: cache: Don't needlessly test "cache" keyword in parse_cache_flt()
    - MINOR: config: Don't dump keywords if argument is NULL
    - MEDIUM: checks: Make post-41 the default mode for mysql checks
    - BUG/MINOR: logs: prevent double line returns in some events.
    - MEDIUM: sink: build header in sink_write for log formats
    - MEDIUM: logs: buffer targets now rely on new sink_write
    - MEDIUM: sink: add global statement to create a new ring (sink buffer)
    - MEDIUM: hpack: use a pool for the hpack table
    - BUG/MAJOR: mux-fcgi: Stop sending loop if FCGI stream is blocked for any reason
    - BUG/MEDIUM: ring: write-lock the ring while attaching/detaching
    - MINOR: applet: adopt the wait list entry from the CLI
    - MINOR: ring: make the applet code not depend on the CLI
    - Revert "MEDIUM: sink: add global statement to create a new ring (sink buffer)"
    - CI: travis-ci: fix libslz download URL
    - MINOR: ssl: split config and runtime variable for ssl-{min,max}-ver
    - CLEANUP: http_ana: Remove unused TXN flags
    - BUG/MINOR: http-rules: Mark http return rules as final
    - MINOR: http-htx: Add http_reply type based on what is used for http return rules
    - CLEANUP: http-htx: Rename http_error structure into http_error_msg
    - MINOR: http-rules: Use http_reply structure for http return rules
    - MINOR: http-htx: Use a dedicated function to release http_reply objects
    - MINOR: http-htx: Use a dedicated function to parse http reply arguments
    - MINOR: http-htx: Use a dedicated function to check http reply validity
    - MINOR: http-ana: Use a dedicated function to send a response from an http reply
    - MEDIUM: http-rules: Rely on http reply for http deny/tarpit rules
    - MINOR: http-htx: Store default error messages in a global http reply array
    - MINOR: http-htx: Store messages of an http-errors section in a http reply array
    - MINOR: http-htx: Store errorloc/errorfile messages in http replies
    - MINOR: proxy: Add references on http replies for proxy error messages
    - MINOR: http-htx: Use http reply from the http-errors section
    - MINOR: http-ana: Use a TXN flag to prevent after-response ruleset evaluation
    - MEDIUM: http-ana: Use http replies for HTTP error messages
    - CLEANUP: http-htx: Remove unused storage of error messages in buffers
    - MINOR: htx: Add a function to copy a buffer in an HTX message
    - CLEANUP: channel: Remove channel_htx_copy_msg() function
    - MINOR: http-ana: Add a function to write an http reply in an HTX message
    - MINOR: http-htx/proxy: Add http-error directive using http return syntax
    - DOC: Fix "errorfile" description in the configuration manual
    - BUG/MINOR: checks: Respect check-ssl param when a port or an addr is specified
    - BUILD: hpack: make sure the hpack table can still be built standalone
    - CONTRIB: hpack: make use of the simplified standalone HPACK API
    - MINOR: connection: add pp2-never-send-local to support old PP2 behavior
2020-05-22 16:19:04 +02:00
Willy Tarreau
119e50e0cc MINOR: connection: add pp2-never-send-local to support old PP2 behavior
A bug in the PROXY protocol v2 implementation was present in HAProxy up to
version 2.1, causing it to emit a PROXY command instead of a LOCAL command
for health checks. This is particularly minor but confuses some servers'
logs. Sadly, the bug was discovered very late and revealed that some servers
which possibly only tested their PROXY protocol implementation against
HAProxy fail to properly handle the LOCAL command, and permanently remain in
the "down" state when HAProxy checks them. When this happens, it is possible
to enable this global option to revert to the older (bogus) behavior for the
time it takes to contact the affected components' vendors and get them fixed.
This option is disabled by default and acts on all servers having the
"send-proxy-v2" statement.

Older versions were reverted to the old behavior and should not attempt to
be fixed by default again. However a variant of this patch could possibly
be implemented to ask to explicitly send LOCAL if needed by some servers.

More context here:
   https://www.mail-archive.com/haproxy@formilux.org/msg36890.html
   https://www.mail-archive.com/haproxy@formilux.org/msg37218.html
2020-05-22 13:55:32 +02:00
Christopher Faulet
7017067d68 DOC: Fix "errorfile" description in the configuration manual
"errorfile" description is outdated. Now, error messages are parsed at startup
and converted to HTX messages. So they must be valid according to the HTTP
standards. In addition, there is now a comment about the reserved buffer space
that should remain available to allow header rewrites.
2020-05-20 18:27:14 +02:00
Christopher Faulet
3b967c1210 MINOR: http-htx/proxy: Add http-error directive using http return syntax
The http-error directive can now be used instead of errorfile to define an error
message in a proxy section (including default sections). This directive uses the
same syntax that http return rules. The only real difference is the limitation
on status code that may be specified. Only status codes supported by errorfile
directives are supported for this new directive. Parsing of errorfile directive
remains independent from http-error parsing. But functionally, it may be
expressed in terms of http-errors :

  errorfile <status> <file> ==> http-errror status <status> errorfile <file>
2020-05-20 18:27:14 +02:00
Christopher Faulet
5cb513abeb MEDIUM: http-rules: Rely on http reply for http deny/tarpit rules
"http-request deny", "http-request tarpit" and "http-response deny" rules now
use the same syntax than http return rules and internally rely on the http
replies. The behaviour is not the same when no argument is specified (or only
the status code). For http replies, a dummy response is produced, with no
payload. For old deny/tarpit rules, the proxy's error messages are used. Thus,
to be compatible with existing configuration, the "default-errorfiles" parameter
is implied. For instance :

  http-request deny deny_status 404

is now an alias of

  http-request deny status 404 default-errorfiles
2020-05-20 18:27:13 +02:00
Willy Tarreau
d68a6927f7 Revert "MEDIUM: sink: add global statement to create a new ring (sink buffer)"
This reverts commit 957ec59571.

As discussed with Emeric, the current syntax is not extensible enough,
this will be turned to a section instead in a forthcoming patch.
2020-05-20 12:06:16 +02:00
Emeric Brun
957ec59571 MEDIUM: sink: add global statement to create a new ring (sink buffer)
This patch adds the new global statement:
ring <name> [desc <desc>] [format <format>] [size <size>] [maxlen <length>]
  Creates a named ring buffer which could be used on log line for instance.

  <desc> is an optionnal description string of the ring. It will appear on
         CLI. By default, <name> is reused to fill this field.

  <format> is the log format used when generating syslog messages. It may be
           one of the following :

    iso       A message containing only the ISO date, followed by the text.
              The PID, process name and system name are omitted. This is
              designed to be used with a local log server.

    raw       A message containing only the text. The level, PID, date, time,
              process name and system name are omitted. This is designed to be
              used in containers or during development, where the severity only
              depends on the file descriptor used (stdout/stderr). This is
              the default.

    rfc3164   The RFC3164 syslog message format. This is the default.
              (https://tools.ietf.org/html/rfc3164)

    rfc5424   The RFC5424 syslog message format.
              (https://tools.ietf.org/html/rfc5424)

    short     A message containing only a level between angle brackets such as
              '<3>', followed by the text. The PID, date, time, process name
              and system name are omitted. This is designed to be used with a
              local log server. This format is compatible with what the systemd
              logger consumes.

    timed     A message containing only a level between angle brackets such as
              '<3>', followed by ISO date and by the text. The PID, process
              name and system name are omitted. This is designed to be
              used with a local log server.

  <length> is the maximum length of event message stored into the ring,
           including formatted header. If the event message is longer
           than <length>, it would be truncated to this length.

  <name> is the ring identifier, which follows the same naming convention as
         proxies and servers.

  <size> is the optionnal size in bytes. Default value is set to BUFSIZE.

Note: Historically sink's name and desc were refs on const strings. But with new
configurable rings a dynamic allocation is needed.
2020-05-19 11:04:11 +02:00
Christopher Faulet
62f79fe68a MEDIUM: checks: Make post-41 the default mode for mysql checks
MySQL 4.1 is old enough to be the default mode for mysql checks. So now, once a
username is defined, post-41 mode is automatically used. To do mysql checks on
previous MySQL version, the argument "pre-41" must be used.

Note, it is a compatibility breakage for everyone using an antique and
unsupported MySQL version.
2020-05-18 18:32:09 +02:00
Christopher Faulet
3b78809de0 DOC: SPOE is no longer experimental
The SPOE was marked as experiemental since the begining. But, it is no longer
true. This can be an obstacle to its use.
2020-05-18 17:47:18 +02:00
Jerome Magnin
5ce3c14aa9 DOC: retry-on can only be used with mode http
The documentation for retry-on hints at it being meant to be used
in conjuction with mode http, but since we've a had bug report
involving mode tcp and retry-on, lets make it explicit in the
documentation that it only works with mode http and will be
ignored otherwise.
2020-05-14 09:43:13 +02:00
Patrick Gansterer
8e36651ed3 MINOR: sample: Add digest and hmac converters
Make the digest and HMAC function of OpenSSL accessible to the user via
converters. They can be used to sign and validate content.

Reviewed-by: Tim Duesterhus <tim@bastelstu.be>
2020-05-12 10:08:11 +02:00
Ilya Shipitsin
4329a9af5a DOC: assorted typo fixes in the documentation
This is the 8th round of cleanups in various docs
2020-05-11 10:10:15 +02:00
Willy Tarreau
3ba77d29ac MEDIUM: ssl: increase default-dh-param to 2048
For 6 years now we've been seeing a warning suggesting to set dh-param
beyond 1024 if possible when it was not set. It's about time to do it
and get rid of this warning since most users seem to already use 2048.
It will remain possible to set a lower value of course, so only those
who were experiencing the warning and were relying on the default value
may notice a change (higher CPU usage). For more context, please refer
to this thread :

  https://www.mail-archive.com/haproxy@formilux.org/msg37226.html

This commit removes a big chunk of code which happened to be needed
exclusively to figure if it was required to emit a warning or not :-)
2020-05-08 09:36:37 +02:00
Christopher Faulet
d888f0fc6e DOC: Be more explicit about configurable check ok/error/timeout status
It is possible to configure the check status on success, on error and on
timeout, for http-check and tcp-check expect rules. But the documentation relies
on internal names. These names are reported on the stats and are describe in the
management guide. But it is probably a good idea to be more explicit too in the
doc describing these options.
2020-05-07 07:40:18 +02:00
Christopher Faulet
574e7bd7f3 MINOR: checks: Support log-format string to set the body for HTTP send rules
For http-check send rules, it is now possible to use a log-format string to set
the request's body. the keyword "body-lf" should be used instead of "body". If the
string eval fails, no body is added.
2020-05-06 18:04:05 +02:00
Christopher Faulet
7c95f5f22b MINOR: checks: Support log-format string to set the URI for HTTP send rules
For http-check send rules, it is now possible to use a log-format string to set
the request URI. the keyword "uri-lf" should be used instead of "uri". If the
string eval fails, we fall back on the default uri "/".
2020-05-06 18:04:05 +02:00
Christopher Faulet
b5594265d2 MINOR: checks: Simplify matching on HTTP headers in HTTP expect rules
Extra parameters on http-check expect rules, for the header matching method, to
use log-format string or to match full header line have been removed. There is
now separate matching methods to match a full header line or to match each
comma-separated values. "http-check expect fhdr" must be used in the first case,
and "http-check expect hdr" in the second one. In addition, to match log-format
header name or value, "-lf" suffix must be added to "name" or "value"
keyword. For intance:

   http-check expect hdr name "set-cookie" value-lf -m beg "sessid=%[var(check.cookie)]"

Thanks to this changes, each parameter may only be interpreted in one way.
2020-05-06 12:42:36 +02:00
Christopher Faulet
b50b3e6d0a MINOR: checks: Use dedicated actions to send log-format strings in send rules
Following actions have been added to send log-format strings from a tcp-check
ruleset instead the log-format parameter:

  * tcp-check send-lf <fmt>
  * tcp-check send-binary-lf <fmt>

It is easier for tools generating configurations. Each action may only be
interpreted in one way.
2020-05-06 12:41:57 +02:00
Christopher Faulet
e596d184be MEDIUM: checks: Remove dedicated sample fetches and use response ones instead
All sample fetches in the scope "check." have been removed. Response sample
fetches must be used instead. It avoids keyword duplication. So, for instance,
res.hdr() must be now used instead of check.hdr().

To do so, following sample fetches have been added on the response :

  * res.body, res.body_len and res.body_size
  * res.hdrs and res.hdrs_bin

Sample feches dealing with the response's body are only useful in the health
checks context. When called from a stream context, there is no warranty on the
body presence. There is no option to wait the response's body.
2020-05-06 12:37:43 +02:00
Christopher Faulet
af4dc4ccaa DOC: Fix req.body and co documentation to be accurate
Because the HTX is the only mode to represent HTTP data, "option
http-request-buffer" is no longer mandatory to have body data. Without this
option, there is no warranty on the body presence. So it is recommanded to use
it. But it is not a requirement. In addition, the note about chunked body is
removed because outdated.
2020-05-06 09:06:01 +02:00
Christopher Faulet
aaab0836d9 MEDIUM: checks: Add matching on log-format string for expect rules
It is now possible to use log-format string (or hexadecimal string for the
binary version) to match a content in tcp-check based expect rules. For
hexadecimal log-format string, the conversion in binary is performed after the
string evaluation, during health check execution. The pattern keywords to use
are "string-lf" for the log-format string and "binary-lf" for the hexadecimal
log-format string.
2020-05-06 08:31:29 +02:00
Willy Tarreau
fc0b8f39a6 [RELEASE] Released version 2.2-dev7
Released version 2.2-dev7 with the following main changes :
    - MINOR: version: Show uname output in display_version()
    - CI: run weekly OpenSSL "no-deprecated" builds
    - CLEANUP: log: fix comment of parse_logformat_string()
    - DOC: Improve documentation on http-request set-src
    - MINOR: ssl/cli: disallow SSL options for directory in 'add ssl crt-list'
    - MINOR: ssl/cli: restrain certificate path when inserting into a directory
    - MINOR: ssl: add ssl-skip-self-issued-ca global option
    - BUG/MINOR: ssl: default settings for ssl server options are not used
    - MINOR: config: add a global directive to set default SSL curves
    - BUG/MEDIUM: http-ana: Handle NTLM messages correctly.
    - DOC: internals: update the SSL architecture schema
    - BUG/MINOR: tools: fix the i386 version of the div64_32 function
    - BUG/MINOR: mux-fcgi/trace: fix wrong set of trace flags in fcgi_strm_add_eom()
    - BUG/MINOR: http: make url_decode() optionally convert '+' to SP
    - DOC: option logasap does not depend on mode
    - MEDIUM: memory: make pool_gc() run under thread isolation
    - MINOR: contrib: make the peers wireshark dissector a plugin
    - BUG/MINOR: http-ana: Throw a 500 error if after-response ruleset fails on errors
    - BUG/MINOR: check: Update server address and port to execute an external check
    - MINOR: mini-clist: Add functions to iterate backward on a list
    - MINOR: checks: Add a way to send custom headers and payload during http chekcs
    - MINOR: server: respect warning and alert semantic
    - BUG/MINOR: checks: Respect the no-check-ssl option
    - BUG/MEDIUM: server/checks: Init server check during config validity check
    - CLEANUP: checks: Don't export anymore init_check and srv_check_healthcheck_port
    - BUG/MINOR: checks: chained expect will not properly wait for enough data
    - BUG/MINOR: checks: Forbid tcp-check lines in default section as documented
    - MINOR: checks: Use an enum to describe the tcp-check rule type
    - MINOR: checks: Simplify connection flag parsing in tcp-check connect
    - MEDIUM: checks: rewind to the first inverse expect rule of a chain on new data
    - MINOR: checks: simplify tcp expect config parser
    - MINOR: checks: add min-recv tcp-check expect option
    - MINOR: checks: add linger option to tcp connect
    - MINOR: checks: define a tcp expect type
    - MEDIUM: checks: rewrite tcp-check expect block
    - MINOR: checks: Stop xform buffers to null-terminated string for tcp-check rules
    - MINOR: checks: add rbinary expect match type
    - MINOR: checks: Simplify functions to get step id and comment
    - MEDIUM: checks: capture groups in expect regexes
    - MINOR: checks: Don't use a static tcp rule list head
    - MEDIUM: checks: Use a non-comment rule iterator to get next rule
    - MEDIUM: proxy/checks: Register a keyword to parse tcp-check rules
    - MINOR: checks: Set the tcp-check rule index during parsing
    - MINOR: checks: define tcp-check send type
    - MINOR: checks: define a tcp-check connect type
    - MEDIUM: checks: Add implicit tcp-check connect rule
    - MAJOR: checks: Refactor and simplify the tcp-check loop
    - MEDIUM: checks: Associate a session to each tcp-check healthcheck
    - MINOR: checks/vars: Add a check scope for variables
    - MEDIUM: checks: Parse custom action rules in tcp-checks
    - MINOR: checks: Add support to set-var and unset-var rules in tcp-checks
    - MINOR: checks: Add the sni option for tcp-check connect rules
    - MINOR: checks: Add the via-socks4 option for tcp-check connect rules
    - MINOR: checks: Add the alpn option for tcp-check connect rules
    - MINOR: ssl: Export a generic function to parse an alpn string
    - MINOR: checks: Add the default option for tcp-check connect rules
    - MINOR: checks: Add the addr option for tcp-check connect rule
    - MEDIUM: checks: Support expression to set the port
    - MEDIUM: checks: Support log-format strings for tcp-check send rules
    - MINOR: log: Don't depends on a stream to process samples in log-format string
    - MINOR: log: Don't systematically set LW_REQ when a sample expr is added
    - MEDIUM: checks: Add a shared list of tcp-check rules
    - MINOR: sample: add htonl converter
    - MINOR: sample: add cut_crlf converter
    - MINOR: sample: add ltrim converter
    - MINOR: sample: add rtrim converter
    - MINOR: checks: Use a name for the healthcheck status enum
    - MINOR: checks: Add option to tcp-check expect rules to customize error status
    - MINOR: checks: Merge tcp-check comment rules with the others at config parsing
    - MINOR: checks: Add a sample fetch to extract a block from the input check buffer
    - MEDIUM: checks: Add on-error/on-success option on tcp-check expect rules
    - MEDIUM: checks: Add status-code sample expression on tcp-check expect rules
    - MINOR: checks: Relax the default option for tcp-check connect rules
    - MEDIUM: checks: Add a list of vars to set before executing a tpc-check ruleset
    - MINOR: checks: Export the tcpcheck_eval_ret enum
    - MINOR: checks: Use dedicated function to handle onsuccess/onerror messages
    - MINOR: checks: Support custom functions to eval a tcp-check expect rules
    - MEDIUM: checks: Implement redis check using tcp-check rules
    - MEDIUM: checks: Implement ssl-hello check using tcp-check rules
    - MEDIUM: checks: Implement smtp check using tcp-check rules
    - MEDIUM: checks: Implement postgres check using tcp-check rules
    - MEDIUM: checks: Implement MySQL check using tcp-check rules
    - MEDIUM: checks: Implement LDAP check using tcp-check rules
    - MEDIUM: checks: Implement SPOP check using tcp-check rules
    - MINOR: server/checks: Move parsing of agent keywords in checks.c
    - MINOR: server/checks: Move parsing of server check keywords in checks.c
    - MEDIUM: checks: Implement agent check using tcp-check rules
    - REGTEST: Adapt regtests about checks to recent changes
    - MINOR: Produce tcp-check info message for pure tcp-check rules only
    - MINOR: checks: Add an option to set success status of tcp-check expect rules
    - MINOR: checks: Improve log message of tcp-checks on success
    - MINOR: proxy/checks: Move parsing of httpchk option in checks.c
    - MINOR: proxy/checks: Move parsing of tcp-check option in checks.c
    - MINOR: proxy/checks: Register a keyword to parse http-check rules
    - MINOR: proxy/checks: Move parsing of external-check option in checks.c
    - MINOR: proxy/checks: Register a keyword to parse external-check rules
    - MEDIUM: checks: Use a shared ruleset to store tcp-check rules
    - MINOR: checks: Use an indirect string to represent the expect matching string
    - MINOR: checks: Introduce flags to configure in tcp-check expect rules
    - MINOR: standard: Add my_memspn and my_memcspn
    - MINOR: checks: Add a reverse non-comment rule iterator to get last rule
    - MAJOR: checks: Implement HTTP check using tcp-check rules
    - MINOR: checks: Make resume conditions more explicit in tcpcheck_main()
    - MINOR: connection: Add macros to know if a conn or a cs uses an HTX mux
    - MEDIUM: checks: Refactor how data are received in tcpcheck_main()
    - MINOR: checks/obj_type: Add a new object type for checks
    - BUG/MINOR: obj_type: Handle stream object in obj_base_ptr() function
    - MINOR: checks: Use the check as origin when a session is created
    - MINOR: checks: Add a mux proto to health-check and tcp-check connect rule
    - MINOR: connection: Add a function to install a mux for a health-check
    - MAJOR: checks: Use the best mux depending on the protocol for health checks
    - MEDIUM: checks: Implement default TCP check using tcp-check rules
    - MINOR: checks: Remove unused code about pure TCP checks
    - CLEANUP: checks: Reorg checks.c file to be more readable
    - REGTEST: Fix reg-tests about health-checks to adapt them to recent changes
    - MINOR: ist: Add a function to retrieve the ist pointer
    - MINOR: checks: Use ist API as far as possible
    - BUG/MEDIUM: checks: Be sure to subscribe for sends if outgoing data remains
    - MINOR: checks: Use a tree instead of a list to store tcp-check rulesets
    - BUG/MINOR: checks: Send the right amount of outgoing data for HTTP checks
    - REGTEST: Add scripts to test based tcp-check health-checks
    - Revert "MEDIUM: checks: capture groups in expect regexes"
    - DOC: Add documentation about comments for tcp-check and http-check directives
    - DOC: Fix the tcp-check and http-check directives layout
    - BUG/MEDIUM: checks: Use the mux protocol specified on the server line
    - MINOR: checks: Support mux protocol definition for tcp and http health checks
    - BUG/MINOR: mux-fcgi: Be sure to have a connection as session's origin to use it
    - MINOR: checks: Support list of status codes on http-check expect rules
    - BUG/MEDIUM: checks: Unsubscribe to mux events when a conn-stream is destroyed
    - REGTEST: Add a script to validate agent checks
    - BUG/MINOR: server: Fix server_finalize_init() to avoid unused variable
    - BUG/MEDIUM: checks: unsubscribe for events on the old conn-stream on connect
    - BUG/MINOR: checks: Only use ssl_sock_is_ssl() if compiled with SSL support
    - BUG/MINOR: checks/server: use_ssl member must be signed
    - BUG/MEDIUM: sessions: Always pass the mux context as argument to destroy a mux
    - BUG/MEDIUM: checks: Destroy the conn-stream before the session
    - BUG/MINOR: checks: Fix PostgreSQL regex on the authentication packet
    - CI: cirrus-ci: remove reg-tests/checks/tcp-check-ssl.vtc on CentOS 6
    - MINOR: checks: Support HTTP/2 version (without '.0') for http-check send rules
    - MINOR: checks: Use ver keyword to specify the HTTP version for http checks
    - BUG/MINOR: checks: Remove wrong variable redeclaration
    - BUG/MINOR: checks: Properly handle truncated mysql server messages
    - CLEANUP: checks: Remove unused code when ldap server message is parsed
    - MINOR: checks: Make the use of the check's server more explicit on connect
    - BUG/MINOR: checks: Avoid incompatible cast when a binary string is parsed
    - BUG/MINOR: checks: Remove bad call to free() when an expect rule is parsed
    - BUG/MINOR: checks: Don't lose warning on proxy capability
    - MINOR: log: Add "Tu" timer
    - BUG/MINOR: checks: Set the output buffer length before calling parse_binary()
    - BUG/MEDIUM: mux-h1: make sure we always have a timeout on front connections
    - REGTEST: ssl: test the client certificate authentication
    - DOC: give a more accurate description of what check does
    - BUG/MEDIUM: capture: capture-req/capture-res converters crash without a stream
    - BUG/MEDIUM: capture: capture.{req,res}.* crash without a stream
    - BUG/MEDIUM: http: the "http_first_req" sample fetch could crash without a steeam
    - BUG/MEDIUM: http: the "unique-id" sample fetch could crash without a steeam
    - CLEANUP: http: add a few comments on certain functions' assumptions about streams
    - BUG/MEDIUM: sample: make the CPU and latency sample fetches check for a stream
    - MINOR: http-htx: Export functions to update message authority and host
    - MINOR: checks: Don't support multiple host header for http-check send rule
    - MINOR: checks: Skip some headers for http-check send rules
    - MINOR: checks: Keep the Host header and the request uri synchronized
    - CLEANUP: checks: Fix checks includes
    - DOC: Fix send rules in the http-check connect example
    - DOC: Add more info about request formatting in http-check send description
    - REGTEST: http-rules: Require PCRE or PCRE2 option to run map_redirect script
    - REGTEST: ssl: remove curl from the "add ssl crt-list" test
    - REGTEST: ssl: improve the "set ssl cert" test
    - CLEANUP: ssl: silence a build warning when threads are disabled
    - BUG/MEDIUM: listener: mark the thread as not stuck inside the loop
    - MINOR: threads: export the POSIX thread ID in panic dumps
    - BUG/MINOR: debug: properly use long long instead of long for the thread ID
    - BUG/MEDIUM: shctx: really check the lock's value while waiting
    - BUG/MEDIUM: shctx: bound the number of loops that can happen around the lock
    - MINOR: stream: report the list of active filters on stream crashes
    - BUG/MEDIUM: mux-fcgi: Return from detach if server don't keep the connection
    - BUG/MEDIUM: mux_fcgi: Free the FCGI connection at the end of fcgi_release()
    - BUG/MEDIUM: mux-fcgi: Fix wrong test on FCGI_CF_KEEP_CONN in fcgi_detach()
    - BUG/MEDIUM: connections: force connections cleanup on server changes
    - BUG/MEDIUM: h1: Don't compare host and authority if only h1 headers are parsed
    - BUG/MEDIUM: ssl: fix the id length check within smp_fetch_ssl_fc_session_id()
    - CLEANUP: connections: align function declaration
    - BUG/MINOR: sample: Set the correct type when a binary is converted to a string
    - MEDIUM: checks/http-fetch: Support htx prefetch from a check for HTTP samples
    - DOC: Document the log-format parameter for tcp-check send/send-binary rules
    - MINOR: checks: Add support of payload-based sample fetches
    - MINOR: checks: Add support of be_id, be_name, srv_id and srv_name sample fetches
    - MINOR: checks: Add support of server side ssl sample fetches
    - MINOR: checks: Add support of HTTP response sample fetches
    - MINOR: http-htx: Support different methods to look for header names
    - MINOR: checks: Set by default expect rule status to UNKNOWN during parsing
    - BUG/MINOR: checks: Support multiple HTTP expect rules
    - REGTEST: checks: Fix sync condition for agent-check
    - MEDIUM: checks: Support matching on headers for http-check expect rules
    - MINOR: lua: allow changing port with set_addr
    - BUG/MINOR: da: Fix HTX message prefetch
    - BUG/MINOR: wurfl: Fix HTX message prefetch
    - BUG/MINOR: 51d: Fix HTX message prefetch
    - MINOR: ist: add istadv() function
    - MINOR: ist: add istissame() function
    - MINOR: istbuf: add ist2buf() function
    - BUG/MINOR: threads: fix multiple use of argument inside HA_ATOMIC_CAS()
    - BUG/MINOR: threads: fix multiple use of argument inside HA_ATOMIC_UPDATE_{MIN,MAX}()
    - DOC: update intro.txt for 2.2
    - DOC: intro: add a contacts section
2020-05-05 21:49:10 +02:00
Willy Tarreau
6562623f41 DOC: intro: add a contacts section
This indicates where to find help and where to report bugs.
2020-05-05 18:08:07 +02:00
Willy Tarreau
ec8962cb5a DOC: update intro.txt for 2.2
A number of things have changed since last update, for example caching
and fastcgi were not mentioned.
2020-05-05 17:39:16 +02:00
Joseph C. Sible
49bbf528e4 MINOR: lua: allow changing port with set_addr
Add an optional port parameter, which can be either a number or a
string (to support '+' and '-' for port mapping).

This fixes issue #586.
2020-05-05 11:24:39 +02:00
Christopher Faulet
3970819a55 MEDIUM: checks: Support matching on headers for http-check expect rules
It is now possible to add http-check expect rules matching HTTP header names and
values. Here is the format of these rules:

  http-check expect header name [ -m <meth> ] <name> [log-format] \
                           [ value [ -m <meth> ] <value> [log-format] [full] ]

the name pattern (name ...) is mandatory but the value pattern (value ...) is
optionnal. If not specified, only the header presence is verified. <meth> is the
matching method, applied on the header name or the header value. Supported
matching methods are:

  * "str" (exact match)
  * "beg" (prefix match)
  * "end" (suffix match)
  * "sub" (substring match)
  * "reg" (regex match)

If not specified, exact matching method is used. If the "log-format" option is
used, the pattern (<name> or <value>) is evaluated as a log-format string. This
option cannot be used with the regex matching method. Finally, by default, the
header value is considered as comma-separated list. Each part may be tested. The
"full" option may be used to test the full header line. Note that matchings are
case insensitive on the header names.
2020-05-05 11:19:27 +02:00
Christopher Faulet
16032ab44a MINOR: checks: Add support of HTTP response sample fetches
HTPP sample fetches acting on the response can now be called from any sample
expression or log-format string in a tcp-check based ruleset. To avoid any
ambiguities, all these sample fetches are in the check scope, for instance
check.hdr() or check.cook().
2020-05-05 11:06:43 +02:00
Christopher Faulet
d92ea7f5e7 MINOR: checks: Add support of server side ssl sample fetches
SSL sample fetches acting on the server connection can now be called from any
sample expression or log-format string in a tcp-check based ruleset. ssl_bc and
ssl_bc_* sample fetches are concerned.
2020-05-05 11:06:43 +02:00
Christopher Faulet
d1b4464b69 MINOR: checks: Add support of be_id, be_name, srv_id and srv_name sample fetches
It is now possible to call be_id, be_name, srv_id and srv_name sample fetches
from any sample expression or log-format string in a tcp-check based ruleset.
2020-05-05 11:06:43 +02:00
Christopher Faulet
78f371e498 MINOR: checks: Add support of payload-based sample fetches
It is now possible to call check.payload(), check.payload_lv() and check.len()
sample fetches from any sample expression or log-format string in a tcp-check
based ruleset. In fact, check.payload() was already added. But instead of having
a specific function to handle this sample fetch, we use the same than
req.payload().

These sample fetches act on the check input buffer, containing data received for
the server. So it should be part of or after an expect rule, but before any send
rule. Because the input buffer is cleared at this stage.
2020-05-05 11:06:43 +02:00
Christopher Faulet
16fff67e2e DOC: Document the log-format parameter for tcp-check send/send-binary rules
The documentation was not updated when the parameter was added.
2020-05-05 11:06:43 +02:00
Christopher Faulet
9df910c7f3 DOC: Add more info about request formatting in http-check send description
Only one Host header can be defined and some headers are automatically skipped
(Connection, Content-Length and Transfer-Encoding). In addition, a note about
the synchronisation of the Host header value and the request uri has been added.
2020-04-29 14:20:48 +02:00
Christopher Faulet
a5c14ef767 DOC: Fix send rules in the http-check connect example
Method, uri and version arguments must be explicitly named.
2020-04-29 14:19:15 +02:00
Jerome Magnin
90702bc8f0 DOC: give a more accurate description of what check does
The documentation for check implies that without an application
level check configured, it only enables simple tcp checks. What it
actually does is verify that the configured transport layer is available,
and that optional application level checks succeed.
2020-04-28 23:31:41 +02:00
Damien Claisse
57c8eb939d MINOR: log: Add "Tu" timer
It can be sometimes useful to measure total time of a request as seen
from an end user, including TCP/TLS negotiation, server response time
and transfer time. "Tt" currently provides something close to that, but
it also takes client idle time into account, which is problematic for
keep-alive requests as idle time can be very long. "Ta" is also not
sufficient as it hides TCP/TLS negotiationtime. To improve that, introduce
a "Tu" timer, without idle time and everything else. It roughly estimates
time spent time spent from user point of view (without DNS resolution
time), assuming network latency is the same in both directions.
2020-04-28 16:30:13 +02:00
Christopher Faulet
907701bcda MINOR: checks: Use ver keyword to specify the HTTP version for http checks
'ver' keyword is already used by sample fetches while 'vsn' is not used anywhere
else. So better to use 'ver' too for http-check send rules.
2020-04-28 09:37:01 +02:00
Christopher Faulet
8021a5f4a5 MINOR: checks: Support list of status codes on http-check expect rules
It is now possible to match on a comma-separated list of status codes or range
of codes. In addtion, instead of a string comparison to match the response's
status code, a integer comparison is performed. Here is an example:

  http-check expect status 200,201,300-310
2020-04-27 10:46:28 +02:00
Christopher Faulet
edc6ed9778 MINOR: checks: Support mux protocol definition for tcp and http health checks
It is now possible to force the mux protocol for a tcp-check based health check
using the server keyword "check-proto". If set, this parameter overwrites the
server one.

In the same way, a "proto" parameter has been added for tcp-check and http-check
connect rules. If set, this mux protocol overwrites all others for the current
connection.
2020-04-27 10:46:28 +02:00
Christopher Faulet
c52ea4d9cd DOC: Fix the tcp-check and http-check directives layout 2020-04-27 10:46:28 +02:00
Christopher Faulet
4f5c2e2e0f DOC: Add documentation about comments for tcp-check and http-check directives
The documentation about the comment argument for some tcp-check and http-check
directives was missing. As well as the description of "tcp-check comment" and
"http-check comment" directives.
2020-04-27 10:46:28 +02:00