Commit Graph

12104 Commits

Author SHA1 Message Date
Christopher Faulet
8aa825a356 REGTEST: http-rules: Require PCRE or PCRE2 option to run map_redirect script
Only PCRE was specified as required option to execute this script. But PCRE2 is
an valid alternative.
2020-04-29 14:32:28 +02:00
Christopher Faulet
9df910c7f3 DOC: Add more info about request formatting in http-check send description
Only one Host header can be defined and some headers are automatically skipped
(Connection, Content-Length and Transfer-Encoding). In addition, a note about
the synchronisation of the Host header value and the request uri has been added.
2020-04-29 14:20:48 +02:00
Christopher Faulet
a5c14ef767 DOC: Fix send rules in the http-check connect example
Method, uri and version arguments must be explicitly named.
2020-04-29 14:19:15 +02:00
Christopher Faulet
dc75d577b9 CLEANUP: checks: Fix checks includes 2020-04-29 13:32:29 +02:00
Christopher Faulet
39959a75e9 MINOR: checks: Keep the Host header and the request uri synchronized
Because in HTTP, the host header and the request authority, if any, must be
identical, we keep both synchornized. It means the right flags are set on the
HTX statrt-line calling http_update_host(). There is no header when it happens,
but it is not an issue. Then, if a Host header is inserted,
http_update_authority() is called.

Note that for now, the host header is not automatically added when required.
2020-04-29 13:32:29 +02:00
Christopher Faulet
7d765dbe9a MINOR: checks: Skip some headers for http-check send rules
Connection, content-length and transfer-encoding headers are ignored for
http-check send rules. For now, the keep-alive is not supported and the
"connection: close" header is always added to the request. And the
content-length header is automatically added.
2020-04-29 13:32:29 +02:00
Christopher Faulet
13ec87b4c4 MINOR: checks: Don't support multiple host header for http-check send rule
Only one host header definition is supported. There is no reason to define it
several times.
2020-04-29 13:32:29 +02:00
Christopher Faulet
1543d44607 MINOR: http-htx: Export functions to update message authority and host
These functions will be used by HTTP health checks when a request is formatted
before sending it.
2020-04-29 13:32:29 +02:00
Willy Tarreau
e0dd210cea BUG/MEDIUM: sample: make the CPU and latency sample fetches check for a stream
cpu_calls, cpu_ns_avg, cpu_ns_tot, lat_ns_avg and lat_ns_tot depend on the
stream to find the current task and must check for it or they may cause a
crash if misused or used in a log-format string after commit 5f940703b3
("MINOR: log: Don't depends on a stream to process samples in log-format
string").

This must be backported as far as 1.9.
2020-04-29 11:59:02 +02:00
Willy Tarreau
eae837295d CLEANUP: http: add a few comments on certain functions' assumptions about streams
get_http_auth() expects a valid stream but this is not mentioned, though
fortunately it's always called from places which already check this.

smp_prefetch_htx() performs all the required checks and is the key to the
stability of almost all sample fetch functions, so let's make this clearer.
2020-04-29 11:55:36 +02:00
Willy Tarreau
a1062a4de8 BUG/MEDIUM: http: the "unique-id" sample fetch could crash without a steeam
Since commit 5f940703b3 ("MINOR: log: Don't depends on a stream to process
samples in log-format string") it has become quite obvious that a few sample
fetch functions and converters were still heavily dependent on the presence
of a stream without testing for it.

The unique-id sample fetch function, if called without a stream, will result
in a crash.

This fix adds a check for the stream's existence, and should be backported
to all stable versions up to 1.7.
2020-04-29 11:55:36 +02:00
Willy Tarreau
79512b6d8f BUG/MEDIUM: http: the "http_first_req" sample fetch could crash without a steeam
Since commit 5f940703b3 ("MINOR: log: Don't depends on a stream to process
samples in log-format string") it has become quite obvious that a few sample
fetch functions and converters were still heavily dependent on the presence
of a stream without testing for it.

The http_first_req sample fetch function, if called without a stream, will
result in a crash.

This fix adds a check for the stream's existence, and should be backported
to all stable versions up to 1.6.
2020-04-29 11:55:36 +02:00
Willy Tarreau
0898c2d2f2 BUG/MEDIUM: capture: capture.{req,res}.* crash without a stream
Since commit 5f940703b3 ("MINOR: log: Don't depends on a stream to process
samples in log-format string") it has become quite obvious that a few sample
fetch functions and converters were still heavily dependent on the presence
of a stream without testing for it.

The capture.req.hdr, capture.res.hdr, capture.req.method, capture.req.uri,
capture.req.ver and capture.res.ver sample fetches used to assume the
presence of a stream, which is not necessarily the case (especially after
the commit above) and would crash haproxy if incorrectly used. Let's make
sure they check for this stream.

This fix adds a check for the stream's existence, and should be backported
to all stable versions up to 1.6.
2020-04-29 11:44:54 +02:00
Willy Tarreau
5575896ba1 BUG/MEDIUM: capture: capture-req/capture-res converters crash without a stream
Since commit 5f940703b3 ("MINOR: log: Don't depends on a stream to process
samples in log-format string") it has become quite obvious that a few sample
fetch functions and converters were still heavily dependent on the presence
of a stream without testing for it.

The capture-req and capture-res converters were in this case and could
crash the process if misused.

This fix adds a check for the stream's existence, and should be backported
to all stable versions up to 1.6.
2020-04-29 11:29:17 +02:00
Jerome Magnin
90702bc8f0 DOC: give a more accurate description of what check does
The documentation for check implies that without an application
level check configured, it only enables simple tcp checks. What it
actually does is verify that the configured transport layer is available,
and that optional application level checks succeed.
2020-04-28 23:31:41 +02:00
William Lallemand
2e0dbb7412 REGTEST: ssl: test the client certificate authentication
This reg-test tests the client auth feature of HAProxy for both the
backend and frontend section with a CRL list.

This reg-test uses 2 chained listeners because vtest does not handle the
SSL. Test the frontend client auth and the backend side at the same
time.

It sends 3 requests: one with a correct certificate, one with an expired
one and one which was revoked. The client then checks if we received the
right one with the right error.

Certificates, CA and CRL are expiring in 2050 so it should be fine for
the CI.

This test could be backported as far as HAProxy 1.6
2020-04-28 22:04:13 +02:00
Willy Tarreau
ca39747dcf BUG/MEDIUM: mux-h1: make sure we always have a timeout on front connections
Mux-h1 currently heavily relies on the presence of an upper stream, even
when waiting for a new request after one is being finished, and it's that
upper stream that's in charge of request and keep-alive timeouts for now.
But since recent commit 493d9dc6ba ("MEDIUM: mux-h1: do not blindly wake
up the tasklet at end of request anymore") that assumption was broken as
the purpose of this change was to avoid initiating processing of a request
when there's no data in the buffer. The side effect is that there's no more
timeout to handle the front connection, resulting in dead front connections
stacking up as clients get kicked off the net.

This fix makes sure we always enable the timeout when there's no stream
attached to the connection. It doesn't do this for back connections since
they may purposely be left idle.

No backport is needed as this bug was introduced in 2.2-dev4.
2020-04-28 19:48:41 +02:00
Christopher Faulet
9c2cb2deee BUG/MINOR: checks: Set the output buffer length before calling parse_binary()
A bug was introduced in the commit 2edcd4cbd ("BUG/MINOR: checks: Avoid
incompatible cast when a binary string is parsed"). The length of the
destination buffer must be set before call the parse_binary() function.

No backport needed.
2020-04-28 16:43:39 +02:00
Damien Claisse
57c8eb939d MINOR: log: Add "Tu" timer
It can be sometimes useful to measure total time of a request as seen
from an end user, including TCP/TLS negotiation, server response time
and transfer time. "Tt" currently provides something close to that, but
it also takes client idle time into account, which is problematic for
keep-alive requests as idle time can be very long. "Ta" is also not
sufficient as it hides TCP/TLS negotiationtime. To improve that, introduce
a "Tu" timer, without idle time and everything else. It roughly estimates
time spent time spent from user point of view (without DNS resolution
time), assuming network latency is the same in both directions.
2020-04-28 16:30:13 +02:00
Christopher Faulet
528f4811d6 BUG/MINOR: checks: Don't lose warning on proxy capability
When a tcp-check line is parsed, a warning may be reported if the keyword is
used for a frontend. The return value must be used to report it. But this info
is lost before the end of the function.

Partly fixes issue #600. No backport needed.
2020-04-28 10:47:30 +02:00
Christopher Faulet
cd8eb85dfc BUG/MINOR: checks: Remove bad call to free() when an expect rule is parsed
When an error is found during the parsing of an expect rule (tcp or http),
everything is released at the same place, at the end of the function.

Partly fixes issue #600. No backport needed.
2020-04-28 10:42:45 +02:00
Christopher Faulet
2edcd4cbde BUG/MINOR: checks: Avoid incompatible cast when a binary string is parsed
parse_binary() function must be called with a pointer on an integer. So don't
pass a pointer on a size_t element, casting it to a pointer on a integer.

Partly fixes issue #600. No backport needed.
2020-04-28 10:39:52 +02:00
Christopher Faulet
931ae5b103 MINOR: checks: Make the use of the check's server more explicit on connect
The variable s, pointing on the check server, may be null when a connection is
openned. It happens for email alerts. To avoid ambiguities, its use is now more
explicit. Comments have been added at some places and tests on the variable have
been added elsewhere (useless but explicit).

Partly fixes issue #600.
2020-04-28 10:31:54 +02:00
Christopher Faulet
ef3e69d4c4 CLEANUP: checks: Remove unused code when ldap server message is parsed
In tcpcheck_ldap_expect_bindrsp(), wait_more_data label cannot be reached.

Partly fixes issue #600.
2020-04-28 10:29:14 +02:00
Christopher Faulet
733dd73443 BUG/MINOR: checks: Properly handle truncated mysql server messages
If a message is not fully received from a mysql server, depending on last_read
value, an error must be reported or we must wait for more data. The first if
statement must not check last_read.

Partly fixes issue #600. No backport needed.
2020-04-28 10:24:27 +02:00
Christopher Faulet
587c414071 BUG/MINOR: checks: Remove wrong variable redeclaration
When mysql-check option is parsed, the user variable is redeclared without any
reason. thus the redeclared variable is removed.

No backport needed.
2020-04-28 09:46:23 +02:00
Christopher Faulet
907701bcda MINOR: checks: Use ver keyword to specify the HTTP version for http checks
'ver' keyword is already used by sample fetches while 'vsn' is not used anywhere
else. So better to use 'ver' too for http-check send rules.
2020-04-28 09:37:01 +02:00
Christopher Faulet
8bf8fda49b MINOR: checks: Support HTTP/2 version (without '.0') for http-check send rules
The version is partially parsed to set the flag HTX_SL_F_VER_11 on the HTX
message. But exactly 8 chars is expected. So if "HTTP/2" is specified, the flag
is not set. Thus, the version parsing has been updated to handle "HTTP/2" and
"HTTP/2.0" the same way.
2020-04-28 09:24:49 +02:00
Ilya Shipitsin
e50b7dc2e0 CI: cirrus-ci: remove reg-tests/checks/tcp-check-ssl.vtc on CentOS 6
reg-tests/checks/tcp-check-ssl.vtc requires ALPN which is not
available on CentOS 6
2020-04-28 09:08:10 +02:00
Christopher Faulet
b841c74aad BUG/MINOR: checks: Fix PostgreSQL regex on the authentication packet
For PostgreSQL health check, there is a regex on the backend authentication
packet. It must match to succeed. But it exists 6 types of authentication
packets and the regex only matches the first one (AuthenticationOK). This patch
fixes the regex to match all authentication packets.

No backport needed.
2020-04-27 18:29:51 +02:00
Christopher Faulet
b693a0de37 BUG/MEDIUM: checks: Destroy the conn-stream before the session
At the end of a tcp-check based health check, if there is still a connection
attached to the check, it must be closed. But it must be done before releasing
the session, because the session may still be referenced by the mux. For
instance, an h2 stream may still have a reference on the session.

No need to backport.
2020-04-27 15:59:26 +02:00
Christopher Faulet
bfb0f72d52 BUG/MEDIUM: sessions: Always pass the mux context as argument to destroy a mux
This bug was introduced by the commit 2444aa5b ("MEDIUM: sessions: Don't be
responsible for connections anymore."). In session_check_idle_conn(), when the
mux is destroyed, its context must be passed as argument instead of the
connection.

It is de 2.2-dev bug. No need to backport.
2020-04-27 15:53:43 +02:00
Christopher Faulet
4a8c026117 BUG/MINOR: checks/server: use_ssl member must be signed 2020-04-27 12:13:06 +02:00
Christopher Faulet
f73f5cc11f BUG/MINOR: checks: Only use ssl_sock_is_ssl() if compiled with SSL support
ssl_sock_is_ssl() only exists if HAProxy is complied with SSL support.

No backport needed.
2020-04-27 12:06:56 +02:00
Christopher Faulet
06150e4c73 BUG/MEDIUM: checks: unsubscribe for events on the old conn-stream on connect
When a new connection is established, if an old connection is still attached to
the current check, it must be detroyed. When it happens, the old conn-stream
must be used to unsubscribe for events, not the new one.

No backport is needed.
2020-04-27 11:22:57 +02:00
Christopher Faulet
b3b53524ad BUG/MINOR: server: Fix server_finalize_init() to avoid unused variable
The variable 'ret' must only be declared When HAProxy is compiled with the SSL
support (more precisely SSL_CTRL_SET_TLSEXT_HOSTNAME must be defined).

No backport needed.
2020-04-27 11:17:12 +02:00
Christopher Faulet
f97f846413 REGTEST: Add a script to validate agent checks 2020-04-27 10:46:28 +02:00
Christopher Faulet
0b9376ae8d BUG/MEDIUM: checks: Unsubscribe to mux events when a conn-stream is destroyed
Since the tcp-check based heath checks uses the best multuplexer for a
connection, the mux-pt is no longer the only possible choice. So events
subscriptions and unsubscriptions must be done with the mux.

No backport needed.
2020-04-27 10:46:28 +02:00
Christopher Faulet
8021a5f4a5 MINOR: checks: Support list of status codes on http-check expect rules
It is now possible to match on a comma-separated list of status codes or range
of codes. In addtion, instead of a string comparison to match the response's
status code, a integer comparison is performed. Here is an example:

  http-check expect status 200,201,300-310
2020-04-27 10:46:28 +02:00
Christopher Faulet
bb86a0f7be BUG/MINOR: mux-fcgi: Be sure to have a connection as session's origin to use it
When default parameters are set in a request message, we get the client
connection using the session's origin. But it is not necessarily a
conncection. For instance, for health checks, thanks to recent changes, it may
be a check object. At this step, the client connection may be NULL. Thus, we
must be sure to have a client connection before using it.

This patch must be backported to 2.1.
2020-04-27 10:46:28 +02:00
Christopher Faulet
edc6ed9778 MINOR: checks: Support mux protocol definition for tcp and http health checks
It is now possible to force the mux protocol for a tcp-check based health check
using the server keyword "check-proto". If set, this parameter overwrites the
server one.

In the same way, a "proto" parameter has been added for tcp-check and http-check
connect rules. If set, this mux protocol overwrites all others for the current
connection.
2020-04-27 10:46:28 +02:00
Christopher Faulet
12882cfaf8 BUG/MEDIUM: checks: Use the mux protocol specified on the server line
First, when a server health check is initialized, it inherits the mux protocol
from the server if it is not already specified. Because there is no option to
specify the mux protocol for the checks, it is always inherited from the server
for now.

Then, if the connect rule is configured to use the server options, the mux
protocol of the check is used, if defined. Of course, if a mux protocol is
already defined for the connect rule, it is used in priority. But for now, it is
not possible.

Thus, if a server is configured to use, for instance, the h2 protocol, it is
possible to do the same for the health-checks.

No backport needed.
2020-04-27 10:46:28 +02:00
Christopher Faulet
c52ea4d9cd DOC: Fix the tcp-check and http-check directives layout 2020-04-27 10:46:28 +02:00
Christopher Faulet
4f5c2e2e0f DOC: Add documentation about comments for tcp-check and http-check directives
The documentation about the comment argument for some tcp-check and http-check
directives was missing. As well as the description of "tcp-check comment" and
"http-check comment" directives.
2020-04-27 10:46:28 +02:00
Christopher Faulet
88d939c831 Revert "MEDIUM: checks: capture groups in expect regexes"
This reverts commit 1979943c30ef285ed04f07ecf829514de971d9b2.

Captures in comment was only used when a tcp-check expect based on a negative
regex matching failed to eventually report what was captured while it was not
expected. It is a bit far-fetched to be useable IMHO. on-error and on-success
log-format strings are far more usable. For now there is few check sample
fetches (in fact only one...). But it could be really powerful to report info in
logs.
2020-04-27 10:46:28 +02:00
Christopher Faulet
aec7f76af8 REGTEST: Add scripts to test based tcp-check health-checks
These scripts have been added to validate the health-checks based on tcp-check
rules (http, redis, MySQL...).
2020-04-27 10:46:28 +02:00
Christopher Faulet
6d47121ddc BUG/MINOR: checks: Send the right amount of outgoing data for HTTP checks
HTTP health-checks now use HTX multiplexers. So it is important to really send
the amount of outgoing data for such checks because the HTX buffers appears
always full.

No backport needed.
2020-04-27 10:46:28 +02:00
Christopher Faulet
d7cee71e77 MINOR: checks: Use a tree instead of a list to store tcp-check rulesets
Since all tcp-check rulesets are globally stored, it is a problem to use
list. For configuration with many backends, the lookups in list may be costly
and slow downs HAProxy startup. To solve this problem, tcp-check rulesets are
now stored in a tree.
2020-04-27 10:46:28 +02:00
Christopher Faulet
815516d6fd BUG/MEDIUM: checks: Be sure to subscribe for sends if outgoing data remains
When some data are scheduled to be sent, we must be sure to subscribe for sends
if nothing was sent. Because of a bug, when nothing was sent, connection errors
are checks. If no error is found, we exit, waiting for more data, without any
subcription on send events.

No need to backport.
2020-04-27 10:46:28 +02:00
Christopher Faulet
b61caf4061 MINOR: checks: Use ist API as far as possible
Instead of accessing directly to the ist fields, the ist API is used instead. To
get its length or its pointer, to release it or to duplicate it. It is more
readable this way.
2020-04-27 10:46:28 +02:00