Commit Graph

4042 Commits

Author SHA1 Message Date
Christopher Faulet
7a1e2e1823 MEDIUM: checks: Add a list of vars to set before executing a tpc-check ruleset
A list of variables is now associated to each tcp-check ruleset. It is more a
less a list of set-var expressions. This list may be filled during the
configuration parsing. The listed variables will then be set during each
execution of the tcp-check healthcheck, at the begining, before execution of the
the first tcp-check rule.

This patch is mandatory to convert all protocol checks to tcp-checks. It is a
way to customize shared tcp-check rulesets.
2020-04-27 09:39:37 +02:00
Christopher Faulet
bb591a1a11 MINOR: checks: Relax the default option for tcp-check connect rules
Now this option may be mixed with other options. This way, options on the server
line are used but may be overridden by tcp-check connect options.
2020-04-27 09:39:37 +02:00
Christopher Faulet
98cc57cf5c MEDIUM: checks: Add status-code sample expression on tcp-check expect rules
This option defines a sample expression, evaluated as an integer, to set the
status code (check->code) if a tcp-check healthcheck ends on the corresponding
expect rule.
2020-04-27 09:39:37 +02:00
Christopher Faulet
be52b4de66 MEDIUM: checks: Add on-error/on-success option on tcp-check expect rules
These options define log-format strings used to produce the info message if a
tcp-check expect rule fails (on-error option) or succeeds (on-success
option). For this last option, it must be the ending rule, otherwise the
parameter is ignored.
2020-04-27 09:39:37 +02:00
Christopher Faulet
cf80f2f263 MINOR: checks: Add option to tcp-check expect rules to customize error status
It is now possible to specified the healthcheck status to use on error or on
timeout for tcp-check expect rules. First, to define the error status, the
option "error-status" must be used followed by "L4CON", "L6RSP", "L7RSP" or
"L7STS". Then, to define the timeout status, the option "tout-status" must be
used followed by "L4TOUT", "L6TOUT" or "L7TOUT".

These options will be used to convert specific protocol healthchecks (redis,
pgsql...) to tcp-check ones.
x
2020-04-27 09:39:37 +02:00
Christopher Faulet
1032059bd0 MINOR: checks: Use a name for the healthcheck status enum
The enum defining all healthcheck status (HCHK_STATUS_*) is now named.
2020-04-27 09:39:37 +02:00
Christopher Faulet
5d503fcf5b MEDIUM: checks: Add a shared list of tcp-check rules
A global list to tcp-check ruleset can now be used to share common rulesets with
all backends without any duplication. It is mandatory to convert all specific
protocol checks (redis, pgsql...) to tcp-check healthchecks.

To do so, a flag is now attached to each tcp-check ruleset to know if it is a
shared ruleset or not. tcp-check rules defined in a backend are still directly
attached to the proxy and not shared. In addition a second flag is used to know
if the ruleset is inherited from the defaults section.
2020-04-27 09:39:37 +02:00
Christopher Faulet
f50f4e956f MEDIUM: checks: Support log-format strings for tcp-check send rules
An extra parameter for tcp-check send rules can be specified to handle the
string or the hexa string as a log-format one. Using "log-format" option,
instead of considering the data to send as raw data, it is parsed as a
log-format string. Thus it is possible to call sample fetches to customize data
sent to a server. Of course, because we have no stream attached to healthchecks,
not all sample fetches are available. So be careful.

    tcp-check set-var(check.port) int(8000)
    tcp-check set-var(check.uri) str(/status)
    tcp-check connect port var(check.port)
    tcp-check send "GET %[check.uri] HTTP/1.0\r\n" log-format
    tcp-check send "Host: %[srv_name]\r\n" log-format
    tcp-check send "\r\n"
2020-04-27 09:39:37 +02:00
Christopher Faulet
b7d30098f3 MEDIUM: checks: Support expression to set the port
Since we have a session attached to tcp-check healthchecks, It is possible use
sample expression and variables. In addition, it is possible to add tcp-check
set-var rules to define custom variables. So, now, a sample expression can be
used to define the port to use to establish a connection for a tcp-check connect
rule. For instance:

    tcp-check set-var(check.port) int(8888)
    tcp-check connect port var(check.port)
2020-04-27 09:39:37 +02:00
Christopher Faulet
5c28874a69 MINOR: checks: Add the addr option for tcp-check connect rule
With this option, it is now possible to use a specific address to open the
connection for a tcp-check connect rule. If the port option is also specified,
it is used in priority.
2020-04-27 09:39:37 +02:00
Christopher Faulet
d75f57e94c MINOR: ssl: Export a generic function to parse an alpn string
Parsing of an alpn string has been moved in a dedicated function and exposed to
be used from outside the ssl_sock module.
2020-04-27 09:39:37 +02:00
Christopher Faulet
085426aea9 MINOR: checks: Add the via-socks4 option for tcp-check connect rules
With this option, it is possible to establish the connection opened by a
tcp-check connect rule using upstream socks4 proxy. Info from the socks4
parameter on the server are used.
2020-04-27 09:39:37 +02:00
Christopher Faulet
79b31d4ee5 MINOR: checks: Add the sni option for tcp-check connect rules
With this option, it is possible to specify the SNI to be used for SSL
conncection opened by a tcp-check connect rule.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
707b52f17e MEDIUM: checks: Parse custom action rules in tcp-checks
Register the custom action rules "set-var" and "unset-var", that will
call the parse_store() command upon parsing.

These rules are thus built and integrated to the tcp-check ruleset, but
have no further effect for the moment.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
13a5043a9e MINOR: checks/vars: Add a check scope for variables
Add a dedicated vars scope for checks. This scope is considered as part of the
session scope for accounting purposes.

The scope can be addressed by a valid session, even embryonic. The stream is not
necessary.

The scope is initialized after the check session is created. All variables are
then pruned before the session is destroyed.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
05d692dc09 MEDIUM: checks: Associate a session to each tcp-check healthcheck
Create a session for each healthcheck relying on a tcp-check ruleset. When such
check is started, a session is allocated, which will be freed when the check
finishes. A dummy static frontend is used to create these sessions. This will be
useful to support variables and sample expression. This will also be used,
later, by HTTP healthchecks to rely on HTTP muxes.
2020-04-27 09:39:37 +02:00
Christopher Faulet
b2c2e0fcca MAJOR: checks: Refactor and simplify the tcp-check loop
The loop in tcpcheck_main() function is quite hard to understand. Depending
where we are in the loop, The current_step is the currentely executed rule or
the one to execute on the next call to tcpcheck_main(). When the check result is
reported, we rely on the rule pointed by last_started_step or the one pointed by
current_step. In addition, the loop does not use the common list_for_each_entry
macro and it is thus quite confusing.

So the loop has been totally rewritten and splitted to several functions to
simplify its reading and its understanding. Tcp-check rules are evaluated in
dedicated functions. And a common for_each loop is used and only one rule is
referenced, the current one.
2020-04-27 09:39:37 +02:00
Christopher Faulet
a202d1d4c1 MEDIUM: checks: Add implicit tcp-check connect rule
After the configuration parsing, when its validity check, an implicit tcp-check
connect rule is added in front of the tcp-check ruleset if the first non-comment
rule is not a connect one. This implicit rule is flagged to use the default
check parameter.

This means now, all tcp-check rulesets begin with a connect and are never
empty. When tcp-check healthchecks are used, all connections are thus handled by
tcpcheck_main() function.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
06d963aeca MINOR: checks: define a tcp-check connect type
The check rule itself is not changed, only its representation.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
48219dc50e MINOR: checks: define tcp-check send type
The check rule itself is not changed, only its representation.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
5301b01f99 MINOR: checks: Set the tcp-check rule index during parsing
Now the position of a tcp-check rule in a chain is set during the parsing. This
simplify significantly the function retrieving the current step id.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
04578dbf37 MINOR: checks: Don't use a static tcp rule list head
To allow reusing these blocks without consuming more memory, their list
should be static and share-able accross uses. The head of the list will
be shared as well.

It is thus necessary to extract the head of the rule list from the proxy
itself. Transform it into a pointer instead, that can be easily set to
an external dynamically allocated head.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
9dcb09fc98 MEDIUM: checks: capture groups in expect regexes
Parse back-references in comments of tcp-check expect rules.  If references are
made, capture groups in the match and replace references to it within the
comment when logging the error. Both text and binary regex can caputre groups
and reference them in the expect rule comment.

[Cf: I slightly updated the patch. exp_replace() function is used instead of a
custom one. And if the trash buffer is too small to contain the comment during
the substitution, the comment is ignored.]
2020-04-27 09:39:37 +02:00
Gaetan Rivet
efab6c61d9 MINOR: checks: add rbinary expect match type
The rbinary match works similarly to the rstring match type, however the
received data is rewritten as hex-string before the match operation is
done.

This allows using regexes on binary content even with the POSIX regex
engine.

[Cf: I slightly updated the patch. mem2hex function was removed and dump_binary
is used instead.]
2020-04-27 09:39:37 +02:00
Gaetan Rivet
b616add793 MINOR: checks: define a tcp expect type
Extract the expect definition from its tcpcheck ; create a standalone type.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
f8ba6773e5 MINOR: checks: add linger option to tcp connect
Allow declaring tcpcheck connect commands with a new parameter,
"linger". This option will configure the connection to avoid using an
RST segment to close, instead following the four-way termination
handshake. Some servers would otherwise log each healthcheck as
an error.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
1afd826ae4 MINOR: checks: add min-recv tcp-check expect option
Some expect rules cannot be satisfied due to inherent ambiguity towards
the received data: in the absence of match, the current behavior is to
be forced to wait either the end of the connection or a buffer full,
whichever comes first. Only then does the matching diagnostic is
considered  conclusive. For instance :

    tcp-check connect
    tcp-check expect !rstring "^error"
    tcp-check expect string "valid"

This check will only succeed if the connection is closed by the server before
the check timeout. Otherwise the first expect rule will wait for more data until
"^error" regex matches or the check expires.

Allow the user to explicitly define an amount of data that will be
considered enough to determine the value of the check.

This allows succeeding on negative rstring rules, as previously
in valid condition no match happened, and the matching was repeated
until the end of the connection. This could timeout the check
while no error was happening.

[Cf: I slighly updated the patch. The parameter was renamed and the value is a
signed integer to support -1 as default value to ignore the parameter.]
2020-04-27 09:39:37 +02:00
Gaetan Rivet
4038b94706 MEDIUM: checks: rewind to the first inverse expect rule of a chain on new data
When receiving additional data while chaining multiple tcp-check expects,
previous inverse expects might have a different result with the new data. They
need to be evaluated again against the new data.

Add a pointer to the first inverse expect rule of the current expect chain
(possibly of length one) to each expect rule. When receiving new data, the
currently evaluated tcp-check rule is set back to this pointed rule.

Fonctionnaly speaking, it is a bug and it exists since the introduction of the
feature. But there is no way for now to hit it because when an expect rule does
not match, we wait for more data, independently on the inverse flag. The only
way to move to the following rule is to be sure no more data will be received.

This patch depends on the commit "MINOR: mini-clist: Add functions to iterate
backward on a list".

[Cf: I slightly updated the patch. First, it only concerns inverse expect
rule. Normal expect rules are not concerned. Then, I removed the BUG tag
because, for now, it is not possible to move to the following rule when the
current one does not match while more data can be received.]
2020-04-27 09:39:37 +02:00
Gaetan Rivet
dd66732ffe MINOR: checks: Use an enum to describe the tcp-check rule type
Replace the generic integer with an enumerated list. This allows light
type check and helps debugging (seeing action = 2 in the struct is not
helpful).
2020-04-27 09:39:37 +02:00
Christopher Faulet
31c30fdf1e CLEANUP: checks: Don't export anymore init_check and srv_check_healthcheck_port
These functions are no longer called outside the checks.
2020-04-27 09:39:37 +02:00
Christopher Faulet
f61f33a1b2 BUG/MINOR: checks: Respect the no-check-ssl option
This options is used to force a non-SSL connection to check a SSL server or to
invert a check-ssl option inherited from the default section. The use_ssl field
in the check structure is used to know if a SSL connection must be used
(use_ssl=1) or not (use_ssl=0). The server configuration is used by default.

The problem is that we cannot distinguish the default case (no specific SSL
check option) and the case of an explicit non-SSL check. In both, use_ssl is set
to 0. So the server configuration is always used. For a SSL server, when
no-check-ssl option is set, the check is still performed using a SSL
configuration.

To fix the bug, instead of a boolean value (0=TCP, 1=SSL), we use a ternary value :

  * 0  = use server config
  * 1  = force SSL
  * -1 = force non-SSL

The same is done for the server parameter. It is not really necessary for
now. But it is a good way to know is the server no-ssl option is set.

In addition, the PR_O_TCPCHK_SSL proxy option is no longer used to set use_ssl
to 1 for a check. Instead the flag is directly tested to prepare or destroy the
server SSL context.

This patch should be backported as far as 1.8.
2020-04-27 09:39:37 +02:00
Christopher Faulet
8acb1284bc MINOR: checks: Add a way to send custom headers and payload during http chekcs
The 'http-check send' directive have been added to add headers and optionnaly a
payload to the request sent during HTTP healthchecks. The request line may be
customized by the "option httpchk" directive but there was not official way to
add extra headers. An old trick consisted to hide these headers at the end of
the version string, on the "option httpchk" line. And it was impossible to add
an extra payload with an "http-check expect" directive because of the
"Connection: close" header appended to the request (See issue  for details).

So to make things official and fully support payload additions, the "http-check
send" directive have been added :

    option httpchk POST /status HTTP/1.1

    http-check send hdr Content-Type "application/json;charset=UTF-8" \
        hdr X-test-1 value1 hdr X-test-2 value2 \
        body "{id: 1, field: \"value\"}"

When a payload is defined, the Content-Length header is automatically added. So
chunk-encoded requests are not supported yet. For now, there is no special
validity checks on the extra headers.

This patch is inspired by Kiran Gavali's work. It should fix the issue  and
as far as possible, it may be backported, at least as far as 1.8.
2020-04-27 09:39:37 +02:00
Christopher Faulet
bc1f54b0fc MINOR: mini-clist: Add functions to iterate backward on a list
list_for_each_entry_rev() and list_for_each_entry_from_rev() and corresponding
safe versions have been added to iterate on a list in the reverse order. All
these functions work the same way than the forward versions, except they use the
.p field to move for an element to another.
2020-04-27 09:39:37 +02:00
Christopher Faulet
aaae9a0e99 BUG/MINOR: check: Update server address and port to execute an external check
Server address and port may change at runtime. So the address and port passed as
arguments and as environment variables when an external check is executed must
be updated. The current number of connections on the server was already updated
before executing the command. So the same mechanism is used for the server
address and port. But in addition, command arguments are also updated.

This patch must be backported to all stable versions. It should fix the
issue .
2020-04-27 09:39:13 +02:00
Willy Tarreau
62ba9ba6ca BUG/MINOR: http: make url_decode() optionally convert '+' to SP
The url_decode() function used by the url_dec converter and a few other
call points is ambiguous on its processing of the '+' character which
itself isn't stable in the spec. This one belongs to the reserved
characters for the query string but not for the path nor the scheme,
in which it must be left as-is. It's only in argument strings that
follow the application/x-www-form-urlencoded encoding that it must be
turned into a space, that is, in query strings and POST arguments.

The problem is that the function is used to process full URLs and
paths in various configs, and to process query strings from the stats
page for example.

This patch updates the function to differentiate the situation where
it's parsing a path and a query string. A new argument indicates if a
query string should be assumed, otherwise it's only assumed after seeing
a question mark.

The various locations in the code making use of this function were
updated to take care of this (most call places were using it to decode
POST arguments).

The url_dec converter is usually called on path or url samples, so it
needs to remain compatible with this and will default to parsing a path
and turning the '+' to a space only after a question mark. However in
situations where it would explicitly be extracted from a POST or a
query string, it now becomes possible to enforce the decoding by passing
a non-null value in argument.

It seems to be what was reported in issue . This fix may be
backported to older stable releases.
2020-04-23 20:03:27 +02:00
Willy Tarreau
09568fd54d BUG/MINOR: tools: fix the i386 version of the div64_32 function
As reported in issue , the edx register isn't marked as clobbered
in div64_32(), which could technically allow gcc to try to reuse it
if it needed a copy of the 32 highest bits of the o1 register after
the operation.

Two attempts were tried, one using a dummy 32-bit local variable to
store the intermediary edx and another one switching to "=A" and making
result a long long. It turns out the former makes the resulting object
code significantly dirtier while the latter makes it better and was
kept. This is due to gcc's difficulties at working with register pairs
mixing 32- and 64- bit values on i386. It was verified that no code
change happened at all on x86_64, armv7, aarch64 nor mips32.

In practice it's only used by the frequency counters so this bug
cannot even be triggered but better fix it.

This may be backported to stable branches though it will not fix any
issue.
2020-04-23 17:21:37 +02:00
Ilya Shipitsin
856aabcda5 CLEANUP: assorted typo fixes in the code and comments
This is 8th iteration of typo fixes
2020-04-17 09:37:36 +02:00
Willy Tarreau
bb86986253 MINOR: init: report the haproxy version and executable path once on errors
If haproxy fails to start and emits an alert, then it can be useful
to have it also emit the version and the path used to load it. Some
users may be mistakenly launching the wrong binary due to a misconfigured
PATH variable and this will save them some troubleshooting time when it
reports that some keywords are not understood.

What we do here is that we *try* to extract the binary name from the
AUX vector on glibc, and we report this as a NOTICE tag before the
very first alert is emitted.
2020-04-16 10:52:41 +02:00
Ilya Shipitsin
d425950c68 CLEANUP: assorted typo fixes in the code and comments
This is 7th iteration of typo fixes
2020-04-16 10:04:36 +02:00
Willy Tarreau
3eb10b8e98 MINOR: init: add -dW and "zero-warning" to reject configs with warnings
Since some systems switched to service managers which hide all warnings
by default, some users are not aware of some possibly important warnings
and get caught too late with errors that could have been detected earlier.

This patch adds a new global keyword, "zero-warning" and an equivalent
command-line option "-dW" to refuse to start in case any warning is
detected. It is recommended to use these with configurations that are
managed by humans in order to catch mistakes very early.
2020-04-15 16:42:39 +02:00
Willy Tarreau
bebd212064 MINOR: init: report in "haproxy -c" whether there were warnings or not
This helps quickly checking if the config produces any warning. For
this we reuse the "warned" bit field to add a new WARN_ANY bit that is
set by ha_warning(). The rest of the bit field was also cleaned from
unused bits.
2020-04-15 16:42:00 +02:00
Fr�d�ric L�caille
8ba10fea69 BUG/MINOR: peers: Incomplete peers sections should be validated.
Before supporting "server" line in "peers" section, such sections without
any local peer were removed from the configuration to get it validated.

This patch fixes the issue where a "server" line without address and port which
is a remote peer without address and port makes the configuration parsing fail.
When encoutering such cases we now ignore such lines remove them from the
configuration.

Thank you to J�r�me Magnin for having reported this bug.

Must be backported to 2.1 and 2.0.
2020-04-15 10:47:39 +02:00
William Lallemand
b7296c42bd CLEANUP: ssl: remove a commentary in struct ckch_inst
The struct ckch_inst now handles the ssl_bind_conf so this commentary is
obsolete
2020-04-09 16:13:42 +02:00
William Lallemand
caa161982f CLEANUP: ssl/cli: use the list of filters in the crtlist_entry
In 'commit ssl cert', instead of trying to regenerate a list of filters
from the SNIs, use the list provided by the crtlist_entry used to
generate the ckch_inst.

This list of filters doesn't need to be free'd anymore since they are
always reused from the crtlist_entry.
2020-04-08 16:52:51 +02:00
William Lallemand
02e19a5c7b CLEANUP: ssl: use the refcount for the SSL_CTX'
Use the refcount of the SSL_CTX' to free them instead of freeing them on
certains conditions. That way we can free the SSL_CTX everywhere its
pointer is used.
2020-04-08 16:52:51 +02:00
William Lallemand
c69f02d0f0 MINOR: ssl/cli: replace dump/show ssl crt-list by '-n' option
The dump and show ssl crt-list commands does the same thing, they dump
the content of a crt-list, but the 'show' displays an ID in the first
column. Delete the 'dump' command so it is replaced by the 'show' one.
The old 'show' command is replaced by an '-n' option to dump the ID.
And the ID which was a pointer is replaced by a line number and placed
after colons in the filename.

Example:
  $ echo "show ssl crt-list -n kikyo.crt-list" | socat /tmp/sock1 -
  # kikyo.crt-list
  kikyo.pem.rsa:1 secure.domain.tld
  kikyo.pem.ecdsa:2 secure.domain.tld
2020-04-06 19:33:33 +02:00
Fr�d�ric L�caille
876ed55d9b BUG/MINOR: protocol_buffer: Wrong maximum shifting.
This patch fixes a bad stop condition when decoding a protocol buffer variable integer
whose maximum lenghts are 10, shifting a uint64_t value by more than 63.

Thank you to Ilya for having reported this issue.

Must be backported to 2.1 and 2.0.
2020-04-02 15:09:46 +02:00
Olivier Houchard
4a0e7fe4f7 MINOR: connections: Don't mark conn flags 0x00000001 and 0x00000002 as unused.
Remove the comments saying 0x00000001 and 0x00000002 are unused, they are
now used by CO_FL_SAFE_LIST and CO_FL_IDLE_LIST.
2020-03-31 23:04:20 +02:00
William Lallemand
fa8cf0c476 MINOR: ssl: store a ptr to crtlist in crtlist_entry
Store a pointer to crtlist in crtlist_entry so we can re-insert a
crtlist_entry in its crtlist ebpt after updating its key.
2020-03-31 12:32:17 +02:00
William Lallemand
23d61c00b9 MINOR: ssl: add a list of crtlist_entry in ckch_store
When updating a ckch_store we may want to update its pointer in the
crtlist_entry which use it. To do this, we need the list of the entries
using the store.
2020-03-31 12:32:17 +02:00