Commit Graph

44 Commits

Author SHA1 Message Date
Christopher Faulet
dc75d577b9 CLEANUP: checks: Fix checks includes 2020-04-29 13:32:29 +02:00
Christopher Faulet
61cc852230 CLEANUP: checks: Reorg checks.c file to be more readable
The patch is not obvious at the first glance. But it is just a reorg. Functions
have been grouped and ordered in a more logical way. Some structures and flags
are now private to the checks module (so moved from the .h to the .c file).
2020-04-27 10:46:28 +02:00
Christopher Faulet
6f5579160a MINOR: proxy/checks: Move parsing of external-check option in checks.c
Parsing of the proxy directive "option external-check" have been moved in checks.c.
2020-04-27 09:39:38 +02:00
Christopher Faulet
430e480510 MINOR: proxy/checks: Move parsing of tcp-check option in checks.c
Parsing of the proxy directive "option tcp-check" have been moved in checks.c.
2020-04-27 09:39:38 +02:00
Christopher Faulet
6c2a743538 MINOR: proxy/checks: Move parsing of httpchk option in checks.c
Parsing of the proxy directive "option httpchk" have been moved in checks.c.
2020-04-27 09:39:38 +02:00
Christopher Faulet
0ae3d1dbdf MEDIUM: checks: Implement agent check using tcp-check rules
A shared tcp-check ruleset is now created to support agent checks. The following
sequence is used :

    tcp-check send "%[var(check.agent_string)] log-format
    tcp-check expect custom

The custom function to evaluate the expect rule does the same that it was done
to handle agent response when a custom check was used.
2020-04-27 09:39:38 +02:00
Christopher Faulet
267b01b761 MEDIUM: checks: Implement SPOP check using tcp-check rules
A share tcp-check ruleset is now created to support SPOP checks. This way no
extra memory is used if several backends use a SPOP check.

The following sequence is used :

    tcp-check send-binary SPOP_REQ
    tcp-check expect custom min-recv 4

The spop request is the result of the function
spoe_prepare_healthcheck_request() and the expect rule relies on a custom
function calling spoe_handle_healthcheck_response().
2020-04-27 09:39:38 +02:00
Christopher Faulet
1997ecaa0c MEDIUM: checks: Implement LDAP check using tcp-check rules
A shared tcp-check ruleset is now created to support LDAP check. This way no
extra memory is used if several backends use a LDAP check.

The following sequance is used :

    tcp-check send-binary "300C020101600702010304008000"

    tcp-check expect rbinary "^30" min-recv 14 \
        on-error "Not LDAPv3 protocol"

    tcp-check expect custom

The last expect rule relies on a custom function to check the LDAP server reply.
2020-04-27 09:39:38 +02:00
Christopher Faulet
f2b3be5c27 MEDIUM: checks: Implement MySQL check using tcp-check rules
A share tcp-check ruleset is now created to support MySQL checks. This way no
extra memory is used if several backends use a MySQL check.

One for the following sequence is used :

    ## If no extra params are set
    tcp-check connect default linger
    tcp-check expect custom  ## will test the initial handshake

    ## If the username is defined
    tcp-check connect default linger
    tcp-check send-binary MYSQL_REQ log-format
    tcp-check expect custom  ## will test the initial handshake
    tcp-check expect custom  ## will test the reply to the client message

The log-format hexa string MYSQL_REQ depends on 2 preset variables, the packet
header containing the packet length and the sequence ID (check.header) and the
username (check.username). If is also different if the "post-41" option is set
or not. Expect rules relies on custom functions to check MySQL server packets.
2020-04-27 09:39:38 +02:00
Christopher Faulet
ce355074f1 MEDIUM: checks: Implement postgres check using tcp-check rules
A shared tcp-check ruleset is now created to support postgres check. This way no
extra memory is used if several backends use a pgsql check.

The following sequence is used :

    tcp-check connect default linger

    tcp-check send-binary PGSQL_REQ log-format

    tcp-check expect !rstring "^E" min-recv 5 \
        error-status "L7RSP" on-error "%[check.payload(6,0)]"

    tcp-check expect rbinary "^520000000800000000 min-recv "9" \
        error-status "L7STS" \
        on-success "PostgreSQL server is ok" \
        on-error "PostgreSQL unknown error"

The log-format hexa string PGSQL_REQ depends on 2 preset variables, the packet
length (check.plen) and the username (check.username).
2020-04-27 09:39:38 +02:00
Christopher Faulet
fbcc77c6ba MEDIUM: checks: Implement smtp check using tcp-check rules
A share tcp-check ruleset is now created to support smtp checks. This way no
extra memory is used if several backends use a smtp check.

The following sequence is used :

    tcp-check connect default linger

    tcp-check expect rstring "^[0-9]{3}[ \r]" min-recv 4 \
        error-status "L7RSP" on-error "%[check.payload(),cut_crlf]"

    tcp-check expect rstring "^2[0-9]{2}[ \r]" min-recv 4 \
        error-status "L7STS" \
        on-error %[check.payload(4,0),ltrim(' '),cut_crlf] \
        status-code "check.payload(0,3)"

    tcp-echeck send "%[var(check.smtp_cmd)]\r\n" log-format

    tcp-check expect rstring "^2[0-9]{2}[- \r]" min-recv 4 \
        error-status "L7STS" \
        on-error %[check.payload(4,0),ltrim(' '),cut_crlf] \
        on-success "%[check.payload(4,0),ltrim(' '),cut_crlf]" \
        status-code "check.payload(0,3)"

The variable check.smtp_cmd is by default the string "HELO localhost" by may be
customized setting <helo> and <domain> parameters on the option smtpchk
line. Note there is a difference with the old smtp check. The server gretting
message is checked before send the HELO/EHLO comand.
2020-04-27 09:39:38 +02:00
Christopher Faulet
811f78ced1 MEDIUM: checks: Implement ssl-hello check using tcp-check rules
A shared tcp-check ruleset is now created to support ssl-hello check. This way
no extra memory is used if several backends use a ssl-hello check.

The following sequence is used :

    tcp-check send-binary SSLV3_CLIENT_HELLO log-format

    tcp-check expect rbinary "^1[56]" min-recv 5 \
        error-status "L6RSP" tout-status "L6TOUT"

SSLV3_CLIENT_HELLO is a log-format hexa string representing a SSLv3 CLIENT HELLO
packet. It is the same than the one used by the old ssl-hello except the sample
expression "%[date(),htonl,hex]" is used to set the date field.
2020-04-27 09:39:38 +02:00
Christopher Faulet
33f05df650 MEDIUM: checks: Implement redis check using tcp-check rules
A share tcp-check ruleset is now created to support redis checks. This way no
extra memory is used if several backends use a redis check.

The following sequence is used :

  tcp-check send "*1\r\n$4\r\nPING\r\n"

  tcp-check expect string "+PONG\r\n" error-status "L7STS" \
      on-error "%[check.payload(),cut_crlf]" on-success "Redis server is ok"
2020-04-27 09:39:38 +02:00
Christopher Faulet
7a1e2e1823 MEDIUM: checks: Add a list of vars to set before executing a tpc-check ruleset
A list of variables is now associated to each tcp-check ruleset. It is more a
less a list of set-var expressions. This list may be filled during the
configuration parsing. The listed variables will then be set during each
execution of the tcp-check healthcheck, at the begining, before execution of the
the first tcp-check rule.

This patch is mandatory to convert all protocol checks to tcp-checks. It is a
way to customize shared tcp-check rulesets.
2020-04-27 09:39:37 +02:00
Christopher Faulet
5d503fcf5b MEDIUM: checks: Add a shared list of tcp-check rules
A global list to tcp-check ruleset can now be used to share common rulesets with
all backends without any duplication. It is mandatory to convert all specific
protocol checks (redis, pgsql...) to tcp-check healthchecks.

To do so, a flag is now attached to each tcp-check ruleset to know if it is a
shared ruleset or not. tcp-check rules defined in a backend are still directly
attached to the proxy and not shared. In addition a second flag is used to know
if the ruleset is inherited from the defaults section.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
707b52f17e MEDIUM: checks: Parse custom action rules in tcp-checks
Register the custom action rules "set-var" and "unset-var", that will
call the parse_store() command upon parsing.

These rules are thus built and integrated to the tcp-check ruleset, but
have no further effect for the moment.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
13a5043a9e MINOR: checks/vars: Add a check scope for variables
Add a dedicated vars scope for checks. This scope is considered as part of the
session scope for accounting purposes.

The scope can be addressed by a valid session, even embryonic. The stream is not
necessary.

The scope is initialized after the check session is created. All variables are
then pruned before the session is destroyed.
2020-04-27 09:39:37 +02:00
Christopher Faulet
31c30fdf1e CLEANUP: checks: Don't export anymore init_check and srv_check_healthcheck_port
These functions are no longer called outside the checks.
2020-04-27 09:39:37 +02:00
Christopher Faulet
2a944ee16b BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix
This remove any name conflicts, especially on Solaris.
2017-11-07 11:10:24 +01:00
Emeric Brun
9f0b458525 MEDIUM: threads/server: Use the server lock to protect health check and cli concurrency 2017-10-31 13:58:33 +01:00
Christopher Faulet
0108bb3e40 MEDIUM: mailers: Init alerts during conf parsing and refactor their processing
Email alerts relies on checks to send emails. The link between a mailers section
and a proxy was resolved during the configuration parsing, But initialization was
done when the first alert is triggered. This implied memory allocations and
tasks creations. With this patch, everything is now initialized during the
configuration parsing. So when an alert is triggered, only the memory required
by this alert is dynamically allocated.

Moreover, alerts processing had a flaw. The task handler used to process alerts
to be sent to the same mailer, process_email_alert, was designed to give back
the control to the scheduler when an alert was sent. So there was a delay
between the sending of 2 consecutives alerts (the min of
"proxy->timeout.connect" and "mailer->timeout.mail"). To fix this problem, now,
we try to process as much queued alerts as possible when the task is woken up.
2017-10-31 11:36:12 +01:00
Baptiste Assmann
201c07f681 MAJOR/REORG: dns: DNS resolution task and requester queues
This patch is a major upgrade of the internal run-time DNS resolver in
HAProxy and it brings the following 2 main changes:

1. DNS resolution task

Up to now, DNS resolution was triggered by the health check task.
From now, DNS resolution task is autonomous. It is started by HAProxy
right after the scheduler is available and it is woken either when a
network IO occurs for one of its nameserver or when a timeout is
matched.

From now, this means we can enable DNS resolution for a server without
enabling health checking.

2. Introduction of a dns_requester structure

Up to now, DNS resolution was purposely made for resolving server
hostnames.
The idea, is to ensure that any HAProxy internal object should be able
to trigger a DNS resolution. For this purpose, 2 things has to be done:
  - clean up the DNS code from the server structure (this was already
    quite clean actually) and clean up the server's callbacks from
    manipulating too much DNS resolution
  - create an agnostic structure which allows linking a DNS resolution
    and a requester of any type (using obj_type enum)

3. Manage requesters through queues

Up to now, there was an uniq relationship between a resolution and it's
owner (aka the requester now). It's a shame, because in some cases,
multiple objects may share the same hostname and may benefit from a
resolution being performed by a third party.
This patch introduces the notion of queues, which are basically lists of
either currently running resolution or waiting ones.

The resolutions are now available as a pool, which belongs to the resolvers.
The pool has has a default size of 64 resolutions per resolvers and is
allocated at configuration parsing.
2017-06-02 11:58:54 +02:00
Christopher Faulet
8ef75251e3 MAJOR: spoe: refactor the filter to clean up the code
The SPOE code is now pretty big and it was the good time to clean it up. It is
not perfect, some parts remains a bit ugly. But it is far better now.
2017-03-09 15:32:55 +01:00
Willy Tarreau
865c5148e6 CLEANUP: checks: make use of the post-init registration to start checks
Instead of calling the checks directly from the init code, we now
register the start_checks() function to be run at this point. This
also allows to unexport the check init function and to remove one
include from haproxy.c.
2016-12-21 21:30:54 +01:00
Christopher Faulet
ba7bc164f7 MINOR: spoe/checks: Add support for SPOP health checks
A new "option spop-check" statement has been added to enable server health
checks based on SPOP HELLO handshake. SPOP is the protocol used by SPOE filters
to talk to servers.
2016-11-09 22:57:02 +01:00
Baptiste Assmann
95db2bcfee MAJOR: check: find out which port to use for health check at run time
HAProxy used to deduce port used for health checks when parsing configuration
at startup time.
Because of this way of working, it makes it complicated to change the port at
run time.

The current patch changes this behavior and makes HAProxy to choose the
port used for health checking when preparing the check task itself.

A new type of error is introduced and reported when no port can be found.

There won't be any impact on performance, since the process to find out the
port value is made of a few 'if' statements.

This patch also introduces a new check state CHK_ST_PORT_MISS: this flag is
used to report an error in the case when HAProxy needs to establish a TCP
connection to a server, to perform a health check but no TCP ports can be
found for it.

And last, it also introduces a new stream termination condition:
SF_ERR_CHK_PORT. Purpose of this flag is to report an error in the event when
HAProxy has to run a health check but no port can be found to perform it.
2016-09-11 08:12:13 +02:00
Baptiste Assmann
a68ca96375 MAJOR: server: add DNS-based server name resolution
Relies on the DNS protocol freshly implemented in HAProxy.
It performs a server IP addr resolution based on a server hostname.
2015-06-13 22:07:35 +02:00
Simon Horman
64e3416662 MEDIUM: Allow suppression of email alerts by log level
This patch adds a new option which allows configuration of the maximum
log level of messages for which email alerts will be sent.

The default is alert which is more restrictive than
the current code which sends email alerts for all priorities.
That behaviour may be configured using the new configuration
option to set the maximum level to notice or greater.

	email-alert level notice

Signed-off-by: Simon Horman <horms@verge.net.au>
2015-02-06 07:59:58 +01:00
Simon Horman
0ba0e4ac07 MEDIUM: Support sending email alerts
Signed-off-by: Simon Horman <horms@verge.net.au>
2015-02-03 00:24:16 +01:00
Simon Horman
bfb5d33fe6 MEDIUM: Add free_check() helper
Add free_check() helper to free the memory allocated by init_check().

Signed-off-by: Simon Horman <horms@verge.net.au>
2015-02-03 00:24:15 +01:00
Simon Horman
b1900d55df MEDIUM: Refactor init_check and move to checks.c
Refactor init_check so that an error string is returned
rather than alerts being printed by it. Also
init_check to checks.c and provide a prototype to allow
it to be used from multiple C files.

Signed-off-by: Simon Horman <horms@verge.net.au>
2015-02-03 00:24:15 +01:00
Willy Tarreau
af54958d72 MEDIUM: checks: simplify server up/down/nolb transitions
We don't have to handle the maintenance transition here anymore so we
can simplify the functions and conditions. This also means that we don't
need the disable/enable functions but only a function to switch to each
new state.

It's worth mentionning that at this stage there are still confusions
between the server state and the checks states. For example, the health
check's state is adjusted from tracked servers changing state, while it
should not be.
2014-05-23 14:29:11 +02:00
Willy Tarreau
9fe7aae6eb MINOR: checks: use an inline function for health_adjust()
This function is called twice per request, and does almost always nothing.
Better use an inline version to avoid entering it when we can.

About 0.5% additional performance was gained this way.
2013-12-31 23:47:37 +01:00
Simon Horman
4a741432be MEDIUM: Paramatise functions over the check of a server
Paramatise the following functions over the check of a server

* set_server_down
* set_server_up
* srv_getinter
* server_status_printf
* set_server_check_status
* set_server_disabled
* set_server_enabled

Generally the server parameter of these functions has been removed.
Where it is still needed it is obtained using check->server.

This is in preparation for associating a agent check
with a server which runs as well as the server's existing check.
By paramatising these functions they may act on each of the checks
without further significant modification.

Explanation of the SSP_O_HCHK portion of this change:

* Prior to this patch SSP_O_HCHK serves a single purpose which
  is to tell server_status_printf() weather it should print
  the details of the check of a server or not.

  With the paramatisation that this patch adds there are two cases.
  1) Printing the details of the check in which case a
     valid check parameter is needed.
  2) Not printing the details of the check in which case
     the contents check parameter are unused.

  In case 1) we could pass SSP_O_HCHK and a valid check and;
  In case 2) we could pass !SSP_O_HCHK and any value for check
  including NULL.

  If NULL is used for case 2) then SSP_O_HCHK becomes supurfulous
  and as NULL is used for case 2) SSP_O_HCHK has been removed.

Signed-off-by: Simon Horman <horms@verge.net.au>
2013-11-19 09:35:54 +01:00
Willy Tarreau
bc174aa144 MINOR: cli: report connection status in "show sess xxx"
Connection flags, targets and transport layers are now reported in
"show sess $PTR", as it is an absolute requirement in debugging.
2012-11-19 16:22:22 +01:00
Simon Horman
63a4a822c1 CLEANUP: Make check_statuses, analyze_statuses and process_chk static
These symbols are only used inside src/checks.c
2012-03-24 21:54:19 +01:00
Cyril Bonté
cd19e51b05 [MEDIUM] add a maintenance mode to servers
This is a first attempt to add a maintenance mode on servers, using
the stat socket (in admin level).

It can be done with the following command :
   - disable server <backend>/<server>
   - enable  server <backend>/<server>

In this mode, no more checks will be performed on the server and it
will be marked as a special DOWN state (MAINT).

If some servers were tracking it, they'll go DOWN until the server
leaves the maintenance mode. The stats page and the CSV export also
display this special state.

This can be used to disable the server in haproxy before doing some
operations on this server itself. This is a good complement to the
"http-check disable-on-404" keyword and works in TCP mode.
2010-01-31 23:33:18 +01:00
Krzysztof Piotr Oledzki
97f07b832f [MEDIUM] Decrease server health based on http responses / events, version 3
Implement decreasing health based on observing communication between
HAProxy and servers.

Changes in this version 2:
 - documentation
 - close race between a started check and health analysis event
 - don't force fastinter if it is not set
 - better names for options
 - layer4 support

Changes in this version 3:
 - add stats
 - port to the current 1.4 tree
2009-12-16 00:29:27 +01:00
Krzysztof Piotr Oledzki
0960541e49 [MEDIUM] Collect & show information about last health check, v3
Collect information about last health check result,
including L7 code if possible (for example http or smtp
return code) and time took to finish last check.

Health check info is provided on both stats pages (html & csv)
and logged when a server is marked UP or DOWN. Currently active
check are marked with an asterisk, but only in html mode.

Currently there are 14 status codes:
  UNK     -> unknown

  INI     -> initializing
  SOCKERR -> socket error

  L4OK    -> check passed on layer 4, no upper layers testing enabled
  L4TOUT  -> layer 1-4 timeout
  L4CON   -> layer 1-4 connection problem, for example "Connection refused"
              (tcp rst) or "No route to host" (icmp)

  L6OK    -> check passed on layer 6
  L6TOUT  -> layer 6 (SSL) timeout
  L6RSP   -> layer 6 invalid response - protocol error

  L7OK    -> check passed on layer 7
  L7OKC   -> check conditionally passed on layer 7, for example
               404 with disable-on-404
  L7TOUT  -> layer 7 (HTTP/SMTP) timeout
  L7RSP   -> layer 7 invalid response - protocol error
  L7STS   -> layer 7 response error, for example HTTP 5xx
2009-09-23 23:15:36 +02:00
Willy Tarreau
26c250683f [MEDIUM] minor update to the task api: let the scheduler queue itself
All the tasks callbacks had to requeue the task themselves, and update
a global timeout. This was not convenient at all. Now the API has been
simplified. The tasks callbacks only have to update their expire timer,
and return either a pointer to the task or NULL if the task has been
deleted. The scheduler will take care of requeuing the task at the
proper place in the wait queue.
2009-03-08 09:38:41 +01:00
Krzysztof Oledzki
b304dc7fd7 [MEDIUM] Spread health checks even more
When one server appears at the same position in multiple backends, it
receives all the checks from all the backends exactly at the same time
because the health-checks are only spread within a backend but not
globally.

Attached patch implements per-server start delay in a different way.
Checks are now spread globally - not locally to one backend. It also makes
them start faster - IMHO there is no need to add a 'server->inter' when
calculating first execution. Calculation were moved from cfgparse.c to
checks.c. There is a new function start_checks() and now it is not called
when haproxy is started in MODE_CHECK.

With this patch it is also possible to set a global 'spread-checks'
parameter. It takes a percentage value (1..50, probably something near
5..10 is a good idea) so haproxy adds or removes that many percent to the
original interval after each check. My test shows that with 18 backends,
54 servers total and 10000ms/5% it takes about 45m to mix them completely.

I decided to use rand/srand pseudo-random number generator. I am aware it
is not recommend for a good randomness but a) we do not need a good random
generator here b) it is probably the most portable one.
2007-10-15 09:33:10 +02:00
Willy Tarreau
d825eef9c5 [MAJOR] replaced all timeouts with struct timeval
The timeout functions were difficult to manipulate because they were
rounding results to the millisecond. Thus, it was difficult to compare
and to check what expired and what did not. Also, the comparison
functions were heavy with multiplies and divides by 1000. Now, all
timeouts are stored in timevals, reducing the number of operations
for updates and leading to cleaner and more efficient code.
2007-05-12 22:35:00 +02:00
Willy Tarreau
e3ba5f0aaa [CLEANUP] included common/version.h everywhere 2006-06-29 18:54:54 +02:00
Willy Tarreau
baaee00406 [BIGMOVE] exploded the monolithic haproxy.c file into multiple files.
The files are now stored under :
  - include/haproxy for the generic includes
  - include/types.h for the structures needed within prototypes
  - include/proto.h for function prototypes and inline functions
  - src/*.c for the C files

Most include files are now covered by LGPL. A last move still needs
to be done to put inline functions under GPL and not LGPL.

Version has been set to 1.3.0 in the code but some control still
needs to be done before releasing.
2006-06-26 02:48:02 +02:00