Commit Graph

11995 Commits

Author SHA1 Message Date
Christopher Faulet
b6ea17c6fc CLEANUP: http-htx: Rename http_error structure into http_error_msg
The structure owns an error message, most of time loaded from a file, and
converted to HTX. It is created when an errorfile or errorloc directive is
parsed. It is renamed to avoid ambiguities with http_reply structure.
2020-05-20 18:27:13 +02:00
Christopher Faulet
7bd3de06e7 MINOR: http-htx: Add http_reply type based on what is used for http return rules
The http_reply structure is added. It represents a generic HTTP message used as
internal response by HAProxy. It is based on the structure used to store http
return rules. The aim is to store all error messages using this structure, as
well as http return and http deny rules.
2020-05-20 18:27:13 +02:00
Christopher Faulet
ba946bf23a BUG/MINOR: http-rules: Mark http return rules as final
For HTTP rules, this flag is only used to trigger a warning during HAProxy
startup when a final rule without ACL is not the last one. So this patch is
marked as a bug, but its impact is really limited.

No backport needed because http return rules were introduced in the 2.2.
2020-05-20 18:27:13 +02:00
Christopher Faulet
a53abad42d CLEANUP: http_ana: Remove unused TXN flags
TX_CLDENY, TX_CLALLOW, TX_SVDENY and TX_SVALLOW flags are unused. Only
TX_CLTARPIT is used to make the difference between an http deny rule and an http
tarpit rule. So these unused flags are removed.
2020-05-20 18:27:13 +02:00
William Lallemand
8177ad9895 MINOR: ssl: split config and runtime variable for ssl-{min,max}-ver
In the CLI command 'show ssl crt-list', the ssl-min-ver and the
ssl-min-max arguments were always displayed because the dumped versions
were the actual version computed and used by haproxy, instead of the
version found in the configuration.

To fix the problem, this patch separates the variables to have one with
the configured version, and one with the actual version used. The dump
only shows the configured version.
2020-05-20 16:49:02 +02:00
Ilya Shipitsin
13dd45178e CI: travis-ci: fix libslz download URL
let us switch to github mirror, travis cannot download original libslz

error is:
fatal: unable to access 'http://git.1wt.eu/git/libslz.git/': Failed
to connect to git.1wt.eu port 80: Connection timed out The command
"git clone http://git.1wt.eu/git/libslz.git/" failed and exited with
128...
2020-05-20 12:53:48 +02:00
Willy Tarreau
d68a6927f7 Revert "MEDIUM: sink: add global statement to create a new ring (sink buffer)"
This reverts commit 957ec59571.

As discussed with Emeric, the current syntax is not extensible enough,
this will be turned to a section instead in a forthcoming patch.
2020-05-20 12:06:16 +02:00
Willy Tarreau
928068a74b MINOR: ring: make the applet code not depend on the CLI
The ring to applet communication was only made to deal with CLI functions
but it's generic. Let's have generic appctx functions and have the CLI
rely on these instead. This patch introduces ring_attach_appctx() and
ring_detach_appctx().
2020-05-19 19:37:12 +02:00
Willy Tarreau
9597cbd17a MINOR: applet: adopt the wait list entry from the CLI
A few fields, including a generic list entry, were added to the CLI context
by commit 300decc8d9 ("MINOR: cli: extend the CLI context with a list and
two offsets"). It turns out that the list entry (l0) is solely used to
consult rings and that the generic ring_write() code is restricted to a
consumer on the CLI due to this, which was not the initial intent. Let's
make it a general purpose wait_entry field that is properly initialized
during appctx_init(). This will allow any applet to wait on a ring, not
just the CLI.
2020-05-19 19:37:12 +02:00
Willy Tarreau
223ddedb46 BUG/MEDIUM: ring: write-lock the ring while attaching/detaching
The LIST_ADDQ() and LIST_DEL_INIT() calls made to attach/detach a waiter
to the ring were made under a read lock which was sufficient in front of
the writer's write lock. But it's not sufficient against other readers!
Thus theorically multiple "show events" on the same ring buffer on the
CLI could result in a crash, even though for now I couldn't manage to
reproduce it.

This fixes commit 1d181e489c ("MEDIUM: ring: implement a wait mode for
watchers") so it must be backported to 2.1, possibly further if the ring
code gets backported.
2020-05-19 19:37:12 +02:00
Christopher Faulet
fe410d685a BUG/MAJOR: mux-fcgi: Stop sending loop if FCGI stream is blocked for any reason
Because of a typo error in conditions to exit the sending loop, it is possible
to loop infinitely in fcgi_snd_buf() function. Instead of checking the FCGI
stream is not blocked to continue sending data, the FCGI connection is used. So
it is possible to have a stream blocked because there is not enough space in the
mux buffers to copy more data but continue to loop to send more data.

This patch should fix the issue #637. It must be backported to 2.1.
2020-05-19 16:00:55 +02:00
Willy Tarreau
2bdcc70fa7 MEDIUM: hpack: use a pool for the hpack table
Instead of using malloc/free to allocate an HPACK table, let's declare
a pool. However the HPACK size is configured by the H2 mux, so it's
also this one which allocates it after post_check.
2020-05-19 11:40:39 +02:00
Emeric Brun
957ec59571 MEDIUM: sink: add global statement to create a new ring (sink buffer)
This patch adds the new global statement:
ring <name> [desc <desc>] [format <format>] [size <size>] [maxlen <length>]
  Creates a named ring buffer which could be used on log line for instance.

  <desc> is an optionnal description string of the ring. It will appear on
         CLI. By default, <name> is reused to fill this field.

  <format> is the log format used when generating syslog messages. It may be
           one of the following :

    iso       A message containing only the ISO date, followed by the text.
              The PID, process name and system name are omitted. This is
              designed to be used with a local log server.

    raw       A message containing only the text. The level, PID, date, time,
              process name and system name are omitted. This is designed to be
              used in containers or during development, where the severity only
              depends on the file descriptor used (stdout/stderr). This is
              the default.

    rfc3164   The RFC3164 syslog message format. This is the default.
              (https://tools.ietf.org/html/rfc3164)

    rfc5424   The RFC5424 syslog message format.
              (https://tools.ietf.org/html/rfc5424)

    short     A message containing only a level between angle brackets such as
              '<3>', followed by the text. The PID, date, time, process name
              and system name are omitted. This is designed to be used with a
              local log server. This format is compatible with what the systemd
              logger consumes.

    timed     A message containing only a level between angle brackets such as
              '<3>', followed by ISO date and by the text. The PID, process
              name and system name are omitted. This is designed to be
              used with a local log server.

  <length> is the maximum length of event message stored into the ring,
           including formatted header. If the event message is longer
           than <length>, it would be truncated to this length.

  <name> is the ring identifier, which follows the same naming convention as
         proxies and servers.

  <size> is the optionnal size in bytes. Default value is set to BUFSIZE.

Note: Historically sink's name and desc were refs on const strings. But with new
configurable rings a dynamic allocation is needed.
2020-05-19 11:04:11 +02:00
Emeric Brun
e709e1e777 MEDIUM: logs: buffer targets now rely on new sink_write
Before this path, they rely directly on ring_write bypassing
a part of the sink API.

Now the maxlen parameter of the log will apply only on the text
message part (and not the header, for this you woud prefer
to use the maxlen parameter on the sink/ring).

sink_write prototype was also reviewed to return the number of Bytes
written to be compliant with the other write functions.
2020-05-19 11:04:11 +02:00
Emeric Brun
bd163817ed MEDIUM: sink: build header in sink_write for log formats
This patch extends the sink_write prototype and code to
handle the rfc5424 and rfc3164 header.

It uses header building tools from log.c. Doing this some
functions/vars have been externalized.

facility and minlevel have been removed from the struct sink
and passed to args at sink_write because they depends of the log
and not of the sink (they remained unused by rest of the code
until now).
2020-05-19 11:04:11 +02:00
Emeric Brun
9e8ea0ae6f BUG/MINOR: logs: prevent double line returns in some events.
Historically some messages used to already contain the trailing LF but
not all, and __do_send_log adds a new one in needed cases. It also does
trim a trailing LF in certain cases while computing the max message
length, as a result of subtracting 1 to the available room in the
destination buffer. But the way it's done is wrong since some messages
still contain it.

So the code was fixed to always trim the trailing LF from messages if
present, and then only subtract 1 from the destination buffer room
instead of the size..

Note: new sink API is not designed to receive a trailing LF on
event messages

This could be backported to relevant stable versions with particular
care since the logic of the code changed a bit since 1.6 and there
may be other locations that need to be adjusted.
2020-05-19 10:59:53 +02:00
Christopher Faulet
62f79fe68a MEDIUM: checks: Make post-41 the default mode for mysql checks
MySQL 4.1 is old enough to be the default mode for mysql checks. So now, once a
username is defined, post-41 mode is automatically used. To do mysql checks on
previous MySQL version, the argument "pre-41" must be used.

Note, it is a compatibility breakage for everyone using an antique and
unsupported MySQL version.
2020-05-18 18:32:09 +02:00
Christopher Faulet
784063eeb2 MINOR: config: Don't dump keywords if argument is NULL
Helper functions are used to dump bind, server or filter keywords. These
functions are used to report errors during the configuration parsing. To have a
coherent API, these functions are now prepared to handle a null pointer as
argument. If so, no action is performed and functions immediately return.

This patch should fix the issue #631. It is not a bug. There is no reason to
backport it.
2020-05-18 18:30:06 +02:00
Christopher Faulet
2a37cdbe6b BUG/MINOR: cache: Don't needlessly test "cache" keyword in parse_cache_flt()
parse_cache_flt() is the registered callback for the "cache" filter keyword. It
is only called when the "cache" keyword is found on a filter line. So, it is
useless to test the filter name in the callback function.

This patch should fix the issue #634. It may be backported as far as 1.9.
2020-05-18 17:47:18 +02:00
Christopher Faulet
3b78809de0 DOC: SPOE is no longer experimental
The SPOE was marked as experiemental since the begining. But, it is no longer
true. This can be an obstacle to its use.
2020-05-18 17:47:18 +02:00
William Lallemand
5520d6f3c7 BUILD: ssl: fix build without OPENSSL_NO_ENGINE
The CI revealed that the boringssl build is still broken because of some
ifdef misplacement.

Bug introduced by dad3105 ("REORG: ssl: move ssl configuration to
cfgparse-ssl.c").

No backport needed.

Fix issue #633.
2020-05-18 14:19:26 +02:00
Aleksandar Lazi
6112f5ccd2 DOC/MINOR: halog: Add long help info for ic flag
Add missing long help text for the ic (ip count) flag
2020-05-18 09:30:43 +02:00
Ilya Shipitsin
bd7212be10 CLEANUP: acl: remove unused assignment
Coverity found unused variable assignment

    CID 1299671 (#1 of 1): Unused value (UNUSED_VALUE)assigned_pointer:
    Assigning value from args[arg + 1] to word here, but that stored
    value is overwritten before it can be used.
 958                        word = args[arg + 1];
 959                        arg = arg_end;
2020-05-18 09:21:27 +02:00
William Lallemand
212e993e71 BUILD: ssl: include errno.h in ssl_crtlist.c
In issue #632 boringssl build were broken by the lack of errno.h
include in ssl_crtlist.c

Bug introduced by 6e9556b ("REORG: ssl: move crtlist functions to src/ssl_crtlist.c").

No backport needed.
2020-05-18 08:39:31 +02:00
William Dauchy
1665c43fd8 BUILD: ssl: include buffer common headers for ssl_sock_ctx
since commit c0cdaffaa3 ("REORG: ssl: move ssl_sock_ctx and fix
cross-dependencies issues"), `struct ssl_sock_ctx` was moved in
ssl_sock.h. As it contains a `struct buffer`, including
`common/buffer.h` is now mandatory. I encountered an issue while
including ssl_sock.h on another patch:

include/types/ssl_sock.h:240:16: error: field ‘early_buf’ has incomplete type
  240 |  struct buffer early_buf;      /* buffer to store the early data received */

no backport needed.

Fixes: c0cdaffaa3 ("REORG: ssl: move ssl_sock_ctx and fix
cross-dependencies issues")
Signed-off-by: William Dauchy <w.dauchy@criteo.com>
2020-05-18 08:29:32 +02:00
Marcin Deranek
a8dbdf3c4b MEDIUM: stats: Enable more accurate moving average calculation for stats
Enable more accurate generation of moving averages for partially
populated samples window.
2020-05-16 22:40:06 +02:00
Marcin Deranek
3c27ddabec MINOR: stats: Expose native cum_req metric for a server
Expose native cum_req metric for a server: so far it was calculated as a
sum or all responses. Rename it from Cum. HTTP Responses to Cum. HTTP
Requests to be consistent with Frontend and Backend.
2020-05-16 22:40:03 +02:00
Marcin Deranek
4dc2b57d51 MINOR: stats: Prepare for more accurate moving averages
Add swrate_add_dynamic function which is similar to swrate_add, but more
accurate when calculating moving averages when not enough samples have
been processed yet.
2020-05-16 22:40:00 +02:00
Martin Tzvetanov Grigorov
5eb5e253e6 CI: travis-ci: switch arm64 builds to use openssl from distro
there are ongoing arm64 failures in travis-ci. we suspect that
build stucks on scripts/build-ssl.sh which is called during build.
let us try to use openssl from ubuntu distro and not build it
ourselves. to achive that we add "install" section without
build-ssl.sh
2020-05-15 20:52:52 +02:00
William Lallemand
2f14aeb0cc DOC: ssl: update MAINTAINERS file
Update the MAINTAINERS file with new files and maintainer.
2020-05-15 14:11:54 +02:00
William Lallemand
6a66a5ec9b REORG: ssl: move utility functions to src/ssl_utils.c
These functions are mainly used to extract information from
certificates.
2020-05-15 14:11:54 +02:00
William Lallemand
15e169447d REORG: ssl: move sample fetches to src/ssl_sample.c
Move all SSL sample fetches to src/ssl_sample.c.
2020-05-15 14:11:54 +02:00
William Lallemand
c0cdaffaa3 REORG: ssl: move ssl_sock_ctx and fix cross-dependencies issues
In order to move all SSL sample fetches in another file, moving the
ssl_sock_ctx definition in a .h file is required.

Unfortunately it became a cross dependencies hell to solve, because of
the struct wait_event field, so <types/connection.h> is needed which
created other problems.
2020-05-15 14:11:54 +02:00
William Lallemand
ef76107a4b MINOR: ssl: remove static keyword in some SSL utility functions
In order to move the the sample fetches to another file, remove the
static keyword of some utility functions in the SSL fetches.
2020-05-15 14:11:54 +02:00
William Lallemand
dad3105157 REORG: ssl: move ssl configuration to cfgparse-ssl.c
Move all the configuration parsing of the ssl keywords in cfgparse-ssl.c
2020-05-15 14:11:54 +02:00
William Lallemand
da8584c1ea REORG: ssl: move the CLI 'cert' functions to src/ssl_ckch.c
Move the 'ssl cert' CLI functions to src/ssl_ckch.c.
2020-05-15 14:11:54 +02:00
William Lallemand
c756bbd3df REORG: ssl: move the crt-list CLI functions in src/ssl_crtlist.c
Move the crtlist functions for the CLI to src/ssl_crtlist.c
2020-05-15 14:11:54 +02:00
William Lallemand
fa1d8b4eaa REORG: ssl: move ckch_inst functions to src/ssl_ckch.c
Move ckch_inst_new() and ckch_inst_free() to src/ssl_ckch.c
2020-05-15 14:11:54 +02:00
William Lallemand
03c331c80a REORG: ssl: move the ckch_store related functions to src/ssl_ckch.c
Move the cert_key_and_chain functions:

int ssl_sock_load_files_into_ckch(const char *path, struct cert_key_and_chain *ckch, char **err);
int ssl_sock_load_pem_into_ckch(const char *path, char *buf, struct cert_key_and_chain *ckch , char **err);
void ssl_sock_free_cert_key_and_chain_contents(struct cert_key_and_chain *ckch);

int ssl_sock_load_key_into_ckch(const char *path, char *buf, struct cert_key_and_chain *ckch , char **err);
int ssl_sock_load_ocsp_response_from_file(const char *ocsp_path, char *buf, struct cert_key_and_chain *ckch, char **err);
int ssl_sock_load_sctl_from_file(const char *sctl_path, char *buf, struct cert_key_and_chain *ckch, char **err);
int ssl_sock_load_issuer_file_into_ckch(const char *path, char *buf, struct cert_key_and_chain *ckch, char **err);

And the utility ckch_store functions:

void ckch_store_free(struct ckch_store *store)
struct ckch_store *ckch_store_new(const char *filename, int nmemb)
struct ckch_store *ckchs_dup(const struct ckch_store *src)
ckch_store *ckchs_lookup(char *path)
ckch_store *ckchs_load_cert_file(char *path, int multi, char **err)
2020-05-15 14:11:54 +02:00
William Lallemand
c1c50b46e9 CLEANUP: ssl: avoid circular dependencies in ssl_crtlist.h
Add forward declarations in types/ssl_crtlist.h in order to avoid
circular dependencies. Also remove the listener.h include which is not
needed anymore.
2020-05-15 14:11:54 +02:00
William Lallemand
6e9556b635 REORG: ssl: move crtlist functions to src/ssl_crtlist.c
Move the crtlist functions to src/ssl_crtlist.c and their definitions to
proto/ssl_crtlist.h.

The following functions were moved:

/* crt-list entry functions */
void ssl_sock_free_ssl_conf(struct ssl_bind_conf *conf);
char **crtlist_dup_filters(char **args, int fcount);
void crtlist_free_filters(char **args);
void crtlist_entry_free(struct crtlist_entry *entry);
struct crtlist_entry *crtlist_entry_new();

/* crt-list functions */
void crtlist_free(struct crtlist *crtlist);
struct crtlist *crtlist_new(const char *filename, int unique);

/* file loading */
int crtlist_parse_line(char *line, char **crt_path, struct crtlist_entry *entry, const char *file, int linenum, char **err);
int crtlist_parse_file(char *file, struct bind_conf *bind_conf, struct proxy *curproxy, struct crtlist **crtlist, char **err);
int crtlist_load_cert_dir(char *path, struct bind_conf *bind_conf, struct crtlist **crtlist, char **err);
2020-05-15 14:11:54 +02:00
William Lallemand
c69973f7eb CLEANUP: ssl: add ckch prototypes in proto/ssl_ckch.h
Remove the static definitions of the ckch functions and add them to
ssl_ckch.h in order to use them outside ssl_sock.c.
2020-05-15 14:11:54 +02:00
William Lallemand
d4632b2b6d REORG: ssl: move the ckch structures to types/ssl_ckch.h
Move all the structures used for loading the SSL certificates in
ssl_ckch.h
2020-05-15 14:11:54 +02:00
William Lallemand
be21b663cd REORG: move the crt-list structures in their own .h
Move the structure definitions specifics to the crt-list in
types/ssl_crtlist.h.
2020-05-15 14:11:54 +02:00
William Lallemand
336c4bbb08 CLEANUP: ssl: remove the shsess_* macros
The shsess_* macros where already defined in proto/ssl_sock.h, remove
them from ssl_sock.c
2020-05-15 14:11:54 +02:00
William Lallemand
7fd8b4567e REORG: ssl: move macros and structure definitions to ssl_sock.h
The ssl_sock.c file contains a lot of macros and structure definitions
that should be in a .h. Move them to the more appropriate
types/ssl_sock.h file.
2020-05-15 14:11:54 +02:00
Dragan Dosen
2dec6a3bf1 MEDIUM: ssl: use ssl_sock_get_ssl_object() in fetchers where appropriate
Doing this also makes sure that conn->xprt_ctx is always checked before
using it.
2020-05-14 13:13:14 +02:00
Dragan Dosen
eb607fe6a1 MINOR: ssl: add a new function ssl_sock_get_ssl_object()
This one can be used later to get a SSL object from connection. It will
return NULL if connection is not established over SSL.
2020-05-14 13:13:14 +02:00
Dragan Dosen
9ac9809cb9 MEDIUM: ssl: split ssl_sock_msgcbk() and use a new callback mechanism
Make use of ssl_sock_register_msg_callback(). Function ssl_sock_msgcbk()
is now split into two dedicated functions for heartbeat and clienthello.
They are both registered by using a new callback mechanism for SSL/TLS
protocol messages.
2020-05-14 13:13:14 +02:00
Dragan Dosen
1e7ed04665 MEDIUM: ssl: allow to register callbacks for SSL/TLS protocol messages
This patch adds the ability to register callbacks for SSL/TLS protocol
messages by using the function ssl_sock_register_msg_callback().

All registered callback functions will be called when observing received
or sent SSL/TLS protocol messages.
2020-05-14 13:13:14 +02:00