Commit Graph

12104 Commits

Author SHA1 Message Date
Willy Tarreau
af613e8359 CLEANUP: thread: rename __decl_hathreads() to __decl_thread()
I can never figure whether it takes an "s" or not, and in the end it's
better if it matches the file's naming, so let's call it "__decl_thread".
2020-06-11 10:18:56 +02:00
Willy Tarreau
3f567e4949 REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h
This splits the hathreads.h file into types+macros and functions. Given
that most users of this file used to include it only to get the definition
of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h
(i.e. types and macros).

All the thread management was left to haproxy/thread.h. It's worth noting
the drop of the trailing "s" in the name, to remove the permanent confusion
that arises between this one and the system implementation (no "s") and the
makefile's option (no "s").

For consistency, src/hathreads.c was also renamed thread.c.

A number of files were updated to only include thread-t which is the one
they really needed.

Some future improvements are possible like replacing empty inlined
functions with macros for the thread-less case, as building at -O0 disables
inlining and causes these ones to be emitted. But this really is cosmetic.
2020-06-11 10:18:56 +02:00
Willy Tarreau
5775d0964a CLEANUP: threads: remove a few needless includes of hathreads.h
A few files were including it while not needing it (anymore). Some
only required access to the atomic ops and got haproxy/atomic.h in
exchange. Others didn't need it at all. A significant number of
files still include it only for THREAD_LOCAL definition.
2020-06-11 10:18:56 +02:00
Willy Tarreau
9453ecd670 REORG: threads: extract atomic ops from hathreads.h
The hathreads.h file has quickly become a total mess because it contains
thread definitions, atomic operations and locking operations, all this for
multiple combinations of threads, debugging and architectures, and all this
done with random ordering!

This first patch extracts all the atomic ops code from hathreads.h to move
it to haproxy/atomic.h. The code there still contains several sections
based on non-thread vs thread, and GCC versions in the latter case. Each
section was arranged in the exact same order to ease finding.

The redundant HA_BARRIER() which was the same as __ha_compiler_barrier()
was dropped in favor of the latter which follows the naming convention
of all other barriers. It was only used in freq_ctr.c which was updated.
Additionally, __ha_compiler_barrier() was defined inconditionally but
used only for thread-related operations, so it was made thread-only like
HA_BARRIER() used to be. We'd still need to have two types of compiler
barriers, one for the general case (e.g. signals) and another one for
concurrency, but this was not addressed here.

Some comments were added at the beginning of each section to inform about
the use case and warn about the traps to avoid.

Some files which continue to include hathreads.h solely for atomic ops
should now be updated.
2020-06-11 10:18:56 +02:00
Willy Tarreau
853b297c9b REORG: include: split mini-clist into haproxy/list and list-t.h
Half of the users of this include only need the type definitions and
not the manipulation macros nor the inline functions. Moves the various
types into mini-clist-t.h makes the files cleaner. The other one had all
its includes grouped at the top. A few files continued to reference it
without using it and were cleaned.

In addition it was about time that we'd rename that file, it's not
"mini" anymore and contains a bit more than just circular lists.
2020-06-11 10:18:56 +02:00
Willy Tarreau
f0f1c80daf REORG: include: move istbuf.h to haproxy/
This one now relies on two files that were already cleaned up and is
only used by buffer.h.
2020-06-11 10:18:56 +02:00
Willy Tarreau
8dabda7497 REORG: include: split buf.h into haproxy/buf-t.h and haproxy/buf.h
File buf.h is one common cause of pain in the dependencies. Many files in
the code need it to get the struct buffer definition, and a few also need
the inlined functions to manipulate a buffer, but the file used to depend
on a long chain only for BUG_ON() (addressed by last commit).

Now buf.h is split into buf-t.h which only contains the type definitions,
and buf.h for all inlined functions. Callers who don't care can continue
to use buf.h but files in types/ must only use buf-t.h. sys/types.h had
to be added to buf.h to get ssize_t as used by b_move(). It's worth noting
that ssize_t is only supposed to be a size_t supporting -1, so b_move()
ought to be rethought regarding this.

The files were moved to haproxy/ and all their users were updated
accordingly. A dependency issue was addressed on fcgi whose C file didn't
include buf.h.
2020-06-11 10:18:56 +02:00
Willy Tarreau
025beea507 CLEANUP: debug: drop unused function p_malloc()
This one was introduced 5 years ago for debugging and never really used.
It is the one which used to cause circular dependencies issues. Let's drop
it instead of starting to split the debug include in two.
2020-06-11 10:18:56 +02:00
Willy Tarreau
2a83d60662 REORG: include: move debug.h from common/ to haproxy/
The debug file is cleaner now and does not depend on much anymore.
2020-06-11 10:18:56 +02:00
Willy Tarreau
58017eef3f REORG: include: move the BUG_ON() code to haproxy/bug.h
This one used to be stored into debug.h but the debug tools got larger
and require a lot of other includes, which can't use BUG_ON() anymore
because of this. It does not make sense and instead this macro should
be placed into the lower includes and given its omnipresence, the best
solution is to create a new bug.h with the few surrounding macros needed
to trigger bugs and place assertions anywhere.

Another benefit is that it won't be required to add include <debug.h>
anymore to use BUG_ON, it will automatically be covered by api.h. No
less than 32 occurrences were dropped.

The FSM_PRINTF macro was dropped since not used at all anymore (probably
since 1.6 or so).
2020-06-11 10:18:56 +02:00
Willy Tarreau
eb6f701b99 REORG: include: move ist.h from common/ to import/
Fortunately that file wasn't made dependent upon haproxy since it was
integrated, better isolate it before it's too late. Its dependency on
api.h was the result of the change from config.h, which in turn wasn't
correct. It was changed back to stddef.h for size_t and sys/types.h for
ssize_t. The recently added reference to MAX() was changed as it was
placed only to avoid a zero length in the non-free-standing version and
was causing a build warning in the hpack encoder.
2020-06-11 10:18:56 +02:00
Willy Tarreau
6019faba50 REORG: include: move openssl-compat.h from common/ to haproxy/
This file is to openssl what compat.h is to the libc, so it makes sense
to move it to haproxy/. It could almost be part of api.h but given the
amount of openssl stuff that gets loaded I fear it could increase the
build time.

Note that this file contains lots of inlined functions. But since it
does not depend on anything else in haproxy, it remains safe to keep
all that together.
2020-06-11 10:18:56 +02:00
Willy Tarreau
8d36697dee REORG: include: move base64.h, errors.h and hash.h from common to to haproxy/
These ones do not depend on any other file. One used to include
haproxy/api.h but that was solely for stddef.h.
2020-06-11 10:18:56 +02:00
Willy Tarreau
d678805783 REORG: include: move version.h to haproxy/
Few files were affected. The release scripts was updated.
2020-06-11 10:18:56 +02:00
Willy Tarreau
fd4bffe7c0 REORG: include: move the base files from common/ to haproxy/
The files currently covered by api-t.h and api.h (compat, compiler,
defaults, initcall) are now located inside haproxy/.
2020-06-11 10:18:56 +02:00
Willy Tarreau
b9082a93e5 CLEANUP: include: remove unused common/tools.h
Let's definitely get rid of this old file.
2020-06-11 10:18:56 +02:00
Willy Tarreau
4d653a6285 REORG: include: move SWAP/MID_RANGE/MAX_RANGE from tools.h to standard.h
Tools.h doesn't make sense for these 3 macros alone anymore, let's move
them to standard.h which will ultimately become again tools.h once moved.
2020-06-11 10:18:56 +02:00
Willy Tarreau
5ae5006dde REORG: include: move MIN/MAX from tools.h to compat.h
Given that these macros are usually provided by sys/param.h, better move
them to compat.h.
2020-06-11 10:18:56 +02:00
Willy Tarreau
57bb71e83a CLEANUP: include: remove unused template.h
There is one "template.h" per include subdirectory to show how to create
a new file but in practice nobody knows they're here so they're useless.
Let's simply remove them.
2020-06-11 10:18:56 +02:00
Willy Tarreau
86556a5377 CLEANUP: include: remove common/config.h
It was already an indirection to load other files, it's not used
anymore.
2020-06-11 10:18:56 +02:00
Willy Tarreau
4c7e4b7738 REORG: include: update all files to use haproxy/api.h or api-t.h if needed
All files that were including one of the following include files have
been updated to only include haproxy/api.h or haproxy/api-t.h once instead:

  - common/config.h
  - common/compat.h
  - common/compiler.h
  - common/defaults.h
  - common/initcall.h
  - common/tools.h

The choice is simple: if the file only requires type definitions, it includes
api-t.h, otherwise it includes the full api.h.

In addition, in these files, explicit includes for inttypes.h and limits.h
were dropped since these are now covered by api.h and api-t.h.

No other change was performed, given that this patch is large and
affects 201 files. At least one (tools.h) was already freestanding and
didn't get the new one added.
2020-06-11 10:18:42 +02:00
Willy Tarreau
7ab7031e34 REORG: include: create new file haproxy/api.h
This file includes everything that must be guaranteed to be available to
any buildable file in the project (including the contrib/ subdirs). For
now it includes <haproxy/api-t.h> so that standard integer types and
compiler macros are known, <common/initcall.h> to ease dynamic registration
of init functions, and <common/tools.h> for a few MIN/MAX macros.

version.h should probably also be added, though at the moment it doesn't
bring a great value.

All files which currently include the ones above should now switch to
haproxy/api.h or haproxy/api-t.h instead. This should also reduce build
time by having a single guard for several files at once.
2020-06-11 09:31:11 +02:00
Willy Tarreau
ca1765713b REORG: include: create new file haproxy/api-t.h
This file is at the lowest level of the include tree. Its purpose is to
make sure that common types are known pretty much everywhere, particularly
in structure declarations. It will essentially cover integer types such as
uintXX_t via inttypes.h, "size_t" and "ptrdiff_t" via stddef.h, and various
type modifiers such as __maybe_unused or ALIGN() via compiler.h, compat.h
and defaults.h.

It could be enhanced later if required, for example if some macros used
to compute array sizes are needed.
2020-06-11 09:31:11 +02:00
Willy Tarreau
fc80e30217 REORG: ebtree: clean up remains of the ebtree/ directory
The only leftovers were the unused compiler.h file and the LICENSE file
which is already mentioned in each and every ebtree file header.

A few build paths were updated in the contrib/ directory not to mention
this directory anymore, and all its occurrences were dropped from the
main makefile. From now on no other include path but include/ will be
needed anymore to build any file.
2020-06-11 09:31:11 +02:00
Willy Tarreau
8d2b777fe3 REORG: ebtree: move the include files from ebtree to include/import/
This is where other imported components are located. All files which
used to directly include ebtree were touched to update their include
path so that "import/" is now prefixed before the ebtree-related files.

The ebtree.h file was slightly adjusted to read compiler.h from the
common/ subdirectory (this is the only change).

A build issue was encountered when eb32sctree.h is loaded before
eb32tree.h because only the former checks for the latter before
defining type u32. This was addressed by adding the reverse ifdef
in eb32tree.h.

No further cleanup was done yet in order to keep changes minimal.
2020-06-11 09:31:11 +02:00
Willy Tarreau
ff0e8a44a4 REORG: ebtree: move the C files from ebtree/ to src/
As part of the include files cleanup, we're going to kill the ebtree
directory. For this we need to host its C files in a different location
and src/ is the right one.
2020-06-11 09:31:11 +02:00
Tim Duesterhus
f38175cf6e MINOR: sample: Add secure_memcmp converter
secure_memcmp compares two binary strings in constant time. It's only
available when haproxy is compiled with USE_OPENSSL.
2020-06-09 22:04:13 +02:00
William Lallemand
9fc6c97fb3 BUG/MINOR: mworker: fix a memleak when execvp() failed
Free next_argv when execvp() failed.

Must be backported as far as 1.8.

Should fix issue #668.
2020-06-08 10:01:13 +02:00
William Lallemand
e5ff4addb2 BUG/MINOR: ssl: fix a trash buffer leak in some error cases
Fix a trash buffer leak when we can't take the lock of the ckch, or when
"set ssl cert" is wrongly used.

The bug was mentionned in this thread:
https://www.mail-archive.com/haproxy@formilux.org/msg37539.html

The bug was introduced by commit bc6ca7c ("MINOR: ssl/cli: rework 'set
ssl cert' as 'set/commit'").

Must be backported in 2.1.
2020-06-08 09:51:14 +02:00
William Lallemand
0041741ef7 BUG/MEDIUM: mworker: fix the reload with an -- option
When HAProxy is started with a '--' option, all following parameters are
considered configuration files. You can't add new options after a '--'.

The current reload system of the master-worker adds extra options at the
end of the arguments list. Which is a problem if HAProxy was started wih
'--'.

This patch fixes the issue by copying the new option at the beginning of
the arguments list instead of the end.

This patch must be backported as far as 1.8.
2020-06-05 14:30:53 +02:00
William Lallemand
a6b3249935 BUG/MINOR: init: -S can have a parameter starting with a dash
There is no reason the -S option can't take an argument which starts with
a -. This limitation must only be used for options that take a
non-finite list of parameters (-sf/-st)

This can be backported only if the previous patch which fixes
copy_argv() is backported too.

Could be backported as far as 1.9.
2020-06-05 14:30:49 +02:00
William Lallemand
4f71d304aa BUG/MINOR: init: -x can have a parameter starting with a dash
There is no reason the -x option can't take an argument which starts with
a -. This limitation must only be used for options that take a
non-finite list of parameters (-sf/-st)

This can be backported only if the previous patch which fixes
copy_argv() is backported too.

Could be backported as far as 1.8.
2020-06-05 14:30:45 +02:00
William Lallemand
df6c5a8ffa BUG/MEDIUM: mworker: fix the copy of options in copy_argv()
The copy_argv() function, which is used to copy and remove some of the
arguments of the command line in order to re-exec() the master process,
is poorly implemented.

The function tries to remove the -x and the -sf/-st options but without
taking into account that some of the options could take a parameter
starting with a dash.

In issue #644, haproxy starts with "-L -xfoo" which is perfectly
correct. However, the re-exec is done without "-xfoo" because the master
tries to remove the "-x" option. Indeed, the copy_argv() function does
not know how much arguments an option can have, and just assume that
everything starting with a dash is an option. So haproxy is exec() with
"-L" but without a parameter, which is wrong and leads to the exit of
the master, with usage().

To fix this issue, copy_argv() must know how much parameters an option
takes, and copy or skip the parameters correctly.

This fix is a first step but it should evolve to a cleaner way of
declaring the options to avoid deduplication of the parsing code, so we
avoid new bugs.

Should be backported with care as far as 1.8, by removing the options
that does not exists in the previous versions.
2020-06-05 14:30:34 +02:00
Christopher Faulet
efde955bdb BUG/MEDIUM: contrib/prometheus-exporter: Properly set flags to dump metrics
When the metrics are dumped, in the main function promex_dump_metrics(), the
appctx flags are set before entering in a new scope, among others things to know
which metrics names and descriptions to use. But, those flags are not restored
when the dump is interrupted because of a full output buffer. If this happens
after the dump of global metrics, it may only lead to extra #TYPE and #HELP
lines. But if this happens during the dump of global metrics, the following
dumps of frontends, backends and servers metrics use names and descriptions of
global ones with the unmatching indexes. This first leads to unexisting metrics
names. For instance, "haproxy_frontend_nbproc". But also to out-of-bound
accesses to name and description arrays because there are more stats fields than
info fields.

It is easy to reproduce the bug using small buffers, setting tune.bufsize to
8192 for instance.

This patch should fix the issue #666. It must be backported as far as 2.0.
2020-06-05 09:41:46 +02:00
Christopher Faulet
c0fcbe4150 BUG/MINOR: checks: Fix test on http-check rulesets during config validity check
When checking the config validity of the http-check rulesets, the test on the
ruleset type is inverted. So a warning about ignored directive is emitted when
the config is valid and omitted when it should be reported.

No backport needed.
2020-06-03 19:08:35 +02:00
Christopher Faulet
5cf2dfc5fd BUG/MEDIUM: hlua: Lock pattern references to perform set/add/del operations
The pattern references lock must be hold to perform set/add/del
operations. Unfortunately, it is not true for the lua functions manipulating acl
and map files.

This patch should fix the issue #664. It must be backported as far as 1.8.
2020-06-03 19:08:35 +02:00
William Lallemand
7c26ed73f7 DOC: add a line about comments in crt-list
Add a line about comments in crt-list.

Fix issue #514.
2020-06-03 17:36:39 +02:00
Christopher Faulet
23308ebf0d BUG/MEDIUM: lua: Reset analyse expiration timeout before executing a lua action
Before executing a lua action, the analyse expiration timeout of the
corresponding channel must be reset. Otherwise, when it expires, for instance
because of a call to core.msleep(), if the action yields, an expired timeout
will be used for the stream's task, leading to a loop.

This patch should fix the issue #661. It must be backported in all versions
supporting the lua.
2020-06-03 10:24:58 +02:00
Christopher Faulet
89aed32bff MINOR: mux-h1/proxy: Add a proxy option to disable clear h2 upgrade
By default, HAProxy is able to implicitly upgrade an H1 client connection to an
H2 connection if the first request it receives from a given HTTP connection
matches the HTTP/2 connection preface. This way, it is possible to support H1
and H2 clients on a non-SSL connections. It could be a problem if for any
reason, the H2 upgrade is not acceptable. "option disable-h2-upgrade" may now be
used to disable it, per proxy. The main puprose of this option is to let an
admin to totally disable the H2 support for security reasons. Recently, a
critical issue in the HPACK decoder was fixed, forcing everyone to upgrade their
HAProxy version to fix the bug. It is possible to disable H2 for SSL
connections, but not on clear ones. This option would have been a viable
workaround.
2020-06-03 10:23:39 +02:00
William Lallemand
f187ce68b1 Revert "MINOR: ssl: rework add cert chain to CTX to be libssl independent"
This reverts commit 4fed93eb72.

This commit was simplifying the certificate chain loading with
SSL_CTX_add_extra_chain_cert() which is available in all SSL libraries.
Unfortunately this function is not compatible with the
multi-certificates bundles, which have the effect of concatenating the
chains of all certificate types instead of creating a chain for each
type (RSA, ECDSA etc.)

Should fix issue #655.
2020-06-02 18:37:42 +02:00
Willy Tarreau
39bd740d00 CLEANUP: regex: remove outdated support for regex actions
The support for reqrep and friends was removed in 2.1 but the
chain_regex() function and the "action" field in the regex struct
was still there. This patch removes them.

One point worth mentioning though. There is a check_replace_string()
function whose purpose was to validate the replacement strings passed
to reqrep. It should also be used for other replacement regex, but is
never called. Callers of exp_replace() should be checked and a call to
this function should be added to detect the error early.
2020-06-02 17:17:13 +02:00
Emeric Brun
530ba38a70 BUG/MINOR: peers: fix internal/network key type mapping.
Network types were directly and mistakenly mapped on sample types:

This patch fix the doc with values effectively used to keep backward
compatiblitiy on existing implementations.

In addition it adds an internal/network mapping for key types to avoid
further mistakes adding or modifying internals types.

This patch should be backported on all maintained branches,
particularly until v1.8 included for documentation part.
2020-06-02 16:25:19 +02:00
Willy Tarreau
570a22b11b BUILD: sink: address build warning on 32-bit architectures
The recently added ring section post-processing added this bening
warning on 32-bit archs:

  src/sink.c: In function 'cfg_post_parse_ring':
  src/sink.c:994:15: warning: format '%lu' expects argument of type 'long unsigned int', but argument 4 has type 'size_t {aka unsigned int}' [-Wformat=]
      ha_warning("ring '%s' event max length '%u' exceeds size, forced to size '%lu'.\n",
               ^
Let's just cast b_size() to unsigned long here.
2020-06-02 12:01:51 +02:00
William Lallemand
55af9e90f8 CLEANUP: ssl: remove comment from dump_crtlist_sslconf()
Since 8177ad9 ("MINOR: ssl: split config and runtime variable for
ssl-{min,max}-ver"), the dump for ssl-min-ver and ssl-max-ver is fixed,
so we can remove the comment.
2020-06-02 11:54:46 +02:00
William Lallemand
50df1cb1e5 MINOR: ssl: set ssl-min-ver in ambiguous configurations
Using ssl-max-ver without ssl-min-ver is ambiguous.

When the ssl-min-ver is not configured, and ssl-max-ver is set to a
value lower than the default ssl-min-ver (which is TLSv1.2 currently),
set the ssl-min-ver to the value of ssl-max-ver, and emit a warning.
2020-06-02 11:13:12 +02:00
Emeric Brun
975564784f MEDIUM: ring: add new srv statement to support octet counting forward
log-proto <logproto>
  The "log-proto" specifies the protocol used to forward event messages to
  a server configured in a ring section. Possible values are "legacy"
  and "octet-count" corresponding respectively to "Non-transparent-framing"
  and "Octet counting" in rfc6587. "legacy" is the default.

Notes: a separated io_handler was created to avoid per messages test
and to prepare code to set different log protocols such as
request- response based ones.
2020-05-31 10:49:43 +02:00
Emeric Brun
494c505703 MEDIUM: ring: add server statement to forward messages from a ring
This patch adds new statement "server" into ring section, and the
related "timeout connect" and "timeout server".

server <name> <address> [param*]
  Used to configure a syslog tcp server to forward messages from ring buffer.
  This supports for all "server" parameters found in 5.2 paragraph.
  Some of these parameters are irrelevant for "ring" sections.

timeout connect <timeout>
  Set the maximum time to wait for a connection attempt to a server to succeed.

  Arguments :
    <timeout> is the timeout value specified in milliseconds by default, but
              can be in any other unit if the number is suffixed by the unit,
              as explained at the top of this document.

timeout server <timeout>
  Set the maximum time for pending data staying into output buffer.

  Arguments :
    <timeout> is the timeout value specified in milliseconds by default, but
              can be in any other unit if the number is suffixed by the unit,
              as explained at the top of this document.

  Example:
    global
        log ring@myring local7

    ring myring
        description "My local buffer"
        format rfc3164
        maxlen 1200
        size 32764
        timeout connect 5s
        timeout server 10s
        server mysyslogsrv 127.0.0.1:6514
2020-05-31 10:46:13 +02:00
Emeric Brun
9f2ff3a700 BUG/MINOR: error on unknown statement in ring section.
Previously unknown statements were silently ignored.
2020-05-31 10:37:54 +02:00
Emeric Brun
dcd58afaf1 MINOR: ring: re-work ring attach generic API.
Attach is now independent on appctx, which was unused anyway.
2020-05-31 10:37:31 +02:00
Willy Tarreau
6fab3e6d91 SCRIPTS: publish-release: pass -n to gzip to remove timestamp
It just appeared that the tar.gz we put online are not reproducible
because a timestamp is put by default into the archive. Passing "-n"
to gzip is sufficient to remove this timestamp, so let's do it, and
also make the gzip command configurable for more flexibility. Now
issuing the commands multiple times finally results in the same
archives being produced.

This should be backported to supported stable branches.
2020-05-30 06:59:07 +02:00