Commit Graph

603 Commits

Author SHA1 Message Date
Willy Tarreau
8dd672523f BUILD: makefile: move default verbosity settings to include/make/verbose.mk
The $(Q), $(V), $(cmd_xx) handling needs to be reused in sub-project
makefiles and it's a pain to maintain inside the main makefile. Let's
just move that into a new subdir include/make/ with a dedicated file
"verbose.mk". It slightly cleans up the makefile in addition.
2022-11-17 10:56:35 +01:00
Willy Tarreau
d575661d40 BUILD: makefile: properly pass CC to sub-projects
The "poll" and "tcploop" sub-projects have their own makefiles. But
since the cmd_* commands were migrated from "echo" to $(info) with
make 3.81, the command is confusingly displayed in the top-level
makefile before entering the directory, even making one think that
the build occurred.

Let's instead propagate the verbosity level through the sub-projects
and let them adapt their own cmd_CC. For now this peans a little bit
of duplication for poll and tcploop.
2022-11-17 10:56:35 +01:00
Willy Tarreau
59b50bd04a BUILD: makefile: mark poll and tcploop targets as phony
Since these ones come with their own makefiles, the top-level makefile
cannot decide when they have to be rebuilt, it should always defer the
decision to the compoent's makefile, so we must mark them as phony.
Because of these, they were not updated after a change without calling
a "clean" first.
2022-11-17 10:56:35 +01:00
William Lallemand
83e9bcaa87 BUILD: Makefile: add "USE_SHM_OPEN" on the linux-musl target
The startup-logs with the shm works correctly with Alpine and Musl,
enable the feature by default for the linux-musl target.
2022-10-21 10:35:37 +02:00
William Lallemand
eba6a54cd4 MINOR: logs: startup-logs can use a shm for logging the reload
When compiled with USE_SHM_OPEN=1 the startup-logs are now able to use
an shm which is used to keep the logs when switching to mworker wait
mode. This allows to keep the failed reload logs.

When allocating the startup-logs at first start of the process, haproxy
will do a shm_open with a unique path using the PID of the process, the
file is unlink immediatly so we don't let unwelcomed files be. The fd
resulting from this shm is stored in the HAPROXY_STARTUPLOGS_FD
environment variable so it can be mmap again when switching to wait
mode.

When forking children, the process is copying the mmap to a a mallocated
ring so we never share the same memory section between the master and
the workers. When switching to wait mode, the shm is not used anymore as
it is also copied to a mallocated structure.

This allow to use the "show startup-logs" command over the master CLI,
to get the logs of the latest startup or reload. This way the logs of
the latest failed reload are also kept.

This is only activated on the linux-glibc target for now.
2022-10-13 16:50:22 +02:00
Amaury Denoyelle
92fa63f735 CLEANUP: quic: create a dedicated quic_conn module
xprt_quic module was too large and did not reflect the true architecture
by contrast to the other protocols in haproxy.

Extract code related to XPRT layer and keep it under xprt_quic module.
This code should only contains a simple API to communicate between QUIC
lower layer and connection/MUX.

The vast majority of the code has been moved into a new module named
quic_conn. This module is responsible to the implementation of QUIC
lower layer. Conceptually, it overlaps with TCP kernel implementation
when comparing QUIC and HTTP1/2 stacks of haproxy.

This should be backported up to 2.6.
2022-10-03 16:25:17 +02:00
Amaury Denoyelle
d80fbcaca2 REORG: mux-quic: export HTTP related function in a dedicated file
Extract function dealing with HTX outside of MUX QUIC. For the moment,
only rcv_buf stream operation is concerned.

The main objective is to be able to support both TCP and HTTP proxy mode
with a common base and add specialized modules on top of it.

This should be backported up to 2.6.
2022-09-20 15:35:23 +02:00
Amaury Denoyelle
36d50bff22 REORG: mux-quic: extract traces in a dedicated source file
QUIC MUX implements several APIs to interface with stream, quic-conn and
app-ops layers. It is planified to better separate this roles, possibly
by using several files.

The first step is to extract QUIC MUX traces in a dedicated source
files. This will allow to reuse traces in multiple files.

The main objective is to be
able to support both TCP and HTTP proxy mode with a common base and add
specialized modules on top of it.

This should be backported up to 2.6.
2022-09-20 15:35:09 +02:00
Frédéric Lécaille
ef2d2340e6 BUILD: udp-perturb: Add a make target for udp-perturb tool
This is only to rely on make to build this tool.
2022-09-08 20:47:28 +02:00
Brad Smith
2f105b8a45 BUILD: makefile: enable crypt(3) for NetBSD
Allow NetBSD to support encrypted passwords in Userlists.
2022-09-03 06:11:08 +02:00
Willy Tarreau
75014fcd4d DEV: haring: add a simple utility to read file-backed rings
With the ability to back a memory ring into an mmapped file, it makes
sense to be able to dump these files. That's what this utility does.
The entire ring is dumped to stdout. It's well suited to large dumps,
it converts roughly 6 GB of logs per second.

The utility is really meant for developers at the moment. It might
evolve into a more general tool but at the moment it's still possible
that it might need to be run under gdb to process certain crash dumps.

Also at the moment it must not be used on a ring being actively written
to or it will dump garbage.

The code is made so that we can envision later to attach to a live
ring and dump live contents, but this requires that the utility is
built with the exact same options (threads etc), and that the file
is opened read-write. For now these parts have been commented out,
waiting for a reasonably balanced and non-intrusive solution to be
found (e.g. signals must be intercepted so that the tool cannot
leave the ring with a watcher present).

If it is detected that the memory layout of the ring struct differs,
a warning is emitted. At the end, if an error occurs, a warning is
printed as well (this does happen when the process is not cleanly
stopped, but it indicates the end was reached).
2022-08-12 11:48:32 +02:00
Willy Tarreau
87aff021db MINOR: thread: provide an alternative to pthread's rwlock
Since version 1.1.0, OpenSSL's libcrypto ignores the provided locking
mechanism and uses pthread's rwlocks instead. The problem is that for
some code paths (e.g. async engines) this results in a huge amount of
syscalls on systems facing a bit of contention, to the point where more
than 80% of the CPU can be spent in the system dealing with spinlocks
just for futex_wake().

This patch provides an alternative by redefining the relevant pthread
rwlocks from the low-overhead version of the progressive rw locks. This
way there will be no more syscalls in case of contention, and CPU will
be burnt in userland. Doing this saves massive amounts of CPU, where
the locks only take 12-15% vs 80% before, which allows SSL to work much
faster on large thread counts (e.g. 24 or more).

The tryrdlock and trywrlock variants have been implemented using a CAS
since their goal is only to succeed on no contention and never to wait.
The pthread_rwlock API is complete except that the timed versions of
the rdlock and wrlock do not wait and simply fall back to trylock
versions.

Since the gains have only been observed with async engines for now,
this option remains disabled by default. It can be enabled at build
time using USE_PTHREAD_EMULATION=1.
2022-07-30 10:17:22 +02:00
Frédéric Lécaille
1c9c2f6c02 MEDIUM: quic: Cubic congestion control algorithm implementation
Cubic is the congestion control algorithm used by default by the Linux kernel
since 2.6.15 version. This algorithm is supposed to achieve good scalability and
fairness between flows using the same network path, it should also be used by QUIC
by default. This patch implements this algorithm and select it as default algorithm
for the congestion control.

Must be backported to 2.6.
2022-07-29 17:32:05 +02:00
Brad Smith
bc50e0d0fb BUILD: makefile: Fix install(1) handling for OpenBSD/NetBSD/Solaris/AIX
Add a new INSTALL variable to allow overridiing the flags passed to
install(1). install(1) on OpenBSD/NetBSD/Solaris/AIX does not support
the -v flag. With the new INSTALL variable and handling only use the
-v flag with the Linux targets.
2022-07-16 18:51:13 +02:00
Christian Ruppert
3214b44702 BUILD: Makefile: Add Lua 5.4 autodetect
This patch is based on:
https://www.mail-archive.com/haproxy@formilux.org/msg39689.html
Thanks to Callum Farmer!

Signed-off-by: Christian Ruppert <idl0r@qasl.de>
2022-07-04 17:28:48 +02:00
Christopher Faulet
2b6777021d MEDIUM: bwlim: Add support of bandwith limitation at the stream level
This patch adds a filter to limit bandwith at the stream level. Several
filters can be defined. A filter may limit incoming data (upload) or
outgoing data (download). The limit can be defined per-stream or shared via
a stick-table. For a given stream, the bandwith limitation filters can be
enabled using the "set-bandwidth-limit" action.

A bandwith limitation filter can be used indifferently for HTTP or TCP
stream. For HTTP stream, only the payload transfer is limited. The filter is
pretty simple for now. But it was designed to be extensible. The current
design tries, as far as possible, to never exceed the limit. There is no
burst.
2022-06-24 14:06:26 +02:00
Amaury Denoyelle
c003f50122 MINOR: qpack: implement standalone decoder tool
Implement a standalone binary to be able to easily a hex-string QPACK
stream. The binary must be compiled via the Makefile. Hex-strings are
specified on stdin.
2022-06-15 15:42:10 +02:00
Willy Tarreau
453d60be17 BUILD: makefile: reorder objects by build time
As usual, let's sort objects by inverse build time at -O2. It will
still vary based on the options but keeps them optimally sorted for
parallel builds.
2022-05-30 19:24:27 +02:00
Frédéric Lécaille
6f7607ef1f MINOR: h3: Add a statistics module for h3
Add ->inc_err_cnt new callback to qcc_app_ops struct which can
be called from xprt to increment the application level error code counters.
It take the application context as first parameter to be generic and support
new QUIC applications to come.
Add h3_stats.c module with counters for all the frame types and error codes.
2022-05-30 09:59:26 +02:00
Frédéric Lécaille
748ece68b8 MINOR: quic: QUIC transport parameters split.
Make the transport parameters be standlone as much as possible as
it consists only in encoding/decoding data into/from buffers.
Reduce the size of xprt_quic.h. Unfortunalety, I think we will
have to continue to include <xprt_quic-t.h> to use the trace API
into this module.
2022-05-30 09:59:26 +02:00
Willy Tarreau
cb086c6de1 REORG: stconn: rename conn_stream.{c,h} to stconn.{c,h}
There's no more reason for keepin the code and definitions in conn_stream,
let's move all that to stconn. The alphabetical ordering of include files
was adjusted.
2022-05-27 19:33:35 +02:00
Frédéric Lécaille
a58cafeb89 MINOR: quic_stats: Add a new stats module for QUIC
This is a very minimalist frontend only stats module with only one gauge for the
QUIC establishing connections count.
2022-05-20 17:11:13 +02:00
Amaury Denoyelle
1b5f77fc18 MINOR: ncbuf: define non-contiguous buffer
Define the new type ncbuf. It can be used as a buffer with
non-contiguous data and wrapping support.

To reduce as much as possible the memory footprint, size of data and
gaps are stored in the gaps themselves. This put some limitation on the
buffer usage. A reserved space is present just before the head to store
the size of the first data block. Also, add and delete operations will
be constrained to ensure minimal gap sizes are preserved.

The sizes stored in the gaps are represented by a custom type named
ncb_sz_t. This type is a typedef to easily change it : this has a
direct impact on the maximum buffer size (MAX(ncb_sz_t) - sizeof(ncb_sz_t))
and the minimal gap sizes (sizeof(ncb_sz_t) * 2)).
Currently, it is set to uint32_t.
2022-05-12 18:13:21 +02:00
Willy Tarreau
6d868a2c65 BUILD: makefile: add -Wfatal-errors to the default flags
Some error reports are misleading on some recent versions of gcc because
it goes on to build for a very long time after it meets an error. Not
only this makes it hard to scroll back to the beginning of the error,
but it also hides the cause of the error when it's prominently printed
in a "#error" statement. This typically happens when building with QUIC
and without OPENSSL where there can be 4 pages of unknown types and such
errors after the "Must define USE_OPENSSL" suggestion.

The flag -Wfatal-errors serves exactly this purpose, to stop after the
first error, and it's supported on all the compilers we support, so let's
enable this now.
2022-05-11 11:32:41 +02:00
Amaury Denoyelle
0cc02a345b REORG: quic: use a dedicated module for qc_stream_desc
Regroup all type definitions and functions related to qc_stream_desc in
the source file src/quic_stream.c.

qc_stream_desc complexity will be increased with the development of Tx
multi-buffers. Having a dedicated module is useful to mix it with
pure transport/quic-conn code.
2022-04-21 11:05:27 +02:00
Christopher Faulet
6b0a0fb2f9 CLEANUP: tree-wide: Remove any ref to stream-interfaces
Stream-interfaces are gone. Corresponding files can be safely be removed. In
addition, comments are updated accordingly.
2022-04-13 15:10:16 +02:00
Willy Tarreau
6ea6ed7418 BUILD: makefile: silence unbearable OpenSSL deprecation warnings
OpenSSL 3.0 emits tons of deprecation warnings for the engine API, and
it becomes a real problem because these hide other real warnings and
will prevent distros from building with -Werror. Fortunately there's a
macro to shut this one, OPENSSL_SUPPRESS_DEPRECATED, that is sufficient
to get things back to normal, so let's define it when USE_ENGINE is set.
This way we still get a chance to see other deprecation warnings when
engines are not used.
2022-04-11 19:31:47 +02:00
Willy Tarreau
a88f3c24d0 BUILD: makefile: pass USE_ENGINE to cflags
Previous patch forgot to add USE_ENGINE to the list of options to be
transferred to CFLAGS, so USE_ENGINE had no effect and engines would
remain disabled.
2022-04-11 18:54:09 +02:00
William Lallemand
d7bfbe2333 BUILD: ssl: add USE_ENGINE and disable the openssl engine by default
The OpenSSL engine API is deprecated starting with OpenSSL 3.0.

In order to have a clean build this feature is now disabled by default.
It can be reactivated with USE_ENGINE=1 on the build line.
2022-04-11 18:41:24 +02:00
Amaury Denoyelle
d96361b270 CLEANUP: qpack: suppress by default stdout traces
Remove the definition of DEBUG_HPACK on qpack-dec.c which forces the
QPACK decoding traces on stderr. Also change the name to use a dedicated
one for QPACK decoding as DEBUG_QPACK.
2022-03-25 15:22:40 +01:00
Amaury Denoyelle
0c2d964280 REORG: quic: use a dedicated quic_loss.c
Move all inline functions with trace from quic_loss.h to a dedicated
object file. This let to remove the TRACE_SOURCE macro definition
outside of the include file.

This change is required to be able to define another TRACE_SOUCE inside
the mux_quic.c for a dedicated trace module.
2022-03-25 14:45:45 +01:00
David Carlier
43a568575f BUILD: fix kFreeBSD build.
kFreeBSD needs to be treated as a distinct target from FreeBSD
since the underlying system libc is the GNU one. Thus, relying
only on __GLIBC__ no longer suffice.

- freebsd-glibc new target, key difference is including crypt.h
  and linking to libdl like linux.
- cpu affinity available but the api is still the FreeBSD's.
- enabling auxiliary data access only for Linux.

Patch based on preliminary work done by @bigon.

closes #1555
2022-03-04 17:19:12 +01:00
Willy Tarreau
86bcc53084 DEBUG: implement 4 levels of choices between warn and crash.
We used to have DEBUG_STRICT_NOCRASH to disable crashes on BUG_ON().
Now we have other levels (WARN_ON(), CHECK_IF()) so we need something
finer-grained.

This patch introduces DEBUG_STRICT_ACTION which takes an integer value.
0 disables crashes and is the equivalent of DEBUG_STRICT_NOCRASH. 1 is
the default and only enables crashes on BUG_ON(). 2 also enables crashes
on WARN_ON(), and 3 also enables warnings on CHECK_IF(), and is suited
to developers and CI.
2022-02-28 15:00:55 +01:00
Christopher Faulet
1329f2a12a REORG: conn_stream: move conn-stream stuff in dedicated files
Move code dealing with the conn-streams in dedicated files.
2022-02-24 11:00:02 +01:00
Willy Tarreau
8de7f2822b BUILD: makefile: enable both DEBUG_STRICT and DEBUG_MEMORY_POOLS by default
The first one will enable all currently deployed BUG_ON() checks. These
ones are safe from a performance perspective and from a reliability
perspective. New ones may be added later with different categories
(hot path, detection of uncertain events, etc).

DEBUG_MEMORY_POOLS enables the "tag" pool debugging option by default,
so that pools may be better traced in dumps. This one alone results in
almost imperceptible performance difference, and 8 extra bytes per
allocated object.

Both options are safe for production use (they're among those enabled
all the time on haproxy.org) and allow to produce much more trustable
bug reports which should save a few round trips with the reporters.
2022-02-23 18:14:49 +01:00
Willy Tarreau
5ea9fa62e2 BUILD: makefile: only compute alternative options if required
Currently, the way the "cc-opt-alt" macro works consists in always
pre-calculating the alternative value for the case the main one would
not work, and pass both to an "if" clause in shell. Most of the time
we evaluate the second one for no reason.

Let's change this to use an internal "if" function instead, and directly
pass both option names to cc-opt-alt instead of passing a pre-calculated
expression. This saves one fork/exec per option and makes the option
easier to use.
2022-01-31 21:00:35 +01:00
Willy Tarreau
d521b8b61f BUILD: makefile: validate support for extra warnings by batches
The makefile takes quite some time to check supported warning options
and that's getting quite annoying. Most of the time all the tested ones
are quite legacy and well supported, so let's first try to validate
them all at once, and only if they fail, test them individually.

Doing so reduces the number of calls to the compiler to ~4 during the
startup, which is much better.
2022-01-31 21:00:35 +01:00
Willy Tarreau
7a1e6fe128 BUILD: makefile: avoid testing all -Wno-* options when not needed
We already have 9 different warning shutup options and this list grows
with each new version. Testing for their support takes some time at the
makefile's initialisation which is visible on all options (make clean
etc). Some compilers like clang are extremely slow to validate them all
and spend roughly half a second on modern machines to validate all
options. And some compilers are happier than others when passed a -Wno-*
option they do not know:
  - gcc < 4 complains loudly
  - gcc 4 and above do not say anything, unless there is already another
    warning, in which case they will report about the unknown option as
    well, but without affecting the return code
  - clang by default rejects unknown options but supports a special option
    -Wno-unknown-warning-option to silently ignore them

This patch improves the situation a bit by detecting if the compiler
already supports random options, only supports them when called with
-Wno-unknown-warning-option, or not at all. Based on this, a variable
is set to indicate if we can avoid testing for all unknown options and
assume they are supported, and another one is set to hold the optionally
required option to shut the warning. This results in almost halving the
makefile's startup time, which is particularly appreciable with latest
compilers which become really fat (the other half is caused by the same
tests on various cc-opt).
2022-01-31 21:00:35 +01:00
David Carlier
b81483cf2d MEDIUM: da: update doc and build for new scheduler mode service.
Mentions of the new database update runtime mode and update of
the legit module and the dummy part too.
Note the DeviceAtlas C API version 2.4.0 minimum required
alongside with libCURL, libzip and libgz.
2022-01-28 07:28:53 +01:00
Willy Tarreau
add43fa43e DEBUG: pools: add new build option DEBUG_POOL_TRACING
This new option, when set, will cause the callers of pool_alloc() and
pool_free() to be recorded into an extra area in the pool that is expected
to be helpful for later inspection (e.g. in core dumps). For example it
may help figure that an object was released to a pool with some sub-fields
not yet released or that a use-after-free happened after releasing it,
with an immediate indication about the exact line of code that released
it (possibly an error path).

This only works with the per-thread cache, and even objects refilled from
the shared pool directly into the thread-local cache will have a NULL
there. That's not an issue since these objects have not yet been freed.
It's worth noting that pool_alloc_nocache() continues not to set any
caller pointer (e.g. when the cache is empty) because that would require
a possibly undesirable API change.

The extra cost is minimal (one pointer per object) and this completes
well with DEBUG_POOL_INTEGRITY.
2022-01-24 16:40:48 +01:00
Willy Tarreau
0575d8fd76 DEBUG: pools: add new build option DEBUG_POOL_INTEGRITY
When enabled, objects picked from the cache are checked for corruption
by comparing their contents against a pattern that was placed when they
were inserted into the cache. Objects are also allocated in the reverse
order, from the oldest one to the most recent, so as to maximize the
ability to detect such a corruption. The goal is to detect writes after
free (or possibly hardware memory corruptions). Contrary to DEBUG_UAF
this cannot detect reads after free, but may possibly detect later
corruptions and will not consume extra memory. The CPU usage will
increase a bit due to the cost of filling/checking the area and for the
preference for cold cache instead of hot cache, though not as much as
with DEBUG_UAF. This option is meant to be usable in production.
2022-01-21 19:07:48 +01:00
David Carlier
4831db2f32 BUILD/MINOR: fix solaris build with clang.
clang 9 sets the level to POSIX 2004.
2022-01-17 17:41:08 +01:00
Amaury Denoyelle
c8b4ce4a47 MINOR: quic: add config parse source file
Create a new dedicated source file for QUIC related options parsing on
the bind line.
2022-01-12 11:08:48 +01:00
Willy Tarreau
790169fe69 BUILD: makefile: add -Wno-atomic-alignment to work around clang abusive warning
As reported in github issue #1502, clang, when building for i386, will
try to use CMPXCHG8B-based loops for 64-bit atomic operations, and emits
warnings for all 64-bit operands that are not 64-bit aligned, an alignment
that is *not* required by the ABI, that the compiler itself does not
enforce, and that the intel SDM clearly says is not required on this
32-bit platform for this operation. But this is likely an excessive
outcome of the same code being used in 64-bit for CMPXCHG16B which does
require proper alignment. Firefox already gave up on this one 3 years
ago, let's not waste our time arguing and just shut up the warning
instead. It might hide some real bugs in the future but till now
experience showed that overall it's unlikely.

This should be backported to all maintained branches that use 64-bit
atomic ops (e.g. for counters).

Thanks to Brad Smith for reporting it and confirming that shutting the
warning addresses it.
2022-01-07 14:58:48 +01:00
Willy Tarreau
e5f5948858 BUILD: makefile: reorder objects by build time
This is the usual pre-release reordering. It saves roughly one
second of build time at -O2 on my machine, which is always nice to
have.
2021-11-19 11:24:33 +01:00
Willy Tarreau
9e3093c48a BUILD: makefile: stop opening sub-shells for each and every command
We're spending ~8% of the total build time calling a shell to display
"CC" using the "echo" command! We don't really need this, as make also
knows a "$(info ...)" command to print a message. However there's a catch,
this command trims leading spaces, so we need to use an invisible space
using "$ ". Furthermore, in GNU make 3.80 and older, $(info) doesn't show
anything, so we only do that for 3.81 and above, older versions continue
to use echo.

This measurably speeds up build time especially at -O0 that developers
use most of the time for quick checks.
2021-11-19 11:24:19 +01:00
Amaury Denoyelle
154bc7f864 MINOR: quic: support hq-interop
Implement a new app_ops layer for quic interop. This layer uses HTTP/0.9
on top of QUIC. Implementation is minimal, with the intent to be able to
pass interoperability test suite from
https://github.com/marten-seemann/quic-interop-runner.

It is instantiated if the negotiated ALPN is "hq-interop".
2021-11-18 10:50:58 +01:00
Willy Tarreau
70c268b4da BUILD: makefile: simplify detection of libatomic
We've had libatomic enabled on arm and aarch64 for some Raspberry PI
while usually it's not needed, but it was a bit arbitrary and in
issue #1455 it was reported that RISCV requires it for single-byte
atomics.

This changes the approach to detect the explicit requirement of
external functions for the builtins, as reported with *_LOCK_FREE=1.
If any of the atomics requires libatomic, it will be used. Older
compilers do not report any such atomic as they use sync_* instead
and will not match it nor include libatomic (which usually is not
present there).

On x86, the rules depend on -march. i386 uses LOCK_FREE=1 for all of
them. i486 uses it only for the 8-byte CAS and i586 doesn't require
it at all. For this reason, the build flags are used during the test.

This was tested with armv7, aarch64, mips, riscv, i
2021-11-14 15:37:29 +01:00
Remi Tricot-Le Breton
7feb361776 MINOR: jwt: Parse JWT alg field
The full list of possible algorithms used to create a JWS signature is
defined in section 3.1 of RFC7518. This patch adds a helper function
that converts the "alg" strings into an enum member.
2021-10-14 16:38:08 +02:00
Willy Tarreau
5554264f31 REORG: time: move time-keeping code and variables to clock.c
There is currently a problem related to time keeping. We're mixing
the functions to perform calculations with the os-dependent code
needed to retrieve and adjust the local time.

This patch extracts from time.{c,h} the parts that are solely dedicated
to time keeping. These are the "now" or "before_poll" variables for
example, as well as the various now_*() functions that make use of
gettimeofday() and clock_gettime() to retrieve the current time.

The "tv_*" functions moved there were also more appropriately renamed
to "clock_*".

Other parts used to compute stolen time are in other files, they will
have to be picked next.
2021-10-08 17:22:26 +02:00
William Lallemand
dbee2e57fd MINOR: Makefile: add MEMORY_POOLS to the list of DEBUG_xxx options
Add the DEBUG_MEMORY_POOLS option to the list of DEBUG_xxx options in
the Makefile.
2021-09-29 16:55:39 +02:00
Amaury Denoyelle
4652a59255 MINOR: qpack: create qpack-enc module 2021-09-23 15:27:25 +02:00
Frédéric Lécaille
c6bc185c18 MINOR: quic: Add a ring buffer implementation for QUIC
This implementation is inspired from Linux kernel circular buffer implementation
(see include/linux/circ-buf.h). Such buffers may be used at the same time both
by writer and reader (lock-free).
2021-09-23 15:27:25 +02:00
Frédéric Lécaille
11c304da0a MINOR: quic: Enable some quic, h3 and qpack modules compilation.
Enable src/h3.c compilation for h3, src/qpack-dec.c and sr/qpack-tbl.c
compilation for qpack, and src/mux_quic.c for the QUIC mux.
2021-09-23 15:27:25 +02:00
Miroslav Zagorac
4cb2c83f46 BUILD: opentracing: exclude the use of haproxy variables for the OpenTracing context
Due to a recent change in the handling of haproxy variables, their use for
OpenTracing context transfer has been excluded from the compilation process.

The use of variables can be re-enabled if the newly defined variable
OT_USE_VARS is set to 1 when calling the 'make' utility.  However,
this should not be used for now as the compilation will end in error.

This change prevents the use of haproxy variables to convey the OpenTracing
context.  This means that the 'use-vars' parameter cannot be used in the
OpenTracing filter configuration for 'inject' and 'extract' operations.

An example configuration that uses this feature is in the test/ctx
directory, while the script to run that test is test/run-ctx.sh.

Then, the 'sess.ot.uuid' variable is no longer set when initializing the
OpenTracing session.  This means that this variable can still be used in
the OpenTracing configuration, but its contents will be empty.
2021-09-12 07:07:13 +02:00
Willy Tarreau
28e295d92c BUILD: globally enable -Wundef
As seen in issue #1369, supporting #if with unknown macros can silently
hide typos that may result in suboptimal code paths to be used, or even
possibly bugs. It looks like our code base does not rely that much on
this, so it's worth enabling -Wundef to catch future ones and have them
turned to more explicit "#if defined()" or #ifdef.
2021-08-30 10:16:30 +02:00
devnexen@gmail.com
21185970c1 MINOR: proc: setting the process to produce a core dump on FreeBSD.
using the procctl api to set the current process as traceable, thus being able to produce a core dump as well.

making it as compile option if not wished or using freebsd prior to 11.x (last no EOL release).
2021-08-25 05:14:27 +02:00
William Lallemand
83614a9fbe MINOR: httpclient: initialize the proxy
Initialize a proxy which contain a server for the raw HTTP, and another
one for the HTTPS. This proxy will use the global server log definition
and the 'option httplog' directive.

This proxy is internal and will only be used for the HTTP Client API.
2021-08-18 17:35:48 +02:00
Willy Tarreau
6807c7f6e1 ADMIN: dyncookie: implement a simple dynamic cookie calculator
This utility can be useful to figure what cookie value a server will
have based on the secret, its IP and its port.
2021-08-11 14:07:45 +02:00
Willy Tarreau
66243b4273 REORG: config: move the condition preprocessing code to its own file
The .if/.else/.endif and condition evaluation code is quite dirty and
was dumped into cfgparse.c because it was easy. But it should be tidied
quite a bit as it will need to evolve.

Let's move all that to cfgcond.{c,h}.
2021-07-16 19:18:41 +02:00
David Carlier
953771b0e3 BUILD: Makefile: fix linkage for Haiku.
At runtime, the haiku's loader displays `could not resolve symbol: __start_i_STG_ALLOC`
thus using linker setting fallback.
2021-06-25 08:57:42 +02:00
Willy Tarreau
6fd0450b47 CLEANUP: shctx: remove the different inter-process locking techniques
With a single process, we don't need to USE_PRIVATE_CACHE, USE_FUTEX
nor USE_PTHREAD_PSHARED anymore. Let's only keep the basic spinlock
to lock between threads.
2021-06-15 16:52:42 +02:00
Willy Tarreau
8715dec6f9 MEDIUM: pools: remove the locked pools implementation
Now that the modified lockless variant does not need a DWCAS anymore,
there's no reason to keep the much slower locked version, so let's
just get rid of it.
2021-06-10 17:46:50 +02:00
Amaury Denoyelle
ce986e1ce8 REORG: errors: split errors reporting function from log.c
Move functions related to errors output on stderr from log.c to a newly
created errors.c file. It targets print_message and
ha_alert/warning/notice/diag functions and related startup_logs feature.
2021-06-07 16:58:15 +02:00
Willy Tarreau
40a871f09d BUILD: makefile: add a few popular ARMv8 CPU targets
This adds the following CPUs to the makefile:
  - armv81    : modern ARM cores (Cortex A55/A75/A76/A78/X1, Neoverse, Graviton2)
  - a72       : ARM Cortex-A72 or A73 (e.g. RPi4, Odroid N2, VIM3, AWS Graviton)
  - a53       : ARM Cortex-A53 or any of its successors in 64-bit mode (e.g. RPi3)
  - armv8-auto: both older and newer ARMv8 cores, with a minor runtime penalty

The reasons for these ones are:
  - a53 is the common denominator of all of its successors, and does
    support CRC32 which is used by the gzip compression, that the generic
    armv8-a does not ;

  - a72 supports the same features but is an out-of-order one that deserves
    better optimizations; it's found in a number of high-performance
    multi-core CPUs mainly oriented towards I/O and network processing
    (Armada 8040, NXP LX2160A, AWS Graviton), and more recently the
    Raspberry Pi 4. The A73 found in VIM3 and Odroid-N2 can use the same
    optimizations ;

  - armv81 is for generic ARMv8.1-A and above, automatically enables LSE
    atomics which are way more scalable, and CRC32. This one covers modern
    ARMv8 cores such as Cortex A55/A75/A76/A77/A78/X1 and the Neoverse
    family such as found in AWS's Graviton2. The LSE instructions are
    essential for large numbers of cores (8 and above).

  - armv8-auto dynamically enables support for LSE extensions when
    detected while still being compatible with older cores. There is a
    small performance penalty in doing this (~3%) but a same executable
    will perform optimally on a wider range of hardware. This should be
    the best option for distros. It requires gcc-10 or gcc-9.4 and above.

When no CPU is specified, GCC version 10.2 and above will automatically
implement the wrapper used to detect the LSE extensions.
2021-05-12 14:23:22 +02:00
Willy Tarreau
1d14262a69 BUILD: makefile: reorder object files by build time
This is the per-release reordering to improve build parallelism.
It didn't change much, mostly dns+resolvers inflated this time.
Nowadays build times are mostly dominated by the long dependencies
of include files, no less than 170MB of preprocessed code has to be
built, and half of this is SSL support is disabled. Includes should
likely be reworked to be smaller with less dependencies each,
possibly splitting what's the core of each of them and what is used
to interface with other ones. Each split of a .C file in two adds
0.3s of build time just because of this.
2021-05-08 23:40:47 +02:00
Willy Tarreau
f6818d637a BUILD: makefile: add new option USE_MEMORY_PROFILING
It is not enabled by default, and may only work on linux-glibc for now,
though maybe other platforms could adopt it, possibly with certain
restrictions.
2021-05-05 19:09:19 +02:00
Amaury Denoyelle
f75c640f7b MINOR: cpuset: define a platform-independent cpuset type
This module can be used to manipulate a cpu sets in a platform agnostic
way. Use the type cpu_set_t/cpuset_t if available on the platform, or
fallback to unsigned long, which limits de facto the maximum cpu index
to LONGBITS.
2021-04-23 16:06:49 +02:00
Willy Tarreau
12840be005 BUILD: compression: switch SLZ from out-of-tree to in-tree
Now that SLZ is merged, let's update the makefile and compression
files to use it. As a result, SLZ_INC and SLZ_LIB are neither defined
nor used anymore.

USE_SLZ is enabled by default ("USE_SLZ=default") and can be disabled
by passing "USE_SLZ=" or by enabling USE_ZLIB=1.

The doc was updated to reflect the changes.
2021-04-22 16:08:25 +02:00
Willy Tarreau
af6ae6395f BUILD: makefile: fix the "make clean" target on strict bourne shells
As reported by @axinojolais in issue #1217, some older bourne shells do
not expand on braces so some files were not cleaned since the recent
splitting of the contrib/ subdir. Let's fix that by explicitly listing
the patterns to be cleared (which are in much smaller quantity now that
contrib was removed), and for grouping them with their respective dirs.

At some point, some recursive makefiles would probably help there.
2021-04-21 17:22:33 +02:00
Willy Tarreau
580727f3af CLEANUP: contrib: remove the last references to the now dead contrib/ directory
Now with the last SPOA modules gone, contrib/ doesn't exist anymore
and does not need to be referenced in the Makefile nor .gitignore.
2021-04-21 15:13:58 +02:00
Willy Tarreau
2d6f628d34 MINOR: pools: rename CONFIG_HAP_LOCAL_POOLS to CONFIG_HAP_POOLS
We're going to make the local pool always present unless pools are
completely disabled. This means that pools are always enabled by
default, regardless of the use of threads. Let's drop this notion
of "local" pools and make it just "pool". The equivalent debug
option becomes DEBUG_NO_POOLS instead of DEBUG_NO_LOCAL_POOLS.

For now this changes nothing except the option and dropping the
dependency on USE_THREAD.
2021-04-19 15:24:33 +02:00
Tim Duesterhus
dbd25c34de MINOR: uri_normalizer: Add uri_normalizer module
This is in preparation for future patches.
2021-04-19 09:05:57 +02:00
Christopher Faulet
1615064ea0 BUILD: makefile: Redirect stderr to /dev/null when probing options
It is a workaround to avoid a clang 11 bug that exits with SIGABRT when
stderr is redirected to stdin. This bug was already reported few weeks ago:

  https://bugs.llvm.org/show_bug.cgi?id=49463

But because it is pretty annoying, the standard error is now redirected to
/dev/null.
2021-04-19 08:52:17 +02:00
Alex
41007a6835 MINOR: sample: converter: Add mjson library.
This library is required for the subsequent patch which adds
the JSON query possibility.

It is necessary to change the include statement in "src/mjson.c"
because the imported includes in haproxy are in "include/import"

orig: #include "mjson.h"
new:  #include <import/mjson.h>
2021-04-15 17:05:38 +02:00
Willy Tarreau
a63d1a0863 CONTRIB: move contrib/opentracing to addons/ot
This one is the last optional module to build with haproxy, so let's move
it to addons/. It was renamed to "ot" as it was the only one whose USE_*
option did not match the directory name, now this is consistent.

Few changes were required, only the Makefile, and doc were adjusted, as
the directory was already self-contained and relocatable.
2021-04-02 17:48:42 +02:00
Willy Tarreau
57610c694e CONTRIB: move src/wurfl.c and contrib/wurfl to addons/wurfl
Both the source file and the dummy library are now at the same place.
Maybe the build howto could be moved there as well to make things even
cleaner.

The Makefile, MAINTAINERS, doc, and vtest matrix were updated.
2021-04-02 17:48:42 +02:00
Willy Tarreau
f8d9ec57f0 CONTRIB: move src/da.c and contrib/deviceatlas to addons/deviceatlas
Both the source file and the dummy library are now at the same place.
Maybe the build howto could be moved there as well to make things even
cleaner.

The Makefile, MAINTAINERS, doc, github build matrix, coverity checks
and travis CI's build were updated.
2021-04-02 17:48:42 +02:00
Willy Tarreau
977209d1d8 CONTRIB: move 51Degrees to addons/51degrees
Now it's much cleaner, both 51d.c and the dummy library live together and
are easier to spot and maintain. The build howto probably ought to be moved
there as well. Makefile, docs and MAINTAINERS were updated, as well as
the github CI's build matrix, travis CI's, and coverity checks.
2021-04-02 17:48:42 +02:00
Willy Tarreau
3dfadc7ae7 CONTRIB: move prometheus-exporter to addons/promex
Let's start to better organize the addons by moving promex there (and
with an easier directory name). The makefile and maintainers files were
updated, as well as the CI's build matrix.
2021-04-02 17:48:42 +02:00
Willy Tarreau
92dc78605a BUILD: makefile: add a "USE_PROMEX" variable to ease building prometheus-exporter
The Prometheus exporter has gained in popularity and deserves to be easier
to build. Let's add a standard "USE_PROMEX" variable to enable it without
having to modify EXTRA_OBJS nor fiddling with the build path. The readme
was updated to reflect this.
2021-04-02 17:48:42 +02:00
Willy Tarreau
a9f16c6495 BUILD: makefile: build halog with the correct flags
halog currently emits lots of warnings because it does not benefit from
the default flags. Let's update the main makefile to build it by itself
and remove the other one. The sub-project's makefile was replaced with
A readme indicating how to build it.
2021-04-02 17:48:42 +02:00
Willy Tarreau
d798889a5d CONTRIB: move halog to admin/
halog is an admin tool, so let's move it to admin/ as well. The makefile
was updated to build from the new directory.
2021-04-02 17:48:42 +02:00
Willy Tarreau
233e868895 CONTRIB: move some admin-related sub-projects to admin/
The following components were moved to admin/ because they're generally
used in field by admins:

  iprange/  netsnmp-perl/  selinux/  systemd/  wireshark-dissectors/
  syntax-highlight/ release-estimator/
2021-04-02 17:48:42 +02:00
Willy Tarreau
21ef8b9064 BUILD: makefile: integrate the hpack tools
The few hpack development tools are now integrated into the main
makefile, which allows to remove the original one which was causing
lots of build warnings. A README was added to explain how to build
instead.
2021-04-02 17:48:42 +02:00
Willy Tarreau
888bda0ace BUILD: makefile: always build the flags utility
This utility is absolutely required for developers and not having it
built by default is a real pain that tends to encourage keeping an
outdated copy somewhere else. Let's have it built by default then,
since it has no dependency and is ultra-small.
2021-04-02 17:48:42 +02:00
Willy Tarreau
074ebcde29 CONTRIB: move some dev-specific tools to dev/
The following directories were moved from contrib/ to dev/ to make their
use case a bit clearer. In short, only developers are expected to ever
go there. The makefile was updated to build and clean from these ones.

base64/  flags/  hpack/  plug_qdisc/  poll/  tcploop/  trace/
2021-04-02 17:48:42 +02:00
Willy Tarreau
34b4369709 CONTRIB: debug: split poll from flags
Now poll is its own project and doesn't share the "flags" Makefile
any more. One of the issues was that it was making references to the
haproxy include path which is not needed here.
2021-04-02 17:48:42 +02:00
Amaury Denoyelle
5a6926dcf0 MINOR: diag: create cfgdiag module
This module is intended to serve as a placeholder for various
diagnostics executed after the configuration file has been fully loaded.
2021-04-01 18:03:37 +02:00
Amaury Denoyelle
ce44482fe5 REORG: global: move initcall register code in a dedicated file
Create a new module init which contains code related to REGISTER_*
macros for initcalls. init.h is included in api.h to make init code
available to all modules.

It's a step to clean up a bit haproxy.c/global.h.
2021-03-26 15:28:33 +01:00
Emeric Brun
8af3bb0abf BUG/MINOR: protocol: add missing support of dgram unix socket.
The proto "uxdg" (UNIX DGRAM) was not declared, causing an error trying
to put a socket unix on "dgram-bind" into a log-forward section.

This patch introduces the missing "uxdg" protocol by adding proto_uxdg.c
which was fully created based on the code available for the other
protocols.

This patch should be backported to version 2.3 and above.
2021-03-18 18:30:29 +01:00
Bertrand Jacquin
57647cac62 MINOR: build: force CC to set a return code when probing options
gcc returns non zero code if an option is not supported (tested
from 6.5 to 10.2).

  $ gcc -Wfoobar -E -xc - -o /dev/null < /dev/null > /dev/null 2>&1 ; echo $?
  1

clang always return 0 if an option in not recognized unless
-Werror is also passed, preventing a correct probing of options
supported by the compiler (tested with clang 6.0.1 to 11.1.0).

  $ clang -Wfoobar -E -xc - -o /dev/null < /dev/null > /dev/null 2>&1 ; echo $?
  0
  $ clang -Werror -Wfoobar -E -xc - -o /dev/null < /dev/null > /dev/null 2>&1 ; echo $?
  1

Please note today this is not visible since clang 11 exit with SIGABRT
or with return code 1 on older version due to bad file descriptor from
file descriptor handling

  $ clang -Wfoobar -E -xc - -o /dev/null < /dev/null 2>&0 ; echo $?
  Aborted (core dumped)
  134
  $ clang -Wfoobar -E -xc - -o /dev/null < /dev/null ; echo $?
  warning: unknown warning option '-Wfoobar'; did you mean '-Wformat'? [-Wunknown-warning-option]
  1 warning generated.
  0
  $ clang-11 -Werror -Wfoobar -E -xc - -o /dev/null < /dev/null ; echo $?
  error: unknown warning option '-Wfoobar'; did you mean '-Wformat'? [-Werror,-Wunknown-warning-option]
  1

This specific issue is being tracked with clang upstream in https://bugs.llvm.org/show_bug.cgi?id=49463
2021-03-09 15:24:47 +01:00
Christopher Faulet
3e3d3be708 REORG: server-state: Move functions to deal with server-state in its own file
All functions dealing with the server-state files are moved to
server_state.c.

srv_update_state() function was renammed to srv_state_srv_update().
2021-02-25 10:02:39 +01:00
David Carlier
019dbd7884 BUILD/MEDIUM: da Adding pcre2 support.
The DeviceAtlas Detection API now supports also the pcre2 library,
 and some users wish to have exclusively this version in their
environment.
Also, there is no longer new development happening in the legacy
 pcre(1) counterpart.
Simple check in the build process as the mutual exclusivity check between the
 two are already taking care of early on. Moving the check to the part
only when we build haproxy + the API from source as the other case the API is
 already built with the chosen regex library separately.
2021-02-18 14:58:43 +01:00
Willy Tarreau
b23f04260b MINOR: tasks: add DEBUG_TASK to report caller info in a task
The idea is to know who woke a task up, by recording the last two
callers in a rotating mode. For now it's trivial with task_wakeup()
but tasklet_wakeup_on() will require quite some more changes.

This typically gives this from the debugger:

  (gdb) p t->debug
  $2 = {
    caller_file = {0x0, 0x8c0d80 "src/task.c"},
    caller_line = {0, 260},
    caller_idx = 1
  }

or this:

  (gdb) p t->debug
  $6 = {
    caller_file = {0x7fffe40329e0 "", 0x885feb "src/stream.c"},
    caller_line = {284, 284},
    caller_idx = 1
  }

But it also provides a trivial macro allowing to simply place a call in
a task/tasklet handler that needs to be observed:

   DEBUG_TASK_PRINT_CALLER(t);

Then starting haproxy this way would trivially yield such info:

  $ ./haproxy -db -f test.cfg | sort | uniq -c | sort -nr
   199992 h1_io_cb woken up from src/sock.c:797
    51764 h1_io_cb woken up from src/mux_h1.c:3634
       65 h1_io_cb woken up from src/connection.c:169
       45 h1_io_cb woken up from src/sock.c:777
2021-02-18 10:42:07 +01:00
Emeric Brun
c943799c86 MEDIUM: resolvers/dns: split dns.c into dns.c and resolvers.c
This patch splits current dns.c into two files:

The first dns.c contains code related to DNS message exchange over UDP
and in future other TCP. We try to remove depencies to resolving
to make it usable by other stuff as DNS load balancing.

The new resolvers.c inherit of the code specific to the actual
resolvers.

Note:
It was really difficult to obtain a clean diff dur to the amount
of moved code.

Note2:
Counters and stuff related to stats is not cleany separated because
currently counters for both layers are merged and hard to separate
for now.
2021-02-13 10:03:46 +01:00
William Lallemand
c1ddcafdf9 BUILD: Makefile: move REGTESTST_TYPE default setting
In patch 3bad3d5 ("BUILD: Makefile: exclude broken tests by default"),
the default setting of the REGTESTST_TYPE variable was set in the
Makefile instead of the run-regtests.sh script.

Doing it in the Makefile was breaking the use of this environment
varible with make ( REGTESTS_TYPES=slow,default make reg-tests )

This patch move the default setting from the Makefile to
run-regtests.sh. It also change the documentation in `make
reg-tests-help` about the default value.

This patch should be backported where 3bad3d5 is backported.
2021-02-05 11:41:16 +01:00
Bertrand Jacquin
b399a992f3 MINOR: build: discard echoing in help target
When V=1 is used in conjuction with help, the output becomes pretty
difficult to read properly.

  $ make TARGET=linux-glibc V=1 help
  ..
    DEBUG_USE_ABORT: use abort() for program termination, see include/haproxy/bug.h for details
  echo; \
     if [ -n "" ]; then \
       if [ -n "" ]; then \
          echo "Current TARGET: "; \
       else \
          echo "Current TARGET:  (custom target)"; \
       fi; \
     else \
       echo "TARGET not set, you may pass 'TARGET=xxx' to set one among :";\
       echo "  linux-glibc, linux-glibc-legacy, solaris, freebsd, dragonfly, netbsd,"; \
       echo "  osx, openbsd, aix51, aix52, aix72-gcc, cygwin, haiku, generic,"; \
       echo "  custom"; \
     fi

  TARGET not set, you may pass 'TARGET=xxx' to set one among :
    linux-glibc, linux-glibc-legacy, solaris, freebsd, dragonfly, netbsd,
    osx, openbsd, aix51, aix52, aix72-gcc, cygwin, haiku, generic,
    custom
  echo;echo "Enabled features for TARGET '' (disable with 'USE_xxx=') :"

  Enabled features for TARGET '' (disable with 'USE_xxx=') :
  set --        POLL                                  ; echo "  $*" | (fmt || cat) 2>/dev/null
    POLL
  echo;echo "Disabled features for TARGET '' (enable with 'USE_xxx=1') :"

  Disabled features for TARGET '' (enable with 'USE_xxx=1') :
  set -- EPOLL KQUEUE NETFILTER PCRE PCRE_JIT PCRE2 PCRE2_JIT  PRIVATE_CACHE THREAD PTHREAD_PSHARED BACKTRACE STATIC_PCRE STATIC_PCRE2 TPROXY LINUX_TPROXY LINUX_SPLICE LIBCRYPT CRYPT_H GETADDRINFO OPENSSL LUA FUTEX ACCEPT4 CLOSEFROM ZLIB SLZ CPU_AFFINITY TFO NS DL RT DEVICEATLAS 51DEGREES WURFL SYSTEMD OBSOLETE_LINKER PRCTL THREAD_DUMP EVPORTS OT QUIC; echo "  $*" | (fmt || cat) 2>/dev/null
    EPOLL KQUEUE NETFILTER PCRE PCRE_JIT PCRE2 PCRE2_JIT PRIVATE_CACHE

This commit ensure the help target always discard line echoing
regardless of V variable as done for reg-tests-help target.
2021-01-18 08:58:33 +01:00
Thayne McCombs
8f0cc5c4ba CLEANUP: Fix spelling errors in comments
This is from the output of codespell. It's done at once over a bunch
of files and only affects comments, so there is nothing user-visible.
No backport needed.
2021-01-08 14:56:32 +01:00
William Dauchy
3bad3d5ee6 BUILD: Makefile: exclude broken tests by default
it could be sometimes a bit confusing to have tests which are known to
be broken executed in the default `make reg-tests` command, especially
for not frequent contributors which are not necessarily aware of all our
quirks.

without this patch, this test is failing on my side:
  #    top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (2.228) exit=2
  1 tests failed, 0 tests skipped, 107 tests passed

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-01-08 14:43:27 +01:00