Define the new type ncbuf. It can be used as a buffer with
non-contiguous data and wrapping support.
To reduce as much as possible the memory footprint, size of data and
gaps are stored in the gaps themselves. This put some limitation on the
buffer usage. A reserved space is present just before the head to store
the size of the first data block. Also, add and delete operations will
be constrained to ensure minimal gap sizes are preserved.
The sizes stored in the gaps are represented by a custom type named
ncb_sz_t. This type is a typedef to easily change it : this has a
direct impact on the maximum buffer size (MAX(ncb_sz_t) - sizeof(ncb_sz_t))
and the minimal gap sizes (sizeof(ncb_sz_t) * 2)).
Currently, it is set to uint32_t.
Some error reports are misleading on some recent versions of gcc because
it goes on to build for a very long time after it meets an error. Not
only this makes it hard to scroll back to the beginning of the error,
but it also hides the cause of the error when it's prominently printed
in a "#error" statement. This typically happens when building with QUIC
and without OPENSSL where there can be 4 pages of unknown types and such
errors after the "Must define USE_OPENSSL" suggestion.
The flag -Wfatal-errors serves exactly this purpose, to stop after the
first error, and it's supported on all the compilers we support, so let's
enable this now.
Regroup all type definitions and functions related to qc_stream_desc in
the source file src/quic_stream.c.
qc_stream_desc complexity will be increased with the development of Tx
multi-buffers. Having a dedicated module is useful to mix it with
pure transport/quic-conn code.
OpenSSL 3.0 emits tons of deprecation warnings for the engine API, and
it becomes a real problem because these hide other real warnings and
will prevent distros from building with -Werror. Fortunately there's a
macro to shut this one, OPENSSL_SUPPRESS_DEPRECATED, that is sufficient
to get things back to normal, so let's define it when USE_ENGINE is set.
This way we still get a chance to see other deprecation warnings when
engines are not used.
Previous patch forgot to add USE_ENGINE to the list of options to be
transferred to CFLAGS, so USE_ENGINE had no effect and engines would
remain disabled.
The OpenSSL engine API is deprecated starting with OpenSSL 3.0.
In order to have a clean build this feature is now disabled by default.
It can be reactivated with USE_ENGINE=1 on the build line.
Remove the definition of DEBUG_HPACK on qpack-dec.c which forces the
QPACK decoding traces on stderr. Also change the name to use a dedicated
one for QPACK decoding as DEBUG_QPACK.
Move all inline functions with trace from quic_loss.h to a dedicated
object file. This let to remove the TRACE_SOURCE macro definition
outside of the include file.
This change is required to be able to define another TRACE_SOUCE inside
the mux_quic.c for a dedicated trace module.
kFreeBSD needs to be treated as a distinct target from FreeBSD
since the underlying system libc is the GNU one. Thus, relying
only on __GLIBC__ no longer suffice.
- freebsd-glibc new target, key difference is including crypt.h
and linking to libdl like linux.
- cpu affinity available but the api is still the FreeBSD's.
- enabling auxiliary data access only for Linux.
Patch based on preliminary work done by @bigon.
closes#1555
We used to have DEBUG_STRICT_NOCRASH to disable crashes on BUG_ON().
Now we have other levels (WARN_ON(), CHECK_IF()) so we need something
finer-grained.
This patch introduces DEBUG_STRICT_ACTION which takes an integer value.
0 disables crashes and is the equivalent of DEBUG_STRICT_NOCRASH. 1 is
the default and only enables crashes on BUG_ON(). 2 also enables crashes
on WARN_ON(), and 3 also enables warnings on CHECK_IF(), and is suited
to developers and CI.
The first one will enable all currently deployed BUG_ON() checks. These
ones are safe from a performance perspective and from a reliability
perspective. New ones may be added later with different categories
(hot path, detection of uncertain events, etc).
DEBUG_MEMORY_POOLS enables the "tag" pool debugging option by default,
so that pools may be better traced in dumps. This one alone results in
almost imperceptible performance difference, and 8 extra bytes per
allocated object.
Both options are safe for production use (they're among those enabled
all the time on haproxy.org) and allow to produce much more trustable
bug reports which should save a few round trips with the reporters.
Currently, the way the "cc-opt-alt" macro works consists in always
pre-calculating the alternative value for the case the main one would
not work, and pass both to an "if" clause in shell. Most of the time
we evaluate the second one for no reason.
Let's change this to use an internal "if" function instead, and directly
pass both option names to cc-opt-alt instead of passing a pre-calculated
expression. This saves one fork/exec per option and makes the option
easier to use.
The makefile takes quite some time to check supported warning options
and that's getting quite annoying. Most of the time all the tested ones
are quite legacy and well supported, so let's first try to validate
them all at once, and only if they fail, test them individually.
Doing so reduces the number of calls to the compiler to ~4 during the
startup, which is much better.
We already have 9 different warning shutup options and this list grows
with each new version. Testing for their support takes some time at the
makefile's initialisation which is visible on all options (make clean
etc). Some compilers like clang are extremely slow to validate them all
and spend roughly half a second on modern machines to validate all
options. And some compilers are happier than others when passed a -Wno-*
option they do not know:
- gcc < 4 complains loudly
- gcc 4 and above do not say anything, unless there is already another
warning, in which case they will report about the unknown option as
well, but without affecting the return code
- clang by default rejects unknown options but supports a special option
-Wno-unknown-warning-option to silently ignore them
This patch improves the situation a bit by detecting if the compiler
already supports random options, only supports them when called with
-Wno-unknown-warning-option, or not at all. Based on this, a variable
is set to indicate if we can avoid testing for all unknown options and
assume they are supported, and another one is set to hold the optionally
required option to shut the warning. This results in almost halving the
makefile's startup time, which is particularly appreciable with latest
compilers which become really fat (the other half is caused by the same
tests on various cc-opt).
Mentions of the new database update runtime mode and update of
the legit module and the dummy part too.
Note the DeviceAtlas C API version 2.4.0 minimum required
alongside with libCURL, libzip and libgz.
This new option, when set, will cause the callers of pool_alloc() and
pool_free() to be recorded into an extra area in the pool that is expected
to be helpful for later inspection (e.g. in core dumps). For example it
may help figure that an object was released to a pool with some sub-fields
not yet released or that a use-after-free happened after releasing it,
with an immediate indication about the exact line of code that released
it (possibly an error path).
This only works with the per-thread cache, and even objects refilled from
the shared pool directly into the thread-local cache will have a NULL
there. That's not an issue since these objects have not yet been freed.
It's worth noting that pool_alloc_nocache() continues not to set any
caller pointer (e.g. when the cache is empty) because that would require
a possibly undesirable API change.
The extra cost is minimal (one pointer per object) and this completes
well with DEBUG_POOL_INTEGRITY.
When enabled, objects picked from the cache are checked for corruption
by comparing their contents against a pattern that was placed when they
were inserted into the cache. Objects are also allocated in the reverse
order, from the oldest one to the most recent, so as to maximize the
ability to detect such a corruption. The goal is to detect writes after
free (or possibly hardware memory corruptions). Contrary to DEBUG_UAF
this cannot detect reads after free, but may possibly detect later
corruptions and will not consume extra memory. The CPU usage will
increase a bit due to the cost of filling/checking the area and for the
preference for cold cache instead of hot cache, though not as much as
with DEBUG_UAF. This option is meant to be usable in production.
As reported in github issue #1502, clang, when building for i386, will
try to use CMPXCHG8B-based loops for 64-bit atomic operations, and emits
warnings for all 64-bit operands that are not 64-bit aligned, an alignment
that is *not* required by the ABI, that the compiler itself does not
enforce, and that the intel SDM clearly says is not required on this
32-bit platform for this operation. But this is likely an excessive
outcome of the same code being used in 64-bit for CMPXCHG16B which does
require proper alignment. Firefox already gave up on this one 3 years
ago, let's not waste our time arguing and just shut up the warning
instead. It might hide some real bugs in the future but till now
experience showed that overall it's unlikely.
This should be backported to all maintained branches that use 64-bit
atomic ops (e.g. for counters).
Thanks to Brad Smith for reporting it and confirming that shutting the
warning addresses it.
We're spending ~8% of the total build time calling a shell to display
"CC" using the "echo" command! We don't really need this, as make also
knows a "$(info ...)" command to print a message. However there's a catch,
this command trims leading spaces, so we need to use an invisible space
using "$ ". Furthermore, in GNU make 3.80 and older, $(info) doesn't show
anything, so we only do that for 3.81 and above, older versions continue
to use echo.
This measurably speeds up build time especially at -O0 that developers
use most of the time for quick checks.
Implement a new app_ops layer for quic interop. This layer uses HTTP/0.9
on top of QUIC. Implementation is minimal, with the intent to be able to
pass interoperability test suite from
https://github.com/marten-seemann/quic-interop-runner.
It is instantiated if the negotiated ALPN is "hq-interop".
We've had libatomic enabled on arm and aarch64 for some Raspberry PI
while usually it's not needed, but it was a bit arbitrary and in
issue #1455 it was reported that RISCV requires it for single-byte
atomics.
This changes the approach to detect the explicit requirement of
external functions for the builtins, as reported with *_LOCK_FREE=1.
If any of the atomics requires libatomic, it will be used. Older
compilers do not report any such atomic as they use sync_* instead
and will not match it nor include libatomic (which usually is not
present there).
On x86, the rules depend on -march. i386 uses LOCK_FREE=1 for all of
them. i486 uses it only for the 8-byte CAS and i586 doesn't require
it at all. For this reason, the build flags are used during the test.
This was tested with armv7, aarch64, mips, riscv, i
The full list of possible algorithms used to create a JWS signature is
defined in section 3.1 of RFC7518. This patch adds a helper function
that converts the "alg" strings into an enum member.
There is currently a problem related to time keeping. We're mixing
the functions to perform calculations with the os-dependent code
needed to retrieve and adjust the local time.
This patch extracts from time.{c,h} the parts that are solely dedicated
to time keeping. These are the "now" or "before_poll" variables for
example, as well as the various now_*() functions that make use of
gettimeofday() and clock_gettime() to retrieve the current time.
The "tv_*" functions moved there were also more appropriately renamed
to "clock_*".
Other parts used to compute stolen time are in other files, they will
have to be picked next.
This implementation is inspired from Linux kernel circular buffer implementation
(see include/linux/circ-buf.h). Such buffers may be used at the same time both
by writer and reader (lock-free).
Due to a recent change in the handling of haproxy variables, their use for
OpenTracing context transfer has been excluded from the compilation process.
The use of variables can be re-enabled if the newly defined variable
OT_USE_VARS is set to 1 when calling the 'make' utility. However,
this should not be used for now as the compilation will end in error.
This change prevents the use of haproxy variables to convey the OpenTracing
context. This means that the 'use-vars' parameter cannot be used in the
OpenTracing filter configuration for 'inject' and 'extract' operations.
An example configuration that uses this feature is in the test/ctx
directory, while the script to run that test is test/run-ctx.sh.
Then, the 'sess.ot.uuid' variable is no longer set when initializing the
OpenTracing session. This means that this variable can still be used in
the OpenTracing configuration, but its contents will be empty.
As seen in issue #1369, supporting #if with unknown macros can silently
hide typos that may result in suboptimal code paths to be used, or even
possibly bugs. It looks like our code base does not rely that much on
this, so it's worth enabling -Wundef to catch future ones and have them
turned to more explicit "#if defined()" or #ifdef.
using the procctl api to set the current process as traceable, thus being able to produce a core dump as well.
making it as compile option if not wished or using freebsd prior to 11.x (last no EOL release).
Initialize a proxy which contain a server for the raw HTTP, and another
one for the HTTPS. This proxy will use the global server log definition
and the 'option httplog' directive.
This proxy is internal and will only be used for the HTTP Client API.
The .if/.else/.endif and condition evaluation code is quite dirty and
was dumped into cfgparse.c because it was easy. But it should be tidied
quite a bit as it will need to evolve.
Let's move all that to cfgcond.{c,h}.
With a single process, we don't need to USE_PRIVATE_CACHE, USE_FUTEX
nor USE_PTHREAD_PSHARED anymore. Let's only keep the basic spinlock
to lock between threads.
Now that the modified lockless variant does not need a DWCAS anymore,
there's no reason to keep the much slower locked version, so let's
just get rid of it.
Move functions related to errors output on stderr from log.c to a newly
created errors.c file. It targets print_message and
ha_alert/warning/notice/diag functions and related startup_logs feature.
This adds the following CPUs to the makefile:
- armv81 : modern ARM cores (Cortex A55/A75/A76/A78/X1, Neoverse, Graviton2)
- a72 : ARM Cortex-A72 or A73 (e.g. RPi4, Odroid N2, VIM3, AWS Graviton)
- a53 : ARM Cortex-A53 or any of its successors in 64-bit mode (e.g. RPi3)
- armv8-auto: both older and newer ARMv8 cores, with a minor runtime penalty
The reasons for these ones are:
- a53 is the common denominator of all of its successors, and does
support CRC32 which is used by the gzip compression, that the generic
armv8-a does not ;
- a72 supports the same features but is an out-of-order one that deserves
better optimizations; it's found in a number of high-performance
multi-core CPUs mainly oriented towards I/O and network processing
(Armada 8040, NXP LX2160A, AWS Graviton), and more recently the
Raspberry Pi 4. The A73 found in VIM3 and Odroid-N2 can use the same
optimizations ;
- armv81 is for generic ARMv8.1-A and above, automatically enables LSE
atomics which are way more scalable, and CRC32. This one covers modern
ARMv8 cores such as Cortex A55/A75/A76/A77/A78/X1 and the Neoverse
family such as found in AWS's Graviton2. The LSE instructions are
essential for large numbers of cores (8 and above).
- armv8-auto dynamically enables support for LSE extensions when
detected while still being compatible with older cores. There is a
small performance penalty in doing this (~3%) but a same executable
will perform optimally on a wider range of hardware. This should be
the best option for distros. It requires gcc-10 or gcc-9.4 and above.
When no CPU is specified, GCC version 10.2 and above will automatically
implement the wrapper used to detect the LSE extensions.
This is the per-release reordering to improve build parallelism.
It didn't change much, mostly dns+resolvers inflated this time.
Nowadays build times are mostly dominated by the long dependencies
of include files, no less than 170MB of preprocessed code has to be
built, and half of this is SSL support is disabled. Includes should
likely be reworked to be smaller with less dependencies each,
possibly splitting what's the core of each of them and what is used
to interface with other ones. Each split of a .C file in two adds
0.3s of build time just because of this.
It is not enabled by default, and may only work on linux-glibc for now,
though maybe other platforms could adopt it, possibly with certain
restrictions.
This module can be used to manipulate a cpu sets in a platform agnostic
way. Use the type cpu_set_t/cpuset_t if available on the platform, or
fallback to unsigned long, which limits de facto the maximum cpu index
to LONGBITS.
Now that SLZ is merged, let's update the makefile and compression
files to use it. As a result, SLZ_INC and SLZ_LIB are neither defined
nor used anymore.
USE_SLZ is enabled by default ("USE_SLZ=default") and can be disabled
by passing "USE_SLZ=" or by enabling USE_ZLIB=1.
The doc was updated to reflect the changes.
As reported by @axinojolais in issue #1217, some older bourne shells do
not expand on braces so some files were not cleaned since the recent
splitting of the contrib/ subdir. Let's fix that by explicitly listing
the patterns to be cleared (which are in much smaller quantity now that
contrib was removed), and for grouping them with their respective dirs.
At some point, some recursive makefiles would probably help there.
