Commit Graph

21673 Commits

Author SHA1 Message Date
Christopher Faulet
a4dcd3e54b MEDIUM: stats: Don't interrupt processing on partial post
We no longer test the opposite stream-connector to detect aborted partial
post. Applets must not try to access to info ouside their scope. This make
the code more sensitive to changes and it is a common source of bug.

Tests on the sedesc flags at the begining of the I/O handler should be
enough.
2024-02-07 15:04:09 +01:00
Christopher Faulet
6ac119ba2d MINOR: applet: Automatically handle applets having more data for the stream
This should simplify applets implementation. Of course an applet should
still do it by itself if conditions to set this flag differ.
2024-02-07 15:04:06 +01:00
Christopher Faulet
39b6f5b04c MEDIUM: applet: Add support for zero-copy forwarding from an applet
Thanks to this patch, it is possible to an applet to directly send data to
the opposite endpoint. To do so, it must implement <fastfwd> appctx callback
function and set SE_FL_MAY_FASTFWD flag.

Everything will be handled by appctx_fastfwd() function. The applet is only
responsible to transfer data. If it sets <to_forward> value, it is used to
limit the amount of data to forward.
2024-02-07 15:04:01 +01:00
Christopher Faulet
62a81cb6a6 MINOR: applet: Add callback function to deal with zero-copy forwarding
This patch introduces the support for the callback function responsible to
produce data via the zero-copy forwarding mechanism. There is no
implementation for now. But <to_forward> field was added in the appctx
structure to let an applet inform how much data it want to forward. It is
not mandatory but it will be used during the zero-copy forwarding
negociation.
2024-02-07 15:03:57 +01:00
Christopher Faulet
7ec544b217 MEDIUM: applet: Use appctx flags to report EOS/EOI/ERROR to SE
We have indroduced flags to deal with end of input, end of stream and errors
at the applet level. With this patch we make the link with the endpoint
descriptor.

In appctx_rcv_buf(), applet flags are converted to SE flags.
2024-02-07 15:03:54 +01:00
Christopher Faulet
cc7b141e1c MINOR: applet: Add an appctx flag to report shutdown to applets
There is no shutdown for reads and send with applets. Both are performed
when the appctx is released. So instead of 2 flags, like for
muxes/connections, only one flag is used. But the idea is the same:
acknowledge the event at the applet level.
2024-02-07 15:03:50 +01:00
Christopher Faulet
14bd091fd7 MINOR: applet: Remove appctx state field to only used the flags
The appctx state was never really used as a state. It is only used to know
when an applet should be freed on the next wakeup. This can be converted to
a flag and the state can be removed. This is what this patch does.
2024-02-07 15:03:46 +01:00
Christopher Faulet
4434b03358 MINIOR: applet: Add flags to deal with ends of input, ends of stream and errors
Dedicated appctx flags to report EOI, EOS and errors (pending or terminal) were
added with the functions to set these flags. It is pretty similar to what it
done on most of muxes.
2024-02-07 15:03:42 +01:00
Christopher Faulet
e8655546b7 MINOR: applet: Add flags on the appctx and stop abusing its state
Till now, we've extended the appctx state to add some flags. However, the
field name is misleading. So a bitfield was added to handle real flags. And
helper functions to manipulate this bitfield were added.
2024-02-07 15:03:34 +01:00
Christopher Faulet
c0527261cf MINOR: applet: Show IN/OUT buffers in trace messages when used
The function dumping applet trace messages was updated to dump info about
in/out buffers instead of channel buffers when it is relevant.
2024-02-07 15:03:30 +01:00
Christopher Faulet
4ad8192ce4 MEDIM: applet: Add the applet handler based on IN/OUT buffers
A dedicated function to run applets was introduced, in addition to the old
one, to deal with applets that use their own buffers. The main differnce
here is that this handler does not use channels at all. It performs a
synchronous send before calling the applet and performs a synchronous
receive just after.

No applets are plugged on this handler for now.
2024-02-07 15:03:26 +01:00
Christopher Faulet
f81b704d01 MEDIUM: stconn: Add functions to handle applets I/O from the SC layer
There is no tasklet to handle I/O subscriptions for applets, but functions
to deal with receives and sends from the SC layer were added. it meanse a
function to retrieve data from an applet with this synchronous version and a
function to push data to an applet wit this synchronous version.

It is pretty similar to the functions used for muxes but there are some
differences. So for now, we keep them separated.

Zero-copy forwarding is not supported for now. In addition, there is no
subscription mechanism.
2024-02-07 15:03:23 +01:00
Christopher Faulet
525ec12305 MINOR: applet: Implement default functions to exchange data with channels
In this patch, we add default functions to copy data from a channel to the
<inbuf> buffer of an applet (appctx_rcv_buf) and another on to copy data
from <outbuf> buffer of an applet to a channel (appctx_snd_buf).

These functions are not used for now, but they will be used by applets to
define their <rcv_buf> and <snd_buf> callback functions. Of course, it will
be possible for a specific applet to implement its own functions but these
ones should be good enough for most of applets. HTX and RAW buffers are
supported.
2024-02-07 15:03:18 +01:00
Christopher Faulet
361b81bfca MINOR: applet: Add support for callback functions to exchange data with channels
For now, it is not usable, but this patch introduce the support of callback
functions, in the applet structure, to exchange data between channels and
applets. It is pretty similar to callback functions defined by muxes.
2024-02-07 15:03:14 +01:00
Christopher Faulet
04eca50f49 MINOR: applet: Add traces to debug receive/send and block/wake events
New traces events are added to be able to debug receives and sends.
2024-02-07 15:03:09 +01:00
Christopher Faulet
ab9d2c6ca8 MINOR: applet: Add dedicated IN/OUT buffers for appctx
It is the first patch of a series aimed to align applets on connections.
Here, dedicated buffers are added for applets. For now, buffers are
initialized and helpers function to deal with allocation are added. In
addition, flags to report allocation failures or full buffers are also
introduced. <inbuf> will be used to push data to the applet from the stream
and <outbuf> will be used to push data from the applet to the stream.
2024-02-07 15:03:01 +01:00
Christopher Faulet
45ca9dadcd MINOR: stconn: Be prepared to handle error when a SC is attached to an applet
sc_attach_applet() was changed to be able to fail and callers were updated
accordingly. For now it cannot fail but if this changes, callers will be
prepared to handle errors.
2024-02-07 15:02:27 +01:00
Christopher Faulet
ad937372f3 MINOR: stconn: Explicitly use an appctx to attach a stconn on it
In sc_attach_applet, an untyped pointer (void *) was used to attach a SC on
an applet. There is no reason to not use the right type here. So now a
pointer on an appctx is explicitly used.
2024-02-07 15:02:22 +01:00
Christopher Faulet
0dd7ff0d67 MINOR: stconn: Be able to detect applets using HTX
IS_HXT_SC() macro is only usable if the stream-connector is attached to a
connection. It is a bit restrictive because this cannot work if the SC is
attached to an applet. So let's fix that be adding the support of applets
too.
2024-02-07 15:02:19 +01:00
Christopher Faulet
6734e56514 MINOR: task: Move wait_event in the task header file
wait_event structure was in connection header file because it is only used
by connections and muxes. But, this may change. For instance applets may be
good candidates to use it too. So, the structure is moved to the task header
file instead.
2024-02-07 15:02:13 +01:00
Lukas Tribus
e9bfc25581 DOC: install: clarify WolfSSL chroot requirements 2024-02-07 11:17:53 +01:00
Ilya Shipitsin
a68a2891b3 CI: ssl: add yet another OpenSSL download fallback 2024-02-07 11:05:45 +01:00
Ilya Shipitsin
7b881003b1 CI: github: abandon asan matrix.py helper
that helper was useful in days when we beleived that gcc and clang
require different flags. those days are gone.
2024-02-07 11:05:28 +01:00
Frederic Lecaille
c977b9aa15 MINOR: quic: Stop using 1024th of a second.
Use milliseconds in place of 1024th of a second.

Should be backported as far as 2.6.
2024-02-07 08:44:31 +01:00
Frederic Lecaille
19a66b290e BUG/MINOR: quic: fix possible integer wrap around in cubic window calculation
Avoid loss of precision when computing K cubic value.
Same issue when computing the congestion window value from cubic increase function
formula with possible integer varaiable wrap around.

Depends on this commit:

	MINOR: quic: Code clarifications for QUIC CUBIC (RFC 9438)

Must be backported as far as 2.6.
2024-02-07 08:44:31 +01:00
Frederic Lecaille
88d13caa38 CLEANUP: quic: Code clarifications for QUIC CUBIC (RFC 9438)
The first version of our QUIC CUBIC implementation is confusing because relying on
TCP CUBIC linux kernel implementation and with references to RFC 8312 which is
obsoleted by RFC 9438 (August 2023) after our implementation. RFC 8312 is a little
bit hard to understand. RFC 9438 arrived with much more clarifications.

So, RFC 9438 is about "CUBIC for Fast Long-Distance Networks". Our implementation
for QUIC is not very well documented. As it was difficult to reread this
code, this patch adds only some comments at complicated locations and rename
some macros, variables without logic modifications at all.

So, the aim of this patch is to add first some comments and variables/macros renaming
to avoid embedding too much code modifications in the same big patch.

Some code modifications will come to adapt this CUBIC implementation to this new
RFC 9438.

Rename some macros:
  CUBIC_BETA -> CUBIC_BETA_SCALED
  CUBIC_C    -> CUBIC_C_SCALED
  CUBIC_BETA_SCALE_SHIFT -> CUBIC_SCALE_FACTOR_SHIFT (this is the scaling factor
which is used only for CUBIC_BETA_SCALED)
  CUBIC_DIFF_TIME_LIMIT -> CUBIC_TIME_LIMIT

CUBIC_ONE_SCALED was added (scaled value of 1).

These cubic struct members were renamed:
	->tcp_wnd -> ->W_est
	->origin_point -> ->W_target
	->epoch_start -> ->t_epoch
	->remaining_tcp_inc -> remaining_W_est_inc

Local variables to quic_cubic_update() were renamed:
    t -> elapsed_time
    diff ->t
    delta -> W_cubic_t

Add a grahpic curve about the CUBIC Increase function.
Add big copied & pasted RFC 9438 extracts in relation with the 3 different increase
function regions.
Same thing for the fast convergence.
Fix a typo about the reference to QUIC RFC 9002.

Must be backported as far as 2.6 to ease any further modifications to come.
2024-02-07 08:44:31 +01:00
Willy Tarreau
7cba015c85 DEBUG: make the "debug dev {debug|warn|check}" command print a message
In order to test the new message output capability, these commands will
now explicitly mention that the bug was triggered from the CLI.
2024-02-05 17:09:00 +01:00
Willy Tarreau
25968c186a MINOR: debug: add an optional message argument to the BUG_ON() family
This commit adds support for an optional second argument to BUG_ON(),
WARN_ON(), CHECK_IF(), that can be a constant string. When such an
argument is given, it will be printed on a second line after the
existing first message that contains the condition.

This can be used to provide more human-readable explanations about
what happened, such as "too low on memory" or "memory corruption
detected" that may help a user resolve the incident by themselves.
2024-02-05 17:09:00 +01:00
Willy Tarreau
d417863828 MINOR: debug: support passing an optional message in ABORT_NOW()
The ABORT_NOW() macro is not much used since we have BUG_ON(), but
there are situations where it makes sense, typically if the program
must always die regardless od DEBUG_STRICT, or if the condition must
always be evaluated (e.g. decompress something and check it).

It's not convenient not to have any hint about what happened there. But
providing too much info also results in wiping some registers, making
the trace less exploitable, so a compromise must be found.

What this patch does is to provide the support for an optional argument
to ABORT_NOW(). When an argument is passed (a string), then a message
will be emitted with the file name, line number, the message and a
trailing LF, before the stack dump and the crash. It should be used
reasonably, for example in functions that have multiple calls that need
to be more easily distinguished.
2024-02-05 17:09:00 +01:00
Remi Tricot-Le Breton
73705ac701 BUG/MINOR: ssl: Fix error message after ssl_sock_load_ocsp call
If we were to enable 'ocsp-update' on a certificate that does not have
an OCSP URI, we would exit ssl_sock_load_ocsp with a negative error code
which would raise a misleading error message ("<cert> has an OCSP URI
and OCSP auto-update is set to 'on' ..."). This patch simply fixes the
error message but an error is still raised.

This issue was raised in GitHub #2432.
It can be backported up to branch 2.8.
2024-02-05 15:59:16 +01:00
Willy Tarreau
bc70b385fd MINOR: debug: make BUG_ON() catch build errors even without DEBUG_STRICT
As seen in previous commit 59acb27001 ("BUILD: quic: Variable name typo
inside a BUG_ON()."), it can sometimes happen that with DEBUG forced
without DEBUG_STRICT, BUG_ON() statements are ignored. Sadly, it means
that typos there are not even build-tested.

This patch makes these statements reference sizeof(cond) to make sure
the condition is parsed. This doesn't result in any code being emitted,
but makes sure the expression is correct so that an issue such as the one
above will fail to build (which was verified).

This may be backported as it can help spot failed backports as well.
2024-02-05 15:09:37 +01:00
Aurelien DARRAGON
be0165b249 BUILD: debug: remove leftover parentheses in ABORT_NOW()
Since d480b7b ("MINOR: debug: make ABORT_NOW() store the caller's line
number when using abort"), building with 'DEBUG_USE_ABORT' fails with:

  |In file included from include/haproxy/api.h:35,
  |                 from include/haproxy/activity.h:26,
  |                 from src/ev_poll.c:20:
  |include/haproxy/thread.h: In function ‘ha_set_thread’:
  |include/haproxy/bug.h:107:47: error: expected ‘;’ before ‘_with_line’
  |  107 | #define ABORT_NOW() do { DUMP_TRACE(); abort()_with_line(__LINE__); } while (0)
  |      |                                               ^~~~~~~~~~
  |include/haproxy/bug.h:129:25: note: in expansion of macro ‘ABORT_NOW’
  |  129 |                         ABORT_NOW();                                    \
  |      |                         ^~~~~~~~~
  |include/haproxy/bug.h:123:9: note: in expansion of macro ‘__BUG_ON’
  |  123 |         __BUG_ON(cond, file, line, crash, pfx, sfx)
  |      |         ^~~~~~~~
  |include/haproxy/bug.h:174:30: note: in expansion of macro ‘_BUG_ON’
  |  174 | #  define BUG_ON(cond)       _BUG_ON     (cond, __FILE__, __LINE__, 3, "FATAL: bug ",     "")
  |      |                              ^~~~~~~
  |include/haproxy/thread.h:201:17: note: in expansion of macro ‘BUG_ON’
  |  201 |                 BUG_ON(!thr->ltid_bit);
  |      |                 ^~~~~~
  |compilation terminated due to -Wfatal-errors.
  |make: *** [Makefile:1006: src/ev_poll.o] Error 1

This is because of a leftover: abort()_with_line(__LINE__);
                                    ^^
Fixing it by removing the extra parentheses after 'abort' since the
abort() call is now performed under abort_with_line() helper function.

This was raised by Ilya in GH #2440.

No backport is needed, unless the above commit gets backported.
2024-02-05 14:55:04 +01:00
Aurelien DARRAGON
92b2edb42e MINOR: stream: add "txn.redispatch" fetch
Fetch will return true if the stream underwent a redispatch according to
"option redispatch" setting upon retries.

Documentation was added, and the "%rc" logformat alternative now mentions
the new fetch to properly emulate the logformat behavior.
2024-02-05 14:54:37 +01:00
Aurelien DARRAGON
8ede0fb1d1 DOC: config: fix ordering for "txn.*" fetches
Properly re-order "txn.*" fetches by alphabetical name.
2024-02-05 14:54:37 +01:00
Aurelien DARRAGON
4455756c26 DOC: config: fix typo for '%ms' log format alternative
As found in the documentation: '%ms' can be replaced with
%[accept_date(ms),ms_utime("%3N")] alternative, however the closing
bracket was missing.
2024-02-05 14:54:37 +01:00
Frederic Lecaille
59acb27001 BUILD: quic: Variable name typo inside a BUG_ON().
This build issued was introduced by this previous commit which is a bugfix:

   BUG/MINOR: quic: Wrong ack ranges handling when reaching the limit.

A BUG_ON() referenced <fist> variable in place of <first>.

Must be backported as far as 2.6 as the previous commit.
2024-02-05 14:31:21 +01:00
Frederic Lecaille
0ce61d2f6d BUG/MINOR: quic: Wrong ack ranges handling when reaching the limit.
Acknowledgements ranges are used to build ACK frames. To avoid allocating too
much such objects, a limit was set to 32(QUIC_MAX_ACK_RANGES) by this commit:

	MINOR: quic: Do not allocate too much ack ranges

But there is an inversion when removing the oldest range from its tree.
eb64_first() must be used in place of eb64_last(). Note that this patch
only does this modification in addition to rename <last> variable to <first>.

This bug leads such a h2load command to block when a request ends up not
being acknowledged by haproxy even if correctly served:

	/opt/nghttp2/build/bin/h2load --alpn-list h3 -t 1 -c 1 -m 1 -n 100 \
		https://127.0.0.1/?s=5m

There is a remaining question to be answered. In such a case, haproxy refuses to
reopen the stream, this is a good thing but should not haproxy ackownledge the
request (because correctly parsed again).

Note that to be easily reproduced, this setting had to be applied to the client
network interface:

    tc qdisc add dev eth1 root netem delay 100ms 1s loss random

Must be backported as far as 2.6.
2024-02-05 14:26:52 +01:00
Willy Tarreau
52cc45dfa5 MINOR: acl: add extra diagnostics about suspicious string patterns
As noticed in this thread, some bogus configurations are not always easy
to spot: https://www.mail-archive.com/haproxy@formilux.org/msg44558.html
Here it was about config keywords being used in ACL patterns where strings
were expected, hence they're always valid.

Since we have the diag mode (-dD) we can perform some extra checks when
it's used, and emit them to suggest the user there might be an issue.

Here we detect a few common words (logic such as "and"/"or"/"||" etc),
C++/JS comments mistakenly used to try to isolate final args, and words
that have the exact name of a sample fetch or an ACL keyword. These checks
are only done in diag mode of course.
2024-02-03 12:08:11 +01:00
Willy Tarreau
75d64c0d4c BUG/MINOR: diag: run the final diags before quitting when using -c
Final diags were added in 2.4 by commit 5a6926dcf ("MINOR: diag: create
cfgdiag module"), but it's called too late in the startup process,
because when "-c" is passed, the call is not made, while it's its primary
use case. Let's just move the call earlier.

Note that currently the check in this function is limited to verifying
unicity of server cookies in a backend, so it can be backported as far
as 2.4, but there is little value in insisting if it doesn't backport
easily.
2024-02-03 12:08:11 +01:00
Willy Tarreau
ced4148401 BUG/MINOR: diag: always show the version before dumping a diag warning
Diag warnings were added in 2.4 by commit 7b01a8dbd ("MINOR: global:
define diagnostic mode of execution") but probably due to the split
function that checks for the mode, they did not reuse the emission of
the version string before the first warning, as was brought in 2.2 by
commit bebd21206 ("MINOR: init: report in "haproxy -c" whether there
were warnings or not"). The effet is that diag warnings are emitted
before the version string if there is no other warning nor error. Let's
just proceed like for the two other ones.

This can be backported to 2.4, though this is of very low importance.
2024-02-03 12:08:11 +01:00
Willy Tarreau
d480b7be96 MINOR: debug: make ABORT_NOW() store the caller's line number when using abort
Placing DO_NOT_FOLD() before abort() only works in -O2 but not in -Os which
continues to place only 5 calls to abort() in h3.o for call places. The
approach taken here is to replace abort() with a new function that wraps
it and stores the line number in the stack. This slightly increases the
code size (+0.1%) but when unwinding a crash, the line number remains
present now. This is a very low cost, especially if we consider that
DEBUG_USE_ABORT is almost only used by code coverage tools and occasional
debugging sessions.
2024-02-02 17:12:06 +01:00
Willy Tarreau
2bb192ba91 MINOR: debug: make sure calls to ha_crash_now() are never merged
As indicated in previous commit, we don't want calls to ha_crash_now()
to be merged, since it will make gdb return a wrong line number. This
was found to happen with gcc 4.7 and 4.8 in h3.c where 26 calls end up
as only 5 to 18 "ud2" instructions depending on optimizations. By
calling DO_NOT_FOLD() just before provoking the trap, we can reliably
avoid this folding problem. Note that this does not address the case
where abort() is used instead (DEBUG_USE_ABORT).
2024-02-02 17:12:06 +01:00
Willy Tarreau
e06e8a2390 MINOR: compiler: add a new DO_NOT_FOLD() macro to prevent code folding
Modern compilers sometimes perform function tail merging and identical
code folding, which consist in merging identical occurrences of same
code paths, generally final ones (e.g. before a return, a jump or an
unreachable statement). In the case of ABORT_NOW(), it can happen that
the compiler merges all of them into a single one in a function,
defeating the purpose of the check which initially was to figure where
the bug occurred.

Here we're creating a DO_NO_FOLD() macro which makes use of the line
number and passes it as an integer argument to an empty asm() statement.
The effect is a code position dependency which prevents the compiler
from merging the code till that point (though it may still merge the
following code). In practice it's efficient at stopping the compilers
from merging calls to ha_crash_now(), which was the initial purpose.

It may also be used to force certain optimization constructs since it
gives more control to the developer.
2024-02-02 17:12:06 +01:00
Christopher Faulet
dfb1cea693 REGTESTS: promex: Adapt script to be less verbose
First, checks on the resolver scope were added. Then, because of the recent
changes, the logs emitted by vtest are now too big and this makes the script
fails. So tests on NaN values are now performed on a smaller request. This
reduces enough the logs to pass.

In fact some checks were removed
2024-02-02 09:11:34 +01:00
Christopher Faulet
83070a08a2 MEDIUM: promex: Add support for filters on metric names
It is now possible to filter the metrics on their name, by listing
explicitly metrics to dump or on the opposite to exclude only some metrics
from the dump. To do so, a comma-separated list of metrics must be
specified. If a name is preceded by a minus (-), the metric is excluded from
the dump. If at least one metric is specified to be explicitly dumped, all
metrics are no longer dumped, but only those explicitly listed.

The list is specified via one or more "metrics" parameters in the uri
query-string. For insance:

  # Dumped all metrics, except "haproxy_server_check_status"
  /metrics?metrics=-haproxy_server_check_status

  # Only dump frontends, backends and servers status
  /metrics?metrics=haproxy_frontend_status,haproxy_backend_status,haproxy_server_status

Included and Excluded metrics can be mixed. Only the intersection will be
dumped.

This patch should fix the issue #770.
2024-02-02 09:11:34 +01:00
Christopher Faulet
1e13796792 MINOR: promex: Always pass the final name and description to promex_dmp_ts()
It is easier this way, especially for promex modules. And because name and
description are now explicitly passed to this function, there is no reason
to still pass the metric, its type is enough. The function is easier to read
this way.
2024-02-02 09:11:34 +01:00
Christopher Faulet
c4b723df99 MINOR: promex: Rename dump functions to use the right wording
In Prometheus, a time series a stream of timestamped values belonging to the
same metric and the same set of labeled dimensions. Thus the exporter dump
time-series and not metrics.

Thus, promex_dump_metric(), promex_dump_metric_header() and
promex_metric_to_str() functions were renamed to replace "metric"
2024-02-02 09:11:34 +01:00
Christopher Faulet
ca6f0ca82b MEDIUM: promex/resolvers: Dump resolvers metrics via a promex module
Just like for stick-tables, this patch adds a promex module to dump
resolvers metrics. It adds the "resolver" scope and for now, it dumps
folloowing metrics:

  * haproxy_resolver_sent
  * haproxy_resolver_send_error
  * haproxy_resolver_valid
  * haproxy_resolver_update
  * haproxy_resolver_cname
  * haproxy_resolver_cname_error
  * haproxy_resolver_any_err
  * haproxy_resolver_nx
  * haproxy_resolver_timeout
  * haproxy_resolver_refused
  * haproxy_resolver_other
  * haproxy_resolver_invalid
  * haproxy_resolver_too_big
  * haproxy_resolver_outdated
2024-02-02 09:11:34 +01:00
Christopher Faulet
3e55b3da30 MEDIUM: promex/stick-table: Dump stick-table metrics via a promex module
Create a promex module to dump stick-table metrics. Thanks to this patch,
all references to stick tables were removed from the promex service.
2024-02-02 09:11:34 +01:00
Christopher Faulet
5f875414eb MEDIUM: promex: Dump metrics of registered modules with a way to filter them
This patch adds a dump loop on the registered modules. It is very similar to
other dump loops. When a module registered, a implicit scope is created with
the module's name. It means a module name must be unique. It also means,
metrics dump of modules can be filtered via the "scope" parameter.
2024-02-02 09:11:34 +01:00