Commit Graph

20722 Commits

Author SHA1 Message Date
Willy Tarreau
0f9a10c7f1 DEBUG: pools: also print the value of the tag when it doesn't match
Sometimes the tag's value may reveal a recognizable pattern, so let's
print it when it doesn't match a known pool.
2023-09-12 18:14:05 +02:00
Willy Tarreau
96c1a24224 DEBUG: pools: also print the item's pointer when crashing
It's important to inspect a core or recognize some values to have the
item pointer, it was not provided.
2023-09-12 18:14:05 +02:00
Frédéric Lécaille
84757e32e6 BUG/MEDIUM: quic: quic_cc_conn ->cntrs counters unreachable
This bug arrived with this commit in 2.9-dev3:

    MEDIUM: quic: Allow the quic_conn memory to be asap released.

When sending packets from quic_cc_conn_io_cb(), e.g. when the quic_conn
object has been released and replaced by a lighter one (quic_cc_conn),
some counters may have to be incremented. But they were not reachable
because not shared between quic_conn and quic_cc_conn struct.

To fix this, one has only to move the ->cntrs counters from quic_conn
to QUIC_CONN_COMMON struct which is shared between both quic_cc_conn

Thank you to Tristan for having reported this in GH #2247.

No need to backport.
2023-09-12 18:13:36 +02:00
Willy Tarreau
efc46dede9 DEBUG: pools: inspect pools on fatal error and dump information found
It's a bit frustrating sometimes to see pool checks catch a bug but not
provide exploitable information without a core.

Here we're adding a function "pool_inspect_item()" which is called just
before aborting in pool_check_pattern() and POOL_DEBUG_CHECK_MARK() and
which will display the error type, the pool's pointer and name, and will
try to check if the item's tag matches the pool, and if not, will iterate
over all pools to see if one would be a better candidate, then will try
to figure the last known caller and possibly other likely candidates if
the pool's tag is not sufficiently trusted. This typically helps better
diagnose corruption in use-after-free scenarios, or freeing to a pool
that differs from the one the object was allocated from, and will also
indicate calling points that may help figure where an object was last
released or allocated. The info is printed on stderr just before the
backtrace.

For example, the recent off-by-one test in the PPv2 changes would have
produced the following output in vtest logs:

  ***  h1    debug|FATAL: pool inconsistency detected in thread 1: tag mismatch on free().
  ***  h1    debug|  caller: 0x62bb87 (conn_free+0x147/0x3c5)
  ***  h1    debug|  pool: 0x2211ec0 ('pp_tlv_256', size 304, real 320, users 1)
  ***  h1    debug|Tag does not match. Possible origin pool(s):
  ***  h1    debug|  tag: @0x2565530 = 0x2216740 (pp_tlv_128, size 176, real 192, users 1)
  ***  h1    debug|Recorded caller if pool 'pp_tlv_128':
  ***  h1    debug|  @0x2565538 (+0184) = 0x62c76d (conn_recv_proxy+0x4cd/0xa24)

A mismatch in the allocated/released pool is already visible, and the
callers confirm it once resolved, where the allocator indeed allocates
from pp_tlv_128 and conn_free() releases to pp_tlv_256:

  $ addr2line -spafe ./haproxy <<< $'0x62bb87\n0x62c76d'
  0x000000000062bb87: conn_free at connection.c:568
  0x000000000062c76d: conn_recv_proxy at connection.c:1177
2023-09-11 15:46:14 +02:00
Willy Tarreau
f6bee5a50b DEBUG: pools: make pool_check_pattern() take a pointer to the pool
This will be useful to report detailed bug traces.
2023-09-11 15:19:49 +02:00
Willy Tarreau
e92e96b00f DEBUG: pools: pass the caller pointer to the check functions and macros
In preparation for more detailed pool error reports, let's pass the
caller pointers to the check functions. This will be useful to produce
messages indicating where the issue happened.
2023-09-11 15:19:49 +02:00
Willy Tarreau
baf2070421 DEBUG: pools: always record the caller for uncached allocs as well
When recording the caller of a pool_alloc(), we currently store it only
when the object comes from the cache and never when it comes from the
heap. There's no valid reason for this except that the caller's pointer
was not passed to pool_alloc_nocache(), so it used to set NULL there.
Let's just pass it down the chain.
2023-09-11 15:19:49 +02:00
Frédéric Lécaille
2dedbe76c9 BUG/MINOR: quic: fdtab array underflow access
When using the listener socket as file descriptor, qc->fd value is -1.
In this case one must not access fdtab[qc->fd] element to change its value.

This bug could have been detected by asan with such a backtrace:

=================================================================
==402222==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7fa8ecf417ex7fa8e915cf90 sp 0x7fa8e915cf88
WRITE of size 8 at 0x7fa8ecf417e8 thread T6
    #0 0x55707a0bf18a in qc_new_cc_conn src/quic_conn.c:838
    #1 0x55707a0c6dc0 in quic_conn_release src/quic_conn.c:1408
    #2 0x55707a10916f in quic_close src/xprt_quic.c:35
    #3 0x55707a0cec77 in conn_xprt_close include/haproxy/connection.h:153
    #4 0x55707a0ceed0 in conn_full_close include/haproxy/connection.h:197
    #5 0x55707a0ec253 in qcc_release src/mux_quic.c:2412
    #6 0x55707a0ec7d0 in qcc_io_cb src/mux_quic.c:2443
    #7 0x55707a63ff2a in run_tasks_from_lists src/task.c:596
    #8 0x55707a641cc9 in process_runnable_tasks src/task.c:876
    #9 0x55707a56f7b2 in run_poll_loop src/haproxy.c:2954
    #10 0x55707a5705fd in run_thread_poll_loop src/haproxy.c:3153
    #11 0x7fa8f9450ea6 in start_thread nptl/pthread_create.c:477
    #12 0x7fa8f936ea2e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0xfba2e)

0x7fa8ecf417e8 is located 24 bytes to the left of 134217728-byte region [0x7fa8e
allocated by thread T0 here:
    #0 0x7fa8f9a37037 in __interceptor_calloc ../../../../src/libsanitizer/asan/
    #1 0x55707a71a61d in init_pollers src/fd.c:1161
    #2 0x55707a56cdf1 in init src/haproxy.c:2672
    #3 0x55707a5714c2 in main src/haproxy.c:3298
    #4 0x7fa8f9296d09 in __libc_start_main ../csu/libc-start.c:308

Thread T6 created by T0 here:
    #0 0x7fa8f99e22a2 in __interceptor_pthread_create ../../../../src/libsanitizpp:214
    #1 0x55707a748a21 in setup_extra_threads src/thread.c:252
    #2 0x55707a5735c9 in main src/haproxy.c:3844
    #3 0x7fa8f9296d09 in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: heap-buffer-overflow src/quic_conn.c:838 in qc_new_cc
Shadow bytes around the buggy address:
  0x0ff59d9e02a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0ff59d9e02b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0ff59d9e02c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0ff59d9e02d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0ff59d9e02e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0ff59d9e02f0: fa fa fa fa fa fa fa fa fa fa fa fa fa[fa]fa fa
  0x0ff59d9e0300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff59d9e0310: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff59d9e0320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff59d9e0330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff59d9e0340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==402222==ABORTING
Aborted

Thank you to @Tristan971 for having reported this bug in GH #2247.

No need to backport.
2023-09-11 15:14:22 +02:00
Willy Tarreau
8b7841ff7a [RELEASE] Released version 2.9-dev5
Released version 2.9-dev5 with the following main changes :
    - BUG/MEDIUM: mux-h2: fix crash when checking for reverse connection after error
    - BUILD: import: guard plock.h against multiple inclusion
    - BUILD: pools: import plock.h to build even without thread support
    - BUG/MINOR: ssl/cli: can't find ".crt" files when replacing a certificate
    - BUG/MINOR: stream: protect stream_dump() against incomplete streams
    - DOC: config: mention uid dependency on the tune.quic.socket-owner option
    - MEDIUM: capabilities: enable support for Linux capabilities
    - CLEANUP/MINOR: connection: Improve consistency of PPv2 related constants
    - MEDIUM: connection: Generic, list-based allocation and look-up of PPv2 TLVs
    - MEDIUM: sample: Add fetch for arbitrary TLVs
    - MINOR: sample: Refactor fc_pp_authority by wrapping the generic TLV fetch
    - MINOR: sample: Refactor fc_pp_unique_id by wrapping the generic TLV fetch
    - MINOR: sample: Add common TLV types as constants for fc_pp_tlv
    - MINOR: ssl_sock: avoid iterating realloc(+1) on stored context
    - DOC: ssl: add some comments about the non-obvious session allocation stuff
    - CLEANUP: ssl: keep a pointer to the server in ssl_sock_init()
    - MEDIUM: ssl_sock: always use the SSL's server name, not the one from the tid
    - MEDIUM: server/ssl: place an rwlock in the per-thread ssl server session
    - MINOR: server/ssl: maintain an index of the last known valid SSL session
    - MINOR: server/ssl: clear the shared good session index on failure
    - MEDIUM: server/ssl: pick another thread's session when we have none yet
    - MINOR: activity: report the current run queue size
    - BUG/MINOR: checks: do not queue/wake a bounced check
    - MINOR: checks: start the checks in sleeping state
    - MINOR: checks: pin the check to its thread upon wakeup
    - MINOR: check: remember when we migrate a check
    - MINOR: check/activity: collect some per-thread check activity stats
    - MINOR: checks: maintain counters of active checks per thread
    - MINOR: check: also consider the random other thread's active checks
    - MEDIUM: checks: search more aggressively for another thread on overload
    - MEDIUM: checks: implement a queue in order to limit concurrent checks
    - MINOR: checks: also consider the thread's queue for rebalancing
    - DEBUG: applet: Properly report opposite SC expiration dates in traces
    - BUG/MEDIUM: stconn: Update stream expiration date on blocked sends
    - BUG/MINOR: stconn: Don't report blocked sends during connection establishment
    - BUG/MEDIUM: stconn: Wake applets on sending path if there is a pending shutdown
    - BUG/MEDIUM: stconn: Don't block sends if there is a pending shutdown
    - BUG/MINOR: quic: Possible skipped RTT sampling
    - MINOR: quic: Add a trace to quic_release_frm()
    - BUG/MAJOR: quic: Really ignore malformed ACK frames.
    - BUG/MINOR: quic: Unchecked pointer to packet number space dereferenced
    - BUG/MEDIUM: connection: fix pool free regression with recent ppv2 TLV patches
    - BUG/MEDIUM: h1-htx: Ensure chunked parsing with full output buffer
    - BUG/MINOR: stream: further protect stream_dump() against incomplete sessions
    - DOC: configuration: update examples for req.ver
    - MINOR: properly mark the end of the CLI command in error messages
    - BUILD: ssl: Build with new cryptographic library AWS-LC
    - REGTESTS: ssl: skip ssl_dh test with AWS-LC
    - BUILD: bug: make BUG_ON() void to avoid a rare warning
    - BUILD: checks: shut up yet another stupid gcc warning
    - MINOR: cpuset: add ha_cpuset_isset() to check for the presence of a CPU in a set
    - MINOR: cpuset: add ha_cpuset_or() to bitwise-OR two CPU sets
    - MINOR: cpuset: centralize a reliable bound cpu detection
    - MEDIUM: threads: detect incomplete CPU bindings
    - MEDIUM: threads: detect excessive thread counts vs cpu-map
    - BUILD: quic: Compilation issue on 32-bits systems with quic_may_send_bytes()
    - BUG/MINOR: quic: Unchecked pointer to Handshake packet number space
    - MINOR: global: export the display_version() symbol
    - MEDIUM: mworker: display a more accessible message when a worker crash
    - MINOR: httpclient: allow to configure the retries
    - MINOR: httpclient: allow to configure the timeout.connect
    - BUG/MINOR: quic: Wrong RTT adjusments
    - BUG/MINOR: quic: Wrong RTT computation (srtt and rrt_var)
    - BUG/MINOR: stconn: Don't inhibit shutdown on connection on error
    - BUG/MEDIUM: applet: Fix API for function to push new data in channels buffer
    - BUG/MEDIUM: stconn: Report read activity when a stream is attached to front SC
    - BUG/MEDIUM: applet: Report an error if applet request more room on aborted SC
    - BUG/MEDIUM: stconn/stream: Forward shutdown on write timeout
    - NUG/MEDIUM: stconn: Always update stream's expiration date after I/O
    - BUG/MINOR: applet: Always expect data when CLI is waiting for a new command
    - BUG/MINOR: ring/cli: Don't expect input data when showing events
    - BUG/MINOR: quic: Dereferenced unchecked pointer to Handshke packet number space
    - BUG/MINOR: hlua/action: incorrect message on E_YIELD error
    - MINOR: http_ana: position the FINAL flag for http_after_res execution
    - CI: scripts: add support to build-ssl.sh to download and build AWS-LC
    - CI: add support to matrix.py to determine the latest AWS-LC release
    - CI: Update matrix.py so all code is contained in functions.
    - CI: github: Add a weekly CI run building with AWS-LC
    - MINOR: ring: add a function to compute max ring payload
    - BUG/MEDIUM: ring: adjust maxlen consistency check
    - MINOR: sink: simplify post_sink_resolve function
    - MINOR: log/sink: detect when log maxlen exceeds sink size
    - MINOR: sink: inform the user when logs will be implicitly truncated
    - MEDIUM: sink: don't perform implicit truncations when maxlen is not set
    - MINOR: log: move log-forwarders cleanup in log.c
    - MEDIUM: httpclient/logs: rely on per-proxy post-check instead of global one
    - MINOR: log: add dup_logsrv() helper function
    - MEDIUM: log/sink: make logsrv postparsing more generic
    - MEDIUM: fcgi-app: properly postresolve logsrvs
    - MEDIUM: spoe-agent: properly postresolve log rings
    - MINOR: sink: add helper function to deallocate sink struct
    - MEDIUM: sink/ring: introduce high level ring creation helper function
    - MEDIUM: sink: add sink_finalize() function
    - CLEANUP: log: remove unnecessary trim in __do_send_log
    - MINOR: cache: Change hash function in default normalizer used in case of "vary"
    - MINOR: tasks/stats: report the number of niced tasks in "show info"
    - CI: Update to actions/checkout@v4
    - MINOR: ssl: add support for 'curves' keyword on server lines
    - BUG/MINOR: quic: Wrong cluster secret initialization
    - CLEANUP: quic: Remove useless free_quic_tx_pkts() function.
    - MEDIUM: init: initialize the trash earlier
    - MINOR: tools: add function read_line_to_trash() to read a line of a file
    - MINOR: cfgparse: use read_line_from_trash() to read from /sys
    - MEDIUM: cfgparse: assign NUMA affinity to cpu-maps
    - MINOR: cpuset: dynamically allocate cpu_map
    - REORG: cpuset: move parse_cpu_set() and parse_cpumap() to cpuset.c
    - CI: musl: highlight section if there are coredumps
    - CI: musl: drop shopt in workflow invocation
2023-09-08 19:21:45 +02:00
Ilya Shipitsin
d89734bcc5 CI: musl: drop shopt in workflow invocation
"shopt" is bash specific, while musl uses bourne shell.

/__w/_temp/1b0f5f5d-c71b-4a66-8be3-e1fe51c10993.sh: line 7: shopt: not found
2023-09-08 19:05:04 +02:00
Ilya Shipitsin
8382d71808 CI: musl: highlight section if there are coredumps
previously, section was collapsed, thus it was harder to find that
there's something to look at
2023-09-08 19:05:04 +02:00
Willy Tarreau
4a18d9e560 REORG: cpuset: move parse_cpu_set() and parse_cpumap() to cpuset.c
These ones were still in cfgparse.c but they're not specific to the
config at all and may actually be used even when parsing cpu list
entries in /sys. Better move them where they can be reused.
2023-09-08 16:25:19 +02:00
Willy Tarreau
5119109e3f MINOR: cpuset: dynamically allocate cpu_map
cpu_map is 8.2kB/entry and there's one such entry per group, that's
~520kB total. In addition, the init code is still in haproxy.c enclosed
in ifdefs. Let's make this a dynamically allocated array in the cpuset
code and remove that init code.

Later we may even consider reallocating it once the number of threads
and groups is known, in order to shrink it a little bit, as the typical
setup with a single group will only need 8.2kB, thus saving half a MB
of RAM. This would require that the upper bound is placed in a variable
though.
2023-09-08 16:25:19 +02:00
Willy Tarreau
b0f20ed79b MEDIUM: cfgparse: assign NUMA affinity to cpu-maps
Do not force affinity on the process, instead let's just apply it to
cpu-map, it will automatically be used later in the init process. We
can do this because we know that cpu-map was not set when we're using
this detection code.

This is much saner, as we don't need to manipulate the process' affinity
at this point in time, and just update the info that the user omitted to
set by themselves, which guarantees a better long-term consistency with
the documented feature.
2023-09-08 16:25:19 +02:00
Willy Tarreau
809a49da96 MINOR: cfgparse: use read_line_from_trash() to read from /sys
It's easier to use this function now to natively support variable
fields in the file's path. This also removes read_file_from_trash()
that was only used here and was static.
2023-09-08 16:25:19 +02:00
Willy Tarreau
1f2433fb6a MINOR: tools: add function read_line_to_trash() to read a line of a file
This function takes on input a printf format for the file name, making
it particularly suitable for /proc or /sys entries which take a lot of
numbers. It also automatically trims the trailing CR and/or LF chars.
2023-09-08 16:25:19 +02:00
Willy Tarreau
5f10176e2c MEDIUM: init: initialize the trash earlier
More and more utility functions rely on the trash while most of the init
code doesn't have access to it because it's initialized very late (in
PRE_CHECK for the initial one). It's a pool, and it purposely supports
being reallocated, so let's initialize it in STG_POOL so that early
STG_INIT code can at least use it.
2023-09-08 16:25:19 +02:00
Frédéric Lécaille
e3e218b98e CLEANUP: quic: Remove useless free_quic_tx_pkts() function.
This function define but no more used since this commit:
    BUG/MAJOR: quic: Really ignore malformed ACK frames.
2023-09-08 10:17:25 +02:00
Frédéric Lécaille
292dfdd78d BUG/MINOR: quic: Wrong cluster secret initialization
The function generate_random_cluster_secret() which initializes the cluster secret
when not supplied by configuration is buggy. There 1/256 that the cluster secret
string is empty.

To fix this, one stores the cluster as a reduced size first 128 bits of its own
SHA1 (160 bits) digest, if defined by configuration. If this is not the case, it
is initialized with a 128 bits random value. Furthermore, thus the cluster secret
is always initialized.

As the cluster secret is always initialized, there are several tests which
are for now on useless. This patch removes such tests (if(global.cluster_secret))
in the QUIC code part and at parsing time: no need to check that a cluster
secret was initialized with "quic-force-retry" option.

Must be backported as far as 2.6.
2023-09-08 09:50:58 +02:00
William Lallemand
15e591b6e0 MINOR: ssl: add support for 'curves' keyword on server lines
This patch implements the 'curves' keyword on server lines as well as
the 'ssl-default-server-curves' keyword in the global section.

It also add the keyword on the server line in the ssl_curves reg-test.

These keywords allow the configuration of the curves list for a server.
2023-09-07 23:29:10 +02:00
Tim Duesterhus
5c923f1869 CI: Update to actions/checkout@v4
No functional change, but we should keep this current.

see 5f4ddb54b0
2023-09-06 17:46:02 +02:00
Willy Tarreau
28ff1a5d56 MINOR: tasks/stats: report the number of niced tasks in "show info"
We currently know the number of tasks in the run queue that are niced,
and we don't expose it. It's too bad because it can give a hint about
what share of the load is relevant. For example if one runs a Lua
script that was purposely reniced, or if a stats page or the CLI is
hammered with slow operations, seeing them appear there can help
identify what part of the load is not caused by the traffic, and
improve monitoring systems or autoscalers.
2023-09-06 17:44:44 +02:00
Remi Tricot-Le Breton
e03d060aa3 MINOR: cache: Change hash function in default normalizer used in case of "vary"
When building the secondary signature for cache entries when vary is
enabled, the referer part of the signature was a simple crc32 of the
first referer header.
This patch changes it to a 64bits hash based of xxhash algorithm with a
random seed built during init. This will prevent "malicious" hash
collisions between entries of the cache.
2023-09-06 16:11:31 +02:00
Aurelien DARRAGON
7a71801af6 CLEANUP: log: remove unnecessary trim in __do_send_log
Since both sink_write and fd_write_frag_line take the maxlen parameter
as argument, there is no added value for the trim before passing the
msg parameter to those functions.
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
8e6339aa29 MEDIUM: sink: add sink_finalize() function
To further clean the code and remove duplication, some sink postparsing
and sink->sft finalization is now performed in a dedicated function
named sink_finalize().
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
b2879e3502 MEDIUM: sink/ring: introduce high level ring creation helper function
ease code maintenance.
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
5a8755681d MINOR: sink: add helper function to deallocate sink struct
In this patch we move sink freeing logic outside of sink_deinit() function
in order to create the sink_free() helper function that could be used
on error paths for example.
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
6049a478e4 MEDIUM: spoe-agent: properly postresolve log rings
Now that we have sink_postresolve_logsrvs() function, we make use of it
for spoe-agent log postparsing logic.

This will allow this kind of config to work:
  |spoe-agent test
  |        log tcp@127.0.0.1:514 local0
  |        use-backend xxx

Plus, consistency checks will also be performed as for regular log
directives used from global, log-forward or proxy sections.
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
486aa01204 MEDIUM: fcgi-app: properly postresolve logsrvs
Now that we have postresolve_logsrv_list() function, we make use of it
for fcgi-app log postparsing logic.

This will allow this kind of config to work:
  |fcgi-app test
  |        docroot /
  |        log-stderr tcp@127.0.0.1:514 local0

Plus, consistency checks will also be performed as for regular log
directives used from global, log-forward or proxy sections.
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
d9b81e5b49 MEDIUM: log/sink: make logsrv postparsing more generic
We previously had postparsing logic but only for logsrv sinks, but now we
need to make this operation on logsrv directly instead of sinks to prepare
for additional postparsing logic that is not sink-specific.

To do this, we migrated post_sink_resolve() and sink_postresolve_logsrvs()
to their postresolve_logsrvs() and postresolve_logsrv_list() equivalents.

Then, we split postresolve_logsrv_list() so that the sink-only logic stays
in sink.c (sink_resolve_logsrv_buffer() function), and the "generic"
target part stays in log.c as resolve_logsrv().

Error messages formatting was preserved as far as possible but some slight
variations are to be expected.
As for the functional aspect, no change should be expected.
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
969e212c66 MINOR: log: add dup_logsrv() helper function
ease code maintenance by introducing dup_logsrv() helper function to
properly duplicate an existing logsrv struct.
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
7a12e2d369 MEDIUM: httpclient/logs: rely on per-proxy post-check instead of global one
httpclient used to register a global post-check function to iterate over
all known proxies and post-initialize httpclient related ones (mainly
for logs initialization).

But we currently have an issue: post_sink_resolve() function which is
also registered using REGISTER_POST_CHECK() macro conflicts with
httpclient_postcheck() function.

This is because post_sink_resolve() relies on proxy->logsrvs to be
correctly initialized already, and httpclient_postcheck() may create
and insert new logsrvs entries to existing proxies when executed.

So depending on which function runs first, we could run into trouble.

Hopefully, to this day, everything works "by accident" due to
http_client.c file being loaded before sink.c file when compiling source
code.

But as soon as we would move one of the two functions to other files, or
if we rename files or make changes to the Makefile build recipe, we could
break this at any time.

To prevent post_sink_resolve() from randomly failing in the future, we now
make httpclient postcheck rely on per-proxy post-checks by slightly
modifying httpclient_postcheck() function so that it can be registered
using REGISTER_POST_PROXY_CHECK() macro.

As per-proxy post-check functions are executed right after config parsing
for each known proxy (vs global post-check which are executed a bit later
in the init process), we can be certain that functions registered using
global post-check macro, ie: post_sink_resolve(), will always be executed
after httpclient postcheck, effectively resolving the ordering conflict.

This should normally not cause visible behavior changes, and while it
could be considered as a bug, it's probably not worth backporting it
since the only way to trigger the issue is through code refactors,
unless we want to backport it to ease code maintenance of course,
in which case it should easily apply for >= 2.7.
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
e187361b52 MINOR: log: move log-forwarders cleanup in log.c
Move the log-forwarded proxies cleanup from global deinit() function into
log dedicated deinit function.

No backport needed.
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
32f1db6d0d MEDIUM: sink: don't perform implicit truncations when maxlen is not set
maxlen now defaults ~0 (instead of BUFSIZE) to make sure no implicit
truncation will be performed when the option is not specified, since the
doc doesn't mention any default value for maxlen. As such, if the payload
is too big, it will be dropped (this is the default expected behavior).
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
fdf82d058b MINOR: sink: inform the user when logs will be implicitly truncated
Consider the following example:

 |log ring@test-ring len 2000 local0
 |
 |ring test-ring
 |  maxlen 1000

This would result in emitted logs being silently truncated to 1000 because
test-ring maxlen is smaller than the log directive maxlen.

In this patch we're adding an extra check in post_sink_resolve() to detect
this kind of confusing setups and warn the user about the implicit
truncation when DIAG mode is on.

This commit depends on:
 - "MINOR: sink: simplify post_sink_resolve function"
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
ceaa1ddb06 MINOR: log/sink: detect when log maxlen exceeds sink size
To prevent logs from being silently (and unexpectly droppped) at runtime,
we check that the maxlen parameter from the log directives are
strictly inferior to the targeted ring size.

      |global
      |     tune.bufsize 16384
      |     log tcp@127.0.0.1:514 len 32768
      |     log myring@127.0.0.1:514 len 32768
      |ring myring
      |     # no explicit size

On such configs, a diag warning will be reported.

This commit depends on:
 - "MINOR: sink: simplify post_sink_resolve function"
 - "MINOR: ring: add a function to compute max ring payload"
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
d499485aa9 MINOR: sink: simplify post_sink_resolve function
Simplify post_sink_resolve() function to reduce code duplication and
make it easier to maintain.
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
ddd8671b19 BUG/MEDIUM: ring: adjust maxlen consistency check
When user specifies a maxlen parameter that is greater than the size of
a given ring section, a warning is emitted to inform that the max length
exceeds size, and then the maxlen is forced to size.

The logic is good, but imprecise, because it doesn't take into account
the slight overhead from storing payloads into the ring.

In practise, we cannot store a single message which is exactly the same
length than size. Doing so will result in the message being dropped at
runtime.

Thanks to the ring_max_payload() function introduced in "MINOR: ring: add
a function to compute max ring payload", we can now deduce the maximum
value for the maxlen parameter before it could result in messages being
dropped.

When maxlen value is set to an improper value, the warning will be emitted
and maxlen will be forced to the maximum "single" payload len that could
fit in the ring buffer, preventing messages from being dropped
unexpectedly.

This commit depends on:
 - "MINOR: ring: add a function to compute max ring payload"

This may be backported as far as 2.2
2023-09-06 16:06:39 +02:00
Aurelien DARRAGON
5b295ff409 MINOR: ring: add a function to compute max ring payload
Add a helper function to the ring API to compute the maximum payload
length that could fit into the ring based on ring size.
2023-09-06 16:06:39 +02:00
Andrew Hopkins
06d36913bd CI: github: Add a weekly CI run building with AWS-LC
Use determine_latest_aws_lc() from matrix.py to always test with
the latest release of AWS-LC. Run the common "default,bug,devel"
tests.
2023-09-06 13:41:36 +02:00
Andrew Hopkins
b2a7840a28 CI: Update matrix.py so all code is contained in functions.
Refactor matrix.py so all the logic is contained inside either
helper functions or a new main function. Run the new main function
by default. This lets other GitHub actions use functions in the
python code without generating the whole matrix.
2023-09-06 13:41:36 +02:00
Andrew Hopkins
4f77690366 CI: add support to matrix.py to determine the latest AWS-LC release
Refactor the existing OpenSSL tag parsing logic to share some of GitHub
tag logic. OpenSSL and AWS-LC don't follow the same naming convention so
each library has it's own sorting logic.
2023-09-06 13:41:36 +02:00
Andrew Hopkins
c677bb840b CI: scripts: add support to build-ssl.sh to download and build AWS-LC
Relies on a new enviornment variable 'AWS_LC_VERSION' to be set to
the GitHub tag to download and build.
2023-09-06 13:41:36 +02:00
Aurelien DARRAGON
4457783ade MINOR: http_ana: position the FINAL flag for http_after_res execution
Ensure that the ACT_OPT_FINAL flag is always set when executing actions
from http_after_res context.

This will permit lua functions to be executed as http_after_res actions
since hlua_ctx_resume() automatically disables "yielding" when such flag
is set: the hlua handler will only allow 1shot executions at this point
(lua or not, we don't wan't to reschedule http_after_res actions).
2023-09-06 11:42:34 +02:00
Aurelien DARRAGON
967608a432 BUG/MINOR: hlua/action: incorrect message on E_YIELD error
When hlua_action error messages were reworked in d5b073cf1
("MINOR: lua: Improve error message"), an error was made for the
E_YIELD case.

Indeed, everywhere E_YIELD error is handled: "yield is not allowed" or
similar error message is reported to the user. But instead we currently
have: "aborting Lua processing on expired timeout".

It is quite misleading because this error message often refers to the
HLUA_E_ETMOUT case.

Thus, we now report the proper error message thanks to this patch.

This should be backported to all stable versions.
[on 2.0, the patch needs to be slightly adapted]
2023-09-06 11:42:34 +02:00
Frédéric Lécaille
e7240a0ba6 BUG/MINOR: quic: Dereferenced unchecked pointer to Handshke packet number space
This issue was reported by longrtt interop test with quic-go as client
and @chipitsine in GH #2282 when haproxy is compiled against libressl.

Add two checks to prevent a pointer to the Handshake packet number space
to be dereferenced if this packet number space was released.

Thank you to @chipitsine for this report.

No need to backport.
2023-09-06 10:13:40 +02:00
Christopher Faulet
700ca14fc1 BUG/MINOR: ring/cli: Don't expect input data when showing events
The "show events" command may wait for now events if "-w" option is used. In
this case, no timeout must be triggered. So we explicitly state no input
data are expected. This disables the read timeout on the client side.

This patch should be backported to 2.8. It is probably useless to backport
it further. In all cases, it depends on the commit "BUG/MINOR: applet:
Always expect data when CLI is waiting for a new command"
2023-09-06 09:36:29 +02:00
Christopher Faulet
2f1e0a0a46 BUG/MINOR: applet: Always expect data when CLI is waiting for a new command
There is a mechanism for applets to disable the read timeout on the opposite
side if it is now waiting for any data. Of course, there is also a way to
re-activate it. But, it must excplicitly be handle by applets.

For the CLI, some commands may state no input data are expected. So we must
be sure to reset its state when the applet is waiting for a new command. For
now, it is not a bug because no CLI command uses this mechanism.

This patch must be backported to 2.8.
2023-09-06 09:36:19 +02:00
Christopher Faulet
8073094bf1 NUG/MEDIUM: stconn: Always update stream's expiration date after I/O
It is a revert of following patches:

  * d7111e7ac ("MEDIUM: stconn: Don't requeue the stream's task after I/O")
  * 3479d99d5 ("BUG/MEDIUM: stconn: Update stream expiration date on blocked sends")

Because the first one is reverted, the second one is useless and can be reverted
too.

The issue here is that I/O may be performed without stream wakeup. So if no
expiration date was set on the last call to process_stream(), the stream is
never rescheduled and no timeout can be detected. This especially happens on
TCP streams because fast-forward is enabled very early.

Instead of tracking all places where the stream's expiration data must be
updated, it is now centralized in sc_notify(), as it was performed before
the timeout refactoring.

This patch must be backported to 2.8.
2023-09-06 09:29:27 +02:00
Christopher Faulet
b9c87f8082 BUG/MEDIUM: stconn/stream: Forward shutdown on write timeout
The commit 7f59d68fe2 ("BUG/MEDIIM: stconn: Flush output data before
forwarding close to write side") introduced a regression. When a write
timeout is detected, the shutdown is no longer forwarded. Dependig on the
channels state, it may block the processing, waiting the client or the
server leaves.

The commit above tries to avoid to truncate messages on shutdown but on
write timeout, if the channel is not empty, there is nothing more we can do
to send these data. It means the endpoint is unable to send data. In this
case, we must forward the shutdown.

This patch should be backported as far as 2.2.
2023-09-06 09:29:27 +02:00