Commit Graph

2628 Commits

Author SHA1 Message Date
Willy Tarreau
db09cd6ad4 [RELEASE] Released version 2.9-dev10
Released version 2.9-dev10 with the following main changes :
    - CLEANUP: Re-apply xalloc_size.cocci (3)
    - BUG/MEDIUM: stconn: Report send activity during mux-to-mux fast-forward
    - BUG/MEDIUM: stconn: Don't report rcv/snd expiration date if SC cannot epxire
    - MINOR: stconn: Don't queue stream task in past in sc_notify()
    - BUG/MEDIUM: Don't apply a max value on room_needed in sc_need_room()
    - BUG/MINOR: stconn: Sanitize report for read activity
    - CLEANUP: htx: Properly indent htx_reserve_max_data() function
    - DOC: stconn: Improve comments about lra and fsb usage
    - BUG/MEDIUM: quic: fix actconn on quic_conn alloc failure
    - BUG/MEDIUM: quic: fix sslconns on quic_conn alloc failure
    - BUG/MEDIUM: mux-h1: Be sure xprt support splicing to use it during fast-forward
    - MINOR: proto_reverse_connect: use connect timeout
    - BUG/MINOR: mux-h1: Release empty ibuf during data fast-forwarding
    - BUG/MINOR: stick-table/cli: Check for invalid ipv4 key
    - MEDIUM: stktable/cli: simplify entry key handling
    - MINOR: stktable/cli: support v6tov4 and v4tov6 conversions
    - BUG/MINOR: mux-h1: Properly handle http-request and http-keep-alive timeouts
    - BUG/MEDIUM: freq-ctr: Don't report overshoot for long inactivity period
    - BUG/MEDIUM: pool: fix releasable pool calculation when overloaded
    - BUG/MINOR: pool: check one other random bucket on alloc conflict
    - BUG/MEDIUM: pool: try once to allocate from another bucket if empty
    - MEDIUM: stconn/muxes: Loop on data fast-forwarding to forward at least a buffer
    - MINOR: stconn/mux-h2: Use a iobuf flag to report EOI to consumer side during FF
    - MEDIUM: quic: Heavy task mode during handshake
    - MEDIUM: quic: Heavy task mode with non contiguously bufferized CRYPTO data
    - MINOR: quic: release the TLS context asap from quic_conn_release()
    - MINOR: quic: Add idle timer task pointer to traces
    - BUG/MINOR: quic: idle timer task requeued in the past
    - CLEANUP: quic: Indentation fix in qc_do_build_pkt()
    - MINOR: quic: Avoid zeroing frame structures
    - BUG/MEDIUM: quic: Too short Initial packet sent (enc. level allocation failed)
    - BUG/MEDIUM: quic: Avoid trying to send ACK frames from an empty ack ranges tree
    - BUG/MEDIUM: quic: Possible crashes when sending too short Initial packets
    - BUG/MEDIUM: quic: Avoid some crashes upon TX packet allocation failures
    - BUG/MEDIUM: quic: Possible crashes during secrets allocations (heavy load)
    - BUG/MEDIUM: stconn: Don't update stream expiration date if already expired
    - MINOR: errors: ha_alert() and ha_warning() uses warn_exec_path()
    - MINOR: errors: does not check MODE_STARTING for log emission
    - MEDIUM: errors: move the MODE_QUIET test in print_message()
    - DOC: management: -q is quiet all the time
    - MEDIUM: mworker: -W is mandatory when using -S
    - BUG/MEDIUM: mux-h1: Exit early if fast-forward is not supported by opposite SC
    - MEDIUM: quic: adjust address validation
    - MINOR: quic: reduce half open counters scope
    - MEDIUM: quic: limit handshake per listener
    - MEDIUM: quic: define an accept queue limit
    - BUG/MINOR: quic: fix retry token check inconsistency
    - MINOR: task/debug: explicitly support passing a null caller to wakeup functions
    - MINOR: task/debug: make task_queue() and task_schedule() possible callers
    - OPTIM: mux-h2: don't allocate more buffers per connections than streams
    - BUG/MINOR: quic: remove dead code in error path
    - MEDIUM: quic: respect closing state even on soft-stop
    - MEDIUM: quic: release conn socket before using quic_cc_conn
    - DOC: config: use the word 'backend' instead of 'proxy' in 'track' description
    - BUG/MEDIUM: applet: Remove appctx from buffer wait list on release
    - MINOR: tools: make str2sa_range() directly return type hints
    - BUG/MEDIUM: server: invalid address (post)parsing checks
    - BUG/MINOR: sink: don't learn srv port from srv addr
    - CLEANUP: sink: bad indent in sink_new_from_logger()
    - CLEANUP: sink: useless leftover in sink_add_srv()
    - BUG/MINOR: quic: Useless use of non-contiguous buffer for in order CRYPTO data
    - MINOR: server: always initialize pp_tlvs for default servers
    - BUG/MEDIUM: proxy: always initialize the default settings after init
    - MEDIUM: startup: 'haproxy -c' is quiet when valid
    - BUG/MINOR: sample: Fix bytes converter if offset is bigger than sample length
    - BUG/MINOR: log: keep the ref in dup_logger()
    - BUG/MINOR: quic: fix crash on qc_new_conn alloc failure
    - BUG/MINOR: quic: fix decrement of half_open counter on qc alloc failure
    - BUG/MEDIUM: quic: fix FD for quic_cc_conn
    - DOC: config: Fix name for tune.disable-zero-copy-forwarding global param
    - REGTESTS: startup: -conf-OK requires -V with current VTest
    - BUG/MEDIUM: quic: Non initialized CRYPTO data stream deferencing
    - MINOR: quic: Add a max window parameter to congestion control algorithms
    - MINOR: quic: Maximum congestion control window for each algo
    - DOC: quic: Wrong syntax for "quic-cc-algo" keyword.
    - DOC: quic: Maximum congestion control window configuration
    - BUG/MINOR: quic: maximum window limits do not match the doc
    - BUG/MEDIUM: connection: report connection errors even when no mux is installed
    - BUG/MINOR: stconn: Handle abortonclose if backend connection was already set up
    - MINOR: connection: Add a CTL flag to notify mux it should wait for reads again
    - MEDIUM: mux-h1: Handle MUX_SUBS_RECV flag in h1_ctl() and susbscribe for reads
    - BUG/MEDIUM: stream: Properly handle abortonclose when set on backend only
    - MINOR: stconn: Use SC to detect frontend connections in sc_conn_recv()
    - REGTESTS: http: Improve script testing abortonclose option
    - MINOR: activity: report profiling duration and age in "show profiling"
    - BUG/MEDIUM: mworker: set the master variable earlier
    - BUG/MEDIUM: stream: Don't call mux .ctl() callback if not implemented
    - MINOR: connection: update rhttp flags usage
    - BUG/MINOR: mux_h2: reject passive reverse conn if error on add to idle
    - MINOR: server: force add to idle on reverse
    - MINOR: shctx: Set last_append to NULL when reserving block in hot list
    - MEDIUM: shctx: Move list between hot and avail list in O(1)
    - MEDIUM: shctx: Simplify shctx_row_reserve_hot loop
    - MINOR: shctx: Remove explicit 'from' param from shctx_row_data_append
    - MEDIUM: cache: Use dedicated cache tree lock alongside shctx lock
    - MINOR: cache: Remove expired entry delete in "show cache" command
    - MINOR: cache: Add option to avoid removing expired entries in lookup function
    - MEDIUM: cache: Use rdlock on cache in cache_use
    - MEDIUM: shctx: Remove 'hot' list from shared_context
    - MINOR: cache: Use dedicated trash for "show cache" cli command
    - MEDIUM: cache: Switch shctx spinlock to rwlock and restrict its scope
    - MEDIUM: cache: Add refcount on cache_entry
    - MEDIUM: shctx: Descend shctx_lock calls into the shctx_row_reserve_hot
    - MINOR: shctx: Add new reserve_finish callback call to shctx_row_reserve_hot
    - MAJOR: cache: Delay cache entry delete in reserve_hot function
    - MINOR: shctx: Remove redundant arg from free_block callback
    - MINOR: shctx: Remove 'use_shared_mem' variable
    - DOC: cache: Specify when function expects a cache lock
    - BUG/MEDIUM: stconn: Update fsb date on partial sends
    - MINOR: htx: Use a macro for overhead induced by HTX
    - MINOR: channel: Add functions to get info on buffers and deal with HTX streams
    - BUG/MINOR: stconn: Fix streamer detection for HTX streams
    - BUG/MINOR: stconn: Use HTX-aware channel's functions to get info on buffer
    - BUG/MINOR: stconn/applet: Report send activity only if there was output data
    - BUG/MINOR: stconn: Report read activity on non-indep streams for partial sends
    - BUG/MINOR: shctx: Remove old HA_SPIN_INIT
    - REGTESTS: try to activate again the seamless reload test with the master CLI
    - MINOR: proxy: Add "handshake" new timeout (frontend side)
    - MEDIUM: quic: Add support for "handshake" timeout setting.
    - MINOR: quic: Dump the expiration date of the idle timer task
    - BUG/MINOR: quic: Malformed CONNECTION_CLOSE frame
    - MEDIUM: session: handshake timeout (TCP)
    - DOC: proxy: Add "handshake" timeout documentation.
    - MINOR: quic: Rename "handshake" timeout to "client-hs"
    - CLEANUP: haproxy: remove old comment from 1.1 from the file header
    - BUG/MEDIUM: mux-h2: fail earlier on malloc in takeover()
    - BUG/MEDIUM: mux-h1: fail earlier on malloc in takeover()
    - BUG/MEDIUM: mux-fcgi: fail earlier on malloc in takeover()
    - MINOR: rhttp: remove the unused outgoing connect() function
    - MINOR: backend: without ->connect(), allow to pick another thread's connection
    - BUG/MINOR: stream/cli: report correct stream age in "show sess"
    - MINOR: stream/cli: add an optional "older" filter for "show sess"
    - MINOR: stream/cli: add another filter "susp" to "show sess"
    - MINOR: stktable: add stktable_deinit function
    - BUG/MINOR: proxy/stktable: missing frees on proxy cleanup
    - CLEANUP: backend: removing unused LB param
    - MEDIUM: lbprm: store algo params on 32bits
    - MEDIUM: log/balance: merge tcp/http algo with log ones
    - Revert "MINOR: proxy: report a warning for max_ka_queue in proxy_cfg_ensure_no_http()"
    - Revert "MINOR: tcp_rules: tcp-{request,response} requires TCP or HTTP mode"
    - Revert "MINOR: stktable: "stick" requires TCP or HTTP mode"
    - Revert "MINOR: cfgparse-listen: "http-send-name-header" requires TCP or HTTP mode"
    - Revert "MINOR: cfgparse-listen: "dynamic-cookie-key" requires TCP or HTTP mode"
    - Revert "MINOR: cfgparse-listen: "http-reuse" requires TCP or HTTP mode"
    - Revert "MINOR: fcgi-app: "use-fcgi-app" requires TCP or HTTP mode"
    - Revert "MINOR: http_htx/errors: prevent the use of some keywords when not in tcp/http mode"
    - Revert "MINOR: flt_http_comp: "compression" requires TCP or HTTP mode"
    - Revert "MINOR: filter: "filter" requires TCP or HTTP mode"
    - MINOR: log/backend: ensure log exclusive params are not used in other modes
    - MINOR: log/backend: prevent tcp-{request,response} use with LOG mode
    - MINOR: log/backend: prevent stick table and stick rules with LOG mode
    - MINOR: log/backend: prevent "http-send-name-header" use with LOG mode
    - MINOR: log/backend: prevent "dynamic-cookie-key" use with LOG mode
    - REGTESTS: http: add a test to validate chunked responses delivery
2023-11-18 12:00:37 +01:00
Aurelien DARRAGON
b61147fd2a MEDIUM: log/balance: merge tcp/http algo with log ones
"log-balance" directive was recently introduced to configure the
balancing algorithm to use when in a log backend. However, it is
confusing and it causes issues when used in default section.

In this patch, we take another approach: first we remove the
"log-balance" directive, and instead we rely on existing "balance"
directive to configure log load balancing in log backend.

Some algorithms such as roundrobin can be used as-is in a log backend,
and for log-only algorithms, they are implemented as "log-$name" inside
the "backend" directive.

The documentation was updated accordingly.
2023-11-18 11:16:21 +01:00
Willy Tarreau
6c7771f1b4 MINOR: stream/cli: add another filter "susp" to "show sess"
This one reports streams considered as "suspicious", i.e. those with
no expiration dates or dates in the past, or those without a front
endpoint. More criteria could be added in the future.
2023-11-17 19:30:07 +01:00
Willy Tarreau
3ffcf7beb1 MINOR: stream/cli: add an optional "older" filter for "show sess"
It's often needed to be able to refine "show sess" when debugging, and
very often a first glance at old streams is performed, but that's a
difficult task in large dumps, and it takes lots of resources to dump
everything.

This commit adds "older <age>" to "show sess" in order to specify the
minimum age of streams that will be dumped. This should simplify the
identification of blocked ones.
2023-11-17 19:30:04 +01:00
Frédéric Lécaille
888d1dc3dc MINOR: quic: Rename "handshake" timeout to "client-hs"
Use a more specific name for this timeout to distinguish it from a possible
future one on the server side.
Also update the documentation.
2023-11-17 18:09:41 +01:00
Frédéric Lécaille
957085e85d DOC: proxy: Add "handshake" timeout documentation.
Document the "handshake" timeout new setting available one frontend side.
This should at least be helpful for QUIC client connections to prevent
an attacker from refreshing plenty of connections without completing
the handshake step, leading haproxy to consume memory for nothing.
2023-11-17 17:31:42 +01:00
Frédéric Lécaille
8df7018736 DOC: quic: Maximum congestion control window configuration
Document the optional parameter which may be supplied after the congestion
control algorithm name to set the maximum congestion control window.
2023-11-13 18:17:43 +01:00
Frédéric Lécaille
d9bf1b6c41 DOC: quic: Wrong syntax for "quic-cc-algo" keyword.
As the argument to "quic-cc-algo" is mandatory, brackets must be used here
in the documentation.

Must be backported as far as 2.6.
2023-11-13 18:14:16 +01:00
Christopher Faulet
cb560bf3d7 DOC: config: Fix name for tune.disable-zero-copy-forwarding global param
"disable-" prefix was missing. This param was correctly named in the list of
supported keywords in the global section, but not in the keyword
description.

No backport needed.
2023-11-13 14:31:14 +01:00
Willy Tarreau
9530e7dcd3 DOC: config: use the word 'backend' instead of 'proxy' in 'track' description
User @nwehrman reported in issue #2328 that the used of "proxy" instead
of "backend" in the argument of the "track" server keyword is confusing.
Admittedly, all other places in the doc use "backend/server" instead of
"proxy/server", so let's update it for the sake of consistency.
2023-11-10 16:29:02 +01:00
Amaury Denoyelle
bb28215d9b MEDIUM: quic: define an accept queue limit
QUIC connections are pushed manually into a dedicated listener queue
when they are ready to be accepted. This happens after handshake
finalization or on 0-RTT packet reception. Listener is then woken up to
dequeue them with listener_accept().

This patch comptabilizes the number of connections currently stored in
the accept queue. If reaching a certain limit, INITIAL packets are
dropped on reception to prevent further QUIC connections allocation.
This should help to preserve system resources.

This limit is automatically derived from the listener backlog. Half of
its value is reserved for handshakes and the other half for accept
queues. By default, backlog is equal to maxconn which guarantee that
there can't be no more than maxconn connections in handshake or waiting
to be accepted.
2023-11-09 16:24:00 +01:00
Amaury Denoyelle
3df6a60113 MEDIUM: quic: limit handshake per listener
Implement a limit per listener for concurrent number of QUIC
connections. When reached, INITIAL packets for new connections are
automatically dropped until the number of handshakes is reduced.

The limit value is automatically based on listener backlog, which itself
defaults to maxconn.

This feature is important to ensure CPU and memory resources are not
consume if too many handshakes attempt are started in parallel.

Special care is taken if a connection is released before handshake
completion. In this case, counter must be decremented. This forces to
ensure that member <qc.state> is set early in qc_new_conn() before any
quic_conn_release() invocation.
2023-11-09 16:23:52 +01:00
William Lallemand
705a72fd19 DOC: management: -q is quiet all the time
The documentation about -q seems wrong, it does not output messages
after the startup, it disables all messages. It was always quiet with
the stdio_quiet() function.

Must be backported in all stable versions.
2023-11-09 14:39:11 +01:00
Willy Tarreau
ff3dcb20f2 [RELEASE] Released version 2.9-dev9
Released version 2.9-dev9 with the following main changes :
    - DOC: internal: filters: fix reference to entities.pdf
    - BUG/MINOR: ssl: load correctly @system-ca when ca-base is define
    - MINOR: lua: Add flags to configure logging behaviour
    - MINOR: lua: change tune.lua.log.stderr default from 'on' to 'auto'
    - BUG/MINOR: backend: fix wrong BUG_ON for avail conn
    - BUG/MAJOR: backend: fix idle conn crash under low FD
    - MINOR: backend: refactor insertion in avail conns tree
    - DEBUG: mux-h2/flags: fix list of h2c flags used by the flags decoder
    - BUG/MEDIUM: server/log: "mode log" after server keyword causes crash
    - MINOR: connection: add conn_pr_mode_to_proto_mode() helper func
    - BUG/MEDIUM: server: "proto" not working for dynamic servers
    - MINOR: server: add helper function to detach server from proxy list
    - DEBUG: add a tainted flag when ha_panic() is called
    - DEBUG: lua: add tainted flags for stuck Lua contexts
    - DEBUG: pools: detect that malloc_trim() is in progress
    - BUG/MINOR: quic: do not consider idle timeout on CLOSING state
    - MINOR: frontend: implement a dedicated actconn increment function
    - BUG/MINOR: ssl: use a thread-safe sslconns increment
    - MEDIUM: quic: count quic_conn instance for maxconn
    - MEDIUM: quic: count quic_conn for global sslconns
    - BUG/MINOR: ssl: suboptimal certificate selection with TLSv1.3 and dual ECDSA/RSA
    - REGTESTS: ssl: update the filters test for TLSv1.3 and sigalgs
    - BUG/MINOR: mux-quic: fix early close if unset client timeout
    - BUG/MEDIUM: ssl: segfault when cipher is NULL
    - BUG/MINOR: tcpcheck: Report hexstring instead of binary one on check failure
    - MEDIUM: systemd: be more verbose about the reload
    - MINOR: sample: Add fetcher for getting all cookie names
    - BUG/MINOR: proto_reverse_connect: support SNI on active connect
    - MINOR: proxy/stktable: add resolve_stick_rule helper function
    - BUG/MINOR: stktable: missing free in parse_stick_table()
    - BUG/MINOR: cfgparse/stktable: fix error message on stktable_init() failure
    - MINOR: stktable: stktable_init() sets err_msg on error
    - MINOR: stktable: check if a type should be used as-is
    - MEDIUM: stktable/peers: "write-to" local table on peer updates
    - CI: github: update wolfSSL to 5.6.4
    - DOC: install: update the wolfSSL required version
    - MINOR: server: Add parser support for set-proxy-v2-tlv-fmt
    - MINOR: connection: Send out generic, user-defined server TLVs
    - BUG/MEDIUM: pattern: don't trim pools under lock in pat_ref_purge_range()
    - MINOR: mux-h2: always use h2_send() in h2_done_ff(), not h2_process()
    - OPTIM: mux-h2: call h2_send() directly from h2_snd_buf()
    - BUG/MINOR: server: remove some incorrect free() calls on null elements
2023-11-04 09:38:16 +01:00
Alexander Stephan
ce7501de79 MINOR: connection: Send out generic, user-defined server TLVs
To follow-up the implementation of the new set-proxy-v2-tlv-fmt
keyword in the server, the connection is updated to use the previously
allocated TLVs. If no value was specified, we send out an empty TLV.
As the feature is fully working with this commit, documentation and a
test for the server and default-server are added as well.
2023-11-04 04:56:59 +01:00
Aurelien DARRAGON
5158c0ff69 MEDIUM: stktable/peers: "write-to" local table on peer updates
In this patch, we add the possibility to declare on a table definition
("table" in peer section, or "stick-table" in proxy section) that we
want the remote/peer updates on that table to be pushed on a local
haproxy table in addition to the source table.

Consider this example:

  |peers mypeers
  |        peer local 127.0.0.1:3334
  |        peer clust 127.0.0.1:3333
  |        table t1.local type string size 10m store server_id,server_key expire 30s
  |        table t1.clust type string size 10m store server_id,server_key write-to mypeers/t1.local expire 30s

With this setup, we consider haproxy uses t1.local as cache/local table
for read and write operations, and that t1.clust is a remote table
containing datas processed from t1.local and similar tables from other
haproxy peers in a cluster setup. The t1.clust table will be used to
refresh the local/cache one via the "write-to" statement.

What will happen, is that every time haproxy will see entry updates for
the t1.clust table: it will overwrite t1.local table with fresh data and
will update the entry expiration timer. If t1.local entry doesn't exist
yet (key doesn't exist), it will automatically create it. Note that only
types that cannot be used for arithmetic ops will be handled, and this
to prevent processed values from the remote table from interfering with
computations based on values from the local table. (ie: prevent
cumulative counters from growing indefinitely).

"write-to" will only push supported types if they both exist in the source
and the target table. Be careful with server_id and server_key storage
because they are often declared implicitly when referencing a table in
sticking rules but it is required to declare them explicitly for them to
be pushed between a remote and a local table through "write-to" option.

Also note that the "write-to" target table should have the same type as
the source one, and that the key length should be strictly equal,
otherwise haproxy will raise an error due to the tables being
incompatibles. A table that is already being written to cannot be used
as a source table for a "write-to" target.

Thanks to this patch, it will now be possible to use sticking rules in
peer cluster context by using a local table as a local cache which
will be automatically refreshed by one or multiple remote table(s).

This commit depends on:
 - "MINOR: stktable: stktable_init() sets err_msg on error"
 - "MINOR: stktable: check if a type should be used as-is"
2023-11-03 17:30:30 +01:00
Ruei-Bang Chen
7a1ec235cd MINOR: sample: Add fetcher for getting all cookie names
This new fetcher can be used to extract the list of cookie names from
Cookie request header or from Set-Cookie response header depending on
the stream direction. There is an optional argument that can be used
as the delimiter (which is assumed to be the first character of the
argument) between cookie names. The default delimiter is comma (,).

Note that we will treat the Cookie request header as a semi-colon
separated list of cookies and each Set-Cookie response header as
a single cookie and extract the cookie names accordingly.
2023-11-03 09:57:06 +01:00
Tristan
8da0e45382 MINOR: lua: change tune.lua.log.stderr default from 'on' to 'auto'
After making it configurable in previous commit "MINOR: lua: Add flags
to configure logging behaviour", this patch changes the default value
of tune.lua.log.stderr from 'on' (unconditionally forward LUA logs to
stderr) to 'auto' (only forward LUA logs to stderr if logging via a
standard logger is disabled, or none is configured for the current context)

Since this is a change in behaviour, it shouldn't be backported
2023-10-25 07:49:03 +02:00
Tristan
97dacbbb86 MINOR: lua: Add flags to configure logging behaviour
Until now, messages printed from LUA log functions were sent both to
the any logger configured for the current proxy, and additionally to
stderr (in most cases)

This introduces two flags to configure LUA log handling:
- tune.lua.log.loggers to use standard loggers or not
- tune.lua.log.stderr to use stderr, or not, or only conditionally

This addresses github feature request #2316

This can be backported to 2.8 as it doesn't change previous behaviour.
2023-10-25 07:48:48 +02:00
Aleksandar Lazic
1428e7b66d DOC: internal: filters: fix reference to entities.pdf
In doc/internals/api/filters.txt was the referece to
doc/internals/entities.pdf which was delteted in the
past.
2023-10-23 11:33:45 +02:00
Willy Tarreau
c1ad57f0de [RELEASE] Released version 2.9-dev8
Released version 2.9-dev8 with the following main changes :
    - MINOR: ssl: add an explicit error when 'ciphersuites' are not supported
    - BUILD: ssl: enable 'ciphersuites' for WolfSSL
    - BUILD: ssl: add 'ssl_c_r_dn' fetch for WolfSSL
    - BUILD: ssl: add 'secure_memcmp' converter for WolfSSL and awslc
    - BUILD: ssl: enable keylog for awslc
    - CLEANUP: ssl: remove compat functions for openssl < 1.0.0
    - BUILD: ssl: enable keylog for WolfSSL
    - REGTESTS: pki: add a pki for SSL tests
    - REGTESTS: ssl: update common.pem with the new pki
    - REGTESTS: ssl: disable ssl_dh.vtc for WolfSSL
    - REGTESTS: wolfssl: temporarly disable some failing reg-tests
    - CI: ssl: add wolfssl to build-ssl.sh
    - CI: ssl: add git id support for wolfssl download
    - CI: github: add a wolfssl entry to the CI
    - CI: github: update wolfssl to git revision d83f2fa
    - CI: github: add awslc 1.16.0 to the push CI
    - BUG/MINOR: quic: Avoid crashing with unsupported cryptographic algos
    - REORG: quic: cleanup traces definition
    - BUG/MINOR: quic: reject packet with no frame
    - BUG/MEDIUM: mux-quic: fix RESET_STREAM on send-only stream
    - BUG/MINOR: mux-quic: support initial 0 max-stream-data
    - BUG/MINOR: h3: strengthen host/authority header parsing
    - CLEANUP: connection: drop an uneeded leftover cast
    - BUG/MAJOR: connection: make sure to always remove a connection from the tree
    - BUG/MINOR: quic: fix qc.cids access on quic-conn fail alloc
    - BUG/MINOR: quic: fix free on quic-conn fail alloc
    - BUG/MINOR: mux-quic: fix free on qcs-new fail alloc
    - BUG/MEDIUM: quic-conn: free unsent frames on retransmit to prevent crash
    - MEDIUM: tree-wide: logsrv struct becomes logger
    - MEDIUM: log: introduce log target
    - DOC: config: log <address> becomes log <target> in "log" related doc
    - MEDIUM: sink/log: stop relying on AF_UNSPEC for rings
    - MINOR: log: support explicit log target as argument in __do_send_log()
    - MINOR: log: remove the logger dependency in do_send_log()
    - MEDIUM: log/sink: simplify log header handling
    - MEDIUM: sink: inherit from caller fmt in ring_write() when rings didn't set one
    - MINOR: sink: add sink_new_from_srv() function
    - MAJOR: log: introduce log backends
    - MINOR: log/balance: support for the "sticky" lb algorithm
    - MINOR: log/balance: support for the "random" lb algorithm
    - MINOR: lbprm: support for the "none" hash-type function
    - MINOR: lbprm: compute the hash avalanche in gen_hash()
    - MINOR: sample: add sample_process_cnv() function
    - MEDIUM: log/balance: support for the "hash" lb algorithm
    - REGTEST: add a test for log-backend used as a log target
    - MINOR: server: introduce "log-bufsize" kw
    - BUG/MEDIUM: stconn: Report a send activity everytime data were sent
    - BUG/MEDIUM: applet: Report a send activity everytime data were sent
    - BUG/MINOR: mux-h1: Send a 400-bad-request on shutdown before the first request
    - MINOR: support for http-response set-timeout
    - BUG/MINOR: mux-h2: make up other blocked streams upon removal from list
    - DEBUG: pool: store the memprof bin on alloc() and update it on free()
    - BUG/MEDIUM: quic_conn: let the scheduler kill the task when needed
    - CLEANUP: hlua: Remove dead-code on error path in hlua_socket_new()
    - BUG/MEDIUM: mux-h1: do not forget TLR/EOT even when no data is sent
    - BUG/MINOR: htpp-ana/stats: Specify that HTX redirect messages have a C-L header
    - BUG/MEDIUM: mux-h2: Don't report an error on shutr if a shutw is pending
    - MEDIUM: stconn/channel: Move pipes used for the splicing in the SE descriptors
    - MINOR: stconn: Start to introduce mux-to-mux fast-forwarding notion
    - MINOR: stconn: Extend iobuf to handle a buffer in addition to a pipe
    - MINOR: connection: Add new mux callbacks to perform data fast-forwarding
    - MINOR: stconn: Temporarily remove kernel splicing support
    - MINOR: mux-pt: Temporarily remove splicing support
    - MINOR: mux-h1: Temporarily remove splicing support
    - MINOR: connection: Remove mux callbacks about splicing
    - MEDIUM: stconn: Add mux-to-mux fast-forward support
    - MINOR: mux-h1: Use HTX extra field only for responses with known length
    - MEDIUM: mux-h1: Properly handle state transitions of chunked outgoing messages
    - MEDIUM: raw-sock: Specifiy amount of data to send via snd_pipe callback
    - MINOR: mux-h1: Add function to add size of a chunk to an outgoind message
    - MEDIUM: mux-h1: Simplify zero-copy on sending path
    - MEDIUM: mux-h1: Simplify payload formatting based on HTX blocks on sending path
    - MEDIUM: mux-h1: Add fast-forwarding support
    - MINOR: h2: Set the BODYLESS_RESP flag on the HTX start-line if necessary
    - MEDIUM: mux-h2: Add consumer-side fast-forwarding support
    - MEDIUM: channel: don't look at iobuf to report an empty channel
    - MINOR: tree-wide: Only rely on co_data() to check channel emptyness
    - REGTESTS: Reenable HTTP tests about splicing
    - CLEAN: mux-h1: Remove useless __maybe_unused attribute on h1_make_chunk()
    - MEDIUM: mux-pt: Add fast-forwarding support
    - MINOR: global: Add an option to disable the zero-copy forwarding
    - BUILD: mux-h1: Fix build without kernel splicing support
    - REORG: stconn/muxes: Rename init step in fast-forwarding
    - MINOR: dgram: allow to set rcv/sndbuf for dgram sockets as well
    - BUG/MINOR: mux-h2: fix http-request and http-keep-alive timeouts again
    - BUG/MINOR: trace: fix trace parser error reporting
    - BUG/MEDIUM: peers: Be sure to always refresh recconnect timer in sync task
    - BUG/MEDIUM: peers: Fix synchro for huge number of tables
    - MINOR: cfgparse: forbid mixing reverse and standard listeners
    - MINOR: listener: add nbconn kw for reverse connect
    - MINOR: server: convert @reverse to rev@ standard format
    - MINOR: cfgparse: rename "rev@" prefix to "rhttp@"
    - REGTESTS: remove maxconn from rhttp bind line
    - MINOR: listener: forbid most keywords for reverse HTTP bind
    - MINOR: sample: Added support for Arrays in sample_conv_json_query in sample.c
    - MINOR: mux-h2/traces: explicitly show the error/refused stream states
    - MINOR: mux-h2/traces: clarify the "rejected H2 request" event
    - BUG/MINOR: mux-h2: commit the current stream ID even on reject
    - BUG/MINOR: mux-h2: update tracked counters with req cnt/req err
2023-10-20 21:36:47 +02:00
Jens Popp
f66b9f6018 MINOR: sample: Added support for Arrays in sample_conv_json_query in sample.c
Method now returns the content of Json Arrays, if it is specified in
Json Path as String. The start and end character is a square bracket. Any
complex object in the array is returned as Json, so that you might get Arrays
of Array or objects. Only recommended for Arrays of simple types (e.g.,
String or int) which will be returned as CSV String. Also updated
documentation and fixed issue with parenthesis and other changes from
comments.

This patch was discussed in issue #2281.

Signed-off-by: William Lallemand <wlallemand@haproxy.com>
2023-10-20 18:42:05 +02:00
Amaury Denoyelle
e05edf71df MINOR: cfgparse: rename "rev@" prefix to "rhttp@"
'rev@' was used to specify a bind/server used with reverse HTTP
transport. This notation was deemed not explicit enough. Rename it
'rhttp@' instead.
2023-10-20 14:44:37 +02:00
Amaury Denoyelle
9d4c7c1151 MINOR: server: convert @reverse to rev@ standard format
Remove the recently introduced '@reverse' notation for HTTP reverse
servers. Instead, reuse the 'rev@' prefix already defined for bind
lines.
2023-10-20 14:44:37 +02:00
Amaury Denoyelle
3222047a14 MINOR: listener: add nbconn kw for reverse connect
Previously, maxconn keyword was reused for a specific usage on reverse
HTTP binds to specify the number of active connect to proceed. To avoid
confusion, introduce a new dedicated keyword 'nbconn' which is specific
to reverse HTTP bind.

This new keyword is forbidden for non-reverse listener. A fatal error is
emitted during config parsing if this rule is not respected. It's safe
because it's also forbidden to mix standard and reverse addresses on the
same bind line.

Internally, nbconn value will be reassigned to 'maxconn' member of
bind_conf structure. This ensures that listener layer will automatically
reenable the preconnect task each time a connection is closed.
2023-10-20 14:44:37 +02:00
Willy Tarreau
91ed52976c MINOR: dgram: allow to set rcv/sndbuf for dgram sockets as well
tune.rcvbuf.client and tune.rcvbuf.server are not suitable for shared
dgram sockets because they're per connection so their units are not the
same. However, QUIC's listener and log servers are not connected and
take per-thread or per-process traffic where a socket log buffer might
be too small, causing undesirable packet losses and retransmits in the
case of QUIC. This essentially manifests in listener mode with new
connections taking a lot of time to set up under heavy traffic due to
the small queues causing delays. Let's add a few new settings allowing
to set these shared socket sizes on the frontend and backend side (which
reminds that these are per-front/back and not per client/server hence
not per connection).
2023-10-18 17:01:19 +02:00
Christopher Faulet
023564b685 MINOR: global: Add an option to disable the zero-copy forwarding
The zero-copy forwarding or the mux-to-mux forwarding is a way to
fast-forward data without using the channels buffers. Data are transferred
from a mux to the other one. The kernel splicing is an optimization of the
zero-copy forwarding. But it can also use normal buffers (but not channels
ones). This way, it could be possible to fast-forward data with muxes not
supporting the kernel splicing (H2 and H3 muxes) but also with applets.

However, this mode can introduce regressions or bugs in future (just like
the kernel splicing). Thus, It could be usefull to disable this optim. To do
so, in configuration, the global tune settting
'tune.disable-zero-copy-forwarding' may be set in a global section or the
'-dZ' command line parameter may be used to start HAProxy. Of course, this
also disables the kernel splicing.
2023-10-17 18:51:13 +02:00
Vladimir Vdovin
70d2d9aefc MINOR: support for http-response set-timeout
Added set-timeout action for http-response. Adapted reg-tests and
documentation.
2023-10-17 08:27:33 +02:00
Aurelien DARRAGON
94d0f77deb MINOR: server: introduce "log-bufsize" kw
"log-bufsize" may now be used for a log server (in a log backend) to
configure the bufsize of implicit ring associated to the server (which
defaults to BUFSIZE).
2023-10-13 10:05:07 +02:00
Aurelien DARRAGON
b30bd7adba MEDIUM: log/balance: support for the "hash" lb algorithm
hash lb algorithm can be configured with the "log-balance hash <cnv_list>"
directive. With this algorithm, the user specifies a converter list with
<cnv_list>.

The produced log message will be passed as-is to the provided converter
list, and the resulting hash will be used to select the log server that
will receive the log message.
2023-10-13 10:05:06 +02:00
Aurelien DARRAGON
a7563158f7 MINOR: lbprm: support for the "none" hash-type function
Allow the use of the "none" hash-type function so that the key resulting
from the sample expression is directly used as the hash.

This can be useful to do the hashing manually using available hashing
converters, or even custom ones, and then inform haproxy that it can
directly rely on the sample expression result which is explictly handled
as an integer in this case.
2023-10-13 10:05:06 +02:00
Aurelien DARRAGON
e0b4660015 MINOR: log/balance: support for the "random" lb algorithm
In this patch we add basic support for the random algorithm:

random algorithm picks a random server using the result of the
statistical_prng() function as if it was a hash key to then compute the
related server ID.

There is no support for the <draw> parameter (which is implemented for
tcp/http load-balancing), because we don't have the required metrics to
evaluate server's load in log backends for the moment. Plus it would add
more complexity to the __do_send_log_backend() function so we'll keep it
this way for now but this might be needed in the future.
2023-10-13 10:05:06 +02:00
Aurelien DARRAGON
26f73dbcbb MINOR: log/balance: support for the "sticky" lb algorithm
sticky algorithm always tries to send log messages to the first server in
the farm. The server will stay in front during queue and dequeue
operations (no other server can steal its place), unless it becomes
unavailable, in which case it will be replaced by another server from
the tree.
2023-10-13 10:05:06 +02:00
Aurelien DARRAGON
9a74a6cb17 MAJOR: log: introduce log backends
Using "mode log" in a backend section turns the proxy in a log backend
which can be used to log-balance logs between multiple log targets
(udp or tcp servers)

log backends can be used as regular log targets using the log directive
with "backend@be_name" prefix, like so:

  | log backend@mybackend local0

A log backend will distribute log messages to servers according to the
log load-balancing algorithm that can be set using the "log-balance"
option from the log backend section. For now, only the roundrobin
algorithm is supported and set by default.
2023-10-13 10:05:06 +02:00
Aurelien DARRAGON
b218ce3469 DOC: config: log <address> becomes log <target> in "log" related doc
This is a follow up of the previous commit to emphasize that "log"
directive allows to provide a log target which may directly be a server
address but may also be a log transport facility such as rings. Thus we
use the term "target" instead of "address" to make it more generic.
2023-10-13 10:05:06 +02:00
Willy Tarreau
7f1a3ee5d7 [RELEASE] Released version 2.9-dev7
Released version 2.9-dev7 with the following main changes :
    - MINOR: support for http-request set-timeout client
    - BUG/MINOR: mux-quic: remove full demux flag on ncbuf release
    - CLEANUP: freq_ctr: make all freq_ctr readers take a const
    - CLEANUP: stream: make the dump code not depend on the CLI appctx
    - MINOR: stream: split stats_dump_full_strm_to_buffer() in two
    - CLEANUP: stream: use const filters in the dump function
    - CLEANUP: stream: make strm_dump_to_buffer() take a const stream
    - MINOR: stream: make strm_dump_to_buffer() take an arbitrary buffer
    - MINOR: stream: make strm_dump_to_buffer() show the list of filters
    - MINOR: stream: make stream_dump() always multi-line
    - MINOR: streams: add support for line prefixes to strm_dump_to_buffer()
    - MEDIUM: stream: now provide full stream dumps in case of loops
    - MINOR: debug: use the more detailed stream dump in panics
    - CLEANUP: stream: remove the now unused stream_dump() function
    - Revert "BUG/MEDIUM: quic: missing check of dcid for init pkt including a token"
    - MINOR: stream: fix output alignment of stuck thread dumps
    - BUG/MINOR: proto_reverse_connect: fix FD leak on connection error
    - BUG/MINOR: tcp_act: fix attach-srv rule ACL parsing
    - MINOR: connection: define error for reverse connect
    - MINOR: connection: define mux flag for reverse support
    - MINOR: tcp_act: remove limitation on protocol for attach-srv
    - BUG/MINOR: proto_reverse_connect: fix FD leak upon connect
    - BUG/MAJOR: plock: fix major bug in pl_take_w() introduced with EBO
    - Revert "MEDIUM: sample: Small fix in function check_operator for eror reporting"
    - DOC: sample: Add a comment in 'check_operator' to explain why 'vars_check_arg' should ignore the 'err' buffer
    - DEV: sslkeylogger: handle file opening error
    - MINOR: quic: define quic-socket bind setting
    - MINOR: quic: handle perm error on bind during runtime
    - MINOR: backend: refactor specific source address allocation
    - MINOR: proto_reverse_connect: support source address setting
    - BUILD: pool: Fix GCC error about potential null pointer dereference
    - MINOR: hlua: Set context's appctx when the lua socket is created
    - MINOR: hlua: Don't preform operations on a not connected socket
    - MINOR: hlua: Save the lua socket's timeout in its context
    - MINOR: hlua: Save the lua socket's server in its context
    - MINOR: hlua: Test the hlua struct first when the lua socket is connecting
    - BUG/MEDIUM: hlua: Initialize appctx used by a lua socket on connect only
    - DEBUG: mux-h1: Fix event label from trace messages about payload formatting
    - BUG/MINOR: mux-h1: Handle read0 in rcv_pipe() only when data receipt was tried
    - BUG/MINOR: mux-h1: Ignore C-L when sending H1 messages if T-E is also set
    - BUG/MEDIUM: h1: Ignore C-L value in the H1 parser if T-E is also set
    - REGTESTS: filters: Don't set C-L header in the successful response to CONNECT
    - MINOR: mux-h1: Add flags if outgoing msg contains a header about its payload
    - MINOR: mux-h1: Rely on H1S_F_HAVE_CHNK to add T-E in outgoing messages
    - BUG/MEDIUM: mux-h1: Add C-L header in outgoing message if it was removed
    - BUG/MEDIUM: mux-h1; Ignore headers modifications about payload representation
    - BUG/MINOR: h1-htx: Keep flags about C-L/T-E during HEAD response parsing
    - MINOR: h1-htx: Declare successful tunnel establishment as bodyless
    - BUILD: quic: allow USE_QUIC to work with AWSLC
    - CI: github: add USE_QUIC=1 to aws-lc build
    - BUG/MINOR: hq-interop: simplify parser requirement
    - MEDIUM: cache: Add "Origin" header to secondary cache key
    - MINOR: haproxy: permit to register features during boot
    - MINOR: tcp_rules: tcp-{request,response} requires TCP or HTTP mode
    - MINOR: stktable: "stick" requires TCP or HTTP mode
    - MINOR: filter: "filter" requires TCP or HTTP mode
    - MINOR: backend/balance: "balance" requires TCP or HTTP mode
    - MINOR: flt_http_comp: "compression" requires TCP or HTTP mode
    - MINOR: http_htx/errors: prevent the use of some keywords when not in tcp/http mode
    - MINOR: fcgi-app: "use-fcgi-app" requires TCP or HTTP mode
    - MINOR: cfgparse-listen: "http-send-name-header" requires TCP or HTTP mode
    - MINOR: cfgparse-listen: "dynamic-cookie-key" requires TCP or HTTP mode
    - MINOR: proxy: dynamic-cookie CLIs require TCP or HTTP mode
    - MINOR: cfgparse-listen: "http-reuse" requires TCP or HTTP mode
    - MINOR: proxy: report a warning for max_ka_queue in proxy_cfg_ensure_no_http()
    - MINOR: cfgparse-listen: warn when use-server rules is used in wrong mode
    - DOC: config: unify "log" directive doc
    - MINOR: sink/log: fix some typos around postparsing logic
    - MINOR: sink: remove useless check after sink creation
    - MINOR: sink: don't rely on p->parent in sink appctx
    - MINOR: sink: don't rely on forward_px to init sink forwarding
    - MINOR: sink: refine forward_px usage
    - MINOR: sink: function to add new sink servers
    - BUG/MEDIUM: stconn: Fix comparison sign in sc_need_room()
    - BUG/MEDIUM: actions: always apply a longest match on prefix lookup
2023-10-06 22:03:17 +02:00
Aurelien DARRAGON
e5d23d8676 DOC: config: unify "log" directive doc
"log" directive description was found 2 times in the configuration file:

First, in 3.1 in the "global parameters" chapter, and then in 4.2 in the
per-proxy keyword options.

Both descriptions are almost identical: having to maintain the "same"
documentation in 2 different places is error-prone. Due to this, some
precisions have been added in one of them, and were missing from
the other, and vice-versa, probably because one didn't see that the
"log" directive was also documented elsewhere.

To prevent the 2 descriptions from further diverging, and make it easier
to maintain, we merge them in the per-proxy "log" directive description
(in 4.2 chapter), and we add a pointer to it in the global "log" to
encourage the user to refer to the per-proxy "log" documentation for
usage details.
2023-10-06 15:34:30 +02:00
Remi Tricot-Le Breton
a5e96425a2 MEDIUM: cache: Add "Origin" header to secondary cache key
This patch add a hash of the Origin header to the cache's secondary key.
This enables to manage store responses that have a "Vary: Origin" header
in the cache when vary is enabled.
This cannot be considered as a means to manage CORS requests though, it
only processes the Origin header and hashes the presented value without
any form of URI normalization.

This need was expressed by Philipp Hossner in GitHub issue #251.

Co-Authored-by: Philipp Hossner <philipp.hossner@posteo.de>
2023-10-05 10:53:54 +02:00
Amaury Denoyelle
3ef6df7387 MINOR: quic: define quic-socket bind setting
Define a new bind option quic-socket :
  quic-socket [ connection | listener ]

This new setting works in conjunction with the existing configuration
global tune.quic.socket-owner and reuse the same semantics.

The purpose of this setting is to allow to disable connection socket
usage on listener instances individually. This will notably be useful
when needing to deactivating it when encountered a fatal permission
error on bind() at runtime.
2023-10-03 16:49:26 +02:00
Vladimir Vdovin
f8b81f6eb7 MINOR: support for http-request set-timeout client
Added set-timeout for frontend side of session, so it can be used to set
custom per-client timeouts if needed. Added cur_client_timeout to fetch
client timeout samples.
2023-09-28 08:49:22 +02:00
Willy Tarreau
f75a369009 [RELEASE] Released version 2.9-dev6
Released version 2.9-dev6 with the following main changes :
    - BUG/MINOR: quic: fdtab array underflow access
    - DEBUG: pools: always record the caller for uncached allocs as well
    - DEBUG: pools: pass the caller pointer to the check functions and macros
    - DEBUG: pools: make pool_check_pattern() take a pointer to the pool
    - DEBUG: pools: inspect pools on fatal error and dump information found
    - BUG/MEDIUM: quic: quic_cc_conn ->cntrs counters unreachable
    - DEBUG: pools: also print the item's pointer when crashing
    - DEBUG: pools: also print the value of the tag when it doesn't match
    - DEBUG: pools: print the contents surrounding the expected tag location
    - MEDIUM: pools: refine pool size rounding
    - BUG/MEDIUM: hlua: don't pass stale nargs argument to lua_resume()
    - BUG/MINOR: hlua/init: coroutine may not resume itself
    - BUG/MEDIUM: mux-fcgi: Don't swap trash and dbuf when handling STDERR records
    - BUG/MINOR: promex: fix backend_agg_check_status
    - BUG/MEDIUM: master/cli: Pin the master CLI on the first thread of the group 1
    - MAJOR: import: update mt_list to support exponential back-off
    - CLEANUP: pools: simplify the pool expression when no pool was matched in dump
    - MINOR: samples: implement bytes_in and bytes_out samples
    - DOC: configuration: add %[req.ver] sample to %HV
    - BUG/MINOR: quic: Leak of frames to send.
    - DOC: configuration: add %[query] to %HQ
    - BUG/MINOR: freq_ctr: fix possible negative rate with the scaled API
    - BUG/MAJOR: mux-h2: Report a protocol error for any DATA frame before headers
    - BUILD: quic: fix build on centos 8 and USE_QUIC_OPENSSL_COMPAT
    - Revert "MAJOR: import: update mt_list to support exponential back-off"
    - BUG/MINOR: server: add missing free for server->rdr_pfx
    - REGTESTS: ssl: skip OCSP test w/ WolfSSL
    - REGTESTS: ssl: skip generate-certificates test w/ wolfSSL
    - MINOR: logs: clarify the check of the log range
    - MINOR: log: remove the unused curr_idx in struct smp_log_range
    - CLEANUP: logs: rename a confusing local variable "curr_rg" to "smp_rg"
    - MINOR: logs: use a single index to store the current range and index
    - MEDIUM: logs: atomically check and update the log sample index
    - CLEANUP: ring: rename the ring lock "RING_LOCK" instead of "LOGSRV_LOCK"
    - BUG/MEDIUM: http-ana: Try to handle response before handling server abort
    - MEDIUM: tools/ip: v4tov6() and v6tov4() rework
    - MINOR: pattern/ip: offload ip conversion logic to helper functions
    - MINOR: pattern: fix pat_{parse,match}_ip() function comments
    - MINOR: pattern/ip: simplify pat_match_ip() function
    - BUG/MEDIUM: server/cli: don't delete a dynamic server that has streams
    - MINOR: hlua: Add support for the "http-after-res" action
    - BUG/MINOR: proto_reverse_connect: fix preconnect with startup name resolution
    - MINOR: proto_reverse_connect: prevent transparent server for pre-connect
    - CI: cirrus-ci: display gdb bt if any
    - MEDIUM: sample: Enhances converter "bytes" to take variable names as arguments
    - MEDIUM: sample: Small fix in function check_operator for eror reporting
    - MINOR: quic: handle external extra CIDs generator.
    - BUG/MINOR: proto_reverse_connect: set default maxconn
    - MINOR: proto_reverse_connect: refactor preconnect failure
    - MINOR: proto_reverse_connect: remove unneeded wakeup
    - MINOR: proto_reverse_connect: emit log for preconnect
2023-09-22 23:11:31 +02:00
Lokesh Jindal
915e48675a MEDIUM: sample: Enhances converter "bytes" to take variable names as arguments
Prior to this commit, converter "bytes" takes only integer values as
arguments.  After this commit, it can take variable names as inputs.
This allows us to dynamically determine the offset/length and capture
them in variables.  These variables can then be used with the converter.
Example use case: parsing a token present in a request header.
2023-09-22 08:48:51 +02:00
Sébastien Gross
6a9ba85322 MINOR: hlua: Add support for the "http-after-res" action
This commit introduces support for the "http-after-res" action in
hlua, enabling the invocation of a Lua function in a
"http-after-response" rule. With this enhancement, a Lua action can be
registered using the "http-after-res" action type:

    core.register_action('myaction', {'http-after-res'}, myaction)

A new "lua.myaction" is created and can be invoked in a
"http-after-response" rule:

    http-after-response lua.myaction

This addition provides greater flexibility and extensibility in
handling post-response actions using Lua.

This commit depends on:
 - 4457783 ("MINOR: http_ana: position the FINAL flag for http_after_res execution")

Signed-off-by: Sébastien Gross <sgross@haproxy.com>
2023-09-21 16:31:20 +02:00
Willy Tarreau
6cbb5a057b Revert "MAJOR: import: update mt_list to support exponential back-off"
This reverts commit c618ed5ff4.

The list iterator is broken. As found by Fred, running QUIC single-
threaded shows that only the first connection is accepted because the
accepter relies on the element being initialized once detached (which
is expected and matches what MT_LIST_DELETE_SAFE() used to do before).
However while doing this in the quic_sock code seems to work, doing it
inside the macro show total breakage and the unit test doesn't work
anymore (random crashes). Thus it looks like the fix is not trivial,
let's roll this back for the time it will take to fix the loop.
2023-09-15 17:13:43 +02:00
William Lallemand
148f145d32 DOC: configuration: add %[query] to %HQ
add %[query] to the alternative sample fetch for the logs
2023-09-13 15:56:23 +02:00
William Lallemand
2a19f7c43b DOC: configuration: add %[req.ver] sample to %HV
add %[req.ver] to the alternative sample fetch for the logs
2023-09-13 14:54:50 +02:00
William Lallemand
c7424a1bac MINOR: samples: implement bytes_in and bytes_out samples
%[bytes_in] and %[bytes_out] are equivalent to %U and %B tags in
log-format.
2023-09-13 14:54:50 +02:00
Willy Tarreau
c618ed5ff4 MAJOR: import: update mt_list to support exponential back-off
The new mt_list code supports exponential back-off on conflict, which
is important for use cases where there is contention on a large number
of threads. The API evolved a little bit and required some updates:

  - mt_list_for_each_entry_safe() is now in upper case to explicitly
    show that it is a macro, and only uses the back element, doesn't
    require a secondary pointer for deletes anymore.

  - MT_LIST_DELETE_SAFE() doesn't exist anymore, instead one just has
    to set the list iterator to NULL so that it is not re-inserted
    into the list and the list is spliced there. One must be careful
    because it was usually performed before freeing the element. Now
    instead the element must be nulled before the continue/break.

  - MT_LIST_LOCK_ELT() and MT_LIST_UNLOCK_ELT() have always been
    unclear. They were replaced by mt_list_cut_around() and
    mt_list_connect_elem() which more explicitly detach the element
    and reconnect it into the list.

  - MT_LIST_APPEND_LOCKED() was only in haproxy so it was left as-is
    in list.h. It may however possibly benefit from being upstreamed.

This required tiny adaptations to event_hdl.c and quic_sock.c. The
test case was updated and the API doc added. Note that in order to
keep include files small, the struct mt_list definition remains in
list-t.h (par of the internal API) and was ifdef'd out in mt_list.h.

A test on QUIC with both quictls 1.1.1 and wolfssl 5.6.3 on ARM64 with
80 threads shows a drastic reduction of CPU usage thanks to this and
the refined memory barriers. Please note that the CPU usage on OpenSSL
3.0.9 is significantly higher due to the excessive use of atomic ops
by openssl, but 3.1 is only slightly above 1.1.1 though:

  - before: 35 Gbps, 3.5 Mpps, 7800% CPU
  - after:  41 Gbps, 4.2 Mpps, 2900% CPU
2023-09-13 11:50:33 +02:00
Willy Tarreau
8b7841ff7a [RELEASE] Released version 2.9-dev5
Released version 2.9-dev5 with the following main changes :
    - BUG/MEDIUM: mux-h2: fix crash when checking for reverse connection after error
    - BUILD: import: guard plock.h against multiple inclusion
    - BUILD: pools: import plock.h to build even without thread support
    - BUG/MINOR: ssl/cli: can't find ".crt" files when replacing a certificate
    - BUG/MINOR: stream: protect stream_dump() against incomplete streams
    - DOC: config: mention uid dependency on the tune.quic.socket-owner option
    - MEDIUM: capabilities: enable support for Linux capabilities
    - CLEANUP/MINOR: connection: Improve consistency of PPv2 related constants
    - MEDIUM: connection: Generic, list-based allocation and look-up of PPv2 TLVs
    - MEDIUM: sample: Add fetch for arbitrary TLVs
    - MINOR: sample: Refactor fc_pp_authority by wrapping the generic TLV fetch
    - MINOR: sample: Refactor fc_pp_unique_id by wrapping the generic TLV fetch
    - MINOR: sample: Add common TLV types as constants for fc_pp_tlv
    - MINOR: ssl_sock: avoid iterating realloc(+1) on stored context
    - DOC: ssl: add some comments about the non-obvious session allocation stuff
    - CLEANUP: ssl: keep a pointer to the server in ssl_sock_init()
    - MEDIUM: ssl_sock: always use the SSL's server name, not the one from the tid
    - MEDIUM: server/ssl: place an rwlock in the per-thread ssl server session
    - MINOR: server/ssl: maintain an index of the last known valid SSL session
    - MINOR: server/ssl: clear the shared good session index on failure
    - MEDIUM: server/ssl: pick another thread's session when we have none yet
    - MINOR: activity: report the current run queue size
    - BUG/MINOR: checks: do not queue/wake a bounced check
    - MINOR: checks: start the checks in sleeping state
    - MINOR: checks: pin the check to its thread upon wakeup
    - MINOR: check: remember when we migrate a check
    - MINOR: check/activity: collect some per-thread check activity stats
    - MINOR: checks: maintain counters of active checks per thread
    - MINOR: check: also consider the random other thread's active checks
    - MEDIUM: checks: search more aggressively for another thread on overload
    - MEDIUM: checks: implement a queue in order to limit concurrent checks
    - MINOR: checks: also consider the thread's queue for rebalancing
    - DEBUG: applet: Properly report opposite SC expiration dates in traces
    - BUG/MEDIUM: stconn: Update stream expiration date on blocked sends
    - BUG/MINOR: stconn: Don't report blocked sends during connection establishment
    - BUG/MEDIUM: stconn: Wake applets on sending path if there is a pending shutdown
    - BUG/MEDIUM: stconn: Don't block sends if there is a pending shutdown
    - BUG/MINOR: quic: Possible skipped RTT sampling
    - MINOR: quic: Add a trace to quic_release_frm()
    - BUG/MAJOR: quic: Really ignore malformed ACK frames.
    - BUG/MINOR: quic: Unchecked pointer to packet number space dereferenced
    - BUG/MEDIUM: connection: fix pool free regression with recent ppv2 TLV patches
    - BUG/MEDIUM: h1-htx: Ensure chunked parsing with full output buffer
    - BUG/MINOR: stream: further protect stream_dump() against incomplete sessions
    - DOC: configuration: update examples for req.ver
    - MINOR: properly mark the end of the CLI command in error messages
    - BUILD: ssl: Build with new cryptographic library AWS-LC
    - REGTESTS: ssl: skip ssl_dh test with AWS-LC
    - BUILD: bug: make BUG_ON() void to avoid a rare warning
    - BUILD: checks: shut up yet another stupid gcc warning
    - MINOR: cpuset: add ha_cpuset_isset() to check for the presence of a CPU in a set
    - MINOR: cpuset: add ha_cpuset_or() to bitwise-OR two CPU sets
    - MINOR: cpuset: centralize a reliable bound cpu detection
    - MEDIUM: threads: detect incomplete CPU bindings
    - MEDIUM: threads: detect excessive thread counts vs cpu-map
    - BUILD: quic: Compilation issue on 32-bits systems with quic_may_send_bytes()
    - BUG/MINOR: quic: Unchecked pointer to Handshake packet number space
    - MINOR: global: export the display_version() symbol
    - MEDIUM: mworker: display a more accessible message when a worker crash
    - MINOR: httpclient: allow to configure the retries
    - MINOR: httpclient: allow to configure the timeout.connect
    - BUG/MINOR: quic: Wrong RTT adjusments
    - BUG/MINOR: quic: Wrong RTT computation (srtt and rrt_var)
    - BUG/MINOR: stconn: Don't inhibit shutdown on connection on error
    - BUG/MEDIUM: applet: Fix API for function to push new data in channels buffer
    - BUG/MEDIUM: stconn: Report read activity when a stream is attached to front SC
    - BUG/MEDIUM: applet: Report an error if applet request more room on aborted SC
    - BUG/MEDIUM: stconn/stream: Forward shutdown on write timeout
    - NUG/MEDIUM: stconn: Always update stream's expiration date after I/O
    - BUG/MINOR: applet: Always expect data when CLI is waiting for a new command
    - BUG/MINOR: ring/cli: Don't expect input data when showing events
    - BUG/MINOR: quic: Dereferenced unchecked pointer to Handshke packet number space
    - BUG/MINOR: hlua/action: incorrect message on E_YIELD error
    - MINOR: http_ana: position the FINAL flag for http_after_res execution
    - CI: scripts: add support to build-ssl.sh to download and build AWS-LC
    - CI: add support to matrix.py to determine the latest AWS-LC release
    - CI: Update matrix.py so all code is contained in functions.
    - CI: github: Add a weekly CI run building with AWS-LC
    - MINOR: ring: add a function to compute max ring payload
    - BUG/MEDIUM: ring: adjust maxlen consistency check
    - MINOR: sink: simplify post_sink_resolve function
    - MINOR: log/sink: detect when log maxlen exceeds sink size
    - MINOR: sink: inform the user when logs will be implicitly truncated
    - MEDIUM: sink: don't perform implicit truncations when maxlen is not set
    - MINOR: log: move log-forwarders cleanup in log.c
    - MEDIUM: httpclient/logs: rely on per-proxy post-check instead of global one
    - MINOR: log: add dup_logsrv() helper function
    - MEDIUM: log/sink: make logsrv postparsing more generic
    - MEDIUM: fcgi-app: properly postresolve logsrvs
    - MEDIUM: spoe-agent: properly postresolve log rings
    - MINOR: sink: add helper function to deallocate sink struct
    - MEDIUM: sink/ring: introduce high level ring creation helper function
    - MEDIUM: sink: add sink_finalize() function
    - CLEANUP: log: remove unnecessary trim in __do_send_log
    - MINOR: cache: Change hash function in default normalizer used in case of "vary"
    - MINOR: tasks/stats: report the number of niced tasks in "show info"
    - CI: Update to actions/checkout@v4
    - MINOR: ssl: add support for 'curves' keyword on server lines
    - BUG/MINOR: quic: Wrong cluster secret initialization
    - CLEANUP: quic: Remove useless free_quic_tx_pkts() function.
    - MEDIUM: init: initialize the trash earlier
    - MINOR: tools: add function read_line_to_trash() to read a line of a file
    - MINOR: cfgparse: use read_line_from_trash() to read from /sys
    - MEDIUM: cfgparse: assign NUMA affinity to cpu-maps
    - MINOR: cpuset: dynamically allocate cpu_map
    - REORG: cpuset: move parse_cpu_set() and parse_cpumap() to cpuset.c
    - CI: musl: highlight section if there are coredumps
    - CI: musl: drop shopt in workflow invocation
2023-09-08 19:21:45 +02:00
Willy Tarreau
5f10176e2c MEDIUM: init: initialize the trash earlier
More and more utility functions rely on the trash while most of the init
code doesn't have access to it because it's initialized very late (in
PRE_CHECK for the initial one). It's a pool, and it purposely supports
being reallocated, so let's initialize it in STG_POOL so that early
STG_INIT code can at least use it.
2023-09-08 16:25:19 +02:00