Commit Graph

2387 Commits

Author SHA1 Message Date
Christopher Faulet
c960a3b60f BUG/MINOR: pool/stats: Use ullong to report total pool usage in bytes in stats
The same change was already performed for the cli. The stats applet and the
prometheus exporter are also concerned. Both use the stats API and rely on
pool functions to get total pool usage in bytes. pool_total_allocated() and
pool_total_used() must return 64 bits unsigned integer to avoid any wrapping
around 4G.

This may be backported to all versions.
2022-12-22 13:46:21 +01:00
Remi Tricot-Le Breton
d5d7796637 DOC: ssl: Add documentation for ocsp-update option
This adds the documentation for the ocsp-update option.
2022-12-21 11:21:07 +01:00
Remi Tricot-Le Breton
eeaa29b36b MINOR: ssl: Add "update ssl ocsp-response" cli command
The new "update ssl ocsp-response <certfile>" CLI command allows to
update the stored OCSP response for a given certificate. It relies on
the http_client which is used to send an HTTP request to the OCSP
responder whose URI can be extracted from the certificate.
This command won't work for a certificate that did not have a stored
OCSP response yet.
2022-12-21 11:21:07 +01:00
Willy Tarreau
284cfc67b8 MINOR: pool: make the thread-local hot cache size configurable
Till now it was only possible to change the thread local hot cache size
at build time using CONFIG_HAP_POOL_CACHE_SIZE. But along benchmarks it
was sometimes noticed a huge contention in the lower level memory
allocators indicating that larger caches could be beneficial, especially
on machines with large L2 CPUs.

Given that the checks against this value was no longer on a hot path
anymore, there was no reason for continuing to force it to be tuned at
build time. So this patch allows to set it by tune.memory-hot-size.

It's worth noting that during the boot phase the value remains zero so
that it's possible to know if the value was set or not, which opens the
possibility that we try to automatically adjust it based on the per-cpu
L2 cache size or the use of certain protocols (none of this is done yet).
2022-12-20 14:51:12 +01:00
Thayne McCombs
02cf4ecb5a MINOR: sample: add param converter
Add a converter that extracts a parameter from string of delimited
key/value pairs.

Fixes: #1697
2022-12-14 08:24:15 +01:00
Willy Tarreau
9192d20f02 MINOR: pools: make DEBUG_UAF a runtime setting
Since the massive pools cleanup that happened in 2.6, the pools
architecture was made quite more hierarchical and many alternate code
blocks could be moved to runtime flags set by -dM. One of them had not
been converted by then, DEBUG_UAF. It's not much more difficult actually,
since it only acts on a pair of functions indirection on the slow path
(OS-level allocator) and a default setting for the cache activation.

This patch adds the "uaf" setting to the options permitted in -dM so
that it now becomes possible to set or unset UAF at boot time without
recompiling. This is particularly convenient, because every 3 months on
average, developers ask a user to recompile haproxy with DEBUG_UAF to
understand a bug. Now it will not be needed anymore, instead the user
will only have to disable pools and enable uaf using -dMuaf. Note that
-dMuaf only disables previously enabled pools, but it remains possible
to re-enable caching by specifying the cache after, like -dMuaf,cache.
A few tests with this mode show that it can be an interesting combination
which catches significantly less UAF but will do so with much less
overhead, so it might be compatible with some high-traffic deployments.

The change is very small and isolated. It could be helpful to backport
this at least to 2.7 once confirmed not to cause build issues on exotic
systems, and even to 2.6 a bit later as this has proven to be useful
over time, and could be even more if it did not require a rebuild. If
a backport is desired, the following patches are needed as well:

  CLEANUP: pools: move the write before free to the uaf-only function
  CLEANUP: pool: only include pool-os from pool.c not pool.h
  REORG: pool: move all the OS specific code to pool-os.h
  CLEANUP: pools: get rid of CONFIG_HAP_POOLS
  DEBUG: pool: show a few examples in -dMhelp
2022-12-08 18:54:59 +01:00
Ilya Shipitsin
5fa29b8a74 CLEANUP: assorted typo fixes in the code and comments
This is 34th iteration of typo fixes
2022-12-07 09:08:18 +01:00
Amaury Denoyelle
e30f378236 MINOR: quic: activate socket per conn by default
Activate QUIC connection socket to achieve the best performance. The
previous behavior can be reverted by tune.quic.socket-owner
configuration option.

This change is part of quic-conn owned socket implementation.

Contrary to its siblings patches, I suggest to not backport it to 2.7.
This should ensure that stable releases behavior is perserved. If a user
faces issues with QUIC performance on 2.7, he can nonetheless change the
default configuration.
2022-12-02 14:45:43 +01:00
Amaury Denoyelle
511ddd5785 MINOR: quic: define config option for socket per conn
Define global configuration option "tune.quic.socket-owner". This option
can be used to activate or not socket per QUIC connection mode. The
default value is "listener" which disable this feature. It can be
activated with the option "connection".

This change is part of quic-conn owned socket implementation.
It may be backported to 2.7 after a period of observation.
2022-12-02 14:45:43 +01:00
Aurelien DARRAGON
d06b9c8b99 DOC/MINOR: api: add documentation for event_hdl feature
This is an initial work for the dedicated
event handler API internal documentation.

The file is located at doc/internals/api/event_hdl.txt

event_hdl feature has been introduced with:
	MINOR: event_hdl: add event handler base api
2022-12-02 09:40:52 +01:00
Dragan Dosen
a9800a0f58 MEDIUM: 51d: add support for 51Degrees V4 with Hash algorithm
This patch also adds a set of new global options:

- 51degrees-use-performance-graph { on | off }
- 51degrees-use-predictive-graph { on | off }
- 51degrees-drift <number>
- 51degrees-difference <number>
- 51degrees-allow-unmatched { on | off }

To build using the latest 51Degrees V4 engine with Hash algorithm, set
USE_51DEGREES_V4=1.

Other supported build options are 51DEGREES_INC, 51DEGREES_LIB and
51DEGREES_SRC which needs to be set to the directory that contains
headers and C files. For example:

make TARGET=<target> USE_51DEGREES_V4=1 51DEGREES_SRC='51D_REPO_PATH'/src
2022-12-01 16:12:21 +01:00
Willy Tarreau
eaded987ee [RELEASE] Released version 2.8-dev0
Released version 2.8-dev0 with the following main changes :
    - MINOR: version: mention that it's development again
2022-12-01 15:25:34 +01:00
Willy Tarreau
437fd289f2 [RELEASE] Released version 2.7.0
Released version 2.7.0 with the following main changes :
    - MINOR: ssl: forgotten newline in error messages on ca-file
    - BUG/MINOR: ssl: shut the ca-file errors emitted during httpclient init
    - DOC: config: provide some configuration hints for "http-reuse"
    - DOC: config: refer to section about quoting in the "add_item" converter
    - DOC: halog: explain how to use -ac and -ad in the help message
    - DOC: config: clarify the fact that SNI should not be used in HTTP scenarios
    - DOC: config: mention that a single monitor-uri rule is supported
    - DOC: config: explain how default matching method for ACL works
    - DOC: config: clarify the fact that "retries" is not just for connections
    - BUILD: halog: fix missing double-quote at end of help line
    - DOC: config: clarify the -m dir and -m dom pattern matching methods
    - MINOR: activity: report uptime in "show activity"
    - REORG: activity/cli: move the "show activity" handler to activity.c
    - DEV: poll: add support for epoll
    - DEV: tcploop: centralize the polling code into wait_for_fd()
    - DEV: tcploop: add support for POLLRDHUP when supported
    - DEV: tcploop: do not report an error on POLLERR
    - DEV: tcploop: add optional support for epoll
    - SCRIPTS: announce-release: add a link to the data plane API
    - CLEANUP: stick-table: fill alignment holes in the stktable struct
    - MINOR: stick-table: store a per-table hash seed and use it
    - MINOR: stick-table: show the shard number in each entry's "show table" output
    - CLEANUP: ncbuf: remove ncb_blk args by value
    - CLEANUP: ncbuf: inline small functions
    - CLEANUP: ncbuf: use standard BUG_ON with DEBUG_STRICT
    - BUG/MINOR: quic: Endless loop during retransmissions
    - MINOR: mux-h2: add the expire task and its expiration date in "show fd"
    - BUG/MINOR: peers: always initialize the stksess shard value
    - REGTESTS: fix peers-related regtests regarding "show table"
    - BUG/MEDIUM: mux-h1: Close client H1C on EOS when there is no output data
    - MINOR: stick-table: change the API of the function used to calculate the shard
    - CLEANUP: peers: factor out the key len calculation in received updates
    - BUG/MINOR: peers: always update the stksess shard number on incoming updates
    - CLEANUP: assorted typo fixes in the code and comments
    - MINOR: mux-h1: add the expire task and its expiration date in "show fd"
    - MINOR: debug: improve error handling on the memstats command parser
    - BUILD: quic: allow build with USE_QUIC and USE_OPENSSL_WOLFSSL
    - CLEANUP: anon: clarify the help message on "debug dev hash"
    - MINOR: debug: relax access restrictions on "debug dev hash" and "memstats"
    - SCRIPTS: run-regtests: add a version check
    - MINOR: version: mention that it's stable now
2022-12-01 15:16:46 +01:00
Ilya Shipitsin
6f86eaae4f CLEANUP: assorted typo fixes in the code and comments
This is 33rd iteration of typo fixes
2022-11-30 14:02:36 +01:00
Willy Tarreau
16b282f4b0 MINOR: stick-table: show the shard number in each entry's "show table" output
Stick-tables support sharding to multiple peers but there was no way to
know to what shard an entry was going to be sent. Let's display this in
the "show table" output to ease debugging.
2022-11-29 12:00:49 +01:00
Willy Tarreau
f386a2de92 DOC: config: clarify the -m dir and -m dom pattern matching methods
There's regularly some confusion about them (do they match at the
beginning, end ? do they support multiple components etc). Tim
suggested to improve the doc in issue #61, it's never too late, so
let's do it now wih a few examples.
2022-11-25 12:02:25 +01:00
Willy Tarreau
0b4a622b49 DOC: config: clarify the fact that "retries" is not just for connections
In issue #412 it was rightfully reported that the wording in "retries"
still exclusively speaks about connection attempts, while since L7
retries with "retry-on" it's no longer a limitation. Let's update the
text.
2022-11-25 11:06:20 +01:00
Willy Tarreau
4f4fea417b DOC: config: explain how default matching method for ACL works
In issue #698, it's made apparent that the default matching method for
ACL keywords can be confusing when a converter is applied, because
depending on the converters used, users may think that the default
matching method from the sample fetch name might apply to the whole
expression. It's easier to understand that this doesn't make sense
when thinking about converters turning to completely different types
(e.g. hdr_beg(host),do_resolve() returns an IP, thus it's obvious
that _beg makes no sense at all).  This patch states this in the
doc to avoid future confusion.
2022-11-25 10:49:41 +01:00
Willy Tarreau
7fe0c62516 DOC: config: mention that a single monitor-uri rule is supported
It was reported in issue #1059 that when multiple monitor-uri rules are
specified, only the last one is used. While this was done on purpose
since a single URI is used, it was not clearly mentioned in the doc,
possibly leading to confusion or wasted time trying to establish a
working setup. Let's clarify this point.
2022-11-25 10:27:05 +01:00
Willy Tarreau
d26fb57e81 DOC: config: clarify the fact that SNI should not be used in HTTP scenarios
As reported by Tim in issue #1373 some warnings are deserved to explain
why using the frontend SNI for routing or connecting to a server is
usually not correct, especially since it can be tempting and used to
make sense in pure TCP scenarios.
2022-11-25 10:14:12 +01:00
Willy Tarreau
b143d110bf DOC: config: refer to section about quoting in the "add_item" converter
As requested by Nick in issue #1719, let's add a reference to the section
about quoting there, since add_item() will often be used with commas and
it's easy to mess up.
2022-11-25 09:27:15 +01:00
Willy Tarreau
44fce8bd73 DOC: config: provide some configuration hints for "http-reuse"
This adds some configuration hints regarding various workloads that do
not manage to achieve high reuse rates due to too low a global maxconn
or thread groups.

This fixes github issue #1472.
2022-11-25 09:20:50 +01:00
Willy Tarreau
0279df9e82 [RELEASE] Released version 2.7-dev10
Released version 2.7-dev10 with the following main changes :
    - MEDIUM: tcp-act: add parameter rst-ttl to silent-drop
    - BUG/MAJOR: quic: Crash upon retransmission of dgrams with several packets
    - MINOR: cli: print parsed command when not found
    - BUG/MAJOR: quic: Crash after discarding packet number spaces
    - CLEANUP: quic: replace "choosen" with "chosen" all over the code
    - MINOR: cli/pools: store "show pools" results into a temporary array
    - MINOR: cli/pools: add sorting capabilities to "show pools"
    - MINOR: cli/pools: add pool name filtering capability to "show pools"
    - DOC: configuration: fix quic prefix typo
    - MINOR: quic: report error if force-retry without cluster-secret
    - MINOR: global: generate random cluster.secret if not defined
    - BUG/MINOR: resolvers: do not run the timeout task when there's no resolution
    - BUG/MINOR: server/idle: at least use atomic stores when updating max_used_conns
    - MINOR: server/idle: make the next_takeover index per-tgroup
    - BUILD: listener: fix build warning on global_listener_rwlock without threads
    - BUG/MAJOR: sched: protect task during removal from wait queue
    - BUILD: sched: fix build with DEBUG_THREAD with the previous commit
    - DOC: quic: add note on performance issue with listener contention
    - BUG/MINOR: cfgparse-listen: fix ebpt_next_dup pointer dereference on proxy "from" inheritance
    - BUG/MINOR: log: fix parse_log_message rfc5424 size check
    - CLEANUP: arg: remove extra check in make_arg_list arg escaping
    - CLEANUP: tools: extra check in utoa_pad
    - MINOR: h1: Consider empty port as invalid in authority for CONNECT
    - MINOR: http: Considere empty ports as valid default ports
    - BUG/MINOR: http-htx: Normalized absolute URIs with an empty port
    - BUG/MINOR: h1: Replace authority validation to conform RFC3986
    - REG-TESTS: http: Add more tests about authority/host matching
    - BUG/MINOR: http-htx: Don't consider an URI as normalized after a set-uri action
    - BUG/MEDIUM: mux-h1: Don't release H1C on timeout if there is a SC attached
    - BUG/MEDIUM: mux-h1: Subscribe for reads on error on sending path
    - BUILD: http-htx: Silent build error about a possible NULL start-line
    - DOC: configuration.txt: add default_value for table_idle signature
    - BUILD: ssl-sock: Silent error about NULL deref in ssl_sock_bind_verifycbk()
    - BUG/MEDIUM: mux-h1: Remove H1C_F_WAIT_NEXT_REQ flag on a next request
    - BUG/MINOR: mux-h1: Fix handling of 408-Request-Time-Out
    - MINOR: mux-h1: Remove H1C_F_WAIT_NEXT_REQ in functions handling errors
    - MINOR: mux-h1: Avoid useless call to h1_send() if no error is sent
    - DOC: configuration.txt: fix typo in table_idle signature
    - BUILD: stick-tables: fix build breakage in xxhash on older compilers
    - BUILD: compiler: include compiler's definitions before ours
    - BUILD: quic: global.h is needed in cfgparse-quic
    - CLEANUP: tools: do not needlessly include xxhash nor cli from tools.h
    - BUILD: flags: really restrict the cases where flags are exposed
    - BUILD: makefile: minor reordering of objects by build time
    - BUILD: quic: silence two invalid build warnings at -O1 with gcc-6.5
    - BUILD: quic: use openssl-compat.h instead of openssl/ssl.h
    - MEDIUM: ssl: add minimal WolfSSL support with OpenSSL compatibility mode
    - MINOR: sample: make the rand() sample fetch function use the statistical_prng
    - MINOR: auth: silence null dereference warning in check_user()
    - CLEANUP: peers: fix format string for status messages (int signedness)
    - CLEANUP: qpack: fix format string in debugging code (int signedness)
    - CLEANUP: qpack: properly use the QPACK macros not HPACK ones in debug code
    - BUG/MEDIUM: quic: fix datagram dropping on queueing failed
2022-11-24 17:13:05 +01:00
Aurelien DARRAGON
fd766ddfaf DOC: configuration.txt: fix typo in table_idle signature
An extra ',' was mistakenly added in table_idle converter signature
with commit ed36968 ("DOC: configuration.txt: add default_value for
table_idle signature").
2022-11-23 17:21:18 +01:00
Aurelien DARRAGON
ed36968f16 DOC: configuration.txt: add default_value for table_idle signature
table_idle converter takes optional default_value argument.
The documentation correctly describes this usage but <default_value> was
missing in the converter signature.

It should be backported with bbeec37b3 ("MINOR: stick-table:
Add table_expire() and table_idle() new converters")
2022-11-22 19:41:12 +01:00
Christopher Faulet
84cdbe478a BUG/MINOR: http-htx: Don't consider an URI as normalized after a set-uri action
An abosulte URI is marked as normalized if it comes from an H2 client. This
way, we know we can send a relative URI to an H1 server. But, after a
set-uri action, the URI must no longer be considered as normalized.
Otherwise there is no way to send an absolute URI on the server side.

If it is important to update a normalized absolute URI without altering this
property, the host, path and/or query-string must be set separatly.

This patch should fix the issue #1938. It should be backported as far as
2.4.
2022-11-22 17:49:10 +01:00
Amaury Denoyelle
7078fb1f3a DOC: quic: add note on performance issue with listener contention
Complete quic4/quic6 bind lines by a note on performance issues due to
receiver socket contention. Suggest to use sharding to improve the
situation.

This should be backported up to 2.6.
2022-11-22 11:46:29 +01:00
Amaury Denoyelle
28ea31c7cb MINOR: global: generate random cluster.secret if not defined
If no cluster-secret is defined by the user, a random one is silently
generated.

This ensures that at least QUIC Retry tokens are generated if abnormal
conditions are detected. However, it is advisable to specify it in the
configuration for tokens to be valid even after a reload or across LBs
instances in the same cluster.

This should be backported up to 2.6.
2022-11-21 16:41:34 +01:00
Amaury Denoyelle
996ca7d0fa MINOR: quic: report error if force-retry without cluster-secret
QUIC Retry generation relies on global cluster-secret to produce token
valid even after a process restart and across several LBs instances.

Before this patch, Retry is automatically deactivated if no
cluster-secret is provided. This is the case even if a user has
configured a QUIC listener with quic-force-retry. Change this behavior
by now returning an error during configuration parsing. The user must
provide a cluster-secret if quic-force-retry is used.

This shoud be backported up to 2.6.
2022-11-21 16:34:09 +01:00
Amaury Denoyelle
936c135e05 DOC: configuration: fix quic prefix typo
Replace quicv4/quicv6 -> quic4/quic6 as prefix for bind lines of QUIC
listeners.

This should be backported up to 2.6.
2022-11-21 16:14:46 +01:00
Willy Tarreau
7583c36790 MINOR: cli/pools: add pool name filtering capability to "show pools"
Now it becomes possible to match a pool name's prefix, for example:

  $ socat - /tmp/haproxy.sock <<< "show pools match quic byusage"
  Dumping pools usage. Use SIGQUIT to flush them.
    - Pool quic_conn_r (65560 bytes) : 1337 allocated (87653720 bytes), ...
    - Pool quic_crypto (1048 bytes) : 6685 allocated (7005880 bytes), ...
    - Pool quic_conn (4056 bytes) : 1337 allocated (5422872 bytes), ...
    - Pool quic_rxbuf (262168 bytes) : 8 allocated (2097344 bytes), ...
    - Pool quic_connne (184 bytes) : 9359 allocated (1722056 bytes), ...
    - Pool quic_frame (184 bytes) : 7938 allocated (1460592 bytes), ...
    - Pool quic_tx_pac (152 bytes) : 6454 allocated (981008 bytes), ...
    - Pool quic_tls_ke (56 bytes) : 12033 allocated (673848 bytes), ...
    - Pool quic_rx_pac (408 bytes) : 1596 allocated (651168 bytes), ...
    - Pool quic_tls_se (88 bytes) : 6685 allocated (588280 bytes), ...
    - Pool quic_cstrea (88 bytes) : 4011 allocated (352968 bytes), ...
    - Pool quic_tls_iv (24 bytes) : 12033 allocated (288792 bytes), ...
    - Pool quic_dgram (344 bytes) : 732 allocated (251808 bytes), ...
    - Pool quic_arng (56 bytes) : 4011 allocated (224616 bytes), ...
    - Pool quic_conn_c (152 bytes) : 1337 allocated (203224 bytes), ...
  Total: 15 pools, 109578176 bytes allocated, 109578176 used ...

In this case the reported total only concerns the dumped ones.
2022-11-21 10:14:52 +01:00
Willy Tarreau
2fba08faec MINOR: cli/pools: add sorting capabilities to "show pools"
The "show pools" command is used a lot for debugging but didn't get much
love over the years. This patch brings new capabilities:
  - sorting the output by pool names to ese their finding ("byname").
  - sorting the output by reverse item size to spot the biggest ones("bysize")
  - sorting the output by reverse number of allocated bytes ("byusage")

The last one (byusage) also omits displaying the ones with zero allocation.

In addition, an optional max number of output entries may be passed so as
to dump only the N most relevant ones.
2022-11-21 10:14:52 +01:00
Mathias Weiersmueller
d9b7174d99 MEDIUM: tcp-act: add parameter rst-ttl to silent-drop
The silent-drop action was extended with an additional optional parameter,
[rst-ttl <ttl> ], causing HAProxy to send a TCP RST with the specified TTL
towards the client.

With this behaviour, the connection state on your own client-
facing middle-boxes (load balancers, firewalls) will be purged,
but the client will still assume the TCP connection is up because
the TCP RST packet expires before reaching the client.
2022-11-19 04:53:47 +01:00
Willy Tarreau
a0abec8bc0 [RELEASE] Released version 2.7-dev9
Released version 2.7-dev9 with the following main changes :
    - BUILD: quic: QUIC mux build fix for 32-bit build
    - BUILD: scripts: disable tests build on QuicTLS build
    - BUG/MEDIUM: httpclient: segfault when the httpclient parser fails
    - BUILD: ssl_sock: fix null dereference for QUIC build
    - BUILD: quic: Fix build for m68k cross-compilation
    - BUG/MINOR: quic: fix buffer overflow on retry token generation
    - MINOR: quic: add version field on quic_rx_packet
    - MINOR: quic: extend pn_offset field from quic_rx_packet
    - MINOR: quic: define first packet flag
    - MINOR: quic: extract connection retrieval
    - MINOR: quic: split and rename qc_lstnr_pkt_rcv()
    - MINOR: quic: refactor packet drop on reception
    - MINOR: quic: extend Retry token check function
    - BUG/MINOR: log: Preserve message facility when the log target is a ring buffer
    - BUG/MINOR: ring: Properly parse connect timeout
    - BUG/MEDIUM: httpclient/lua: crash when the lua task timeout before the httpclient
    - BUG/MEDIUM: httpclient: check if the httpclient was released in the IO handler
    - REGTESTS: httpclient/lua: test the lua task timeout with the httpclient
    - CI: github: dump the backtrace of coredumps in the alpine container
    - BUILD: Makefile: add "USE_SHM_OPEN" on the linux-musl target
    - DOC: lua: add a note about compression w/ httpclient
    - CLEANUP: mworker/cli: rename the status function to loadstatus
    - MINOR: mworker/cli: does no try to dump the startup-logs w/o USE_SHM_OPEN
    - MINOR: list: fixing typo in MT_LIST_LOCK_ELT
    - DOC/MINOR: list: fixing MT_LIST_LOCK_ELT macro documentation
    - MINOR: list: adding MT_LIST_APPEND_LOCKED macro
    - BUG/MINOR: mux-quic: complete flow-control for uni streams
    - BUG/MEDIUM: compression: handle rewrite errors when updating response headers
    - MINOR: quic: do not crash on unhandled sendto error
    - MINOR: quic: display unknown error sendto counter on stat page
    - MINOR: peers: Support for peer shards
    - MINOR: peers: handle multiple resync requests using shards
    - BUG/MINOR: sink: Only use backend capability for the sink proxies
    - BUG/MINOR: sink: Set default connect/server timeout for implicit ring buffers
    - MINOR: ssl: add the SSL error string when failing to load a certificate
    - MINOR: ssl: add the SSL error string before the chain
    - MEDIUM: ssl: be stricter about chain error
    - BUG/MAJOR: stick-table: don't process store-response rules for applets
    - MINOR: quic: remove unnecessary quic_session_accept()
    - BUG/MINOR: quic: fix subscribe operation
    - BUG/MINOR: log: fixing bug in tcp syslog_io_handler Octet-Counting
    - MINOR: ssl: dump the SSL string error when SSL_CTX_use_PrivateKey() failed.
    - MINOR: quic: add counter for interrupted reception
    - BUG/MINOR: quic: fix race condition on datagram purging
    - CI: add monthly gcc cross compile jobs
    - CLEANUP: assorted typo fixes in the code and comments
    - CLEANUP: ssl: remove dead code in ssl_sock_load_pem_into_ckch()
    - BUG/MINOR: httpclient: fixed memory allocation for the SSL ca_file
    - BUG/MINOR: ssl: Memory leak of DH BIGNUM fields
    - BUG/MINOR: ssl: Memory leak of AUTHORITY_KEYID struct when loading issuer
    - BUG/MINOR: ssl: ocsp structure not freed properly in case of error
    - CI: switch to the "latest" LibreSSL
    - CI: enable QUIC for LibreSSL builds
    - BUG/MEDIUM: ssl: Verify error codes can exceed 63
    - MEDIUM: ssl: {ca,crt}-ignore-err can now use error constant name
    - MINOR: ssl: x509_v_err_str converter transforms an integer to a X509_V_ERR name
    - CLEANUP: cli: rename dynamic error printing state
    - MINOR: cli: define usermsgs print context
    - MINOR: server: clear prefix on stderr logs after add server
    - BUG/MINOR: ssl: bind_conf is uncorrectly accessed when using QUIC
    - BUILD: ssl_utils: fix build on gcc versions before 8
    - BUILD: debug: remove unnecessary quotes in HA_WEAK() calls
    - CI: emit the compiler's version in the build reports
    - IMPORT: xxhash: update xxHash to version 0.8.1
    - IMPORT: slz: declare len to fix debug build when optimal match is enabled
    - IMPORT: slz: mention the potential header in slz_finish()
    - IMPORT: slz: define and use a __fallthrough statement for switch/case
    - BUILD: compiler: add a macro to detect if another one is set and equals 1
    - BUILD: compiler: add a default definition for __has_attribute()
    - BUILD: compiler: define a __fallthrough statement for switch/case
    - BUILD: sample: use __fallthrough in smp_is_rw() and smp_dup()
    - BUILD: quic: use __fallthrough in quic_connect_server()
    - BUILD: ssl/crt-list: use __fallthrough in cli_io_handler_add_crtlist()
    - BUILD: ssl: use __fallthrough in cli_io_handler_commit_{cert,cafile_crlfile}()
    - BUILD: ssl: use __fallthrough in cli_io_handler_tlskeys_files()
    - BUILD: hlua: use __fallthrough in hlua_post_init_state()
    - BUILD: stream: use __fallthrough in stats_dump_full_strm_to_buffer()
    - BUILD: tcpcheck: use __fallthrough in check_proxy_tcpcheck()
    - BUILD: stats: use __fallthrough in stats_dump_proxy_to_buffer()
    - BUILD: peers: use __fallthrough in peer_io_handler()
    - BUILD: hash: use __fallthrough in hash_djb2()
    - BUILD: tools: use __fallthrough in url_decode()
    - BUILD: args: use __fallthrough in make_arg_list()
    - BUILD: acl: use __fallthrough in parse_acl_expr()
    - BUILD: spoe: use __fallthrough in spoe_handle_appctx()
    - BUILD: logs: use __fallthrough in build_log_header()
    - BUILD: check: use __fallthrough in __health_adjust()
    - BUILD: http_act: use __fallthrough in parse_http_del_header()
    - BUILD: h1_htx: use __fallthrough in h1_parse_chunk()
    - BUILD: vars: use __fallthrough in var_accounting_{diff,add}()
    - BUILD: map: use __fallthrough in cli_io_handler_*()
    - BUILD: compression: use __fallthrough in comp_http_payload()
    - BUILD: stconn: use __fallthrough in various shutw() functions
    - BUILD: prometheus: use __fallthrough in promex_dump_metrics() and IO handler()
    - CLEANUP: ssl: remove printf in bind_parse_ignore_err
    - BUG/MINOR: ssl:  crt-ignore-err memory leak with 'all' parameter
    - BUG/MINOR: ssl: Fix potential overflow
    - CLEANUP: stick-table: remove the unused table->exp_next
    - OPTIM: stick-table: avoid atomic ops in stktable_requeue_exp() when possible
    - BUG/MEDIUM: stick-table: fix a race condition when updating the expiration task
    - MEDIUM: http-ana: remove set-cookie2 support
    - BUG/MEDIUM: wdt/clock: properly handle early task hangs
    - MINOR: deinit: add a "quick-exit" option to bypass the deinit step
    - OPTIM: ebtree: make ebmb_insert_prefix() keep a copy the new node's pfx
    - OPTIM: ebtree: make ebmb_insert_prefix() keep a copy the new node's key
    - MINOR: ssl: ssl_sock_load_cert_chain() display error strings
    - MINOR: ssl: reintroduce ERR_GET_LIB(ret) == ERR_LIB_PEM in ssl_sock_load_pem_into_ckch()
    - BUG/MINOR: http-htx: Fix error handling during parsing http replies
    - BUG/MINOR: resolvers: Don't wait periodic resolution on healthcheck failure
    - BUG/MINOR: resolvers: Set port before IP address when processing SRV records
    - BUG/MINOR: mux-fcgi: Be sure to send empty STDING record in case of zero-copy
    - BUG/MEDIUM: mux-fcgi: Avoid value length overflow when it doesn't fit at once
    - BUG/MINOR: ssl: SSL_load_error_strings might not be defined
    - MINOR: pool/debug: create a new pool_alloc_flag() macro
    - MINOR: dynbuf: switch allocation and release to macros to better track users
    - BUG/MINOR: mux-h1: Do not send a last null chunk on body-less answers
    - REG-TESTS: cache: Remove T-E header for 304-Not-Modified responses
    - DOC: config: fix alphabetical ordering of global section
    - MINOR: trace: split the CLI "trace" parser in CLI vs statement
    - MEDIUM: trace: create a new "trace" statement in the "global" section
    - BUG/MEDIUM: ring: fix creation of server in uninitialized ring
    - BUILD: quic: fix dubious 0-byte overflow on qc_release_lost_pkts
    - BUILD: makefile: mark poll and tcploop targets as phony
    - BUILD: makefile: properly pass CC to sub-projects
    - BUILD: makefile: move default verbosity settings to include/make/verbose.mk
    - BUILD: makefile: use $(cmd_MAKE) in quiet mode
    - BUILD: makefile: move the compiler option detection stuff to compiler.mk
    - DEV: poll: make the connect() step an action as well
    - DEV: poll: strip the "do_" prefix from reported function names
    - DEV: poll: indicate the FD's side in front of its value
    - BUG/MINOR: pool/cli: use ullong to report total pool usage in bytes
    - MINOR: mux-h1: Remove usless code inside shutr callback
    - CLEANUP: mux-h1; Rename H1S_F_ERROR flag into H1S_F_ERROR_MASK
    - REORG: mux-h1: Reorg the H1C structure
    - CLEANUP: mux-h1: Rename H1C_F_ST_ERROR and H1C_F_ST_SILENT_SHUT flags
    - MINOR: mux-h1: Add a dedicated enum to deal with H1 connection state
    - MEDIUM: mux-h1: Handle H1C states via its state field instead of H1C_F_ST_*
    - MINOR: mux-h1: Don't handle subscribe for reads in h1_process_demux()
    - CLEANUP: mux-h1: Rename H1C_F_ERR_PENDING into H1C_F_ABRT_PENDING
    - MINOR: mux-h1: Add flag on H1 stream to deal with internal errors
    - MEDIUM: mux-h1: Rely on the H1C to deal with shutdown for reads
    - CLEANUP: mux-h1: Reorder H1 connection flags to avoid holes
    - MEDIUM: mux-h1: Don't report a final error whe a message is aborted
    - MEDIUM: mux-pt: Don't always set a final error on SE on the sending path
    - MEDIUM: mux-h2: Introduce flags to deal with connection read/write errors
    - CLEANUP: mux-h2: Remove unused fields in h2c structures
    - MEDIUM: mux-fcgi: Introduce flags to deal with connection read/write errors
    - MINOR: sconn: Set SE_FL_ERROR only when there is no more data to read
    - MINOR: mux-h1: Rely on a H1S flag to know a WS key was found or not
    - DOC: lua-api: Remove warning about the lua filters
    - BUG/MEDIUM: listener: Fix race condition when updating the global mngmt task
    - CLEANUP: listener: Remove useless task_queue from manage_global_listener_queue
    - BUG/MINOR: mux-h1: Fix error handling when H1S allocation failed on client side
    - DOC: internal: commit notes about polling states and flags
    - DOC: internal: commit notes about polling states and flags on connect()
    - CLEANUP: mux-h1: Don't test h1c in h1_shutw_conn()
    - BUG/MINOR: http_ana/txn: don't re-initialize txn and req var lists
    - BUG/MEDIUM: raw-sock: Don't report connection error if something was received
    - BUG/MINOR: ssl: don't initialize the keylog callback when not required
    - BUILD: Makefile: enable USE_SHM_OPEN by default on freebsd
    - BUG/MEDIUM: peers: messages about unkown tables not correctly ignored
    - MINOR: cfgparse: Always check the section position
    - MEDIUM: thread: Restric nbthread/thread-group(s) to very first global sections
    - BUILD: peers: Remove unused variables
    - MINOR: ncbuf: complete doc for ncb_advance()
    - BUG/MEDIUM: quic: fix unsuccessful handshakes on ncb_advance error
    - BUG/MEDIUM: quic: fix memleak for out-of-order crypto data
    - MINOR: quic: complete traces/debug for handshake
2022-11-18 17:48:49 +01:00
Willy Tarreau
831d613f54 DOC: internal: commit notes about polling states and flags on connect()
Let's keep these notes as references for later use. Polling on connect()
can sometimes return a few unexpected state combinations that such tests
illustrate. They can serve as reminders for special error handling.
2022-11-17 16:49:00 +01:00
Willy Tarreau
c89f665cca DOC: internal: commit notes about polling states and flags
Some detailed observations were made on polling general and POLLHUP
more specifically, they can be useful later.
2022-11-17 16:49:00 +01:00
Christopher Faulet
b6c31b3cda DOC: lua-api: Remove warning about the lua filters
The lua filters api is no longer experimental. There are some filters
depending on this api, thus it is fair to make it stable now.
2022-11-17 14:33:15 +01:00
Willy Tarreau
9fd0542148 MEDIUM: trace: create a new "trace" statement in the "global" section
The exact same commands as those from the CLI may be pre-loaded at boot
time by passing them one per line after the "trace" keyword in the global
section; i.e. just copy-pasting all commands directly there will do the
job. Note that if a ring is mentioned, it needs to be declared before the
global section. Another option is to append another global section after
"ring".

For now the keyword is marked as experimental to discourage its broad
adoption by default. "expose-experimental-directives" needs to be placed
in the global section to expose it.
2022-11-16 17:55:53 +01:00
Willy Tarreau
8e6ad2548c DOC: config: fix alphabetical ordering of global section
the global section keywords were seriously misordered, and it's visible
that some mistakes have induced other ones over time, so it was about
time to fix this. Roughly 20% of the keywords were misplaced.

This commit only reordered the keywords index and their description,
nothing else was changed. It might be backported because it's a real
pain to find certain options there.
2022-11-16 17:55:53 +01:00
Willy Tarreau
e98d385819 MINOR: deinit: add a "quick-exit" option to bypass the deinit step
Once in a while we spot a bug in the deinit code that is complex,
especially when it has to deal with incomplete initializations, and the
ability to bypass this step has regularly been raised. In addition for
fast-reloading setups it could theoretically save some time. Tests have
shown that very large configs can barely save ~100-150ms by skipping the
deinit step. However the ability not to crash if a bug is encountered can
occasionally help.

This patch adds an option to do exactly this. It's obviously not enabled
by default and the documentation discourages from using it, but this might
be useful in the future.
2022-11-15 09:37:09 +01:00
William Lallemand
9fbc84e571 MINOR: ssl: x509_v_err_str converter transforms an integer to a X509_V_ERR name
The x509_v_err_str converter transforms a numerical X509 verify error
to its constant name.
2022-11-10 13:28:37 +01:00
William Lallemand
960fb74cae MEDIUM: ssl: {ca,crt}-ignore-err can now use error constant name
The ca-ignore-err and crt-ignore-err directives are now able to use the
openssl X509_V_ERR constant names instead of the numerical values.

This allow a configuration to survive an OpenSSL upgrade, because the
numerical ID can change between versions. For example
X509_V_ERR_INVALID_CA was 24 in OpenSSL 1 and is 79 in OpenSSL 3.

The list of errors must be updated when a new major OpenSSL version is
released.
2022-11-10 13:28:37 +01:00
Ilya Shipitsin
4a689dad03 CLEANUP: assorted typo fixes in the code and comments
This is 32nd iteration of typo fixes
2022-10-30 17:17:56 +01:00
Frédéric Lécaille
36d1565640 MINOR: peers: Support for peer shards
Add "shards" new keyword for "peers" section to configure the number
of peer shards attached to such secions. This impact all the stick-tables
attached to the section.
Add "shard" new "server" parameter to configure the peers which participate to
all the stick-tables contents distribution. Each peer receive the stick-tables updates
only for keys with this shard value as distribution hash. The "shard" value
is stored in ->shard new server struct member.
cfg_parse_peers() which is the function which is called to parse all
the lines of a "peers" section is modified to parse the "shards" parameter
stored in ->nb_shards new peers struct member.
Add srv_parse_shard() new callback into server.c to pare the "shard"
parameter.
Implement stksess_getkey_hash() to compute the distribution hash for a
stick-table key as the 64-bits xxhash of the key concatenated to the stick-table
name. This function is called by stksess_setkey_shard(), itself
called by the already implemented function which create a new stick-table
key (stksess_new()).
Add ->idlen new stktable struct member to store the stick-table name length
to not have to compute it each time a stick-table key hash is computed.
2022-10-24 10:55:53 +02:00
William Lallemand
a9256194b8 DOC: lua: add a note about compression w/ httpclient
Decompression is not supported by the httpclient.
2022-10-21 11:50:05 +02:00
Willy Tarreau
ea8aebe8c5 [RELEASE] Released version 2.7-dev8
Released version 2.7-dev8 with the following main changes :
    - BUG/MINOR: checks: update pgsql regex on auth packet
    - DOC: config: Fix pgsql-check documentation to make user param mandatory
    - CLEANUP: mux-quic: remove usage of non-standard ull type
    - CLEANUP: quic: remove global var definition in quic_tls header
    - BUG/MINOR: quic: adjust quic_tls prototypes
    - CLEANUP: quic: fix headers
    - CLEANUP: quic: remove unused function prototype
    - CLEANUP: quic: remove duplicated varint code from xprt_quic.h
    - CLEANUP: quic: create a dedicated quic_conn module
    - BUG/MINOR: mux-quic: ignore STOP_SENDING for locally closed stream
    - BUG/MEDIUM: lua: Don't crash in hlua_lua2arg_check on failure
    - BUG/MEDIUM: lua: handle stick table implicit arguments right.
    - BUILD: h1: silence an initiialized warning with gcc-4.7 and -Os
    - MINOR: fd: add a new function to only raise RLIMIT_NOFILE
    - MINOR: init: do not try to shrink existing RLIMIT_NOFIlE
    - BUG/MINOR: http-fetch: Update method after a prefetch in smp_fetch_meth()
    - BUILD: http_fetch: silence an uninitiialized warning with gcc-4/5/6 at -Os
    - BUG/MINOR: hlua: hlua_channel_insert_data() behavior conflicts with documentation
    - MINOR: quic: limit usage of ssl_sock_ctx in favor of quic_conn
    - MINOR: mux-quic: check quic-conn return code on Tx
    - CLEANUP: quic: fix indentation
    - MEDIUM: quic: retrieve frontend destination address
    - CLEANUP: Reapply ist.cocci (2)
    - CLEANUP: Reapply strcmp.cocci
    - CLEANUP: quic/receiver: remove the now unused tx_qring list
    - BUG/MINOR: quic: set IP_PKTINFO socket option for QUIC receivers only
    - MINOR: hlua: some luaL_checktype() calls were not guarded with MAY_LJMP
    - DOC: configuration: missing 'if' in tcp-request content example
    - MINOR: hlua: removing ambiguous lua_pushvalue with 0 index
    - BUG/MAJOR: stick-tables: do not try to index a server name for applets
    - MINOR: plock: support disabling exponential back-off
    - MINOR: freq_ctr: use the thread's local time whenever possible
    - MEDIUM: stick-table: switch the table lock to rwlock
    - MINOR: stick-table: do not take an exclusive lock when downing ref_cnt
    - MINOR: stick-table: move the write lock inside stktable_touch_with_exp()
    - MEDIUM: stick-table: only take the lock when needed in stktable_touch_with_exp()
    - MEDIUM: stick-table: make stksess_kill_if_expired() avoid the exclusive lock
    - MEDIUM: stick-table: return inserted entry in __stktable_store()
    - MEDIUM: stick-table: free newly allocated stkess if it couldn't be inserted
    - MEDIUM: stick-table: switch to rdlock in stktable_lookup() and lookup_key()
    - MEDIUM: stick-table: make stktable_get_entry() look up under a read lock
    - MEDIUM: stick-table: do not take a lock to update t->current anymore.
    - MEDIUM: stick-table: make stktable_set_entry() look up under a read lock
    - MEDIUM: stick-table: requeue the expiration task out of the exclusive lock
    - MINOR: stick-table: split stktable_store() between key and requeue
    - MEDIUM: stick-table: always use atomic ops to requeue the table's task
    - MEDIUM: stick-table: requeue the wakeup task out of the write lock
    - BUG/MINOR: stick-table: fix build with DEBUG_THREAD
    - REORG: mux-fcgi: Extract flags and enums into mux_fcgi-t.h
    - MINOR: flags/mux-fcgi: Decode FCGI connection and stream flags
    - BUG/MEDIUM: mux-h1: Add connection error handling when reading/sending on a pipe
    - BUG/MEDIUM: mux-h1: Handle abort with an incomplete message during parsing
    - BUG/MINOR: server: make sure "show servers state" hides private bits
    - MINOR: checks: use the lighter PRNG for spread checks
    - MEDIUM: checks: spread the checks load over random threads
    - CI: SSL: use proper version generating when "latest" semantic is used
    - CI: SSL: temporarily stick to LibreSSL=3.5.3
    - MINOR: quic: New quic_cstream object implementation
    - MINOR: quic: Extract CRYPTO frame parsing from qc_parse_pkt_frms()
    - MINOR: quic: Use a non-contiguous buffer for RX CRYPTO data
    - BUG/MINOR: quic: Stalled 0RTT connections with big ClientHello TLS message
    - MINOR: quic: Split the secrets key allocation in two parts
    - CLEANUP: quic: remove unused rxbufs member in receiver
    - CLEANUP: quic: improve naming for rxbuf/datagrams handling
    - MINOR: quic: implement datagram cleanup for quic_receiver_buf
    - MINOR: ring: ring_cast_from_area() cast from an allocated area
    - MINOR: buffers: split b_force_xfer() into b_cpy() and b_force_xfer()
    - MINOR: logs: startup-logs can use a shm for logging the reload
    - MINOR: mworker/cli: reload command displays the startup-logs
    - MEDIUM: quic: respect the threads assigned to a bind line
    - DOC: management: update the "reload" command of the master CLI
    - BUILD: ssl_sock: bind_conf uninitialized in ssl_sock_bind_verifycbk()
    - BUG/MEDIUM: httpclient: Don't set EOM flag on an empty HTX message
    - MINOR: httpclient/lua: Don't set req_payload callback if body is empty
    - DOC/CLEANUP: lua-api: some minor corrections
    - DOC: lua-api: updating toolbox link
    - DOC/CLEANUP: lua-api: removing duplicate core.proxies attribute
    - DOC: management: add forgotten "show startup-logs"
    - DOC: management: "show startup-logs" for master CLI
    - CI: Replace the deprecated `::set-output` command by writing to $GITHUB_OUTPUT in matrix.py
    - CI: Replace the deprecated `::set-output` command by writing to $GITHUB_OUTPUT in workflow definition
2022-10-14 20:45:23 +02:00
William Lallemand
5d1e1317c2 DOC: management: "show startup-logs" for master CLI
"show startup-logs" on the master CLI has a slighly different behavior.

No backport needed.
2022-10-14 15:46:19 +02:00
William Lallemand
f76b3b47ea DOC: management: add forgotten "show startup-logs"
The keyword was never documented.

Must be backported in all maintained versions.
2022-10-14 15:43:28 +02:00
Aurelien DARRAGON
b5684c54d1 DOC/CLEANUP: lua-api: removing duplicate core.proxies attribute
core.proxies attribute was found 2 times in the core Class.
Removing the second occurrence, which is purely duplicate and
does not belong here.
2022-10-14 15:18:25 +02:00
Aurelien DARRAGON
846fc7d950 DOC: lua-api: updating toolbox link
Link to lua toolbox was dead (project has been deprecated).
Adding a legacy link to get old toolbox source code as well as
a link to luarocks that seems to have superseded it.
2022-10-14 15:18:25 +02:00