Commit Graph

2362 Commits

Author SHA1 Message Date
Christopher Faulet
84cdbe478a BUG/MINOR: http-htx: Don't consider an URI as normalized after a set-uri action
An abosulte URI is marked as normalized if it comes from an H2 client. This
way, we know we can send a relative URI to an H1 server. But, after a
set-uri action, the URI must no longer be considered as normalized.
Otherwise there is no way to send an absolute URI on the server side.

If it is important to update a normalized absolute URI without altering this
property, the host, path and/or query-string must be set separatly.

This patch should fix the issue #1938. It should be backported as far as
2.4.
2022-11-22 17:49:10 +01:00
Amaury Denoyelle
7078fb1f3a DOC: quic: add note on performance issue with listener contention
Complete quic4/quic6 bind lines by a note on performance issues due to
receiver socket contention. Suggest to use sharding to improve the
situation.

This should be backported up to 2.6.
2022-11-22 11:46:29 +01:00
Amaury Denoyelle
28ea31c7cb MINOR: global: generate random cluster.secret if not defined
If no cluster-secret is defined by the user, a random one is silently
generated.

This ensures that at least QUIC Retry tokens are generated if abnormal
conditions are detected. However, it is advisable to specify it in the
configuration for tokens to be valid even after a reload or across LBs
instances in the same cluster.

This should be backported up to 2.6.
2022-11-21 16:41:34 +01:00
Amaury Denoyelle
996ca7d0fa MINOR: quic: report error if force-retry without cluster-secret
QUIC Retry generation relies on global cluster-secret to produce token
valid even after a process restart and across several LBs instances.

Before this patch, Retry is automatically deactivated if no
cluster-secret is provided. This is the case even if a user has
configured a QUIC listener with quic-force-retry. Change this behavior
by now returning an error during configuration parsing. The user must
provide a cluster-secret if quic-force-retry is used.

This shoud be backported up to 2.6.
2022-11-21 16:34:09 +01:00
Amaury Denoyelle
936c135e05 DOC: configuration: fix quic prefix typo
Replace quicv4/quicv6 -> quic4/quic6 as prefix for bind lines of QUIC
listeners.

This should be backported up to 2.6.
2022-11-21 16:14:46 +01:00
Willy Tarreau
7583c36790 MINOR: cli/pools: add pool name filtering capability to "show pools"
Now it becomes possible to match a pool name's prefix, for example:

  $ socat - /tmp/haproxy.sock <<< "show pools match quic byusage"
  Dumping pools usage. Use SIGQUIT to flush them.
    - Pool quic_conn_r (65560 bytes) : 1337 allocated (87653720 bytes), ...
    - Pool quic_crypto (1048 bytes) : 6685 allocated (7005880 bytes), ...
    - Pool quic_conn (4056 bytes) : 1337 allocated (5422872 bytes), ...
    - Pool quic_rxbuf (262168 bytes) : 8 allocated (2097344 bytes), ...
    - Pool quic_connne (184 bytes) : 9359 allocated (1722056 bytes), ...
    - Pool quic_frame (184 bytes) : 7938 allocated (1460592 bytes), ...
    - Pool quic_tx_pac (152 bytes) : 6454 allocated (981008 bytes), ...
    - Pool quic_tls_ke (56 bytes) : 12033 allocated (673848 bytes), ...
    - Pool quic_rx_pac (408 bytes) : 1596 allocated (651168 bytes), ...
    - Pool quic_tls_se (88 bytes) : 6685 allocated (588280 bytes), ...
    - Pool quic_cstrea (88 bytes) : 4011 allocated (352968 bytes), ...
    - Pool quic_tls_iv (24 bytes) : 12033 allocated (288792 bytes), ...
    - Pool quic_dgram (344 bytes) : 732 allocated (251808 bytes), ...
    - Pool quic_arng (56 bytes) : 4011 allocated (224616 bytes), ...
    - Pool quic_conn_c (152 bytes) : 1337 allocated (203224 bytes), ...
  Total: 15 pools, 109578176 bytes allocated, 109578176 used ...

In this case the reported total only concerns the dumped ones.
2022-11-21 10:14:52 +01:00
Willy Tarreau
2fba08faec MINOR: cli/pools: add sorting capabilities to "show pools"
The "show pools" command is used a lot for debugging but didn't get much
love over the years. This patch brings new capabilities:
  - sorting the output by pool names to ese their finding ("byname").
  - sorting the output by reverse item size to spot the biggest ones("bysize")
  - sorting the output by reverse number of allocated bytes ("byusage")

The last one (byusage) also omits displaying the ones with zero allocation.

In addition, an optional max number of output entries may be passed so as
to dump only the N most relevant ones.
2022-11-21 10:14:52 +01:00
Mathias Weiersmueller
d9b7174d99 MEDIUM: tcp-act: add parameter rst-ttl to silent-drop
The silent-drop action was extended with an additional optional parameter,
[rst-ttl <ttl> ], causing HAProxy to send a TCP RST with the specified TTL
towards the client.

With this behaviour, the connection state on your own client-
facing middle-boxes (load balancers, firewalls) will be purged,
but the client will still assume the TCP connection is up because
the TCP RST packet expires before reaching the client.
2022-11-19 04:53:47 +01:00
Willy Tarreau
a0abec8bc0 [RELEASE] Released version 2.7-dev9
Released version 2.7-dev9 with the following main changes :
    - BUILD: quic: QUIC mux build fix for 32-bit build
    - BUILD: scripts: disable tests build on QuicTLS build
    - BUG/MEDIUM: httpclient: segfault when the httpclient parser fails
    - BUILD: ssl_sock: fix null dereference for QUIC build
    - BUILD: quic: Fix build for m68k cross-compilation
    - BUG/MINOR: quic: fix buffer overflow on retry token generation
    - MINOR: quic: add version field on quic_rx_packet
    - MINOR: quic: extend pn_offset field from quic_rx_packet
    - MINOR: quic: define first packet flag
    - MINOR: quic: extract connection retrieval
    - MINOR: quic: split and rename qc_lstnr_pkt_rcv()
    - MINOR: quic: refactor packet drop on reception
    - MINOR: quic: extend Retry token check function
    - BUG/MINOR: log: Preserve message facility when the log target is a ring buffer
    - BUG/MINOR: ring: Properly parse connect timeout
    - BUG/MEDIUM: httpclient/lua: crash when the lua task timeout before the httpclient
    - BUG/MEDIUM: httpclient: check if the httpclient was released in the IO handler
    - REGTESTS: httpclient/lua: test the lua task timeout with the httpclient
    - CI: github: dump the backtrace of coredumps in the alpine container
    - BUILD: Makefile: add "USE_SHM_OPEN" on the linux-musl target
    - DOC: lua: add a note about compression w/ httpclient
    - CLEANUP: mworker/cli: rename the status function to loadstatus
    - MINOR: mworker/cli: does no try to dump the startup-logs w/o USE_SHM_OPEN
    - MINOR: list: fixing typo in MT_LIST_LOCK_ELT
    - DOC/MINOR: list: fixing MT_LIST_LOCK_ELT macro documentation
    - MINOR: list: adding MT_LIST_APPEND_LOCKED macro
    - BUG/MINOR: mux-quic: complete flow-control for uni streams
    - BUG/MEDIUM: compression: handle rewrite errors when updating response headers
    - MINOR: quic: do not crash on unhandled sendto error
    - MINOR: quic: display unknown error sendto counter on stat page
    - MINOR: peers: Support for peer shards
    - MINOR: peers: handle multiple resync requests using shards
    - BUG/MINOR: sink: Only use backend capability for the sink proxies
    - BUG/MINOR: sink: Set default connect/server timeout for implicit ring buffers
    - MINOR: ssl: add the SSL error string when failing to load a certificate
    - MINOR: ssl: add the SSL error string before the chain
    - MEDIUM: ssl: be stricter about chain error
    - BUG/MAJOR: stick-table: don't process store-response rules for applets
    - MINOR: quic: remove unnecessary quic_session_accept()
    - BUG/MINOR: quic: fix subscribe operation
    - BUG/MINOR: log: fixing bug in tcp syslog_io_handler Octet-Counting
    - MINOR: ssl: dump the SSL string error when SSL_CTX_use_PrivateKey() failed.
    - MINOR: quic: add counter for interrupted reception
    - BUG/MINOR: quic: fix race condition on datagram purging
    - CI: add monthly gcc cross compile jobs
    - CLEANUP: assorted typo fixes in the code and comments
    - CLEANUP: ssl: remove dead code in ssl_sock_load_pem_into_ckch()
    - BUG/MINOR: httpclient: fixed memory allocation for the SSL ca_file
    - BUG/MINOR: ssl: Memory leak of DH BIGNUM fields
    - BUG/MINOR: ssl: Memory leak of AUTHORITY_KEYID struct when loading issuer
    - BUG/MINOR: ssl: ocsp structure not freed properly in case of error
    - CI: switch to the "latest" LibreSSL
    - CI: enable QUIC for LibreSSL builds
    - BUG/MEDIUM: ssl: Verify error codes can exceed 63
    - MEDIUM: ssl: {ca,crt}-ignore-err can now use error constant name
    - MINOR: ssl: x509_v_err_str converter transforms an integer to a X509_V_ERR name
    - CLEANUP: cli: rename dynamic error printing state
    - MINOR: cli: define usermsgs print context
    - MINOR: server: clear prefix on stderr logs after add server
    - BUG/MINOR: ssl: bind_conf is uncorrectly accessed when using QUIC
    - BUILD: ssl_utils: fix build on gcc versions before 8
    - BUILD: debug: remove unnecessary quotes in HA_WEAK() calls
    - CI: emit the compiler's version in the build reports
    - IMPORT: xxhash: update xxHash to version 0.8.1
    - IMPORT: slz: declare len to fix debug build when optimal match is enabled
    - IMPORT: slz: mention the potential header in slz_finish()
    - IMPORT: slz: define and use a __fallthrough statement for switch/case
    - BUILD: compiler: add a macro to detect if another one is set and equals 1
    - BUILD: compiler: add a default definition for __has_attribute()
    - BUILD: compiler: define a __fallthrough statement for switch/case
    - BUILD: sample: use __fallthrough in smp_is_rw() and smp_dup()
    - BUILD: quic: use __fallthrough in quic_connect_server()
    - BUILD: ssl/crt-list: use __fallthrough in cli_io_handler_add_crtlist()
    - BUILD: ssl: use __fallthrough in cli_io_handler_commit_{cert,cafile_crlfile}()
    - BUILD: ssl: use __fallthrough in cli_io_handler_tlskeys_files()
    - BUILD: hlua: use __fallthrough in hlua_post_init_state()
    - BUILD: stream: use __fallthrough in stats_dump_full_strm_to_buffer()
    - BUILD: tcpcheck: use __fallthrough in check_proxy_tcpcheck()
    - BUILD: stats: use __fallthrough in stats_dump_proxy_to_buffer()
    - BUILD: peers: use __fallthrough in peer_io_handler()
    - BUILD: hash: use __fallthrough in hash_djb2()
    - BUILD: tools: use __fallthrough in url_decode()
    - BUILD: args: use __fallthrough in make_arg_list()
    - BUILD: acl: use __fallthrough in parse_acl_expr()
    - BUILD: spoe: use __fallthrough in spoe_handle_appctx()
    - BUILD: logs: use __fallthrough in build_log_header()
    - BUILD: check: use __fallthrough in __health_adjust()
    - BUILD: http_act: use __fallthrough in parse_http_del_header()
    - BUILD: h1_htx: use __fallthrough in h1_parse_chunk()
    - BUILD: vars: use __fallthrough in var_accounting_{diff,add}()
    - BUILD: map: use __fallthrough in cli_io_handler_*()
    - BUILD: compression: use __fallthrough in comp_http_payload()
    - BUILD: stconn: use __fallthrough in various shutw() functions
    - BUILD: prometheus: use __fallthrough in promex_dump_metrics() and IO handler()
    - CLEANUP: ssl: remove printf in bind_parse_ignore_err
    - BUG/MINOR: ssl:  crt-ignore-err memory leak with 'all' parameter
    - BUG/MINOR: ssl: Fix potential overflow
    - CLEANUP: stick-table: remove the unused table->exp_next
    - OPTIM: stick-table: avoid atomic ops in stktable_requeue_exp() when possible
    - BUG/MEDIUM: stick-table: fix a race condition when updating the expiration task
    - MEDIUM: http-ana: remove set-cookie2 support
    - BUG/MEDIUM: wdt/clock: properly handle early task hangs
    - MINOR: deinit: add a "quick-exit" option to bypass the deinit step
    - OPTIM: ebtree: make ebmb_insert_prefix() keep a copy the new node's pfx
    - OPTIM: ebtree: make ebmb_insert_prefix() keep a copy the new node's key
    - MINOR: ssl: ssl_sock_load_cert_chain() display error strings
    - MINOR: ssl: reintroduce ERR_GET_LIB(ret) == ERR_LIB_PEM in ssl_sock_load_pem_into_ckch()
    - BUG/MINOR: http-htx: Fix error handling during parsing http replies
    - BUG/MINOR: resolvers: Don't wait periodic resolution on healthcheck failure
    - BUG/MINOR: resolvers: Set port before IP address when processing SRV records
    - BUG/MINOR: mux-fcgi: Be sure to send empty STDING record in case of zero-copy
    - BUG/MEDIUM: mux-fcgi: Avoid value length overflow when it doesn't fit at once
    - BUG/MINOR: ssl: SSL_load_error_strings might not be defined
    - MINOR: pool/debug: create a new pool_alloc_flag() macro
    - MINOR: dynbuf: switch allocation and release to macros to better track users
    - BUG/MINOR: mux-h1: Do not send a last null chunk on body-less answers
    - REG-TESTS: cache: Remove T-E header for 304-Not-Modified responses
    - DOC: config: fix alphabetical ordering of global section
    - MINOR: trace: split the CLI "trace" parser in CLI vs statement
    - MEDIUM: trace: create a new "trace" statement in the "global" section
    - BUG/MEDIUM: ring: fix creation of server in uninitialized ring
    - BUILD: quic: fix dubious 0-byte overflow on qc_release_lost_pkts
    - BUILD: makefile: mark poll and tcploop targets as phony
    - BUILD: makefile: properly pass CC to sub-projects
    - BUILD: makefile: move default verbosity settings to include/make/verbose.mk
    - BUILD: makefile: use $(cmd_MAKE) in quiet mode
    - BUILD: makefile: move the compiler option detection stuff to compiler.mk
    - DEV: poll: make the connect() step an action as well
    - DEV: poll: strip the "do_" prefix from reported function names
    - DEV: poll: indicate the FD's side in front of its value
    - BUG/MINOR: pool/cli: use ullong to report total pool usage in bytes
    - MINOR: mux-h1: Remove usless code inside shutr callback
    - CLEANUP: mux-h1; Rename H1S_F_ERROR flag into H1S_F_ERROR_MASK
    - REORG: mux-h1: Reorg the H1C structure
    - CLEANUP: mux-h1: Rename H1C_F_ST_ERROR and H1C_F_ST_SILENT_SHUT flags
    - MINOR: mux-h1: Add a dedicated enum to deal with H1 connection state
    - MEDIUM: mux-h1: Handle H1C states via its state field instead of H1C_F_ST_*
    - MINOR: mux-h1: Don't handle subscribe for reads in h1_process_demux()
    - CLEANUP: mux-h1: Rename H1C_F_ERR_PENDING into H1C_F_ABRT_PENDING
    - MINOR: mux-h1: Add flag on H1 stream to deal with internal errors
    - MEDIUM: mux-h1: Rely on the H1C to deal with shutdown for reads
    - CLEANUP: mux-h1: Reorder H1 connection flags to avoid holes
    - MEDIUM: mux-h1: Don't report a final error whe a message is aborted
    - MEDIUM: mux-pt: Don't always set a final error on SE on the sending path
    - MEDIUM: mux-h2: Introduce flags to deal with connection read/write errors
    - CLEANUP: mux-h2: Remove unused fields in h2c structures
    - MEDIUM: mux-fcgi: Introduce flags to deal with connection read/write errors
    - MINOR: sconn: Set SE_FL_ERROR only when there is no more data to read
    - MINOR: mux-h1: Rely on a H1S flag to know a WS key was found or not
    - DOC: lua-api: Remove warning about the lua filters
    - BUG/MEDIUM: listener: Fix race condition when updating the global mngmt task
    - CLEANUP: listener: Remove useless task_queue from manage_global_listener_queue
    - BUG/MINOR: mux-h1: Fix error handling when H1S allocation failed on client side
    - DOC: internal: commit notes about polling states and flags
    - DOC: internal: commit notes about polling states and flags on connect()
    - CLEANUP: mux-h1: Don't test h1c in h1_shutw_conn()
    - BUG/MINOR: http_ana/txn: don't re-initialize txn and req var lists
    - BUG/MEDIUM: raw-sock: Don't report connection error if something was received
    - BUG/MINOR: ssl: don't initialize the keylog callback when not required
    - BUILD: Makefile: enable USE_SHM_OPEN by default on freebsd
    - BUG/MEDIUM: peers: messages about unkown tables not correctly ignored
    - MINOR: cfgparse: Always check the section position
    - MEDIUM: thread: Restric nbthread/thread-group(s) to very first global sections
    - BUILD: peers: Remove unused variables
    - MINOR: ncbuf: complete doc for ncb_advance()
    - BUG/MEDIUM: quic: fix unsuccessful handshakes on ncb_advance error
    - BUG/MEDIUM: quic: fix memleak for out-of-order crypto data
    - MINOR: quic: complete traces/debug for handshake
2022-11-18 17:48:49 +01:00
Willy Tarreau
831d613f54 DOC: internal: commit notes about polling states and flags on connect()
Let's keep these notes as references for later use. Polling on connect()
can sometimes return a few unexpected state combinations that such tests
illustrate. They can serve as reminders for special error handling.
2022-11-17 16:49:00 +01:00
Willy Tarreau
c89f665cca DOC: internal: commit notes about polling states and flags
Some detailed observations were made on polling general and POLLHUP
more specifically, they can be useful later.
2022-11-17 16:49:00 +01:00
Christopher Faulet
b6c31b3cda DOC: lua-api: Remove warning about the lua filters
The lua filters api is no longer experimental. There are some filters
depending on this api, thus it is fair to make it stable now.
2022-11-17 14:33:15 +01:00
Willy Tarreau
9fd0542148 MEDIUM: trace: create a new "trace" statement in the "global" section
The exact same commands as those from the CLI may be pre-loaded at boot
time by passing them one per line after the "trace" keyword in the global
section; i.e. just copy-pasting all commands directly there will do the
job. Note that if a ring is mentioned, it needs to be declared before the
global section. Another option is to append another global section after
"ring".

For now the keyword is marked as experimental to discourage its broad
adoption by default. "expose-experimental-directives" needs to be placed
in the global section to expose it.
2022-11-16 17:55:53 +01:00
Willy Tarreau
8e6ad2548c DOC: config: fix alphabetical ordering of global section
the global section keywords were seriously misordered, and it's visible
that some mistakes have induced other ones over time, so it was about
time to fix this. Roughly 20% of the keywords were misplaced.

This commit only reordered the keywords index and their description,
nothing else was changed. It might be backported because it's a real
pain to find certain options there.
2022-11-16 17:55:53 +01:00
Willy Tarreau
e98d385819 MINOR: deinit: add a "quick-exit" option to bypass the deinit step
Once in a while we spot a bug in the deinit code that is complex,
especially when it has to deal with incomplete initializations, and the
ability to bypass this step has regularly been raised. In addition for
fast-reloading setups it could theoretically save some time. Tests have
shown that very large configs can barely save ~100-150ms by skipping the
deinit step. However the ability not to crash if a bug is encountered can
occasionally help.

This patch adds an option to do exactly this. It's obviously not enabled
by default and the documentation discourages from using it, but this might
be useful in the future.
2022-11-15 09:37:09 +01:00
William Lallemand
9fbc84e571 MINOR: ssl: x509_v_err_str converter transforms an integer to a X509_V_ERR name
The x509_v_err_str converter transforms a numerical X509 verify error
to its constant name.
2022-11-10 13:28:37 +01:00
William Lallemand
960fb74cae MEDIUM: ssl: {ca,crt}-ignore-err can now use error constant name
The ca-ignore-err and crt-ignore-err directives are now able to use the
openssl X509_V_ERR constant names instead of the numerical values.

This allow a configuration to survive an OpenSSL upgrade, because the
numerical ID can change between versions. For example
X509_V_ERR_INVALID_CA was 24 in OpenSSL 1 and is 79 in OpenSSL 3.

The list of errors must be updated when a new major OpenSSL version is
released.
2022-11-10 13:28:37 +01:00
Ilya Shipitsin
4a689dad03 CLEANUP: assorted typo fixes in the code and comments
This is 32nd iteration of typo fixes
2022-10-30 17:17:56 +01:00
Frdric Lcaille
36d1565640 MINOR: peers: Support for peer shards
Add "shards" new keyword for "peers" section to configure the number
of peer shards attached to such secions. This impact all the stick-tables
attached to the section.
Add "shard" new "server" parameter to configure the peers which participate to
all the stick-tables contents distribution. Each peer receive the stick-tables updates
only for keys with this shard value as distribution hash. The "shard" value
is stored in ->shard new server struct member.
cfg_parse_peers() which is the function which is called to parse all
the lines of a "peers" section is modified to parse the "shards" parameter
stored in ->nb_shards new peers struct member.
Add srv_parse_shard() new callback into server.c to pare the "shard"
parameter.
Implement stksess_getkey_hash() to compute the distribution hash for a
stick-table key as the 64-bits xxhash of the key concatenated to the stick-table
name. This function is called by stksess_setkey_shard(), itself
called by the already implemented function which create a new stick-table
key (stksess_new()).
Add ->idlen new stktable struct member to store the stick-table name length
to not have to compute it each time a stick-table key hash is computed.
2022-10-24 10:55:53 +02:00
William Lallemand
a9256194b8 DOC: lua: add a note about compression w/ httpclient
Decompression is not supported by the httpclient.
2022-10-21 11:50:05 +02:00
Willy Tarreau
ea8aebe8c5 [RELEASE] Released version 2.7-dev8
Released version 2.7-dev8 with the following main changes :
    - BUG/MINOR: checks: update pgsql regex on auth packet
    - DOC: config: Fix pgsql-check documentation to make user param mandatory
    - CLEANUP: mux-quic: remove usage of non-standard ull type
    - CLEANUP: quic: remove global var definition in quic_tls header
    - BUG/MINOR: quic: adjust quic_tls prototypes
    - CLEANUP: quic: fix headers
    - CLEANUP: quic: remove unused function prototype
    - CLEANUP: quic: remove duplicated varint code from xprt_quic.h
    - CLEANUP: quic: create a dedicated quic_conn module
    - BUG/MINOR: mux-quic: ignore STOP_SENDING for locally closed stream
    - BUG/MEDIUM: lua: Don't crash in hlua_lua2arg_check on failure
    - BUG/MEDIUM: lua: handle stick table implicit arguments right.
    - BUILD: h1: silence an initiialized warning with gcc-4.7 and -Os
    - MINOR: fd: add a new function to only raise RLIMIT_NOFILE
    - MINOR: init: do not try to shrink existing RLIMIT_NOFIlE
    - BUG/MINOR: http-fetch: Update method after a prefetch in smp_fetch_meth()
    - BUILD: http_fetch: silence an uninitiialized warning with gcc-4/5/6 at -Os
    - BUG/MINOR: hlua: hlua_channel_insert_data() behavior conflicts with documentation
    - MINOR: quic: limit usage of ssl_sock_ctx in favor of quic_conn
    - MINOR: mux-quic: check quic-conn return code on Tx
    - CLEANUP: quic: fix indentation
    - MEDIUM: quic: retrieve frontend destination address
    - CLEANUP: Reapply ist.cocci (2)
    - CLEANUP: Reapply strcmp.cocci
    - CLEANUP: quic/receiver: remove the now unused tx_qring list
    - BUG/MINOR: quic: set IP_PKTINFO socket option for QUIC receivers only
    - MINOR: hlua: some luaL_checktype() calls were not guarded with MAY_LJMP
    - DOC: configuration: missing 'if' in tcp-request content example
    - MINOR: hlua: removing ambiguous lua_pushvalue with 0 index
    - BUG/MAJOR: stick-tables: do not try to index a server name for applets
    - MINOR: plock: support disabling exponential back-off
    - MINOR: freq_ctr: use the thread's local time whenever possible
    - MEDIUM: stick-table: switch the table lock to rwlock
    - MINOR: stick-table: do not take an exclusive lock when downing ref_cnt
    - MINOR: stick-table: move the write lock inside stktable_touch_with_exp()
    - MEDIUM: stick-table: only take the lock when needed in stktable_touch_with_exp()
    - MEDIUM: stick-table: make stksess_kill_if_expired() avoid the exclusive lock
    - MEDIUM: stick-table: return inserted entry in __stktable_store()
    - MEDIUM: stick-table: free newly allocated stkess if it couldn't be inserted
    - MEDIUM: stick-table: switch to rdlock in stktable_lookup() and lookup_key()
    - MEDIUM: stick-table: make stktable_get_entry() look up under a read lock
    - MEDIUM: stick-table: do not take a lock to update t->current anymore.
    - MEDIUM: stick-table: make stktable_set_entry() look up under a read lock
    - MEDIUM: stick-table: requeue the expiration task out of the exclusive lock
    - MINOR: stick-table: split stktable_store() between key and requeue
    - MEDIUM: stick-table: always use atomic ops to requeue the table's task
    - MEDIUM: stick-table: requeue the wakeup task out of the write lock
    - BUG/MINOR: stick-table: fix build with DEBUG_THREAD
    - REORG: mux-fcgi: Extract flags and enums into mux_fcgi-t.h
    - MINOR: flags/mux-fcgi: Decode FCGI connection and stream flags
    - BUG/MEDIUM: mux-h1: Add connection error handling when reading/sending on a pipe
    - BUG/MEDIUM: mux-h1: Handle abort with an incomplete message during parsing
    - BUG/MINOR: server: make sure "show servers state" hides private bits
    - MINOR: checks: use the lighter PRNG for spread checks
    - MEDIUM: checks: spread the checks load over random threads
    - CI: SSL: use proper version generating when "latest" semantic is used
    - CI: SSL: temporarily stick to LibreSSL=3.5.3
    - MINOR: quic: New quic_cstream object implementation
    - MINOR: quic: Extract CRYPTO frame parsing from qc_parse_pkt_frms()
    - MINOR: quic: Use a non-contiguous buffer for RX CRYPTO data
    - BUG/MINOR: quic: Stalled 0RTT connections with big ClientHello TLS message
    - MINOR: quic: Split the secrets key allocation in two parts
    - CLEANUP: quic: remove unused rxbufs member in receiver
    - CLEANUP: quic: improve naming for rxbuf/datagrams handling
    - MINOR: quic: implement datagram cleanup for quic_receiver_buf
    - MINOR: ring: ring_cast_from_area() cast from an allocated area
    - MINOR: buffers: split b_force_xfer() into b_cpy() and b_force_xfer()
    - MINOR: logs: startup-logs can use a shm for logging the reload
    - MINOR: mworker/cli: reload command displays the startup-logs
    - MEDIUM: quic: respect the threads assigned to a bind line
    - DOC: management: update the "reload" command of the master CLI
    - BUILD: ssl_sock: bind_conf uninitialized in ssl_sock_bind_verifycbk()
    - BUG/MEDIUM: httpclient: Don't set EOM flag on an empty HTX message
    - MINOR: httpclient/lua: Don't set req_payload callback if body is empty
    - DOC/CLEANUP: lua-api: some minor corrections
    - DOC: lua-api: updating toolbox link
    - DOC/CLEANUP: lua-api: removing duplicate core.proxies attribute
    - DOC: management: add forgotten "show startup-logs"
    - DOC: management: "show startup-logs" for master CLI
    - CI: Replace the deprecated `::set-output` command by writing to $GITHUB_OUTPUT in matrix.py
    - CI: Replace the deprecated `::set-output` command by writing to $GITHUB_OUTPUT in workflow definition
2022-10-14 20:45:23 +02:00
William Lallemand
5d1e1317c2 DOC: management: "show startup-logs" for master CLI
"show startup-logs" on the master CLI has a slighly different behavior.

No backport needed.
2022-10-14 15:46:19 +02:00
William Lallemand
f76b3b47ea DOC: management: add forgotten "show startup-logs"
The keyword was never documented.

Must be backported in all maintained versions.
2022-10-14 15:43:28 +02:00
Aurelien DARRAGON
b5684c54d1 DOC/CLEANUP: lua-api: removing duplicate core.proxies attribute
core.proxies attribute was found 2 times in the core Class.
Removing the second occurrence, which is purely duplicate and
does not belong here.
2022-10-14 15:18:25 +02:00
Aurelien DARRAGON
846fc7d950 DOC: lua-api: updating toolbox link
Link to lua toolbox was dead (project has been deprecated).
Adding a legacy link to get old toolbox source code as well as
a link to luarocks that seems to have superseded it.
2022-10-14 15:18:25 +02:00
Aurelien DARRAGON
53901f423d DOC/CLEANUP: lua-api: some minor corrections
This is a minor cleanup before 2.7 release to correct some typos
and errors without changing the meaning.
2022-10-14 15:18:25 +02:00
William Lallemand
ef3e5a1b68 DOC: management: update the "reload" command of the master CLI
Update the "reload" command with the new output format.
2022-10-13 18:16:12 +02:00
Aurelien DARRAGON
d49b559a15 DOC: configuration: missing 'if' in tcp-request content example
An example given for tcp-request content rule with lua
was missing 'if' keyword. Using it "as is" makes haproxy unhappy.

The example was introduced with 579d83b05.
So it may be backported as far as 1.6, but it is a really minor typo.
2022-10-12 09:22:05 +02:00
Christopher Faulet
59307b3e4e DOC: config: Fix pgsql-check documentation to make user param mandatory
The username is required in the Start-up message. Thus, since the 2.2, when
this health-check was refactored, the user parameter is mandatory. On prior
versions, when no username is provided, no pgsql check is performed but only
a basic tcpcheck.

This patch should be backported as far as 2.2.
2022-10-03 15:31:23 +02:00
Willy Tarreau
dd4a2a69dc [RELEASE] Released version 2.7-dev7
Released version 2.7-dev7 with the following main changes :
    - BUG/MEDIUM: mux-quic: fix nb_hreq decrement
    - CLEANUP: httpclient: deleted unused variables
    - MINOR: httpclient: enabled the use of SNI presets
    - OPTIM: hpack-huff: reduce the cache footprint of the huffman decoder
    - BUG/MINOR: mux-quic: do not keep detached qcs with empty Tx buffers
    - REORG: mux-quic: extract traces in a dedicated source file
    - REORG: mux-quic: export HTTP related function in a dedicated file
    - MINOR: mux-quic: refactor snd_buf
    - BUG/MEDIUM: mux-quic: properly trim HTX buffer on snd_buf reset
    - BUG/MINOR: mux-h1: Account consumed output data on synchronous connection error
    - BUG/MINOR: log: improper behavior when escaping log data
    - CLEANUP: tools: removing escape_chunk() function
    - MINOR: clock: split local and global date updates
    - MINOR: pollers: only update the local date during busy polling
    - MINOR: clock: do not update the global date too often
    - REGTESTS: 4be_1srv_smtpchk_httpchk_layer47errors: Return valid SMTP replies
    - MINOR: smtpchk: Update expect rule to fully match replies to EHLO commands
    - BUG/MINOR: smtpchk: SMTP Service check should gracefully close SMTP transaction
    - MINOR: list: documenting mt_list_for_each_entry_safe() macro
    - CLEANUP: list: Fix mt_list_for_each_entry_safe indentation
    - BUG/MINOR: hlua: Remove \n in Lua error message built with memprintf
    - MINOR: hlua: Allow argument on lua-lod(-per-thread) directives
    - BUG/MINOR: anon: memory illegal accesses in tools.c with hash_anon and hash_ipanon
    - MEDIUM: mworker/cli: keep the connection of the FD that ask for a reload
    - BUG/MINOR: hlua: fixing ambiguous sizeof in hlua_load_per_thread
    - MINOR: mworker/cli: replace close() by fd_delete()
    - MINOR: mworker: store and shows loading status
    - MINOR: mworker: mworker_cli_proxy_new_listener() returns a bind_conf
    - MINOR: mworker: stores the mcli_reload bind_conf
    - MINOR: mworker/cli: the mcli_reload bind_conf only send the reload status
    - DOC: management: describe the new reload command behavior
    - CLEANUP: list: fix again some style issues in the recent comments
    - BUG/MINOR: stream: Perform errors handling in right order in stream_new()
    - BUG/MEDIUM: stconn: Reset SE descriptor when we fail to create a stream
    - BUG/MEDIUM: resolvers: Remove aborted resolutions from query_ids tree
    - DOC: management: add timeout on the "reload" command
    - BUG/MINOR: ring: fix the size check in ring_make_from_area()
    - BUG/MINOR: config: don't count trailing spaces as empty arg
    - Revert "BUG/MINOR: config: don't count trailing spaces as empty arg"
    - BUG/MINOR: hlua: fixing hlua_http_msg_del_data behavior
    - BUG/MINOR: hlua: fixing hlua_http_msg_insert_data behavior
    - MINOR: cli: Add anonymization on a missed element for 'show sess all'
    - MINOR: cli: remove error message with 'set anon on|off'
    - MINOR: tools: modify hash_ipanon in order to use it in cli
    - MINOR: cli: use hash_ipanon to anonymized address
    - MINOR: cli: Add an anonymization on a missed element in 'show server state'
    - MINOR: config: correct errors about argument number in condition in cfgparse.c
    - MINOR: config: Add other keywords when dump the anonymized configuration file
    - MINOR: config: Add option line when the configuration file is dumped
    - MINOR: cli: correct commentary and replace 'set global-key' name
    - MINOR: tools: Impprove hash_ipanon to support dgram sockets and port offsets
    - MINOR: tools: Impprove hash_ipanon to not hash FD-based addresses
    - BUG/MINOR: hlua: _hlua_http_msg_delete incorrect behavior when offset is used
    - DOC: management: httpclient can resolve server names in URLs
    - BUG/MINOR: hlua: prevent crash when loading numerous arguments using lua-load(per-thread)
    - DOC/CLEANUP: lua-api: removing duplicate date functions doc
    - MINOR: hlua: ambiguous lua_pushvalue with 0 index
    - BUG/MINOR: config: don't count trailing spaces as empty arg (v2)
    - BUG/MEDIUM: config: count line arguments without dereferencing the output
    - BUG/MAJOR: conn-idle: fix hash indexing issues on idle conns
    - BUG/MINOR: config: insufficient syntax check of the global "maxconn" value
    - BUG/MINOR: backend: only enforce turn-around state when not redispatching
2022-10-03 15:20:38 +02:00
Aurelien DARRAGON
28f0a60a75 DOC/CLEANUP: lua-api: removing duplicate date functions doc
As reported by Thierry, core.asctime_date() and core.rfc850() were
both documented multiple times in lua-api doc.

This does not need to be backported.
2022-09-30 15:21:20 +02:00
William Lallemand
9ae05bb1e0 DOC: management: httpclient can resolve server names in URLs
The httpclient does support DNS resolution since 2.6.

Must be backported to 2.6.
2022-09-29 15:00:15 +02:00
Erwan Le Goas
d78693178c MINOR: cli: correct commentary and replace 'set global-key' name
Correct a commentary in in include/haproxy/global-t.h and include/haproxy/tools.h
Replace the CLI command 'set global-key <key>' by 'set anon global-key <key>' in
order to find it easily when you don't remember it, the recommandation can guide
you when you just tap 'set anon'.

No backport needed, except if anonymization mechanism is backported.
2022-09-29 10:53:15 +02:00
William Lallemand
bb650f2be8 DOC: management: add timeout on the "reload" command
Add some details about timeout during a reload on the master CLI.
2022-09-27 11:48:59 +02:00
William Lallemand
70c5ad4425 DOC: management: describe the new reload command behavior
The master CLI command now allows to do a synchronous reload with a
status.
2022-09-24 16:47:04 +02:00
Thierry Fournier
ae6b56800f MINOR: hlua: Allow argument on lua-lod(-per-thread) directives
Allow per-lua file argument which makes multiples configuration
easier to handle

This patch fixes issue #1609.
2022-09-22 15:24:29 +02:00
Willy Tarreau
4b10a5c439 [RELEASE] Released version 2.7-dev6
Released version 2.7-dev6 with the following main changes :
    - MINOR: Revert part of clarifying samples support per os commit
    - BUILD: makefile: enable crypt(3) for NetBSD
    - BUG/MINOR: quic: Retransmitted frames marked as acknowledged
    - BUG/MINOR: quic: Possible crash with "tls-ticket-keys" on QUIC bind lines
    - MINOR: http-check: Remove support for headers/body in "option httpchk" version
    - BUG/MINOR: h1: Support headers case adjustment for TCP proxies
    - BUG/MINOR: quic: Possible crash when verifying certificates
    - BUILD: quic: add some ifdef around the SSL_ERROR_* for libressl
    - BUILD: ssl: fix ssl_sock_switchtx_cbk when no client_hello_cb
    - BUILD: quic: temporarly ignore chacha20_poly1305 for libressl
    - BUILD: quic: enable early data only with >= openssl 1.1.1
    - BUILD: ssl: fix the ifdef mess in ssl_sock_initial_ctx
    - BUILD: quic: fix the #ifdef in ssl_quic_initial_ctx()
    - MINOR: quic: add QUIC support when no client_hello_cb
    - MINOR: quic: Add traces about sent or resent TX frames
    - MINOR: quic: No TRACE_LEAVE() in retrieve_qc_conn_from_cid()
    - BUG/MINOR: quic: Wrong connection ID to thread ID association
    - BUG/MINOR: task: always reset a new tasklet's call date
    - BUG/MINOR: task: make task_instant_wakeup() work on a task not a tasklet
    - MINOR: task: permanently enable latency measurement on tasklets
    - CLEANUP: task: rename ->call_date to ->wake_date
    - BUG/MINOR: sched: properly account for the CPU time of dying tasks
    - MINOR: sched: store the current profile entry in the thread context
    - BUG/MINOR: stream/sched: take into account CPU profiling for the last call
    - MINOR: tasks: do not keep cpu and latency times in struct task
    - MINOR: tools: add generic pointer hashing functions
    - CLEANUP: activity: make memprof use the generic ptr_hash() function
    - CLEANUP: activity: make taskprof use ptr_hash()
    - MINOR: debug: add struct ha_caller to describe a calling location
    - CLEANUP: debug: use struct ha_caller for memstat
    - DEBUG: task: define a series of wakeup types for tasks and tasklets
    - DEBUG: task: use struct ha_caller instead of arrays of file:line
    - DEBUG: applet: instrument appctx_wakeup() to log the caller's location
    - DEBUG: task: simplify the caller recording in DEBUG_TASK
    - CLEANUP: task: move tid and wake_date into the common part
    - CLEANUP: sched: remove duplicate code in run_tasks_from_list()
    - CLEANUP: activity: make the number of sched activity entries more configurable
    - DEBUG: resolvers: unstatify process_resolvers() to make it appear in profiling
    - DEBUG: quic: export the few task handlers that often appear in task dumps
    - MEDIUM: tasks/activity: combine the called function with the caller
    - MINOR: tasks/activity: improve the caller-callee activity hash
    - MINOR: activity/cli: support aggregating task profiling outputs
    - MINOR: activity/cli: support sorting task profiling by total CPU time
    - BUG/MINOR: signals/poller: set the poller timeout to 0 when there are signals
    - BUG/MINOR: quic: Speed up the handshake completion only one time
    - BUG/MINOR: quic: Trace fix about packet number space information.
    - BUG/MINOR: h3: Crash when h3 trace verbosity is "minimal"
    - MINOR: h3: Add the quic_conn object to h3 traces
    - MINOR: h3: Missing connection argument for a TRACE_LEAVE() argument
    - MINOR: h3: Send the h3 settings with others streams (requests)
    - MINOR: dev/udp: Apply the corruption to both directions
    - BUILD: udp-perturb: Add a make target for udp-perturb tool
    - BUG/MINOR: signals/poller: ensure wakeup from signals
    - CI: cirrus-ci: bump FreeBSD image to 13-1
    - DEV: flags: fix usage message to reflect available options
    - DEV: flags: add missing CO_FL_FDLESS connection flag
    - MINOR: flags: add a new file to host flag dumping macros
    - MINOR: flags: implement a macro used to dump enums inside masks
    - MINOR: flags/channel: use flag dumping for channel flags and analysers
    - MINOR: flags/connection: use flag dumping for connection flags
    - MINOR: flags/stconn: use flag dumping for stconn and sedesc flags
    - MINOR: flags/stream: use flag dumping for stream error type
    - MINOR: flags/stream: use flag dumping for stream flags
    - MINOR: flags/task: use flag dumping for task state
    - MINOR: flags/http_ana: use flag dumping for txn flags
    - DEV: flags: remove the now unused SHOW_FLAG() definition
    - DEV: flags: remove the now useless intermediary functions
    - MINOR: flags/htx: use flag dumping to show htx and start-line flags
    - MINOR: flags/http_ana: use flag dumping to show http msg states
    - BUG/MEDIUM: proxy: ensure pause_proxy() and resume_proxy() own PROXY_LOCK
    - MINOR: listener: small API change
    - MINOR: proxy/listener: support for additional PAUSED state
    - BUG/MINOR: stats: fixing stat shows disabled frontend status as 'OPEN'
    - BUILD: flags: fix build warning in some macros used by show_flags
    - BUILD: flags: fix the fallback macros for missing stdio
    - CLEANUP: pollers: remove dead code in the polling loop
    - BUG/MINOR: mux-h1: Increment open_streams counter when H1 stream is created
    - REGTESTS: healthcheckmail: Relax matching on the healthcheck log message
    - CLEANUP: listener: function comment typo in stop_listener()
    - BUG/MINOR: listener: null pointer dereference suspected by coverity
    - MINOR: flags/fd: decode FD flags states
    - REORG: mux-h2: extract flags and enums into mux_h2-t.h
    - MINOR: flags/mux-h2: decode H2C and H2S flags
    - REGTESTS: log: test the log-forward feature
    - BUG/MEDIUM: sink: bad init sequence on tcp sink from a ring.
    - REGTESTS: ssl/log: test the log-forward with SSL
    - MEDIUM: httpclient: httpclient_create_proxy() creates a proxy for httpclient
    - MEDIUM: httpclient: allow to use another proxy
    - DOC: fix TOC in starter guide for subsection 3.3.8. Statistics
    - MINOR: httpclient: export httpclient_create_proxy()
    - MEDIUM: quic: separate path for rx and tx with set_encryption_secrets
    - BUG/MEDIUM: mux-quic: fix crash on early app-ops release
    - REORG: mux-h1: extract flags and enums into mux_h1-t.h
    - MINOR: flags/mux-h1: decode H1C and H1S flags
    - CLEANUP: mux-quic: remove stconn usage in h3/hq
    - BUG/MINOR: mux-quic: do not remotely close stream too early
    - CLEANUP: exclude udp-perturb with .gitignore
    - BUG/MEDIUM: server: segv when adding server with hostname from CLI
    - CLEANUP: quic,ssl: fix tiny typos in C comments
    - BUG/MEDIUM: captures: free() an error capture out of the proxy lock
    - BUILD: fd: fix a build warning on the DWCAS
    - MINOR: anon: add new macros and functions to anonymize contents
    - MINOR: anon: store the anonymizing key in the global structure
    - MINOR: anon: store the anonymizing key in the CLI's appctx
    - MINOR: cli: anonymize commands 'show sess' and 'show sess all'
    - MINOR: cli: anonymize 'show servers state' and 'show servers conn'
    - MINOR: config: add command-line -dC to dump the configuration file
    - SCRIPTS: announce-release: update some URLs to https
2022-09-17 12:24:53 +02:00
Erwan Le Goas
b0c0501516 MINOR: config: add command-line -dC to dump the configuration file
This commit adds a new command line option -dC to dump the configuration
file. An optional key may be appended to -dC in order to produce an
anonymized dump using this key. The anonymizing process uses the same
algorithm as the CLI so that the same key will produce the same hashes
for the same identifiers. This way an admin may share an anonymized
extract of a configuration to match against live dumps. Note that key 0
will not anonymize the output. However, in any case, the configuration
is dumped after tokenizing, thus comments are lost.
2022-09-17 11:27:09 +02:00
Erwan Le Goas
54966dffda MINOR: anon: store the anonymizing key in the CLI's appctx
In order to allow users to dump internal states using a specific key
without changing the global one, we're introducing a key in the CLI's
appctx. This key is preloaded from the global one when "set anon on"
is used (and if none exists, a random one is assigned). And the key
can optionally be assigned manually for the whole CLI session.

A "show anon" command was also added to show the anon state, and the
current key if the users has sufficient permissions. In addition, a
"debug dev hash" command was added to test the feature.
2022-09-17 11:27:09 +02:00
Erwan Le Goas
fad9da83da MINOR: anon: store the anonymizing key in the global structure
Add a uint32_t key in global to hash words with it. A new CLI command
'set global-key <key>' was added to change the global anonymizing key.
The global may also be set in the configuration using the global
"anonkey" directive. For now this key is not used.
2022-09-17 11:24:53 +02:00
Mathias Weiersmueller
243c2d1822 DOC: fix TOC in starter guide for subsection 3.3.8. Statistics
This subsection has been moved from 3.4.9 to 3.3.8 somewhere along
2.4, but the TOC has not been updated - resulting in a invalid
anchor in the HTML version.

Needs to be backported to 2.4+
2022-09-14 06:01:55 +02:00
Willy Tarreau
e86bc35672 MINOR: activity/cli: support sorting task profiling by total CPU time
The new "bytime" sorting criterion uses the reported CPU time instead of
the usage. This is convenient to spot tasks that are mostly reponsible
for the CPU usage in a running process. It supports both the detailed
and the aggregated format. The output looks like this:

> show profiling tasks bytime
Tasks activity:
  function                      calls   cpu_tot   cpu_avg   lat_tot   lat_avg
  qc_io_cb                     117739   1.961m    999.1us   37.45s    318.1us <- h3_snd_buf@src/h3.c:1084 tasklet_wakeup
  process_stream              7376273   1.384m    11.26us   1.013h    494.2us <- stream_new@src/stream.c:563 task_wakeup
  process_stream              8104400   1.133m    8.389us   1.130h    502.0us <- sc_notify@src/stconn.c:1209 task_wakeup
  qc_io_cb                      43280   45.76s    1.057ms   13.95s    322.3us <- qc_stream_desc_ack@src/quic_stream.c:128 tasklet_wakeup
  h1_io_cb                   11025715   24.82s    2.251us   5.406m    29.42us <- sock_conn_iocb@src/sock.c:869 tasklet_wakeup
  quic_conn_app_io_cb          312861   23.86s    76.27us   2.373s    7.584us <- qc_lstnr_pkt_rcv@src/xprt_quic.c:6184 tasklet_wakeup_after
  qc_io_cb                      37063   12.65s    341.4us   6.409s    172.9us <- qc_treat_acked_tx_frm@src/xprt_quic.c:1695 tasklet_wakeup
  h1_io_cb                    4783520   11.79s    2.463us   1.419h    1.068ms <- conn_subscribe@src/connection.c:732 tasklet_wakeup
  sc_conn_io_cb              12269693   11.51s    938.0ns   2.117h    621.2us <- sc_app_chk_rcv_conn@src/stconn.c:762 tasklet_wakeup
  sc_conn_io_cb               6479006   10.94s    1.689us   7.984m    73.93us <- h1_wake_stream_for_recv@src/mux_h1.c:2600 tasklet_wakeup
  qc_io_cb                      12011   10.72s    892.5us   2.120s    176.5us <- qcc_release_remote_stream@src/mux_quic.c:1200 tasklet_wakeup
  h2_io_cb                     246423   6.225s    25.26us   56.52s    229.4us <- h2_snd_buf@src/mux_h2.c:6712 tasklet_wakeup
  h2_io_cb                     137744   6.076s    44.11us   16.59s    120.4us <- sock_conn_iocb@src/sock.c:869 tasklet_wakeup
  quic_lstnr_dghdlr            323575   3.062s    9.462us   3.424m    634.9us <- quic_lstnr_dgram_dispatch@src/quic_sock.c:255 tasklet_wakeup
  sc_conn_io_cb               1206939   1.616s    1.338us   27.62m    1.373ms <- qcs_notify_send@src/mux_quic.c:529 tasklet_wakeup
  h2_io_cb                     212370   251.2ms   1.182us   6.476s    30.49us <- h2c_restart_reading@src/mux_h2.c:856 tasklet_wakeup
  h1_io_cb                      44109   197.0ms   4.466us   31.89s    723.0us <- h1_takeover@src/mux_h1.c:4085 tasklet_wakeup
  quic_conn_app_io_cb            3029   87.59ms   28.92us   999.0ms   329.8us <- qc_process_timer@src/xprt_quic.c:4635 tasklet_wakeup
  task_run_applet                  40   35.77ms   894.3us   4.407ms   110.2us <- sc_applet_create@src/stconn.c:489 appctx_wakeup
  task_run_applet                  18   27.36ms   1.520ms   19.56us   1.086us <- sc_app_chk_snd_applet@src/stconn.c:996 appctx_wakeup
  sc_conn_io_cb                  2186   11.76ms   5.377us   963.0ms   440.5us <- h1_wake_stream_for_send@src/mux_h1.c:2610 tasklet_wakeup
  qc_io_cb                          8   9.880ms   1.235ms   5.871ms   733.9us <- qcs_consume@src/mux_quic.c:800 tasklet_wakeup
  quic_conn_io_cb                   4   5.951ms   1.488ms   38.85us   9.713us <- qc_lstnr_pkt_rcv@src/xprt_quic.c:6184 tasklet_wakeup_after
  qc_io_cb                        101   4.975ms   49.26us   13.91ms   137.8us <- qc_process_timer@src/xprt_quic.c:4602 tasklet_wakeup
  h1_io_cb                       2186   1.809ms   827.0ns   720.2ms   329.5us <- sock_conn_iocb@src/sock.c:849 tasklet_wakeup
  qc_process_timer               3031   1.735ms   572.0ns   1.153s    380.3us <- wake_expired_tasks@src/task.c:344 task_wakeup
  accept_queue_process            359   1.362ms   3.793us   80.32ms   223.7us <- listener_accept@src/listener.c:1099 tasklet_wakeup
  quic_conn_app_io_cb               2   921.1us   460.6us   203.1us   101.5us <- qc_xprt_start@src/xprt_quic.c:7122 tasklet_wakeup
  h1_timeout_task                2618   526.8us   201.0ns   1.121s    428.4us <- h1_release@src/mux_h1.c:1087 task_wakeup
  process_resolvers               316   283.3us   896.0ns   14.96ms   47.33us <- wake_expired_tasks@src/task.c:429 task_drop_running
  sc_conn_io_cb                   420   235.6us   560.0ns   116.7ms   277.8us <- h2s_notify_recv@src/mux_h2.c:1298 tasklet_wakeup
  qc_idle_timer_task                1   225.5us   225.5us   506.0ns   506.0ns <- wake_expired_tasks@src/task.c:344 task_wakeup
  accept_queue_process             36   153.0us   4.250us   5.834ms   162.1us <- accept_queue_process@src/listener.c:165 tasklet_wakeup
  sc_conn_io_cb                    18   54.05us   3.003us   11.50us   638.0ns <- sock_conn_iocb@src/sock.c:869 tasklet_wakeup
  h2_io_cb                          6   38.88us   6.480us   2.089ms   348.2us <- h2_do_shutw@src/mux_h2.c:4656 tasklet_wakeup
  srv_cleanup_idle_conns           54   37.72us   698.0ns   14.21ms   263.1us <- wake_expired_tasks@src/task.c:429 task_drop_running
  sc_conn_io_cb                    50   32.86us   657.0ns   28.83ms   576.5us <- qcs_notify_recv@src/mux_quic.c:519 tasklet_wakeup
  qc_io_cb                          2   30.25us   15.12us   6.093us   3.046us <- qc_init@src/mux_quic.c:2057 tasklet_wakeup
  srv_cleanup_toremove_conns        1   27.16us   27.16us   905.6us   905.6us <- srv_cleanup_idle_conns@src/server.c:5948 task_wakeup
  task_run_applet                  39   19.61us   502.0ns   818.7us   20.99us <- run_tasks_from_lists@src/task.c:652 task_drop_running
  quic_accept_run                   2   15.46us   7.727us   305.5us   152.8us <- quic_accept_push_qc@src/quic_sock.c:458 tasklet_wakeup
  h2_timeout_task                  32   12.91us   403.0ns   4.207ms   131.5us <- h2_release@src/mux_h2.c:1191 task_wakeup
  quic_conn_app_io_cb               1   9.645us   9.645us   1.445us   1.445us <- qc_process_timer@src/xprt_quic.c:4589 tasklet_wakeup

> show profiling tasks bytime aggr
Tasks activity:
  function                      calls   cpu_tot   cpu_avg   lat_tot   lat_avg
  qc_io_cb                     212301   3.147m    889.5us   1.009m    285.2us
  process_stream             15503573   2.519m    9.747us   2.148h    498.7us
  h1_io_cb                   15916733   36.95s    2.321us   1.535h    347.1us
  quic_conn_app_io_cb          318845   24.21s    75.92us   3.410s    10.70us
  sc_conn_io_cb              20037058   24.19s    1.207us   2.737h    491.8us
  h2_io_cb                     596543   12.55s    21.04us   1.326m    133.4us
  quic_lstnr_dghdlr            326624   3.094s    9.473us   3.462m    635.9us
  task_run_applet                 100   64.43ms   644.3us   5.285ms   52.85us
  quic_conn_io_cb                   4   5.951ms   1.488ms   38.85us   9.713us
  qc_process_timer               3061   1.750ms   571.0ns   1.162s    379.5us
  accept_queue_process            396   1.521ms   3.840us   86.16ms   217.6us
  h1_timeout_task                2618   526.8us   201.0ns   1.121s    428.4us
  process_resolvers               319   286.0us   896.0ns   16.82ms   52.73us
  qc_idle_timer_task                1   225.5us   225.5us   506.0ns   506.0ns
  srv_cleanup_idle_conns           54   37.72us   698.0ns   14.21ms   263.1us
  srv_cleanup_toremove_conns        1   27.16us   27.16us   905.6us   905.6us
  quic_accept_run                   2   15.46us   7.727us   305.5us   152.8us
  h2_timeout_task                  32   12.91us   403.0ns   4.207ms   131.5us
2022-09-08 16:38:10 +02:00
Willy Tarreau
dc89b1806c MINOR: activity/cli: support aggregating task profiling outputs
By default we now dump stats between caller and callee, but by
specifying "aggr" on the command line, stats get aggregated by
callee again as it used to be before the feature was available.
It may sometimes be helpful when comparing total call counts,
though that's about all.
2022-09-08 16:32:17 +02:00
Christopher Faulet
4b5f3029bc MINOR: http-check: Remove support for headers/body in "option httpchk" version
This trick is deprecated since the health-check refactoring, It is now
invalid. It means the following line will trigger an error during the
configuration parsing:

  option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www

It must be replaced by:

  option httpchk OPTIONS * HTTP/1.1
  http-check send hdr Host www
2022-09-06 18:23:14 +02:00
Willy Tarreau
3bb2b5db50 [RELEASE] Released version 2.7-dev5
Released version 2.7-dev5 with the following main changes :
    - BUG/MINOR: mux-quic: Fix memleak on QUIC stream buffer for unacknowledged data
    - BUG/MEDIUM: cpu-map: fix thread 1's affinity affecting all threads
    - MINOR: cpu-map: remove obsolete diag warning about combined ranges
    - BUG/MAJOR: mworker: fix infinite loop on master with no proxies.
    - REGTESTS: launch http_reuse_always in mworker mode
    - BUG/MINOR: quix: Memleak for non in flight TX packets
    - BUG/MINOR: quic: Wrong list_for_each_entry() use when building packets from qc_do_build_pkt()
    - BUG/MINOR: quic: Safer QUIC frame builders
    - MINOR: quic: Replace MT_LISTs by LISTs for RX packets.
    - BUG/MEDIUM: applet: fix incorrect check for abnormal return condition from handler
    - BUG/MINOR: applet: make the call_rate only count the no-progress calls
    - MEDIUM: peers: limit the number of updates sent at once
    - BUILD: tcp_sample: fix build of get_tcp_info() on OpenBSD
    - BUG/MINOR: resolvers: return the correct value in resolvers_finalize_config()
    - BUG/MINOR: mworker: does not create the "default" resolvers in wait mode
    - BUG/MINOR: tcpcheck: Disable QUICKACK only if data should be sent after connect
    - REGTESTS: Fix prometheus script to perform HTTP health-checks
    - MINOR: resolvers: shut the warning when "default" resolvers is implicit
    - Revert "BUG/MINOR: quix: Memleak for non in flight TX packets"
    - BUG/MINOR: quic: Leak in qc_release_lost_pkts() for non in flight TX packets
    - BUG/MINOR: quic: Stalled connections (missing I/O handler wakeup)
    - CLEANUP: quic: No more use ->rx_list MT_LIST entry point (quic_rx_packet)
    - CLEANUP: quic: Remove a useless check in qc_lstnr_pkt_rcv()
    - MINOR: quic: Remove useless traces about references to TX packets
    - Revert "MINOR: quic: Remove useless traces about references to TX packets"
    - DOC: configuration: do-resolve doesn't work with a port in the string
    - MINOR: sample: add the host_only and port_only converters
    - BUG/MINOR: httpclient: fix resolution with port
    - DOC: configuration.txt: do-resolve must use host_only to remove its port.
    - BUG/MINOR: quic: Null packet dereferencing from qc_dup_pkt_frms() trace
    - BUG/MINOR: quic: Frames added to packets even if not built.
    - BUG/MEDIUM: spoe: Properly update streams waiting for a ACK in async mode
    - BUG/MEDIUM: peers: Add connect and server timeut to peers proxy
    - BUG/MEDIUM: peers: Don't use resync timer when local resync is in progress
    - BUG/MEDIUM: peers: Don't start resync on reload if local peer is not up-to-date
    - BUG/MINOR: hlua: Rely on CF_EOI to detect end of message in HTTP applets
    - BUG/MEDIUM: mux-h1: do not refrain from signaling errors after end of input
    - BUG/MINOR: epoll: do not actively poll for Rx after an error
    - MINOR: raw-sock: don't try to send if an error was already reported
    - BUG/MINOR: quic: Missing header protection AES cipher context initialisations (draft-v2)
    - MINOR: quic: Add a trace to distinguish the datagram from the packets inside
    - BUG/MINOR: ssl: fix deinit of the ca-file tree
    - BUG/MINOR: ssl: leak of ckch_inst_link in ckch_inst_free()
    - BUG/MINOR: tcpcheck: Disable QUICKACK for default tcp-check (with no rule)
    - BUG/MEDIUM: ssl: Fix a UAF when old ckch instances are released
    - BUG/MINOR: ssl: revert two wrong fixes with ckhi_link
    - BUG/MINOR: dev/udp: properly preset the rx address size
    - BUILD: debug: make sure debug macros are never empty
    - MINOR: quic: Move traces about RX/TX bytes from QUIC_EV_CONN_PRSAFRM event
    - BUG/MINOR: quic: TX frames memleak
    - BUG/MINOR: ssl: leak of ckch_inst_link in ckch_inst_free() v2
    - MINOR: sink/ring: rotate non-empty file-backed contents only
    - BUG/MINOR: regex: Properly handle PCRE2 lib compiled without JIT support
    - REGTESTS: http_request_buffer: Add a barrier to not mix up log messages
    - BUG/MEDIUM: mux-h1: always use RST to kill idle connections in pools
    - MINOR: backend: always satisfy the first req reuse rule with l7 retries
    - BUG/MINOR: quic: Do not ack when probing
    - MINOR: quic: Add TX frames addresses to traces to several trace events
    - MINOR: quic: Trace typo fix in qc_release_frm()
    - BUG/MINOR: quic: Frames leak during retransmissions
    - BUG/MINOR: h2: properly set the direction flag on HTX response
    - BUG/MEDIUM: httpclient: always detach the caller before self-killing
    - BUG/MINOR: httpclient: only ask for more room on failed writes
    - BUG/MINOR: httpclient: keep-alive was accidentely disabled
    - MEDIUM: httpclient: enable ALPN support on outgoing https connections
    - BUG/MINOR: mux-h2: fix the "show fd" dest buffer for the subscriber
    - BUG/MINOR: mux-h1: fix the "show fd" dest buffer for the subscriber
    - BUG/MINOR: mux-fcgi: fix the "show fd" dest buffer for the subscriber
    - DEBUG: stream: minor rearrangement of a few fields in struct stream.
    - MINOR: debug: report applet pointer and handler in crashes when known
    - MINOR: mux-h2: extract the stream dump function out of h2_show_fd()
    - MINOR: mux-h2: extract the connection dump function out of h2_show_fd()
    - MINOR: muxes: add a "show_sd" helper to complete "show sess" dumps
    - MINOR: mux-h2: provide a "show_sd" helper to output stream debugging info
    - MINOR: mux-h2: insert line breaks in "show sess all" output for legibility
    - MINOR: mux-quic: provide a "show_sd" helper to output stream debugging info
    - MINOR: mux-h1: split "show_fd" into connection and stream
    - MINOR: mux-h1: provide a "show_sd" helper to output stream debugging info
    - BUG/MINOR: http-act: initialize http fmt head earlier
2022-09-02 19:36:50 +02:00
Willy Tarreau
32872db605 MINOR: sink/ring: rotate non-empty file-backed contents only
If the service is rechecked before a reload, that may cause the config
to be parsed twice and file-backed rings to be lost.

Here we make sure that such a ring does contain information before
deciding to rotate it. This way the first process starting after some
writes will cause a rotate but not subsequent ones until new writes
are applied.

An attempt was also made to disable rotations on checks but this was a
bad idea, as the ring is still initialized and this causes the contents
to be lost. The choice of initializing the ring during parsing is
questionable but the config check ought to be as close as possible to a
real start, and we could imagine that the ring is used by some code
during startup (e.g. lua). So this approach was abandonned and config
checks also cause a rotation, as the purpose of this rotation is to
preserve latest information against accidental removal.
2022-09-01 08:25:34 +02:00
William Lallemand
b5c2cd461d DOC: configuration.txt: do-resolve must use host_only to remove its port.
The do-resolve action does not support a port in its parameter string,
the host_only converter must be used.

Must be backported to 2.6.
2022-08-26 17:00:22 +02:00
William Lallemand
dd754cba16 MINOR: sample: add the host_only and port_only converters
Add 2 converters that can manipulate the value of an Host header.
host_only will return the host without any port, and port_only will
return the port.
2022-08-26 17:00:22 +02:00
William Lallemand
1ef2460934 DOC: configuration: do-resolve doesn't work with a port in the string
Fix the documentation about do-resolve to handle the case where a port
is associated to the hostname in the Host header.

Must be backported as far as 2.0.
2022-08-26 16:51:05 +02:00
Willy Tarreau
8bd146d8af MEDIUM: peers: limit the number of updates sent at once
As seen in GH issue #1770, peers synchronization do not cope well with
very large buffers because by default the only two reasons for stopping
the processing of updates is either that the end was reached or that
the buffer is full. This can cause high latencies, and even rightfully
trigger the watchdog when the operations are numerous and slowed down
by competition on the stick-table lock.

This patch introduces a limit to the number of messages one may send
at once, which now defaults to 200, regardless of the buffer size. This
means taking and releasing the lock up to 400 times in a row, which is
costly enough to let some other parts work.

After some observation this could be backported to 2.6. If so, however,
previous commits "BUG/MEDIUM: applet: fix incorrect check for abnormal
return condition from handler" and "BUG/MINOR: applet: make the call_rate
only count the no-progress calls" must be backported otherwise the call
rate might trigger the looping protection.
2022-08-23 20:19:11 +02:00