Commit Graph

16195 Commits

Author SHA1 Message Date
Tim Duesterhus
9b80a6ca8f DEV: coccinelle: Add rule to use isttrim() where possible
This replaces `if (i.len > e) i.len = e;` by `isttrim(i, e)`.
2021-11-08 12:08:26 +01:00
Willy Tarreau
e9f4d67b15 OPTIM: halog: skip fields 64 bits at a time when supported
Some architectures like x86_64 and aarch64 support efficient unaligned
64-bit reads. On such architectures, we already know that each string
passed to field_start() has some margin at the end because it's parsed
using fgets2() which looks for the trailing LF using the same method.
Thus let's skip spaces by packs of 8. This increases the parsing speed
by 35%.
2021-11-08 12:08:26 +01:00
Willy Tarreau
fc76bbc0f5 OPTIM: halog: improve field parser speed for modern compilers
Modern compilers were producing producing less efficient code in the
field_start() loop, by not emitting two conditional jumps for a single
test. However by reordering the test we can merge the optimal case and
the default one and get back to good performance so let's simplify the
test. This improves the parsing speed by 5%.
2021-11-08 12:08:26 +01:00
Willy Tarreau
fce4e69144 CLEANUP: halog: remove unused strl2ui()
strl2ui() isn't used anymore in the code, likely because str2ic() is
often used instead. Let's drop it.
2021-11-08 12:08:26 +01:00
Frédéric Lécaille
c4becf5424 MINOR: quic: Fix potential null pointer dereference
Fix compilation warnings about non initialized pointers.

This partially address #1445 github issue.
2021-11-08 11:31:12 +01:00
Amaury Denoyelle
b9ce14e5a2 MINOR: h3: fix potential NULL dereference
Fix potential allocation failure of HTX start-line during H3 request
decoding. In this case, h3_decode_qcs returns -1 as error code.

This addresses in part github issue #1445.
2021-11-08 09:17:24 +01:00
Amaury Denoyelle
7bb54f9906 MINOR: mux-quic: fix gcc11 warning
Fix minor warnings about an unused variable.

This addresses in part github issue #1445.
2021-11-08 08:59:30 +01:00
Amaury Denoyelle
3cae4049b0 MINOR: h3/qpack: fix gcc11 warnings
Fix minor warnings about unused variables and mixed declarations.

This addresses in part github issue #1445.
2021-11-08 08:59:30 +01:00
Willy Tarreau
87e7eafde4 CLEANUP: halog: make the default usage message fit in small screens
The usage message was starting to have long lines, it's preferable that
it still fits well into a default 80-col display so that options are
easy to find. Also cut that into the 3 parts (input filter, modifier,
output format) for improved legibility.
2021-11-08 08:39:28 +01:00
Tim Duesterhus
16cc16dd82 CLEANUP: Re-apply xalloc_size.cocci
Use a consistent size as the parameter for the *alloc family.
2021-11-08 08:05:39 +01:00
Tim Duesterhus
4c8f75fc31 CLEANUP: Apply ist.cocci
Make use of the new rules to use `istend()`.
2021-11-08 08:05:39 +01:00
Tim Duesterhus
958f50454a DEV: coccinelle: Add rule to use istend() where possible
This replaces `i.ptr + i.len` by `istend()`.
2021-11-08 07:58:18 +01:00
Tim Duesterhus
9c523f1042 DEV: coccinelle: Remove unused expression e
Introduced in ef00c533e1.
2021-11-08 07:58:18 +01:00
Willy Tarreau
08d3220de5 [RELEASE] Released version 2.5-dev13
Released version 2.5-dev13 with the following main changes :
    - SCRIPTS: git-show-backports: re-enable file-based filtering
    - MINOR: jwt: Make invalid static JWT algorithms an error in `jwt_verify` converter
    - MINOR: mux-h2: add trace on extended connect usage
    - BUG/MEDIUM: mux-h2: reject upgrade if no RFC8441 support
    - MINOR: stream/mux: implement websocket stream flag
    - MINOR: connection: implement function to update ALPN
    - MINOR: connection: add alternative mux_ops param for conn_install_mux_be
    - MEDIUM: server/backend: implement websocket protocol selection
    - MINOR: server: add ws keyword
    - BUG/MINOR: resolvers: fix sent messages were counted twice
    - BUG/MINOR: resolvers: throw log message if trash not large enough for query
    - MINOR: resolvers/dns: split dns and resolver counters in dns_counter struct
    - MEDIUM: resolvers: rename dns extra counters to resolvers extra counters
    - BUG/MINOR: jwt: Fix jwt_parse_alg incorrectly returning JWS_ALG_NONE
    - DOC: add QUIC instruction in INSTALL
    - CLEANUP: halog: Remove dead stores
    - DEV: coccinelle: Add ha_free.cocci
    - CLEANUP: Apply ha_free.cocci
    - DEV: coccinelle: Add rule to use `istnext()` where possible
    - CLEANUP: Apply ist.cocci
    - REGTESTS: Use `feature cmd` for 2.5+ tests (2)
    - DOC: internals: move some API definitions to an "api" subdirectory
    - MINOR: quic: Allocate listener RX buffers
    - CLEANUP: quic: Remove useless code
    - MINOR: quic: Enhance the listener RX buffering part
    - MINOR: quic: Remove a useless lock for CRYPTO frames
    - MINOR: quic: Use QUIC_LOCK QUIC specific lock label.
    - MINOR: backend: Get client dst address to set the server's one only if needful
    - MINOR: compression: Warn for 'compression offload' in defaults sections
    - MEDIUM: connection: rename fc_conn_err and bc_conn_err to fc_err and bc_err
    - DOC: configuration: move the default log formats to their own section
    - MINOR: ssl: make the ssl_fc_sni() sample-fetch function always available
    - MEDIUM: log: add the client's SNI to the default HTTPS log format
    - DOC: config: add an example of reasonably complete error-log-format
    - DOC: config: move error-log-format before custom log format
2021-11-06 09:25:57 +01:00
Willy Tarreau
ec5c110e2d DOC: config: move error-log-format before custom log format
All default formats were described before the custom one, except this
one. Better place them all together before the custom log format. This
only swaps and renumbers the sections.
2021-11-06 09:20:07 +01:00
Willy Tarreau
ecc79bbe28 DOC: config: add an example of reasonably complete error-log-format
This commit adds a suggestion of a useful error-log-format that was
tested with success in production.
2021-11-06 09:20:07 +01:00
Willy Tarreau
68574dd492 MEDIUM: log: add the client's SNI to the default HTTPS log format
During a troublehooting it came obvious that the SNI always ought to
be logged on httpslog, as it explains errors caused by selection of
the default certificate (or failure to do so in case of strict-sni).

This expectation was also confirmed on the mailing list.

Since the field may be empty it appeared important not to leave an
empty string in the current format, so it was decided to place the
field before a '/' preceding the SSL version and ciphers, so that
in the worst case a missing field leads to a field looking like
"/TLSv1.2/AES...", though usually a missing element still results
in a "-" in logs.

This will change the log format for users who already deployed the
2.5-dev versions (hence the medium level) but no released version
was using this format yet so there's no harm for stable deployments.
The reg-test was updated to check for "-" there since we don't send
SNI in reg-tests.

Link: https://www.mail-archive.com/haproxy@formilux.org/msg41410.html
Cc: William Lallemand <wlallemand@haproxy.org>
2021-11-06 09:20:07 +01:00
Willy Tarreau
579259d150 MINOR: ssl: make the ssl_fc_sni() sample-fetch function always available
Its definition is enclosed inside an ifdef SSL_CTRL_SET_TLSEXT_HOSTNAME
which is defined since OpenSSL 0.9.8. Having it conditioned like this
prevents us from using it by default in a log format, which could cause
an error on an old or exotic library.

Let's just always define it and make the sample fetch fail to return
anything on such libs instead.
2021-11-06 09:20:07 +01:00
Willy Tarreau
2ed7350f4c DOC: configuration: move the default log formats to their own section
I'm always having a very hard time finding the log-format definition of
httplog, because it's not in the httplog description, and looking for
"httplog" doesn't yield the custom log formats section.

It would make more sense to write these log-formats into their respective
sections where they will be easier to find. That's what this commit does.
2021-11-06 09:20:07 +01:00
Willy Tarreau
6f7497616e MEDIUM: connection: rename fc_conn_err and bc_conn_err to fc_err and bc_err
Commit 3d2093af9 ("MINOR: connection: Add a connection error code sample
fetch") added these convenient sample-fetch functions but it appears that
due to a misunderstanding the redundant "conn" part was kept in their
name, causing confusion, since "fc" already stands for "front connection".

Let's simply call them "fc_err" and "bc_err" to match all other related
ones before they appear in a final release. The VTC they appeared in were
also updated, and the alpha sort in the keywords table updated.

Cc: William Lallemand <wlallemand@haproxy.org>
2021-11-06 09:20:07 +01:00
Christopher Faulet
44d34bfbe7 MINOR: compression: Warn for 'compression offload' in defaults sections
This directive is documented as being ignored if set in a defaults
section. But it is only mentionned in a small note in the configuration
manual. Thus, now, a warning is emitted. To do so, the errors handling in
parse_compression_options() function was slightly changed.

In addition, this directive is now documented apart from the other
compression directives. This way, it is clearly visible that it must not be
used in a defaults section.
2021-11-05 16:36:42 +01:00
Christopher Faulet
34a3eb4c42 MINOR: backend: Get client dst address to set the server's one only if needful
In alloc_dst_address(), the client destination address must only be
retrieved when we are sure to use it. Most of time, this save a syscall to
getsockname(). It is not a bugfix in itself. But it revealed a bug in the
QUIC part. The CO_FL_ADDR_TO_SET flag is not set when the destination
address is create for anew quic client connection.
2021-11-05 15:25:34 +01:00
Frédéric Lécaille
b0006eee09 MINOR: quic: Use QUIC_LOCK QUIC specific lock label.
Very minor modifications without any impact.
2021-11-05 15:20:04 +01:00
Frédéric Lécaille
46ea033be0 MINOR: quic: Remove a useless lock for CRYPTO frames
->frms_rwlock is an old lock supposed to be used when several threads
could handle the same connection. This is no more the case since this
commit:
 "MINOR: quic: Attach the QUIC connection to a thread."
2021-11-05 15:20:04 +01:00
Frédéric Lécaille
324ecdafbb MINOR: quic: Enhance the listener RX buffering part
Add a buffer per QUIC connection. At this time the listener which receives
the UDP datagram is responsible of identifying the underlying QUIC connection
and must copy the QUIC packets to its buffer.
->pkt_list member has been added to quic_conn struct to enlist the packets
in the order they have been copied to the connection buffer so that to be
able to consume this buffer when the packets are freed. This list is locked
thanks to a R/W lock to protect it from concurent accesses.
quic_rx_packet struct does not use a static buffer anymore to store the QUIC
packets contents.
2021-11-05 15:20:04 +01:00
Frédéric Lécaille
c5c69a0ad2 CLEANUP: quic: Remove useless code
Remove old I/O handler implementation (listener and server).
At this time keep a defined but not used function for servers (qc_srv_pkt_rcv()).
2021-11-05 15:20:04 +01:00
Frédéric Lécaille
c1029f6182 MINOR: quic: Allocate listener RX buffers
At this time we allocate an RX buffer by thread.
Also take the opportunity offered by this patch to rename TX related variable
names to distinguish them from the RX part.
2021-11-05 15:20:04 +01:00
Willy Tarreau
a62f184d3d DOC: internals: move some API definitions to an "api" subdirectory
It's not always easy to figure that there are some docs on internal API
stuff, let's move them to their own directory. There's a diagram for
lists that could be placed there but instead would deserve a greppable
description for quick lookups, so it was not moved there.
2021-11-05 11:53:22 +01:00
Tim Duesterhus
41922af957 REGTESTS: Use feature cmd for 2.5+ tests (2)
This patch effectively is identical to 7ba98480cc.
2021-11-05 08:27:32 +01:00
Tim Duesterhus
284fbe1214 CLEANUP: Apply ist.cocci
Make use of the new rules to use `istnext()`.
2021-11-05 07:48:38 +01:00
Tim Duesterhus
ef00c533e1 DEV: coccinelle: Add rule to use istnext() where possible
This matches both `istadv(..., 1)` as well as raw `.ptr++` uses.
2021-11-05 07:48:38 +01:00
Tim Duesterhus
025b93e3a2 CLEANUP: Apply ha_free.cocci
Use `ha_free()` where possible.
2021-11-05 07:48:38 +01:00
Tim Duesterhus
cc17a6e1d3 DEV: coccinelle: Add ha_free.cocci
Taken from 61cfdf4fd8.
2021-11-05 07:48:38 +01:00
Tim Duesterhus
785b84bb8f CLEANUP: halog: Remove dead stores
Found using clang's scan-build.
2021-11-05 07:48:38 +01:00
Amaury Denoyelle
ad3683be36 DOC: add QUIC instruction in INSTALL
Add a new section about QUIC compilation, based on QUICTLS.
2021-11-03 18:32:22 +01:00
Remi Tricot-Le Breton
7266350181 BUG/MINOR: jwt: Fix jwt_parse_alg incorrectly returning JWS_ALG_NONE
jwt_parse_alg would mistakenly return JWT_ALG_NONE for algorithms "",
"n", "no" and "non" because of a strncmp misuse. It now sees them as
unknown algorithms.

No backport needed.

Cc: Tim Duesterhus <tim@bastelstu.be>
2021-11-03 17:19:48 +01:00
Emeric Brun
f8642ee826 MEDIUM: resolvers: rename dns extra counters to resolvers extra counters
This patch renames all dns extra counters and stats functions, types and
enums using the 'resolv' prefix/suffixes.

The dns extra counter domain id used on cli was replaced by "resolvers"
instead of "dns".

The typed extra counter prefix dumping resolvers domain "D." was
also renamed "N." because it points counters on a Nameserver.

This was done to finish the split between "resolver" and "dns" layers
and to avoid further misunderstanding when haproxy will handle dns
load balancing.

This should not be backported.
2021-11-03 17:16:46 +01:00
Emeric Brun
d174f0e59a MINOR: resolvers/dns: split dns and resolver counters in dns_counter struct
This patch add a union and struct into dns_counter struct to split
application specific counters.

The only current existing application is the resolver.c layer but
in futur we could handle different application such as dns load
balancing with others specific counters.

This patch should not be backported.
2021-11-03 17:16:46 +01:00
Emeric Brun
0161d32df2 BUG/MINOR: resolvers: throw log message if trash not large enough for query
Before this patch the sent error counter was increased
for each targeted nameserver as soon as we were unable to build
the query message into the trash buffer. But this counter is here
to count sent errors at dns.c transport layer and this error is not
related to a nameserver.

This patch stops to increase those counters and sent a log message
to signal the trash buffer size is not large enough to build the query.

Note: This case should not happen except if trash size buffer was
customized to a very low value.

The function was also re-worked to return -1 in this error case
as it was specified in comment. This function is currently
called at multiple point in resolver.c but return code
is still not yet handled. So to advert the user of the malfunction
the log message was added.

This patch should be backported on all versions including the
layer split between dns.c and resolver.c (v >= 2.4)
2021-11-03 17:16:46 +01:00
Emeric Brun
c37caab21c BUG/MINOR: resolvers: fix sent messages were counted twice
The sent messages counter was increased at both resolver.c and dns.c
layers.

This patch let the dns.c layer count the sent messages since this
layer handle a retry if transport layer is not ready (EAGAIN on udp
or tcp session ring buffer full).

This patch should be backported on all versions using a split of those
layers for resolving (v >=2.4)
2021-11-03 17:16:46 +01:00
Amaury Denoyelle
f9d5957cd9 MINOR: server: add ws keyword
Implement parsing for the server keyword 'ws'. This is used to configure
the mode of selection for websocket protocol. The configuration
documentation has been updated.

A new regtest has been created to test the proper behavior of the
keyword.
2021-11-03 16:24:48 +01:00
Amaury Denoyelle
9c3251d108 MEDIUM: server/backend: implement websocket protocol selection
Handle properly websocket streams if the server uses an ALPN with both
h1 and h2. Add a new field h2_ws in the server structure. If set to off,
reuse is automatically disable on backend and ALPN is forced to http1.x
if possible. Nothing is done if on.

Implement a mechanism to be able to use a different http version for
websocket streams. A new server member <ws> represents the algorithm to
select the protocol. This can overrides the server <proto>
configuration. If the connection uses ALPN for proto selection, it is
updated for websocket streams to select the right protocol.

Three mode of selection are implemented :
- auto : use the same protocol between non-ws and ws streams. If ALPN is
  use, try to update it to "http/1.1"; this is only done if the server
  ALPN contains "http/1.1".
- h1 : use http/1.1
- h2 : use http/2.0; this requires the server to support RFC8441 or an
  error will be returned by haproxy.
2021-11-03 16:24:48 +01:00
Amaury Denoyelle
ac03ef26e8 MINOR: connection: add alternative mux_ops param for conn_install_mux_be
Add a new parameter force_mux_ops. This will be useful to specify an
alternative to the srv->mux_proto field. If non-NULL, it will be use to
force the mux protocol wether srv->mux_proto is set or not.

This argument will become useful to install a mux for non-standard
streams, most notably websocket streams.
2021-11-03 16:24:48 +01:00
Amaury Denoyelle
2454bda140 MINOR: connection: implement function to update ALPN
Implement a new function to update the ALPN on an existing connection.
on an existing connection. The ALPN from the ssl context can be checked
to update the ALPN only if it is a subset of the context value.

This method will be useful to change a connection ALPN for websocket,
must notably if the server does not support h2 websocket through the
rfc8441 Extended Connect.
2021-11-03 16:24:48 +01:00
Amaury Denoyelle
90ac605ef3 MINOR: stream/mux: implement websocket stream flag
Define a new stream flag SF_WEBSOCKET and a new cs flag CS_FL_WEBSOCKET.
The conn-stream flag is first set by h1/h2 muxes if the request is a
valid websocket upgrade. The flag is then converted to SF_WEBSOCKET on
the stream creation.

This will be useful to properly manage websocket streams in
connect_server().
2021-11-03 16:24:48 +01:00
Amaury Denoyelle
0df043608f BUG/MEDIUM: mux-h2: reject upgrade if no RFC8441 support
The RFC8441 was not respected by haproxy in regards with server support
for Extended CONNECT. The Extended CONNECT method was used to convert an
Upgrade header stream even if no SETTINGS_ENABLE_CONNECT_PROTOCOL was
received, which is forbidden by the RFC8441. In this case, the behavior
of the http/2 server is unspecified.

Fix this by flagging the connection on receiption of the RFC8441
settings SETTINGS_ENABLE_CONNECT_PROTOCOL. Extended CONNECT is thus only
be used if the flag is present. In the other case, the stream is
immediatly closed as there is no way to handle it in http/2. It results
in a http/1.1 502 or http/2 RESET_STREAM to the client side.

The protocol-upgrade regtest has been extended to test that haproxy does
not emit Extended CONNECT on servers without RFC8441 support.

It must be backported up to 2.4.
2021-11-03 16:24:48 +01:00
Amaury Denoyelle
e0c258c84d MINOR: mux-h2: add trace on extended connect usage
Add a state trace to report that a protocol upgrade is converted using
the rfc8441 Extended connect method. This is useful in regards with the
recent changes to improve http/2 websockets.
2021-11-03 11:42:02 +01:00
Tim Duesterhus
ab896ee3f7 MINOR: jwt: Make invalid static JWT algorithms an error in jwt_verify converter
It is not useful to start a configuration where an invalid static string is
provided as the JWT algorithm. Better make the administrator aware of the
suspected typo by failing to start.
2021-11-03 11:15:32 +01:00
Willy Tarreau
0d026edaef SCRIPTS: git-show-backports: re-enable file-based filtering
The early version of the script used to support passing non-branch
arguments but as it evolved we lost that option. Let's use "--" as a
delimiter after the branch(es) to pass optional file names to filter
on. This is convenient to list missing patches on a specific set of
files.
2021-11-03 08:41:01 +01:00
Willy Tarreau
35dc13f224 [RELEASE] Released version 2.5-dev12
Released version 2.5-dev12 with the following main changes :
    - MINOR: httpclient: support payload within a buffer
    - MINOR: httpclient/lua: support more HTTP methods
    - MINOR: httpclient/lua: return an error when it can't generate the request
    - CLEANUP: lua: Remove any ambiguities about lua txn execution context flags
    - BUG/MEDIUM: lua: fix invalid return types in hlua_http_msg_get_body
    - CLEANUP: connection: No longer export make_proxy_line_v1/v2 functions
    - CLEANUP: tools: Use const address for get_net_port() and get_host_port()
    - CLEANUP: lua: Use a const address to retrieve info about a connection
    - MINOR: connection: Add function to get src/dst without updating the connection
    - MINOR: session: Add src and dst addresses to the session
    - MINOR: stream-int: Add src and dst addresses to the stream-interface
    - MINOR: frontend: Rely on client src and dst addresses at stream level
    - MINOR: log: Rely on client addresses at the appropriate level to log messages
    - MINOR: session: Rely on client source address at session level to log error
    - MINOR: http-ana: Rely on addresses at stream level to set xff and xot headers
    - MINOR: http-fetch: Rely on addresses at stream level in HTTP sample fetches
    - MINOR: mux-fcgi: Rely on client addresses at stream level to set default params
    - MEDIUM: tcp-sample: Rely on addresses at the appropriate level in tcp samples
    - MEDIUM: connection: Rely on addresses at stream level to make proxy line
    - MEDIUM: backend: Rely on addresses at stream level to init server connection
    - MEDIUM: connection: Assign session addresses when PROXY line is received
    - MEDIUM: connection: Assign session addresses when NetScaler CIP proto is parsed
    - MEDIUM: tcp-act: Set addresses at the apprioriate level in set-(src/dst) actions
    - MINOR: tcp-act: Add set-src/set-src-port for "tcp-request content" rules
    - DOC: config: Fix alphabetical order of fc_* samples
    - MINOR: tcp-sample: Add samples to get original info about client connection
    - REGTESTS: Add script to test client src/dst manipulation at different levels
    - MINOR: stream: Use backend stream-interface dst address instead of target_addr
    - BUILD: log: Fix compilation without SSL support
    - DEBUG: protocol: yell loudly during registration of invalid sock_domain
    - MINOR: protocols: add a new protocol type selector
    - MINOR: protocols: make use of the protocol type to select the protocol
    - MINOR: protocols: replace protocol_by_family() with protocol_lookup()
    - MINOR: halog: Add -qry parameter allowing to preserve the query string in -uX
    - CLEANUP: jwt: Remove the use of a trash buffer in jwt_jwsverify_hmac()
    - CLEANUP: jwt: Remove the use of a trash buffer in jwt_jwsverify_rsa_ecdsa()
    - DEV: coccinelle: Add realloc_leak.cocci
    - CLEANUP: hlua: Remove obsolete branch in `hlua_alloc()`
    - BUILD: atomic: prefer __atomic_compare_exchange_n() for __ha_cas_dw()
    - BUILD: atomic: fix build on mac/arm64
    - MINOR: atomic: remove the memcpy() call and dependency on string.h
    - MINOR: httpclient: request streaming with a callback
    - MINOR: httpclient/lua: handle the streaming into the lua applet
    - REGTESTS: lua: test httpclient with body streaming
    - DOC: halog: Move the `-qry` parameter into the correct section in help text
    - MINOR: halog: Rename -qry to -query
    - CLEANUP: halog: Use consistent indentation in help()
    - BUG/MINOR: halog: Add missing newlines in die() messages
    - MINOR: halog: Add support for extracting captures using -hdr
    - DOC: Typo fixed "it" should be "is"
    - BUG/MINOR: mux-h1: Save shutdown mode if the shutdown is delayed
    - BUG/MEDIUM: mux-h1: Perform a connection shutdown when the h1c is released
    - BUG/MEDIUM: resolvers: Don't recursively perform requester unlink
    - BUG/MEDIUM: http-ana: Drain request data waiting the tarpit timeout expiration
    - BUG/MINOR: http: Authorization value can have multiple spaces after the scheme
    - BUG/MINOR: http: http_auth_bearer fetch does not work on custom header name
    - BUG/MINOR: httpclient/lua: misplaced luaL_buffinit()
    - BUILD/MINOR: cpuset freebsd build fix
    - BUG/MINOR: httpclient: use a placeholder value for Host header
    - BUG/MEDIUM: stream-int: Block reads if channel cannot receive more data
    - BUG/MEDIUM: resolvers: Track api calls with a counter to free resolutions
    - MINOR: stream: Improve dump of bogus streams
    - DOC/peers: some grammar fixes for peers 2.1 spec
    - MEDIUM: vars: make the var() sample fetch function really return type ANY
    - MINOR: vars: add "set-var" for "tcp-request connection" rules.
2021-11-02 18:05:41 +01:00