Commit Graph

17475 Commits

Author SHA1 Message Date
Willy Tarreau
4201ab791d CLEANUP: muxes: make mux->attach/detach take a conn_stream endpoint
The mux ->detach() function currently takes a conn_stream. This causes
an awkward situation where the caller cs_detach_endp() has to partially
mark it as released but not completely so that ->detach() finds its
endpoint and context, and it cannot be done later since it's possible
that ->detach() deletes the endpoint. As such the endpoint link between
the conn_stream and the mux's stream is in a transient situation while
we'd like it to be clean so that the mux's ->detach() code can call any
regular function it wants that knows the regular semantics of the
relation between the CS and the endpoint.

A better approach consists in slightly modifying the detach() API to
better match the reality, which is that the endpoint is detached but
still alive and that it's the only part the function is interested in.

As such, this patch modifies the function to take an endpoint there,
and by analogy (or simplicity) does the same for ->attach(), even
though it looks less important there since we're always attaching an
endpoint to a conn_stream anyway. It is possible that in the future
the API could evolve to use more endpoints that provide a bit more
flexibility in the API, but at this point we don't need to go further.
2022-05-13 14:28:48 +02:00
Willy Tarreau
cfbfc3f091 MINOR: mux-pt: remove the now unneeded conn_stream from the context
Since we always have a valid endpoint we can safely use it to access
the conn_stream and stop using ctx->cs. That's one less pointer to
care about.
2022-05-13 14:28:48 +02:00
Willy Tarreau
01c2a4a86f MINOR: mux-quic: remove the now unneeded conn_stream from the qcs
Since we always have a valid endpoint we can safely use it to access
the conn_stream and stop using qcs->cs. That's one less pointer to
care about.
2022-05-13 14:28:48 +02:00
Willy Tarreau
b57669e6a4 MINOR: mux-fcgi: remove the now unneeded conn_stream from the fcgi_strm
Since we always have a valid endpoint we can safely use it to access
the conn_stream and stop using fstrm->cs. That's one less pointer to
care about.
2022-05-13 14:28:48 +02:00
Willy Tarreau
c84610c2ac MINOR: mux-fcgi: make sure any stream always has an endpoint
The principle that each mux stream should have an endpoint is not
guaranteed for closed streams that map to the dummy static streams.
Let's have a dummy endpoint for use with such streams. It only has
the DETACHED flag and a NULL conn_stream, and is referenced by all
the closed streams so that we can afford not to test strm->endp when
trying to access the flags or the CS.
2022-05-13 14:28:48 +02:00
Willy Tarreau
cd6bb1a5e5 MINOR: mux-h2: remove the now unneeded conn_stream from the h2s
Since we always have a valid endpoint we can safely use it to access
the conn_stream and stop using h2s->cs. That's one less pointer to
care about.
2022-05-13 14:28:48 +02:00
Willy Tarreau
b22b5f02af MINOR: mux-h2: make sure any h2s always has an endpoint
The principle that each mux stream should have an endpoint is not
guaranteed for closed streams that map to the dummy static streams.
Let's have a dummy endpoint for use with such streams. It only has
the DETACHED flag and a NULL conn_stream, and is referenced by all
the closed streams so that we can afford not to test h2s->endp when
trying to access the flags or the CS.
2022-05-13 14:28:48 +02:00
Willy Tarreau
56d5a819d7 MINOR: mux-h1: remove the now unneeded h1s->cs
There is always an endpoint link in a stream, and this endpoint link
contains a pointer to the conn_stream it's attached to, so the one in
the h1 stream is always duplicate now. Let's always use endp->cs
instead and get rid of it.
2022-05-13 14:28:48 +02:00
Willy Tarreau
efb4618c6e MINOR: conn_stream: add a pointer back to the cs from the endpoint
Muxes and applets need to have both a pointer to the endpoint and to the
conn_stream. It would seem more natural that they only have a pointer to
the endpoint (that is always there) and that this one has an optional
pointer to the conn_stream. This would reduce the number of elements to
manipulate in lower level code. In addition, the conn_stream is not much
used from the lower layers (wake and exceptional events mostly).
2022-05-13 14:28:48 +02:00
Willy Tarreau
66435e5f63 CLEANUP: applet: use the appctx's endp instead of cs->endp
The few applets that set CS_EP_EOI or CS_EP_ERROR used to set it on the
endpoint retrieved from the conn_stream while it's already available on
the appctx itself. Better use the appctx one to limit the unneeded
interactions between the two sides.
2022-05-13 14:28:46 +02:00
Willy Tarreau
15b0721ca5 CLEANUP: mux-quic: always take the endp from the qcs not the cs
At a few places the endpoint pointer was retrieved from the conn_stream
while it's safer and more long-term proof to take it from the qcs. Let's
just do that.
2022-05-13 14:27:57 +02:00
Willy Tarreau
7d299c284b CLEANUP: mux-fcgi: always take the endp from the fstrm not the cs
At a few places the endpoint pointer was retrieved from the conn_stream
while it's safer and more long-term proof to take it from the fstrm.
Let's just do that.
2022-05-13 14:27:57 +02:00
Willy Tarreau
7a2705f921 CLEANUP: mux-pt: always take the endp from the context not the cs
At a few places the endpoint pointer was retrieved from the conn_stream
while it's safer and more long-term proof to take it from the context.
Let's just do that.
2022-05-13 14:27:57 +02:00
Willy Tarreau
aff21f9a96 CLEANUP: mux-h2: always take the endp from the h2s not the cs
At a few places the endpoint pointer was retrieved from the conn_stream
while it's safer and more long-term proof to take it from the h2s. Let's
just do that.
2022-05-13 14:27:57 +02:00
Willy Tarreau
61533d3486 CLEANUP: mux-h1: always take the endp from the h1s not the cs
At a few places the endpoint pointer was retrieved from the conn_stream
while it's safer and more long-term proof to take it from the h1s. Let's
just do that.
2022-05-13 14:27:57 +02:00
Willy Tarreau
386346f5eb MINOR: conn_stream: make cs_set_error() work on the endpoint instead
Wherever we need to report an error, we have an even easier access to
the endpoint than the conn_stream. Let's first adjust the API to use
the endpoint and rename the function accordingly to cs_ep_set_error().
2022-05-13 14:27:57 +02:00
Christopher Faulet
b112b1d02a CLEANUP: mux-h1: Fix comments and error messages for global options
Wrong name was used in comments and error messages for
"h1-header-case-adjust" and "h1-headers-case-adjust-file" global options.
2022-05-13 12:04:24 +02:00
Christopher Faulet
0f9c0f5801 MINOR: mux-h1: Add global option accpet payload for any HTTP/1.0 requests
Since the 2.5, for security reason, HTTP/1.0 GET/HEAD/DELETE requests with a
payload are rejected (See e136bd12a "MEDIUM: mux-h1: Reject HTTP/1.0
GET/HEAD/DELETE requests with a payload" for details). However it may be an
issue for old clients.

To avoid any compatibility issue with such clients,
"h1-accept-payload-with-any-method" global option was added. It must only be
set if there is a good reason to do so because it may lead to a request
smuggling attack on some servers or intermediaries.

This patch should solve the issue #1691. it may be backported to 2.5.
2022-05-13 12:04:24 +02:00
William Lallemand
ae053b30da BUG/MEDIUM: wdt: don't trigger the watchdog when p is unitialized
In wdt_handler(), does not try to trigger the watchdog if the
prev_cpu_time wasn't initialized.

This prevents an unexpected trigger of the watchdog when it wasn't
initialized yet. This case could happen in the master just after loading
the configuration. This would show a trace where the <diff> value is equal
to the <now> value in the trace, and the <poll> value would be 0.

For example:

    Thread 1 is about to kill the process.
    *>Thread 1 : id=0x0 act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
                  stuck=1 prof=0 harmless=0 wantrdv=0
                  cpu_ns: poll=0 now=6005541706 diff=6005541706
                  curr_task=0

Thanks to Christian Ruppert for repporting the problem.

Could be backported in every stable versions.
2022-05-13 11:28:08 +02:00
Boyang Li
60cfe8b823 DOC/MINOR: fix typos in the lua-api document
Fixes a few typos in the Lua API document.
2022-05-13 08:40:08 +02:00
Boyang Li
e0c5435d76 BUG/MEDIUM: lua: fix argument handling in data removal functions
Lua API Channel.remove() and HTTPMessage.remove() expects 1 to 3
arguments (counting the manipulated object), with offset and length
being the 2nd and 3rd argument, respectively.

hlua_{channel,http_msg}_del_data() incorrectly gets the 3rd argument as
offset, and 4th (nonexistent) as length. hlua_http_msg_del_data() also
improperly checks arguments. This patch fixes argument handling in both.

Must be backported to 2.5.
2022-05-13 08:40:03 +02:00
Amaury Denoyelle
eeeeed44ea MINOR: ncbuf: write unit tests
Implement a series of unit test to validate ncbuf. This is written with
a main function which can be compiled independently using the following
command-line :
 $ gcc -DSTANDALONE -lasan -I./include -o ncbuf src/ncbuf.c

The first part tests is used to test ncb_add()/ncb_advance(). After each
call a loop is done on the buffer blocks which should ensure that the
gap infos are correct.

The second part generates random offsets and insert them until the
buffer is full. The buffer is then resetted and all random offsets are
re-inserted in the reverse order : the buffer should be full once again.

The generated binary takes arguments to change the tests execution.
 "usage: ncbuf [-r] [-s bufsize] [-h bufhead] [-p <delay_msec>]"
2022-05-12 18:29:55 +02:00
Amaury Denoyelle
df25acf47f MINOR: ncbuf: implement advance
A new function ncb_advance() is implemented. This is used to advance the
buffer head pointer. This will consume the front data while forming a
new gap at the end for future data.

On success NCB_RET_OK is returned. The operation can be rejected if a
too small new gap is formed in front of the buffer.
2022-05-12 18:29:55 +02:00
Amaury Denoyelle
b830f0d8d9 MINOR: ncbuf: define various insertion modes
Define three different ways to proceed insertion. This configures how
overlapping data is treated.
- NCB_ADD_PRESERVE : in this mode, old data are kept during insertion.
- NCB_ADD_OVERWRT : new data will overwrite old ones.
- NCB_ADD_COMPARE : this mode adds a new test in check stage. The
  overlapping old and new data must be identical or else the insertion
  is not conducted. An error NCB_RET_DATA_REJ is used in this case.

The mode is specified with a new argument to ncb_add() function.
2022-05-12 18:27:05 +02:00
Amaury Denoyelle
077e096b30 MINOR: ncbuf: implement insertion
Implement a new function ncb_add() to insert data in ncbuf. This
operation is conducted in two stages. First, a simulation will be run to
ensure that insertion can be proceeded. If a gap is formed, either
before or after the new data, it must be big enough to store its header,
or else the insertion is aborted.

After this check stage, the insertion is conducted block by block with
the function pair ncb_fill_data_blk()/ncb_fill_gap_blk().

A new type ncb_ret is used as a return value. For the moment, only
success or gap-size error is used. It is planned to add new error types
in the future when insertion will be extended.
2022-05-12 18:27:05 +02:00
Amaury Denoyelle
edeb0a61a2 MINOR: ncbuf: optimize storage for the last gap
Relax the constraint for gap storage when this is the last block.
ncb_blk API functions will consider that if a gap is stored near the end
of the buffer, without the space to store its header, the gap will cover
entirely the buffer end.

For these special cases, the gap size/data size are not write/read
inside the gap to prevent an overflow. Such a gap is designed in
functions as "reduced gap" and will be flagged with the value
NCB_BK_F_FIN.

This should reduce the rejection on future add operation when receiving
data in-order. Without reduced gap handling, an insertion would be
rejected if it covers only partially the last buffer bytes, which can be
a very common case.
2022-05-12 18:18:47 +02:00
Amaury Denoyelle
d5d2ed90f0 MINOR: ncbuf: complete API and define block interal abstraction
Implement two new functions to report the total data stored accross the
whole buffer and the data stored at a specific offset until the next gap
or the buffer end.

To facilitate implementation of these new functions and also future
add/delete operations, a new abstraction is introduced : ncb_blk. This
structure represents a block of either data or gap in the buffer. It
simplifies operation when moving forward in the buffer. The first buffer
block can be retrieved via ncb_blk_first(buf). The block at a specific
offset is accessed via ncb_blk_find(buf, off).

This abstraction is purely used in functions but not stored in the ncbuf
structure per-se. This is necessary to keep the minimal memory
footprint.
2022-05-12 18:18:47 +02:00
Amaury Denoyelle
1b5f77fc18 MINOR: ncbuf: define non-contiguous buffer
Define the new type ncbuf. It can be used as a buffer with
non-contiguous data and wrapping support.

To reduce as much as possible the memory footprint, size of data and
gaps are stored in the gaps themselves. This put some limitation on the
buffer usage. A reserved space is present just before the head to store
the size of the first data block. Also, add and delete operations will
be constrained to ensure minimal gap sizes are preserved.

The sizes stored in the gaps are represented by a custom type named
ncb_sz_t. This type is a typedef to easily change it : this has a
direct impact on the maximum buffer size (MAX(ncb_sz_t) - sizeof(ncb_sz_t))
and the minimal gap sizes (sizeof(ncb_sz_t) * 2)).
Currently, it is set to uint32_t.
2022-05-12 18:13:21 +02:00
Frédéric Lécaille
31fe308acc CLEANUP: quic_tls: QUIC_TLS_IV_LEN defined two times
Hopefully with the same value!
2022-05-12 17:48:35 +02:00
Frédéric Lécaille
4ba3b4ef67 CLEANUP: quic: Useless use of pointer for quic_hkdf_extract()
There is no need to use a pointer to the output buffer length.
2022-05-12 17:48:35 +02:00
Frédéric Lécaille
a54e49d0b1 CLEANUP: quic: wrong use of eb*entry() macro
This wrong use has no consequence because the ->node member fields of
eb*node structs are the first.
2022-05-12 17:48:35 +02:00
Frédéric Lécaille
36b28ed012 MINOR: quic: Short packets always embed a trailing AEAD TAG
We must drop as soon as possible too small 1-RTT packets to be valid QUIC
packets to avoid replying with stateless reset packets.
2022-05-12 17:48:35 +02:00
Frédéric Lécaille
e2fb1bf487 MINOR: quic: Send stateless reset tokens
Add send_stateless_reset() to send a stateless reset packet. It prepares
a packet to build a 1-RTT packet with quic_stateless_reset_token_cpy()
to copy a stateless reset token derived from the cluster secret with
the destination connection ID received as salt.
Also add QUIC_EV_STATELESS_RST new trace event to at least to have a trace
of the connection which are reset.
2022-05-12 17:48:35 +02:00
Frédéric Lécaille
806e6cf392 MINOR: quic: Stateless reset token copy to transport parameters
A server may send the stateless reset token associated to the current
connection from its transport parameters. So, let's copy it from
qc_lstnt_params_init().
2022-05-12 17:48:35 +02:00
Frédéric Lécaille
395a64dd81 MINOR: qc_new_conn() rework for stateless reset
The stateless reset token of a connection is generated from qc_new_conn() when
allocating the first connection ID. A QUIC server can copy it into its transport
parameters to allow the peer to reset the associated connection.
This latter is not easily reachable after having returned from qc_new_conn().
We want to be able to initialize the transport parameters from this function which
has an access to all the information to do so.

Extract the code used to initialize the transport parameters from qc_lstnr_pkt_rcv()
and make it callable from qc_new_conn(). qc_lstnr_params_init() is implemented
to accomplish this task for a haproxy listener.
Modify qc_new_conn() to reduce its the number of parameters.
The source address coming from Initial packets is also copied from qc_new_conn().
2022-05-12 17:48:35 +02:00
Frédéric Lécaille
28a1795515 MINOR: quic: Initialize stateless reset tokens with HKDF secrets
Add quic_stateless_reset_token_init() wrapper function around
quic_hkdf_extract_and_expand() function to derive the stateless reset tokens
attached to the connection IDs from "cluster-secret" configuration setting
and call it each time we instantiate a QUIC connection ID.
2022-05-12 17:48:35 +02:00
Frédéric Lécaille
0226c521b0 MINOR: quic: new_quic_cid() code moving
This function will have to call another one from quic_tls.[ch] soon.
As we do not want to include quic_tls.h from xprt_quic.h because
quic_tls.h already includes xprt_quic.h, let's moving it into
xprt_quic.c.
2022-05-12 17:48:35 +02:00
Frédéric Lécaille
7b92c81e43 MINOR: quic-tls: Add quic_hkdf_extract_and_expand() for HKDF
This is a wrapper function around OpenSSL HKDF API functions to
use the "extract-then-expand" HKDF mode as defined by rfc5869.
This function will be used to derived stateless reset tokens
from secrets ("cluster-secret" conf. keyword) and CIDs (as salts).
2022-05-12 17:48:35 +02:00
Frédéric Lécaille
372508cc42 MINOR: config: Add "cluster-secret" new global keyword
It could be usefull to set a ASCII secret which could be used for different
usages. For instance, it will be used to derive QUIC stateless reset tokens.
2022-05-12 17:48:35 +02:00
Frédéric Lécaille
7cc8b3166a MINOR: quic: Add correct ack delay values to ACK frames
A ->time_received new member is added to quic_rx_packet to store the time the
packet are received. ->largest_time_received is added the the packet number
space structure to store this timestamp for the packet with a new largest
packet number to be acknowledged. QUIC_FL_PKTNS_NEW_LARGEST_PN new flag is
added to mark a packet number space as having to acknowledged a packet wih a
new largest packet number. In this case, the packet number space ack delay
must be recalculated.
Add quic_compute_ack_delay_us() function to compute the ack delay from the value
of the time a packet was received. Used only when a packet with a new largest
packet number.
2022-05-12 15:30:14 +02:00
Frédéric Lécaille
9475d890ee MINOR: quic: Congestion controller event trace fix (loss)
Missing event type (loss).
2022-05-12 15:30:14 +02:00
Frédéric Lécaille
f6e8594469 BUG/MINOR: quic: Wrong unit for ack delay for incoming ACK frames
This ACK frame field value is in microseconds. Everything is interpreted
and stored in milliseconds in our QUIC implementation.
2022-05-12 15:30:14 +02:00
Frédéric Lécaille
5b988ebed1 BUG/MINOR: quic: Dropped peer transport parameters
The call to quic_dflt_transport_params_cpy() is already first done by
quic_transport_params_init() which is a good thing. But this function was also
called each time we parsed a transport parameters with quic_transport_param_decode(),
re-initializing to default values some of them. The transport parameters concerned
by this bug are the following:
   - max_udp_payload_size
   - ack_delay_exponent
   - max_ack_delay
   - active_connection_id_limit
So, let's remove this call to quic_dflt_transport_params_cpy() which has nothing
to do here!
2022-05-12 15:26:10 +02:00
Frédéric Lécaille
8726d633d4 MINOR: quic: Add a debug counter for sendto() errors
As we do not have any task to be wake up by the poller after sendto() error,
we add an sendto() error counter to the quic_conn struct.
Dump its values from qc_send_ppkts().
2022-05-12 15:11:53 +02:00
William Lallemand
cfabb3526b DOC: configuration: add the httpclient keywords to the global keywords index
- httpclient.ssl.verify
    - httpclient.ssl.ca-file
    - httpclient.resolvers.id
    - httpclient.resolvers.prefer
2022-05-12 10:51:15 +02:00
Willy Tarreau
e872f75fc4 MINOR: mux-h2: report a trace event when failing to create a new stream
There are two reasons we can reject the creation of an h2 stream on the
frontend:
  - its creation would violate the MAX_CONCURRENT_STREAMS setting
  - there's no more memory available

And on the backend it's almost the same except that the setting might
have be negotiated after trying to set up the stream.

Let's add traces for such a sitaution so that it's possible to know why
the stream was rejected (currently we only know it was rejected).

It could be nice to backport this to the most recent versions.
2022-05-12 09:29:58 +02:00
Willy Tarreau
198b50770d BUG/MINOR: mux-h2: mark the stream as open before processing it not after
When a client doesn't respect the h2 MAX_CONCURRENT_STREAMS setting, we
rightfully send RST_STREAM to it so that the client closes. But the
max_id is only updated on the successful path of h2c_handle_stream_new(),
which may be reentered for partial frames or CONTINUATION frames, and as
a result we don't increment it if an extraneous stream ID is rejected.

Normally it doesn't have any consequence. But on a POST it can have some
if the DATA frame immediately follows the faulty HEADERS frame: with
max_id not incremented, the stream remains in IDLE state, and the DATA
frame now lands in an invalid state from a protocol's perspective, which
must lead to a connection error instead of a stream error.

This can be tested by modifying the code to send an arbitrarily large
MAX_CONCURRENT_STREAM setting and using h2load to send more concurrent
streams than configured: with a GET, only a tiny fraction of them will
report an error (e.g. 101 streams for 100 accepted will result in ~1%
failure), but when sending data, most of the streams will be reported
as failed because the connection will be closed. By updating the max_id
earlier, the stream is now considered as closed when the DATA frame
arrives and it's silently discarded.

This must be backported to all versions but only if the code is exactly
the same. Under no circumstance this ID may be updated for a partial frame
(i.e. only update it before or just after calling h2c_frt_steam_new()).
2022-05-12 09:29:58 +02:00
Emeric Brun
314e6ec822 BUG/MAJOR: dns: multi-thread concurrency issue on UDP socket
This patch adds a lock on the struct dgram_conn to ensure
that an other thread cannot trash a fd or alter its status
while the current thread processing it on for send/receive/connect
operations.

Starting with the 2.4 version this could cause a crash when a DNS
request is failing, setting the FD of the dgram structure to -1. If the
dgram structure is reused after that, a read access to fdtab[-1] is
attempted. The crash was only triggered when compiled with ASAN.

In previous versions the concurrency issue also exists but is less
likely to crash.

This patch must be backported until v2.4 and should be
adapt for v < 2.4.
2022-05-11 15:20:10 +02:00
vigneshsp
47a4c61d63 BUG/MINOR: server: Make SRV_STATE_LINE_MAXLEN value from 512 to 2kB (2000 bytes).
The statefile before this patch can only parse lines within 512
characters, now as we made the value to 2000, it can support a
line of length of 2kB.

This patch fixes GitHub issue #1530.
It should be backported to all stable releases.
2022-05-11 11:39:06 +02:00
Willy Tarreau
6d868a2c65 BUILD: makefile: add -Wfatal-errors to the default flags
Some error reports are misleading on some recent versions of gcc because
it goes on to build for a very long time after it meets an error. Not
only this makes it hard to scroll back to the beginning of the error,
but it also hides the cause of the error when it's prominently printed
in a "#error" statement. This typically happens when building with QUIC
and without OPENSSL where there can be 4 pages of unknown types and such
errors after the "Must define USE_OPENSSL" suggestion.

The flag -Wfatal-errors serves exactly this purpose, to stop after the
first error, and it's supported on all the compilers we support, so let's
enable this now.
2022-05-11 11:32:41 +02:00