Commit Graph

16901 Commits

Author SHA1 Message Date
Frédéric Lécaille
dcc74ff792 BUG/MINOR: quic: Unsent frame because of qc_build_frms()
There are non already identified rare cases where qc_build_frms() does not manage
to size frames to be encoded in a packet leading qc_build_frm() to fail to add
such frame to the packet to be built. In such cases we must move back such
frames to their origin frame list passed as parameter to qc_build_frms(): <frms>.
because they were added to the packet frame list (but not built). If this
this packet is not retransmitted, the frame is lost for ever! Furthermore we must
not modify the buffer.
2022-03-21 11:29:40 +01:00
Frédéric Lécaille
d64f68fb0a BUG/MINOR: quic: Possible leak in quic_build_post_handshake_frames()
Rework this function to leave the connection passed as parameter in the same state
it was before entering this function.
2022-03-21 11:29:40 +01:00
Frédéric Lécaille
f1f812bfdb BUG/MINOR: quic: Possible crash in parse_retry_token()
We must check the decoded length of this incoming data before copying into our
internal structure. This could lead to crashes.
Reproduced with such a packet captured from QUIC interop.
    {
	    0xc5, 0x00, 0x00, 0x00, 0x01, 0x12, 0xf2, 0x65,
		0x4d, 0x9d, 0x58, 0x90, 0x23, 0x7e, 0x67, 0xef,
		0xf8, 0xef, 0x5b, 0x87, 0x48, 0xbe, 0xde, 0x7a, /* corrupted byte: 0x11, */
		0x01, 0xdc, 0x41, 0xbf, 0xfb, 0x07, 0x39, 0x9f,
		0xfd, 0x96, 0x67, 0x5f, 0x58, 0x03, 0x57, 0x74,
		0xc7, 0x26, 0x00, 0x45, 0x25, 0xdc, 0x7f, 0xf1,
		0x22, 0x1d,
	}
2022-03-21 11:29:40 +01:00
Frédéric Lécaille
e2a1c1b372 MEDIUM: quic: Rework of the TX packets memory handling
The TX packet refcounting had come with the multithreading support but not only.
It is very useful to ease the management of the memory allocated for TX packets
with TX frames attached to. At some locations of the code we have to move TX
frames from a packet to a new one during retranmission when the packet has been
deemed as lost or not. When deemed lost the memory allocated for the paquet must
be released contrary to when its frames are retransmitted when probing (PTO).

For now on, thanks to this patch we handle the TX packets memory this way. We
increment the packet refcount when:
  - we insert it in its packet number space tree,
  - we attache an ack-eliciting frame to it.
And reciprocally we decrement this refcount when:
  - we remove an ack-eliciting frame from the packet,
  - we delete the packet from its packet number space tree.

Note that an optimization WOULD NOT be to fully reuse (without releasing its
memorya TX packet to retransmit its contents (its ack-eliciting frames). Its
information (timestamp, in flight length) to be processed by packet loss detection
and the congestion control.
2022-03-21 11:29:40 +01:00
Frédéric Lécaille
141982a4e1 MEDIUM: quic: Limit the number of ACK ranges
When building a packet with an ACK frame, we store the largest acknowledged
packet number sent in this frame in the packet (quic_tx_packet struc).
When receiving an ack for such a packet we can purge the tree of acknowledged
packet number ranges from the range sent before this largest acknowledged
packet number.
2022-03-21 11:29:40 +01:00
Frédéric Lécaille
8f3ae0272f CLEANUP: quic: "largest_acked_pn" pktns struc member moving
This struct member stores the largest acked packet number which was received. It
is used to build (TX) packet. But this is confusing to store it in the tx packet
of the packet number space structure even if it is used to build and transmit
packets.
2022-03-21 11:29:40 +01:00
Frédéric Lécaille
302c2b1120 MINOR: quic: Code factorization (TX buffer reuse)
Add qc_may_reuse_cbuf() function used by qc_prep_pkts() and qc_prep_app_pkts().
Simplification of the factorized section code: there is no need to check there
is enough room to mark the end of the data in the TX buf. This is done by
the callers (qc_prep_pkts() and qc_prep_app_pkts()). Add a diagram to explain
the conditions which must be verified to be able to reuse a cbuf struct.

This should improve the QUIC stack implementation maintenability.
2022-03-21 11:29:40 +01:00
Tim Duesterhus
fbbbc33df4 REGTESTS: Do not use REQUIRE_VERSION for HAProxy 2.5+
Introduced in:

0657b9338 MINOR: stream: add "last_rule_file" and "last_rule_line" samples
2022-03-21 09:47:47 +01:00
Tim Duesterhus
f4f6c0f6bb CLEANUP: Reapply ist.cocci
This makes use of the newly added:

    - i.ptr = p;
    - i.len = strlen(i.ptr);
    + i = ist(p);

patch.
2022-03-21 08:30:47 +01:00
Tim Duesterhus
14abfb61fb DEV: coccinelle: Add a new pattern to ist.cocci
This was previously ignored in "DEV: coccinelle: Fix incorrect replacement in ist.cocci",
but is now properly replaced by a simple `ist()` call.
2022-03-21 08:30:47 +01:00
Tim Duesterhus
7750850594 CLEANUP: Reapply ist.cocci with --include-headers-for-types --recursive-includes
Previous uses of `ist.cocci` did not add `--include-headers-for-types` and
`--recursive-includes` preventing Coccinelle seeing `struct ist` members of
other structs.

Reapply the patch with proper flags to further clean up the use of the ist API.

The command used was:

    spatch -sp_file dev/coccinelle/ist.cocci -in_place --include-headers --include-headers-for-types --recursive-includes --dir src/
2022-03-21 08:30:47 +01:00
Tim Duesterhus
99f47377c8 DEV: coccinelle: Fix incorrect replacement in ist.cocci
We must not use `ist2()` if the value of `i.len` is derived from the value of
`i.ptr`:

    i.ptr = "foo";
    i.len = strlen(i.ptr);
2022-03-21 08:30:47 +01:00
Christopher Faulet
ab398d8ff9 BUG/MINOR: http-rules: Don't free new rule on allocation failure
If allocation of a new HTTP rule fails, we must not release it calling
free_act_rule(). The regression was introduced by the commit dd7e6c6dc
("BUG/MINOR: http-rules: completely free incorrect TCP rules on error").

This patch must only be backported if the commit above is backported. It should
fix the issues #1627, #1628 and #1629.
2022-03-21 08:24:17 +01:00
Christopher Faulet
9075dbdd84 BUG/MINOR: rules: Initialize the list element when allocating a new rule
dd7e6c6dc ("BUG/MINOR: http-rules: completely free incorrect TCP rules on
error") and 388c0f2a6 ("BUG/MINOR: tcp-rules: completely free incorrect TCP
rules on error") introduced a regression because the list element of a new
rule is not intialized. Thus HAProxy crashes when an incorrect rule is
released.

This patch must be backported if above commits are backported. Note that
new_act_rule() only exists since the 2.5. It relies on the commit d535f807b
("MINOR: rules: add a new function new_act_rule() to allocate act_rules").
2022-03-21 07:55:37 +01:00
Willy Tarreau
15a4733d5d BUG/MEDIUM: mux-h2: make use of http-request and keep-alive timeouts
Christian Ruppert reported an issue explaining that it's not possible to
forcefully close H2 connections which do not receive requests anymore if
they continue to send control traffic (window updates, ping etc). This
will indeed refresh the timeout. In H1 we don't have this problem because
any single byte is part of the stream, so the control frames in H2 would
be equivalent to TCP acks in H1, that would not contribute to the timeout
being refreshed.

What misses from H2 is the use of http-request and keep-alive timeouts.
These were not implemented because initially it was hard to see how they
could map to H2. But if we consider the real use of the keep-alive timeout,
that is, how long do we keep a connection alive with no request, then it's
pretty obvious that it does apply to H2 as well. Similarly, http-request
may definitely be honored as soon as a HEADERS frame starts to appear
while there is no stream. This will also allow to deal with too long
CONTINUATION frames.

This patch moves the timeout update to a new function, h2c_update_timeout(),
which is in charge of this. It also adds an "idle_start" timestamp in the
connection, which is set when nb_cs reaches zero or when a headers frame
start to arrive, so that it cannot be delayed too long.

This patch should be backported to recent stable releases after some
observation time. It depends on previous patch "MEDIUM: mux-h2: slightly
relax timeout management rules".
2022-03-18 17:43:34 +01:00
Willy Tarreau
3439583dd6 MEDIUM: mux-h2: slightly relax timeout management rules
The H2 timeout rules were arranged to cover complex situations In 2.1
with commit c2ea47fb1 ("BUG/MEDIUM: mux-h2: do not enforce timeout on
long connections").

It turns out that such rules while complex, do not perfectly cover all
use cases. The real intent is to say that as long as there are attached
streams, the connection must not timeout. Then once all these streams
have quit (possibly for timeout reasons) then the mux should take over
the management of timeouts.

We do have this nb_cs field which indicates the number of attached
streams, and it's updated even when leaving orphaned streams. So
checking it alone is sufficient to know whether it's the mux or the
streams that are in charge of the timeouts.

In its current state, this doesn't cause visible effects except that
it makes it impossible to implement more subtle parsing timeouts.

This would need to be backported as far as 2.0 along with the next
commit that will depend on it.
2022-03-18 17:43:34 +01:00
Willy Tarreau
6e805dab2a BUG/MEDIUM: trace: avoid race condition when retrieving session from conn->owner
There's a rare race condition possible when trying to retrieve session from
a back connection's owner, that was fixed in 2.4 and described in commit
3aab17bd5 ("BUG/MAJOR: connection: reset conn->owner when detaching from
session list").

It also affects the trace code which does the same, so the same fix is
needed, i.e. check from conn->session_list that the connection is still
enlisted. It's visible when sending a few tens to hundreds of parallel
requests to an h2 backend and enabling traces in parallel.

This should be backported as far as 2.2 which is the oldest version
supporting traces.
2022-03-18 17:43:28 +01:00
Willy Tarreau
d1480cc8a4 BUG/MEDIUM: stream-int: do not rely on the connection error once established
Historically the stream-interface code used to check for connection
errors by itself. Later this was partially deferred to muxes, but
only once the mux is installed or the connection is at least in the
established state. But probably as a safety practice the connection
error tests remained.

The problem is that they are causing trouble on when a response received
from a mux is mixed with an error report. The typical case is an upload
that is interrupted by the server sending an error or redirect without
draining all data, causing an RST to be queued just after the data. In
this case the mux has the data, the CO_FL_ERROR flag is present on the
connection, and unfortunately the stream-interface refuses to retrieve
the data due to this flag, and return an error to the client.

It's about time to only rely on CS_FL_ERROR which is set by the mux, but
the stream-interface is still responsible for the connection during its
setup. However everywhere the CO_FL_ERROR is checked, CS_FL_ERROR is
also checked.

This commit addresses this by:
  - adding a new function si_is_conn_error() that checks the SI state
    and only reports the status of CO_FL_ERROR for states before
    SI_ST_EST.

  - eliminating all checks for CO_FL_ERORR in places where CS_FL_ERROR
    is already checked and either the presence of a mux was already
    validated or the stream-int's state was already checked as being
    SI_ST_EST or higher.

CO_FL_ERROR tests on the send() direction are also inappropriate as they
may cause the loss of pending data. Now this doesn't happen anymore and
such events are only converted to CS_FL_ERROR by the mux once notified of
the problem. As such, this must not cause the loss of any error event.

Now an early error reported on a backend mux doesn't prevent the queued
response from being read and forwarded to the client (the list of syscalls
below was trimmed and epoll_ctl is not represented):

  recvfrom(10, "POST / HTTP/1.1\r\nConnection: clo"..., 16320, 0, NULL, NULL) = 66
  sendto(11, "POST / HTTP/1.1\r\ntransfer-encodi"..., 47, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 47
  epoll_wait(3, [{events=EPOLLIN|EPOLLERR|EPOLLHUP|EPOLLRDHUP, data={u32=11, u64=11}}], 200, 15001) = 1
  recvfrom(11, "HTTP/1.1 200 OK\r\ncontent-length:"..., 16320, 0, NULL, NULL) = 57
  sendto(10, "HTTP/1.1 200 OK\r\ncontent-length:"..., 57, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 57
  epoll_wait(3, [{events=EPOLLIN|EPOLLERR|EPOLLHUP|EPOLLRDHUP, data={u32=11, u64=11}}], 200, 13001) = 1
  epoll_wait(3, [{events=EPOLLIN, data={u32=10, u64=10}}], 200, 13001) = 1
  recvfrom(10, "A\n0123456789\r\n0\r\n\r\n", 16320, 0, NULL, NULL) = 19
  shutdown(10, SHUT_WR)                   = 0
  close(11)                               = 0
  close(10)                               = 0

Above the server is an haproxy configured with the following:

   listen blah
        bind :8002
        mode http
        timeout connect 5s
        timeout client  5s
        timeout server  5s
        option httpclose
        option nolinger
        http-request return status 200 hdr connection close

And the client takes care of sending requests and data in two distinct
parts:

   while :; do
     ./dev/tcploop/tcploop 8001 C T S:"POST / HTTP/1.1\r\nConnection: close\r\nTransfer-encoding: chunked\r\n\r\n" P1 S:"A\n0123456789\r\n0\r\n\r\n" P R F;
   done

With this, a small percentage of the requests will reproduce the behavior
above. Note that this fix requires the following patch to be applied for
the test above to work:

  BUG/MEDIUM: mux-h1: only turn CO_FL_ERROR to CS_FL_ERROR with empty ibuf

This should be backported with after a few weeks of observation, and
likely one version at a time. During the backports, the patch might
need to be adjusted at each check of CO_FL_ERORR to follow the
principles explained above.
2022-03-18 17:00:19 +01:00
Willy Tarreau
99bbdbcc21 BUG/MEDIUM: mux-h1: only turn CO_FL_ERROR to CS_FL_ERROR with empty ibuf
A connection-level error must not be turned to a stream-level error if there
are still pending data for that stream, otherwise it can cause the truncation
of the last pending data.

This must be backported to affected releases, at least as far as 2.4,
maybe further.
2022-03-18 17:00:17 +01:00
Ilya Shipitsin
2ec39ea653 CI: github actions: switch to LibreSSL-3.5.1 2022-03-18 13:48:42 +01:00
William Lallemand
58a81aeb91 BUG/MINOR: httpclient: CF_SHUTW_NOW should be tested with channel_is_empty()
CF_SHUTW_NOW shouldn't be a condition alone to exit the io handler, it
must be tested with the emptiness of the response channel.

Must be backported to 2.5.
2022-03-18 11:34:10 +01:00
William Lallemand
1eca894321 BUG/MINOR: httpclient: process the response when received before the end of the request
A server could reply a response with a shut before the end of the htx
transfer, in this case the httpclient would leave before computing the
received response.

This patch fixes the issue by calling the "process_data" label instead of
the "more" label which don't do the si_shut.

Must be bacported in 2.5.
2022-03-18 11:34:10 +01:00
William Lallemand
a625b03e83 BUG/MINOR: httpclient: only check co_data() instead of HTTP_MSG_DATA
Checking msg >= HTTP_MSG_DATA was useful to check if we received all the
data. However it does not work correctly in case of errors because we
don't reach this state, preventing to catch the error in the httpclient.

The consequence of this problem is that we don't get the status code of
the error response upon an error.

Fix the issue by only checking co_data().

Must be backported to 2.5.
2022-03-18 11:34:10 +01:00
Willy Tarreau
dd7e6c6dc7 BUG/MINOR: http-rules: completely free incorrect TCP rules on error
When a http-request or http-response rule fails to parse, we currently
free only the rule without its contents, which makes ASAN complain.
Now that we have a new function for this, let's completely free the
rule. This relies on this commit:

  MINOR: actions: add new function free_act_rule() to free a single rule

It's probably not needed to backport this since we're on the exit path
anyway.
2022-03-17 20:29:06 +01:00
Willy Tarreau
388c0f2a63 BUG/MINOR: tcp-rules: completely free incorrect TCP rules on error
When a tcp-request or tcp-response rule fails to parse, we currently
free only the rule without its contents, which makes ASAN complain.
Now that we have a new function for this, let's completely free the
rule. Reg-tests are now completely OK with ASAN. This relies on this
commit:

  MINOR: actions: add new function free_act_rule() to free a single rule

It's probably not needed to backport this since we're on the exit path
anyway.
2022-03-17 20:26:54 +01:00
Willy Tarreau
6a783e499c MINOR: actions: add new function free_act_rule() to free a single rule
There was free_act_rules() that frees all rules from a head but nothing
to free a single rule. Currently some rulesets partially free their own
rules on parsing error, and we're seeing some regtests emit errors under
ASAN because of this.

Let's first extract the code to free a rule into its own function so
that it becomes possible to use it on a single rule.
2022-03-17 20:26:19 +01:00
Willy Tarreau
211ea252d9 BUG/MINOR: logs: fix logsrv leaks on clean exit
Log servers are a real mess because:
  - entries are duplicated using memcpy() without their strings being
    reallocated, which results in these ones not being freeable every
    time.

  - a new field, ring_name, was added in 2.2 by commit 99c453df9
    ("MEDIUM: ring: new section ring to declare custom ring buffers.")
    but it's never initialized during copies, causing the same issue

  - no attempt is made at freeing all that.

Of course, running "haproxy -c" under ASAN quickly notices that and
dumps a core.

This patch adds the missing strdup() and initialization where required,
adds a new free_logsrv() function to cleanly free() such a structure,
calls it from the proxy when iterating over logsrvs instead of silently
leaking their file names and ring names, and adds the same logsrv loop
to the proxy_free_defaults() function so that we don't leak defaults
sections on exit.

It looks a bit entangled, but it comes as a whole because all this stuff
is inter-dependent and was missing.

It's probably preferable not to backport this in the foreseable future
as it may reveal other jokes if some obscure parts continue to memcpy()
the logsrv struct.
2022-03-17 19:53:46 +01:00
William Lallemand
43c2ce4d81 BUG/MINOR: server/ssl: free the SNI sample expression
ASAN complains about the SNI expression not being free upon an haproxy
-c. Indeed the httpclient is now initialized with a sni expression and
this one is never free in the server release code.

Must be backported in 2.5 and could be backported in every stable
versions.
2022-03-16 18:03:15 +01:00
William Lallemand
715c101a19 BUILD: httpclient: fix build without SSL
src/http_client.c: In function ‘httpclient_cfg_postparser’:
src/http_client.c:1065:8: error: unused variable ‘errmsg’ [-Werror=unused-variable]
 1065 |  char *errmsg = NULL;
      |        ^~~~~~
src/http_client.c:1064:6: error: unused variable ‘err_code’ [-Werror=unused-variable]
 1064 |  int err_code = 0;
      |      ^~~~~~~~

Fix the build of the httpclient without SSL, the problem was introduced
with previous patch 71e3158 ("BUG/MINOR: httpclient: send the SNI using
the host header")

Must be backported in 2.5 as well.
2022-03-16 16:39:23 +01:00
William Lallemand
71e3158395 BUG/MINOR: httpclient: send the SNI using the host header
Generate an SNI expression which uses the Host header of the request.
This is mandatory for most of the SSL servers nowadays.

Must be backported in 2.5 with the previous patch which export
server_parse_sni_expr().
2022-03-16 15:55:30 +01:00
William Lallemand
0d05867e78 MINOR: server: export server_parse_sni_expr() function
Export the server_parse_sni_expr() function in order to create a SNI
expression in a server which was not parsed from the configuration.
2022-03-16 15:55:30 +01:00
Willy Tarreau
42cef2a18f DEV: udp: add support for random packet corruption
-c sets a corruption rate in % of the packets.
-o sets the start offset of the area to be corrupted.
-w sets the length of the area to be corrupted.

A single byte within that area will then be randomly XORed with a
random value.
2022-03-16 15:09:54 +01:00
Willy Tarreau
04d3c5cd1f DEV: udp: switch parser to getopt() instead of positional arguments
In order to ease addition of new types of perturbations to udp-perturb,
let's first switch to getopt() and get rid of the positional arguments.
The random seed was already a conditional option of the rate which was
a conditional option as well. We add -r and -s for the rate and the
seed, and new options will follow.
2022-03-16 15:09:54 +01:00
Christopher Faulet
53fa787a07 BUG/MEDIUM: sink: Properly get the stream-int in appctx callback functions
The appctx owner is not a stream-interface anymore. It is now a
conn-stream. However, sink code was not updated accordingly. It is now
fixed.

It is 2.6-specific, no backport is needed.
2022-03-16 10:01:30 +01:00
Christopher Faulet
fe14af30ec BUG/MEDIUM: cli/debug: Properly get the stream-int in all debug I/O handlers
The appctx owner is not a stream-interface anymore. It is now a conn-stream.
In the cli I/O handler for the command "debug dev fd", we still handle it as
a stream-interface. It is now fixed.

It is 2.6-specific, no backport is needed.
2022-03-16 09:52:13 +01:00
Christopher Faulet
9affa931cd BUG/MEDIUM: applet: Don't call .release callback function twice
Since the CS/SI refactoring, the .release callback function may be called
twice. The first call when a shutdown for read or for write is performed.
The second one when the applet is detached from its conn-stream. The second
call must be guarded, just like the first one, to only be performed is the
stream-interface is not the in disconnected (SI_ST_DIS) or closed
(SI_ST_CLO) state.

To simplify the fix, we now always rely on si_applet_release() function.

It is 2.6-specific, no backport is needed.
2022-03-15 11:47:53 +01:00
William Lallemand
8f170c7fca BUG/MINOR: httpclient/lua: stuck when closing without data
The httpclient lua code is lacking the end callback, which means it
won't be able to wake up the lua code after a longjmp if the connection
was closed without any data.

Must be backported to 2.5.
2022-03-15 11:42:38 +01:00
Frdric Lcaille
e9a974a37a BUG/MAJOR: quic: Possible crash with full congestion control window
This commit reverts this one:
  "d5066dd9d BUG/MEDIUM: quic: qc_prep_app_pkts() retries on qc_build_pkt() failures"

After having filled the congestion control window, qc_build_pkt() always fails.
Then depending on the relative position of the writer and  reader indexes for the
TX buffer, this could lead this function to try to reuse the buffer even if not full.
In such case, we do not always mark the end of the data in this TX buffer. This
is something the reader cannot understand: it reads a false datagram length,
then a wrong packet address from the TX buffer, leading to an invalid pointer
dereferencing.
2022-03-15 10:38:48 +01:00
Frdric Lcaille
2ee5c8b3dd BUG/MEDIUM: quic: Blocked STREAM when retransmitted
STREAM frames which are not acknowledged in order are inserted in ->tx.acked_frms
tree ordered by the STREAM frame offset values. Then, they are consumed in order
by qcs_try_to_consume(). But, when we retransmit frames, we possibly have to
insert the same STREAM frame node (with the same offset) in this tree.
The problem is when they have different lengths. Unfortunately the restransmitted
frames are not inserted because of the tree nature (EB_ROOT_UNIQUE). If the STREAM
frame which has been successfully inserted has a smaller length than the
retransmitted ones, when it is consumed they are tailing bytes in the STREAM
(retransmitted ones) which indefinitively remains in the STREAM TX buffer
which will never properly be consumed, leading to a blocking state.

At this time this may happen because we sometimes build STREAM frames
with null lengths. But this is another issue.

The solution is to use an EB_ROOT tree to support the insertion of STREAM frames
with the same offset but with different lengths. As qcs_try_to_consume() support
the STREAM frames retransmission this modification should not have any impact.
2022-03-15 10:38:48 +01:00
Christopher Faulet
a4af1b7633 REGTESTS: fix the race conditions in be2hex.vtc
In the same way than for be2hex.vtc, a "Connection: close" header is added
to all responses to avoid any connection reuse. This should avoid any "HTTP
header incomplete" errors.
2022-03-14 15:38:32 +01:00
William Lallemand
97f69c6fb5 BUG/MEDIUM: httpclient: must manipulate head, not first
The httpclient mistakenly use the htx_get_first{_blk}() functions instead
of the htx_get_head{_blk}() functions. Which could stop the httpclient
because it will be without the start line, waiting for data that won't never
come.

Must be backported in 2.5.
2022-03-14 15:10:12 +01:00
William Lallemand
c020b2505d BUG/MINOR: httpclient: remove the UNUSED block when parsing headers
Remove the UNUSED blocks when iterating on headers, we should not stop
when encountering one. We should only stop iterating once we found the
EOH block. It doesn't provoke a problem, since we don't manipulates
the headers before treating them, but it could evolve in the future.

Must be backported to 2.5.
2022-03-14 15:10:12 +01:00
William Lallemand
c8f1eb99b4 BUG/MINOR: httpclient: consume partly the blocks when necessary
Consume partly the blocks in the httpclient I/O handler when there is
not enough room in the destination buffer for the whole block or when
the block is not contained entirely in the channel's output.

It prevents the I/O handler to be stuck in cases when we need to modify
the buffer with a filter for exemple.

Must be backported in 2.5.
2022-03-14 15:10:12 +01:00
William Lallemand
f5ba296ec8 CLEANUP: htx: remove unused co_htx_remove_blk()
Remove the unused co_htx_remove_blk(), this function was confusing
because you need to check the output size from the caller anyway.
2022-03-14 15:10:12 +01:00
William Lallemand
2b7dc4edb0 BUG/MEDIUM: httpclient: don't consume data before it was analyzed
In httpclient_applet_io_handler(), on the response path, we don't check
if the data are in the output part of the channel, and could consume
them before they were analyzed.

To fix this issue, this patch checks for the stline and the headers if
the msg_state is >= HTTP_MSG_DATA which means the stline and headers
were analyzed. For the data part, it checks if each htx blocks is in the
output before copying it.

Must be backported in 2.5.
2022-03-14 15:10:12 +01:00
Willy Tarreau
bc8b7a14ff [RELEASE] Released version 2.6-dev3
Released version 2.6-dev3 with the following main changes :
    - DEBUG: rename WARN_ON_ONCE() to CHECK_IF()
    - DEBUG: improve BUG_ON output message accuracy
    - DEBUG: implement 4 levels of choices between warn and crash.
    - DEBUG: add two new macros to enable debugging in hot paths
    - DEBUG: buf: replace some sensitive BUG_ON() with BUG_ON_HOT()
    - DEBUG: buf: add BUG_ON_HOT() to most buffer management functions
    - MINOR: channel: don't use co_set_data() to decrement output
    - DEBUG: channel: add consistency checks using BUG_ON_HOT() in some key functions
    - MINOR: conn-stream: Improve API to have safe/unsafe accessors
    - MEDIUM: tree-wide: Use unsafe conn-stream API when it is relevant
    - CLEANUP: stream-int: Make si_cs_send() function static
    - REORG: stream-int: Uninline si_sync_recv() and make si_cs_recv() private
    - BUG/MEDIUM: mux-fcgi: Don't rely on SI src/dst addresses for FCGI health-checks
    - BUG/MEDIUM: htx: Fix a possible null derefs in htx_xfer_blks()
    - REGTESTS: fix the race conditions in normalize_uri.vtc
    - DEBUG: stream-int: Fix BUG_ON used to test appctx in si_applet_ops callbacks
    - BUILD: debug: fix build warning on older compilers around DEBUG_STRICT_ACTION
    - CLEANUP: connection: Indicate unreachability to the compiler in conn_recv_proxy
    - MINOR: connection: Transform safety check in PROXYv2 parsing into BUG_ON()
    - DOC: install: it's DEBUG_CFLAGS, not DEBUG, which is set to -g
    - DOC: install: describe the DEP variable
    - DOC: install: describe how to choose options used in the DEBUG variable
    - MINOR: queue: Replace if() + abort() with BUG_ON()
    - CLEANUP: adjust indentation in bidir STREAM handling function
    - MINOR: quic: simplify copy of STREAM frames to RX buffer
    - MINOR: quic: handle partially received buffered stream frame
    - MINOR: mux-quic: define flag for last received frame
    - BUG/MINOR: quic: support FIN on Rx-buffered STREAM frames
    - MEDIUM: quic: rearchitecture Rx path for bidirectional STREAM frames
    - REGTESTS: fix the race conditions in secure_memcmp.vtc
    - CLEANUP: stream: Remove useless tests on conn-stream in stream_dump()
    - BUILD: ssl: another build warning on LIBRESSL_VERSION_NUMBER
    - MINOR: quic: Ensure PTO timer is not set in the past
    - MINOR: quic: Post handshake I/O callback switching
    - MINOR: quic: Drop the packets of discarded packet number spaces
    - CLEANUP: quic: Useless tests in qc_try_rm_hp()
    - CLEANUP: quic: Indentation fix in qc_prep_pkts()
    - MINOR: quic: Assemble QUIC TLS flags at the same level
    - BUILD: conn_stream: avoid null-deref warnings on gcc 6
    - BUILD: connection: do not declare register_mux_proto() inline
    - BUILD: http_rules: do not declare http_*_keywords_registre() inline
    - BUILD: trace: do not declare trace_registre_source() inline
    - BUILD: tcpcheck: do not declare tcp_check_keywords_register() inline
    - DEBUG: reduce the footprint of BUG_ON() calls
    - BUG/MEDIUM: httpclient/lua: infinite appctx loop with POST
    - BUG/MINOR: pool: always align pool_heads to 64 bytes
    - DEV: udp: add a tiny UDP proxy for testing
    - DEV: udp: implement pseudo-random reordering/loss
    - DEV: udp: add an optional argument to set the prng seed
    - BUG/MINOR: quic: fix segfault on CC if mux uninitialized
    - BUG/MEDIUM: pools: fix ha_free() on area in the process of being freed
    - CLEANUP: tree-wide: remove a few rare non-ASCII chars
    - CI: coverity: simplify debugging options
    - CLEANUP: quic: complete ABORT_NOW with a TODO comment
    - MINOR: quic: qc_prep_app_pkts() implementation
    - MINOR: quic: Send short packet from a frame list
    - MINOR: quic: Make qc_build_frms() build ack-eliciting frames from a list
    - MINOR: quic: Export qc_send_app_pkts()
    - MINOR: mux-quic: refactor transport parameters init
    - MINOR: mux-quic: complete functions to detect stream type
    - MINOR: mux-quic: define new unions for flow-control fields
    - MEDIUM: mux-quic: use direct send transport API for STREAMs
    - MINOR: mux-quic: retry send opportunistically for remaining frames
    - MEDIUM: mux-quic: implement MAX_STREAMS emission for bidir streams
    - BUILD: fix kFreeBSD build.
    - MINOR: quic: Retry on qc_build_pkt() failures
    - BUG/MINOR: quic: Missing recovery start timer reset
    - CLEANUP: quic: Remove QUIC path manipulations out of the congestion controller
    - MINOR: quic: Add a "slow start" callback to congestion controller
    - MINOR: quic: Persistent congestion detection outside of controllers
    - CLEANUP: quic: Remove useless definitions from quic_cc_event struct
    - BUG/MINOR: quic: Confusion betwen "in_flight" and "prep_in_flight" in quic_path_prep_data()
    - MINOR: quic: More precise window update calculation
    - CLEANUP: quic: Remove window redundant variable from NewReno algorithm state struct
    - MINOR: quic: Add quic_max_int_by_size() function
    - BUG/MAJOR: quic: Wrong quic_max_available_room() returned value
    - MINOR: pools: add a new global option "no-memory-trimming"
    - BUG/MINOR: add missing modes in proxy_mode_str()
    - BUG/MINOR: cli: shows correct mode in "show sess"
    - BUG/MEDIUM: quic: do not drop packet on duplicate stream/decoding error
    - MINOR: stats: Add dark mode support for socket rows
    - BUILD: fix recent build breakage of freebsd caused by kFreeBSD build fix
    - BUG/MINOR: httpclient: Set conn-stream/channel EOI flags at the end of request
    - BUG/MINOR: hlua: Set conn-stream/channel EOI flags at the end of request
    - BUG/MINOR: stats: Set conn-stream/channel EOI flags at the end of request
    - BUG/MINOR: cache: Set conn-stream/channel EOI flags at the end of request
    - BUG/MINOR: promex: Set conn-stream/channel EOI flags at the end of request
    - BUG/MEDIUM: stream: Use the front analyzers for new listener-less streams
    - DEBUG: cache: Update underlying buffer when loading HTX message in cache applet
    - BUG/MEDIUM: mcli: Properly handle errors and timeouts during reponse processing
    - DEBUG: stream: Add the missing descriptions for stream trace events
    - DEBUG: stream: Fix stream trace message to print response buffer state
    - MINOR: proxy: Store monitor_uri as a `struct ist`
    - MINOR: proxy: Store fwdfor_hdr_name as a `struct ist`
    - MINOR: proxy: Store orgto_hdr_name as a `struct ist`
    - MEDIUM: proxy: Store server_id_hdr_name as a `struct ist`
    - CLEANUP: fcgi: Replace memcpy() on ist by istcat()
    - CLEANUP: fcgi: Use `istadv()` in `fcgi_strm_send_params`
    - BUG/MAJOR: mux-pt: Always destroy the backend connection on detach
    - DOC: sample fetch methods: move distcc_* to the right locations
    - MINOR: rules: record the last http/tcp rule that gave a final verdict
    - MINOR: stream: add "last_rule_file" and "last_rule_line" samples
    - BUG/MINOR: session: fix theoretical risk of memleak in session_accept_fd()
    - MINOR: quic: Add max_idle_timeout advertisement handling
    - MEDIUM: quic: Remove the QUIC connection reference counter
    - BUG/MINOR: quic: ACK_REQUIRED and ACK_RECEIVED flag collision
    - BUG/MINOR: quic: Missing check when setting the anti-amplification limit as reached
    - MINOR: quic: Add a function to compute the current PTO
    - MEDIUM: quic: Implement the idle timeout feature
    - BUG/MEDIUM: quic: qc_prep_app_pkts() retries on qc_build_pkt() failures
    - CLEANUP: quic: Comments fix for qc_prep_(app)pkts() functions
    - MINOR: mux-quic: prevent push frame for unidir streams
    - MINOR: mux-quic: improve opportunistic retry sending for STREAM frames
    - MINOR: quic: implement sending confirmation
    - MEDIUM: mux-quic: improve bidir STREAM frames sending
    - MEDIUM: check: do not auto configure SSL/PROXY for dynamic servers
    - REGTESTS: server: test SSL/PROXY with checks for dynamic servers
    - MEDIUM: server: remove experimental-mode for dynamic servers
    - BUG/MINOR: buffer: fix debugging condition in b_peek_varint()
2022-03-11 18:09:24 +01:00
Willy Tarreau
f1cb4ac745 BUG/MINOR: buffer: fix debugging condition in b_peek_varint()
The BUG_ON_HOT() test condition added to b_peek_varint() by commit
8873b85bd ("DEBUG: buf: add BUG_ON_HOT() to most buffer management
functions") was wrong as <data> in this function is not b->data,
so that was triggering during live dumps of H2 traces on the CLI
when built with -DDEBUG_STRICT=2. No backport is needed.
2022-03-11 16:59:14 +01:00
Amaury Denoyelle
76e8b70e43 MEDIUM: server: remove experimental-mode for dynamic servers
Dynamic servers feature is now judged to be stable enough. Remove the
experimental-mode requirement for "add/del server" commands. This should
facilitate dynamic servers adoption.
2022-03-11 14:28:28 +01:00
Amaury Denoyelle
3881e1775d REGTESTS: server: test SSL/PROXY with checks for dynamic servers
Complete the dynamic servers regtest to ensure there is no implicit
interaction for checks and SSL/PROXY server settings.
2022-03-11 14:28:28 +01:00
Amaury Denoyelle
7d098bea2b MEDIUM: check: do not auto configure SSL/PROXY for dynamic servers
For server checks, SSL and PROXY is automatically inherited from the
server settings if no specific check port is specified. Change this
behavior for dynamic servers : explicit "check-ssl"/"check-send-proxy"
are required for them.

Without this change, it is impossible to add a dynamic server with
SSL/PROXY settings and checks without, if the check port is not
explicit. This is because "no-check-ssl"/"no-check-send-proxy" keywords
are not available for dynamic servers.

This change respects the principle that dynamic servers on the CLI
should not reuse the same shortcuts used during the config file parsing.
Mostly because we expect this feature to be manipulated by automated
tools, contrary to the config file which should aim to be the shortest
possible for human readability.

Update the documentation of the "check" keyword to reflect this change.
2022-03-11 14:28:28 +01:00