Commit Graph

19319 Commits

Author SHA1 Message Date
Willy Tarreau e008402972 DOC: proxy-protocol: fix wrong byte in provided example
There was a mistake in the example of proxy-proto frame
provided, it cannot end with 0x02 but only 0x20 or 0x21
since the version is in the upper 4 bits and the lower ones
are 0 for LOCAL or 1 for PROXY, hence the example should be:

  \x0D\x0A\x0D\x0A\x00\x0D\x0A\x51\x55\x49\x54\x0A\x20

Thanks to Bram Grit for reporting this mistake.
2023-02-12 09:26:48 +01:00
Frdric Lcaille 91376d6134 BUG/MEDIUM: quic: Buffer overflow when looking through QUIC CLI keyword list
This has been detected by libasan as follows:

=================================================================
==3170559==ERROR: AddressSanitizer: global-buffer-overflow on address 0x55cf77faad08 at pc 0x55cf77a87370 bp 0x7ffc01bdba70 sp 0x7ffc01bdba68
READ of size 8 at 0x55cf77faad08 thread T0
    #0 0x55cf77a8736f in cli_find_kw src/cli.c:335
    #1 0x55cf77a8a9bb in cli_parse_request src/cli.c:792
    #2 0x55cf77a8c385 in cli_io_handler src/cli.c:1024
    #3 0x55cf77d19ca1 in task_run_applet src/applet.c:245
    #4 0x55cf77c0b6ba in run_tasks_from_lists src/task.c:634
    #5 0x55cf77c0cf16 in process_runnable_tasks src/task.c:861
    #6 0x55cf77b48425 in run_poll_loop src/haproxy.c:2934
    #7 0x55cf77b491cf in run_thread_poll_loop src/haproxy.c:3127
    #8 0x55cf77b4bef2 in main src/haproxy.c:3783
    #9 0x7fb8b0693d09 in __libc_start_main ../csu/libc-start.c:308
    #10 0x55cf7764f4c9 in _start (/home/flecaille/src/haproxy-untouched/haproxy+0x1914c9)

0x55cf77faad08 is located 0 bytes to the right of global variable 'cli_kws' defined in 'src/quic_conn.c:7834:27' (0x55cf77faaca0) of size 104
SUMMARY: AddressSanitizer: global-buffer-overflow src/cli.c:335 in cli_find_kw
Shadow bytes around the buggy address:

According to cli_find_kw() code and cli_kw_list struct definition, the second
member of this structure ->kw[] must be a null-terminated array.
Add a last element with default initializers to <cli_kws> global variable which
is impacted by this bug.

This bug arrived with this commit:
   15c74702d MINOR: quic: implement a basic "show quic" CLI handler

Must be backported to 2.7 where this previous commit has been already
backported.
2023-02-11 21:08:34 +01:00
Christopher Faulet 341a5783b0 BUG/MEDIUM: stconn: stop to enable/disable reads from streams via si_update_rx
It is not really a bug because it does not fix any known issue. And it is
flagged as MEDIUM because it is sensitive. But if there are some extra calls
to process_stream(), it can be an issue because, in si_update_rx(), we may
disable reading for the SC when outgoing data are blocked in the input
channel. But it is not really the process_stream() job to take care of
that. This may block data receipt.

It is an old code, mainly here to avoid wakeup in loop on the stats
applet. Today, it seems useless and can lead to bugs. An endpoint is
responsible to block the SC if it waits for some room and the opposite
endpoint is responsible to unblock it when some data are sent. The stream
should not interfere on this part.

This patch could be backported to 2.7 after a period of observation. And it
should only be backported to lower versions if an issue is reported.
2023-02-10 17:50:35 +01:00
Willy Tarreau b685ad0774 BUG/MINOR: clock/stats: also use start_time not start_date in HTML info
For an unknown reason in the change of uptime calculation for the HTML
page didn't make it to commit 6093ba47c ("BUG/MINOR: clock: do not mix
wall-clock and monotonic time in uptime calculation"). Let's address it
as well otherwise the stats page will display an incorrect uptime.

No backport needed unless the patch above is backported.
2023-02-10 16:53:35 +01:00
Amaury Denoyelle 2776e775ec BUG/MINOR: mworker: fix uptime for master process
Uptime calculation for master process was incorrect as it used
<start_date> as its timestamp base time. Fix this by using the scheduler
time <start_time> for this.

The impact of this bug is minor as timestamp base time is only used for
"show proc" CLI output. it was highlighted by the following commit.
which caused a negative value to be displayed for the master process
uptime on "show proc" output.

  28360dc53f
  MEDIUM: clock: force internal time to wrap early after boot

This should be backported up to 2.0.
2023-02-10 15:57:33 +01:00
Amaury Denoyelle a9de25a559 BUG/MINOR: quic: fix type bug on "show quic" for 32-bits arch
Incorrect printf format specifier "%lu" was used on "show quic" handler
for uint64_t. This breaks build on 32-bits architecture. To fix this
portability issue, force an explicit cast to unsigned long long with
"%llu" specifier.

This must be backported up to 2.7.
2023-02-10 09:29:37 +01:00
Christopher Faulet 71c486b290 BUG/MEDIUM: stconn: Don't needlessly wake the stream on send during fast-forward
With a connection, when data are received, if these data are sent to the
opposite side because the fast-forwarding is possible, the stream may be
woken up on some conditions (at the end of sc_app_chk_snd_conn()):

  * The channel is shut for write
  * The SC is not in the "established" state
  * The stream must explicitly be woken up on write and all data was sent
  * The connection was just established.

A bug on the last condition was introduced with the commit d89884153
("MEDIUM: channel: Use CF_WRITE_EVENT instead of CF_WRITE_PARTIAL"). The
stream is now woken up on any write events.

This patch fixes this issue and restores the original behavior. No backport
is needed.
2023-02-10 09:09:57 +01:00
Amaury Denoyelle 10a46de620 BUG/MINOR: quic: fix filtering of closing connections on "show quic"
Filtering of closing/draining connections on "show quic" was not
properly implemented. This causes the extra argument "all" to display
all connections to be without effect. This patch fixes this and restores
the output of all connections.

This must be backported up to 2.7.
2023-02-09 18:30:14 +01:00
Amaury Denoyelle 3f9758ecab MINOR: quic: filter closing conn on "show quic"
Reduce default "show quic" output by masking connection on
closing/draing state due to a CONNECTION_CLOSE emission/reception. These
connections can still be displayed using the special argument "all".

This should be backported up to 2.7.
2023-02-09 18:14:40 +01:00
Amaury Denoyelle 2eda63b447 MINOR: quic: display Tx stream info on "show quic"
Complete "show quic" handler by displaying information about
quic_stream_desc entries. These structures are used to emit stream data
and store them until acknowledgment is received.

This should be backported up to 2.7.
2023-02-09 18:14:40 +01:00
Amaury Denoyelle 1b0fc437f3 MINOR: quic: display infos about various encryption level on "show quic"
Complete "show quic" handler by displaying various information related
to each encryption level and packet number space. Most notably, ack
ranges and bytes in flight are present to help debug retransmission
issues.

This should be backported up to 2.7.
2023-02-09 18:14:40 +01:00
Amaury Denoyelle b89c0e243a MINOR: quic: display socket info on "show quic"
Complete "show quic" handler by displaying information related to the
quic_conn owned socket. First, the FD is printed, followed by the
address of the local and remote endpoint.

This should be backported up to 2.7.
2023-02-09 18:14:40 +01:00
Amaury Denoyelle 58d9d5d160 MINOR: quic: display CIDs and state in "show quic"
Complete "show quic" handler. Source and destination CIDs are printed
for every connection. This is complete by a state info to reflect if
handshake is completed and if a CONNECTION_CLOSE has been emitted or
received and the allocation status of the attached MUX. Finally the idle
timer expiration is also printed.

This should be backported up to 2.7.
2023-02-09 18:14:40 +01:00
Amaury Denoyelle 15c74702d5 MINOR: quic: implement a basic "show quic" CLI handler
Implement a basic "show quic" CLI handler. This command will be useful
to display various information on all the active QUIC frontend
connections.

This work is heavily inspired by "show sess". Most notably, a global
list of quic_conn has been introduced to be able to loop over them. This
list is stored per thread in ha_thread_ctx.

Also add three CLI handlers for "show quic" in order to allocate and
free the command context. The dump handler runs on thread isolation.
Each quic_conn is referenced using a back-ref to handle deletion during
handler yielding.

For the moment, only a list of raw quic_conn pointers is displayed. The
handler will be completed over time with more information as needed.

This should be backported up to 2.7.
2023-02-09 18:11:00 +01:00
Willy Tarreau db991c2658 BUG/MEDIUM: quic: fix crash when "option nolinger" is set in the frontend
Commit 0aba11e9e ("MINOR: quic: remove unnecessary quic_session_accept()")
overlooked one problem, in session_accept_fd() at the end, there's a bunch
of FD-specific stuff that either sets up or resets the socket at the TCP
level. The tests are mostly performed for AF_INET/AF_INET6 families but
they're only for one part (i.e. to avoid setting up TCP options on UNIX
sockets). Other pieces continue to configure the socket regardless of its
family. All of this directly acts on the FD, which is not correct since
the FD is not valid here, it corresponds to the QUIC handle. The issue
is much more visible when "option nolinger" is enabled in the frontend,
because the access to fdatb[cfd].state immediately crashes on the first
connection, as can be seen in github issue #2030.

This patch bypasses this setup for FD-less connections, such as QUIC.
However some of them could definitely be relevant to the QUIC stack, or
even to UNIX sockets sometimes. A better long-term solution would consist
in implementing a setsockopt() equivalent at the protocol layer that would
be used to configure the socket, either the FD or the QUIC conn depending
on the case. Some of them would not always be implemented but that would
allow to unify all this code.

This fix must be backported everywhere the commit above is backported,
namely 2.6 and 2.7.

Thanks to github user @twomoses for the nicely detailed report.
2023-02-09 18:04:10 +01:00
Christopher Faulet eb3f26d5a0 BUG/MEDIUM: stconn: Schedule a shutw on shutr if data must be sent first
The commit 7f59d68fe ("BUG/MEDIIM: stconn: Flush output data before
forwarding close to write side") introduced a regression. When the read side
is closed, the close is not forwarded to the write side if there are some
pending outgoind data. The idea is to foward data first and the close the
write side. However, when fast-forwarding is enabled and last data block is
received with the read0, the close is never forwarded.

We cannot revert the commit above because it really fix an issue. However,
we can schedule the shutdown for write by setting CF_SHUTW_NOW flag on the
write side. Indeed, it is the purpose of this flag.

To not replicate ugly and hardly maintainable code block at different places
in stconn.c, an helper function is used. Thus, sc_cond_forward_shutw() must
be called to know if the close can be fowarded or not. It returns 1 if it is
possible. In this case, the caller is responsible to forward the close to
the write side. Otherwise, if the close cannot be forwarded, 0 is
returned. It happens when it should not be performed at all. Or when it
should only be delayed, waiting for the input channel to be flushed. In this
last case, the CF_SHUTW_NOW flag is set in the output channel.

This patch should fix the issue #2033. It must be backported with the commit
above, thus at least as far as 2.2.
2023-02-08 16:35:54 +01:00
Aurelien DARRAGON 86207e782c BUG/MINOR: server/add: ensure minconn/maxconn consistency when adding server
When a new server was added through the cli using "server add" command,
the maxconn/minconn consistency check historically implemented in
check_config_validity() for static servers was missing.

As a result, when adding a server with the maxconn parameter without the
minconn set, the server was unable to handle any connection because
srv_dynamic_maxconn() would always return 0.

Consider the following reproducer:

    |  global
    |    stats socket /tmp/ha.sock mode 660 level admin expose-fd listeners
    |
    |  defaults
    |  timeout client 5s
    |  timeout server 5s
    |  timeout connect 5s
    |
    |  frontend test
    |    mode http
    |    bind *:8081
    |    use_backend farm
    |
    |  listen dummyok
    |    bind localhost:18999
    |    mode http
    |    http-request return status 200 hdr test "ok"
    |
    |  backend farm
    |    mode http

Start haproxy and perform the following :

  echo "add server farm/t1 127.0.0.1:18999 maxconn 100" | nc -U /tmp/ha.sock
  echo "enable server farm/t1" | nc -U /tmp/ha.sock

  curl localhost:8081 # -> 503 after 5s connect timeout

Thanks to ("MINOR: cfgparse/server: move (min/max)conn postparsing logic into
dedicated function"), we are now able to perform the consistency check after
the new dynamic server has been parsed.
This is enough to fix the issue documented here that was reported by
Thomas Pedoussaut on the ML.

This commit depends on:
 - ("MINOR: cfgparse/server: move (min/max)conn postparsing logic into
     dedicated function")

It must be backported to 2.6 and 2.7
2023-02-08 14:48:21 +01:00
Aurelien DARRAGON 3e7a0bb70b MINOR: cfgparse/server: move (min/max)conn postparsing logic into dedicated function
In check_config_validity() function, we performed some consistency checks to
adjust minconn/maxconn attributes for each declared server.

We move this logic into a dedicated function named srv_minmax_conn_apply()
to be able to perform those checks later in the process life when needed
(ie: dynamic servers)
2023-02-08 14:48:21 +01:00
William Lallemand a14686d096 MINOR: ssl/ocsp: add a function to check the OCSP update configuration
Deduplicate the code which checks the OCSP update in the ckch_store and
in the crtlist_entry.

Also, jump immediatly to error handling when the ERR_FATAL is catched.
2023-02-08 11:40:31 +01:00
William Lallemand b4b9caa65f BUILD: ssl/ocsp: ssl_ocsp-t.h depends on ssl_sock-t.h
ssl_ocsp-t.h uses SSL_SOCK_NUM_KEYTYPES which is defined in
ssl_sock-t.h.

No backport needed.
2023-02-08 11:31:03 +01:00
Willy Tarreau 28360dc53f MEDIUM: clock: force internal time to wrap early after boot
GH issue #2034 clearly indicates yet another case of time roll-over
that went badly. Issues that happen only once every 50 days are hard
to detect and debug, and are usually reported more or less synchronized
from multiple sources. This patch finally does what had long been planned
but never done yet, which is to force the time to wrap early after boot
so that any such remaining issue can be spotted quicker. The margin delay
here is 20s (it may be changed by setting BOOT_TIME_WRAP_SEC to another
value). This value seems sufficient to permit failed health checks to
succeed and traffic to come in and possibly start to update some time
stamps (accept dates in logs, freq counters, stick-tables expiration
dates etc).

It could theoretically be helpful to have this in 2.7, but as can be
seen with the two patches below, we've already had incorrect use cases
of the internal monotonic time when the wall-clock one was needed, so
we could expect to detect other ones in the future. Note that this will
*not* induce bugs, it will only make them happen much faster (i.e. no
need to wait for 50 days before seeing them). If it were to eventually
be backported, these two previous patches must also be backported:

    BUG/MINOR: clock: use distinct wall-clock and monotonic start dates
    BUG/MEDIUM: cache: use the correct time reference when comparing dates
2023-02-08 11:10:33 +01:00
Willy Tarreau 9b5d57dfd5 BUG/MEDIUM: cache: use the correct time reference when comparing dates
The cache makes use of dates advertised by external components, such
as "last-modified" or "date". As such these are wall-clock dates, and
not internal dates. However, all comparisons are mistakenly made based
on the internal monotonic date which is designed to drift from the wall
clock one in order to catch up with stolen time (which can sometimes be
intense in VMs). As such after some run time some objects may fail to
validate or fail to expire depending on the direction of the drift. This
is particularly visible when applying an offset to the internal time to
force it to wrap soon after startup, as it will be shifted up to 49.7
days in the future depending on the current date; what happens in this
case is that the reg-test "cache_expires.vtc" fails on the 3rd test by
returning stale contents from the cache at the date of this commit.

It is really important that all external dates are compared against
"date" and not "now" for this reason.

This fix needs to be backported to all versions.
2023-02-08 11:10:33 +01:00
Willy Tarreau 6093ba47c0 BUG/MINOR: clock: do not mix wall-clock and monotonic time in uptime calculation
We've had a start date even before the internal monotonic clock existed,
but once the monotonic clock was added, the start date was not updated
to distinguish the wall clock time units and the internal monotonic time
units. The distinction is important because both clocks do not necessarily
progress at the same speed. The very rare occurrences of the wall-clock
date are essentially for human consumption and communication with third
parties (e.g. report the start date in "show info" for monitoring
purposes). However currently this one is also used to measure the distance
to "now" as being the process' uptime. This is actually not correct. It
only works because for now the two dates are initialized at the exact
same instant at boot but could still be wrong if the system's date shows
a big jump backwards during startup for example. In addition the current
situation prevents us from enforcing an abritrary offset at boot to reveal
some heisenbugs.

This patch adds a new "start_time" at boot that is set from "now" and is
used in uptime calculations. "start_date" instead is now set from "date"
and will always reflect the system date for human consumption (e.g. in
"show info"). This way we're now sure that any drift of the internal
clock relative to the system date will not impact the reported uptime.

This could possibly be backported though it's unlikely that anyone has
ever noticed the problem.
2023-02-08 11:06:55 +01:00
Aleksey Ponomaryov 593802128c BUG/MEDIUM: stick-table: do not leave entries in end of window during purge
At some moments expired stick table records stop being removed. This
happens when the internal time wraps around the 32-bit limit, or every
49.7 days. What precisely happens is that some elements that are collected
close to the end of the time window (2^32 - table's "expire" setting)
might have been updated and will be requeued further, at the beginning
of the next window. Here, three bad situations happen:

  - the incorrect integer-based comparison that is not aware of wrapping
    will result in the scan to restart from the freshly requeued element,
    skipping all those at the end of the window. The net effect of this
    is that at each wakeup of the expiration task, only one element from
    the end of the window will be expired, and other ones will remain
    there for a very long time, especially if they have to wait for all
    the predecessors to be picked one at a time after slow wakeups due
    to a long expiration ; this is what was observed in issue #2034
    making the table fill up and appear as not expiring at all, and it
    seems that issue #2024 reports the same problem at the same moment
    (since such issues happen for everyone roughly at the same time
    when the clock doesn't drift too much).

  - the elements that were placed at the beginning of the next window
    are skipped as well for as long as there are refreshed entries at
    the end of the previous window, so these ones participate to filling
    the table as well. This is cause by the restart from the current,
    updated node that is generally placed after most other less recently
    updated elements.

  - once the last element at the end of the window is picked, suddenly
    there is a large amount of expired entries at the beginning of the
    next window that all have to be requeued. If the expiration delay
    is large, the number can be big and it can take a long time, which
    can very likely explain the periodic crashes reported in issue #2025.
    Limiting the batch size as done in commit dfe79251d ("BUG/MEDIUM:
    stick-table: limit the time spent purging old entries") would make
    sense for process_table_expire() as well.

This patch addresses the incorrect tree scan algorithm to make sure that:
  - there's always a next element to compare against, even when dealing
    with the last one in the tree, the first one must be used ;

  - time comparisons used to decide whether to restart from the current
    element use tick_is_lt() as it is the only case where we know the
    current element will be placed before any other one (since the tree
    respects insertion ordering for duplicates)

In order to reproduce the issue, it was found that injecting traffic on
a random key that spans over half of the size of a table whose expiration
is set to 15s while the date is going to wrap in 20s does exhibit an
increase of the table's size 5s after startup, when entries start to be
pushed to the next window. It's more effective when a second load
generator constantly hammers a same key to be certain that none of them
is ready to expire. This doesn't happen anymore after this patch.

This fix needs to be backported to all stable versions. The bug has been
there for as long as the stick tables were introduced in 1.4-dev7 with
commit 3bd697e07 ("[MEDIUM] Add stick table (persistence) management
functions and types"). A cleanup could consists in deduplicating that
code by having process_table_expire() call __stktable_trash_oldest(),
with that one improved to support an optional time check.
2023-02-08 08:55:02 +01:00
William Lallemand d85227fca2 BUG/MINOR: ssl/crt-list: warn when a line is malformated
Display a warning when some text exists between the filename and the
options. This part is completely ignored so if there are filters here,
they were never parsed.

This could be backported in every versions. In the older versions, the
parsing was done in ssl_sock_load_cert_list_file() in ssl_sock.c.
2023-02-07 17:28:54 +01:00
Willy Tarreau 655a7bcac1 BUG/MEDIUM: listener/thread: bypass shards setting on failed thread resolution
Aurlien reported that the BUG_ON(!new_ts.nbgrp) added in 2.8-dev3 by
commit 50440457e ("MEDIUM: config: restrict shards, not bind_conf to one
group each") can trigger on some invalid configs where the thread_set on
the "bind" line couldn't be resolved. The reason is that we still enter
the parsing loop (as it was done previously) and we possibly have no
group to work on (which was the purpose of this assertion). There we
need to bypass all this block on such a condition.

No backport is needed.
2023-02-06 18:06:14 +01:00
Willy Tarreau f91ab7a08c BUG/MEDIUM: thread: fix extraneous shift in the thread_set parser
Aurlien reported a bug making a statement such as "thread 2-2" fail for
a config made of exactly 2 threads. What happens is that the parser for
the "thread" keyword scans a range of thread numbers from either 1..64
or 0,-1,-2 for special values, and presets the bit masks accordingly in
the thread set, except that due to the 1..64 range, the shift length must
be reduced by one. Not doing this causes empty masks for single-bit values
that are exactly equal to the number of threads in the group and fails to
properly parse.

No backport is needed as this was introduced in 2.8-dev3 by commit
bef43dfa6 ("MINOR: thread: add a simple thread_set API").
2023-02-06 18:01:50 +01:00
Frédéric Lécaille d97d1d7c7c BUG/MINOR: stats: Prevent HTTP "other sessions" counter underflows
Due to multithreading concurrency, it is difficult at this time to figure
out how this counter may become negative. This simple patch only checks this
will never be the case.

This issue arrives with this commit:
 "9969adbcdc MINOR: stats: add by HTTP version cumulated number of sessions and requests"
So, this patch should be backported when the latter has been backported.
2023-02-06 14:04:27 +01:00
Frédéric Lécaille b7a406ac34 MINOR: quic: Update version_information transport parameter to draft-14
This is necessary to make our stack negotiate the QUIC versions with clients.
(See https://author-tools.ietf.org/iddiff?url1=draft-ietf-quic-version-negotiation-13&url2=draft-ietf-quic-version-negotiation-14&difftype=--html)

Must be backported to 2.7.
2023-02-06 11:54:07 +01:00
Aurelien DARRAGON 90304dcdd8 BUG/MINOR: stats: fix STAT_STARTED behavior with full htx
When stats_putchk() fails to peform the dump because available data space in
htx is less than the number of bytes pending in the dump buffer, we wait
for more room in the htx (ie: sc_need_room()) to retry the dump attempt
on the next applet invocation.

To provide consistent output, we have to make sure that the stat ctx is not
updated (or at least correctly reverted) in case stats_putchk() fails so
that the new dumping attempt behaves just like the previous (failed) one.

STAT_STARTED is not following this logic, the flag is set in
stats_dump_fields_json() as soon as some data is written to the output buffer.

It's done too early: we need to delay this step after the stats_putchk() has
successfully returned if we want to correctly handle the retries attempts.

Because of this, JSON output could suffer from extraneous ',' characters which
could make json parsers unhappy.

For example, this is the kind of errors you could get when using
`python -m json.tool` on such badly formatted outputs:

   "Expecting value: line 1 column 2 (char 1)"

Unfortunately, fixing this means that the flag needs to be enabled at
multiple places, which is what we're doing in this patch.
(in stats_dump_proxy_to_buffer() where stats_dump_one_line() is involved
by underlying stats_dump_{fe,li,sv,be} functions)

Thereby, this raises the need for a cleanup to reduce code duplication around
stats_dump_proxy_to_buffer() function and simplify things a bit.

It could be backported to 2.6 and 2.7
2023-02-06 07:53:03 +01:00
Aurelien DARRAGON 28a23617ce BUG/MINOR: stats: fix show stats field ctx for servers
In ("MINOR: stats: introduce stats field ctx"), we forgot
to apply the patch to servers.

This prevents "BUG/MINOR: stats: fix show stat json buffer limitation"
from working with servers dump.

We're adding the missing part related to servers dump.

This commit should be backported with the aforementioned commits.
2023-02-06 07:53:03 +01:00
Aurelien DARRAGON 9b07d4fecd BUG/MINOR: stats: fix ctx->field update in stats_dump_proxy_to_buffer()
When ctx->field was introduced with ("MINOR: stats: introduce stats field ctx")
a mistake was made for the STAT_PX_ST_LI state in stats_dump_proxy_to_buffer():

current_field reset is placed after the for loop, ie: after multiple lines
are dumped. Instead it should be placed right after each li line is dumped.

This could cause some output inconsistencies (missing fields), especially when
http dump is used with JSON output and "socket-stats" option is enabled
on the proxy, because when htx is full we restore the ctx->field with
current_field (which contains outdated value in this case).

This should be backported with ("MINOR: stats: introduce stats field ctx")
2023-02-06 07:53:03 +01:00
Aurelien DARRAGON e5958d0292 BUG/MEDIUM: stats: fix resolvers dump
In ("BUG/MEDIUM: stats: Rely on a local trash buffer to dump the stats"),
we forgot to apply the patch in resolvers.c which provides the
stats_dump_resolvers() function that is involved when dumping with "resolvers"
domain.

As a consequence, resolvers dump was broken because stats_dump_one_line(),
which is used in stats_dump_resolv_to_buffer(), implicitely uses trash_chunk
from stats.c to prepare the dump, and stats_putchk() is then called with
global trash (currently empty) as output data.

Given that trash_dump variable is static and thus only available within stats.c
we change stats_putchk() function prototype so that the function does not take
the output buffer as an argument. Instead, stats_putchk() will implicitly use
the local trash_dump variable declared in stats.c.

It will also prevent further mixups between stats_dump_* functions and
stats_putchk().

This needs to be backported with ("BUG/MEDIUM: stats: Rely on a local trash
buffer to dump the stats")
2023-02-06 07:53:03 +01:00
Aurelien DARRAGON 14656844cc BUG/MINOR: stats: fix source buffer size for http dump
In ("BUG/MINOR: stats: use proper buffer size for http dump"),
we used trash.size as source buffer size before applying the htx
overhead computation.

It is safer to use res->buf.size instead since res_htx (which is <htx> argument
passed to stats_putchk() in http context) is made from res->buf:

in http_stats_io_handler:
    | res_htx = htx_from_buf(&res->buf);

This will prevent the hang bug from showing up again if res->buf.size were to be
less than trash.size (which is set according to tune.bufsize).

This should be backported with ("BUG/MINOR: stats: use proper buffer size for http dump")
2023-02-06 07:53:03 +01:00
Willy Tarreau e74d77b301 [RELEASE] Released version 2.8-dev3
Released version 2.8-dev3 with the following main changes :
    - BUG/MINOR: sink: make sure to always properly unmap a file-backed ring
    - DEV: haring: add a new option "-r" to automatically repair broken files
    - BUG/MINOR: ssl: Fix leaks in 'update ssl ocsp-response' CLI command
    - MINOR: ssl: Remove debug fprintf in 'update ssl ocsp-response' cli command
    - MINOR: connection: add a BUG_ON() to detect destroying connection in idle list
    - MINOR: mux-quic/h3: send SETTINGS as soon as transport is ready
    - BUG/MINOR: h3: fix GOAWAY emission
    - BUG/MEDIUM: mux-quic: fix crash on H3 SETTINGS emission
    - BUG/MEDIUM: hpack: fix incorrect huffman decoding of some control chars
    - BUG/MINOR: log: release global log servers on exit
    - BUG/MINOR: ring: release the backing store name on exit
    - BUG/MINOR: sink: free the forwarding task on exit
    - CLEANUP: trace: remove the QUIC-specific ifdefs
    - MINOR: trace: add a TRACE_ENABLED() macro to determine if a trace is active
    - MINOR: trace: add a trace_no_cb() dummy callback for when to use no callback
    - MINOR: trace: add the long awaited TRACE_PRINTF()
    - MINOR: h2: add h2_phdr_to_ist() to make ISTs from pseudo headers
    - MEDIUM: mux-h2/trace: add tracing support for headers
    - CLEANUP: mux-h2/trace: shorten the name of the header enc/dec functions
    - DEV: hpack: fix `trash` build regression
    - MINOR: http_htx: add http_append_header() to append value to header
    - MINOR: http_htx: add http_prepend_header() to prepend value to header
    - MINOR: sample: add ARGC_OPT
    - MINOR: proxy: introduce http only options
    - MINOR: proxy/http_ext: introduce proxy forwarded option
    - REGTEST: add ifnone-forwardfor test
    - MINOR: proxy: move 'forwardfor' option to http_ext
    - MINOR: proxy: move 'originalto' option to http_ext
    - MINOR: http_ext: introduce http ext converters
    - MINOR: http_ext: add rfc7239_is_valid converter
    - MINOR: http_ext: add rfc7239_field converter
    - MINOR: http_ext: add rfc7239_n2nn converter
    - MINOR: http_ext: add rfc7239_n2np converter
    - REGTEST: add RFC7239 forwarded header tests
    - OPTIM: http_ext/7239: introduce c_mode to save some space
    - MINOR: http_ext/7239: warn the user when fetch is not available
    - MEDIUM: proxy/http_ext: implement dynamic http_ext
    - MINOR: cfgparse/http_ext: move post-parsing http_ext steps to http_ext
    - DOC: config: fix option spop-check proxy compatibility
    - BUG/MINOR: fcgi-app: prevent 'use-fcgi-app' in default section
    - DOC: config: 'http-send-name-header' option may be used in default section
    - BUG/MINOR: mux-h2: Fix possible null pointer deref on h2c in _h2_trace_header()
    - BUG/MINOR: http_ext/7239: ipv6 dumping relies on out of scope variables
    - BUG/MEDIUM: h3: do not crash if no buf space for trailers
    - OPTIM: h3: skip buf realign if no trailer to encode
    - MINOR: mux-quic/h3: define stream close callback
    - BUG/MEDIUM: h3: handle STOP_SENDING on control stream
    - BUG/MINOR: h3: reject RESET_STREAM received for control stream
    - MINOR: h3: add missing traces on closure
    - BUG/MEDIUM: ssl: wrong eviction from the session cache tree
    - BUG/MINOR: h3: fix crash due to h3 traces
    - BUG/MINOR: h3: fix crash due to h3 traces
    - BUG/MEDIUM: thread: consider secondary threads as idle+harmless during boot
    - BUG/MINOR: stats: use proper buffer size for http dump
    - BUILD: makefile: fix PCRE overriding specific lib path
    - MINOR: quic: remove fin from quic_stream frame type
    - MINOR: quic: ensure offset is properly set for STREAM frames
    - MINOR: quic: define new functions for frame alloc
    - MINOR: quic: refactor frame deallocation
    - MEDIUM: quic: implement a retransmit limit per frame
    - MINOR: quic: add config for retransmit limit
    - OPTIM: htx: inline the most common memcpy(8)
    - CLEANUP: quic: no need for atomics on packet refcnt
    - MINOR: stats: add by HTTP version cumulated number of sessions and requests
    - BUG/MINOR: quic: Possible stream truncations under heavy loss
    - BUG/MINOR: quic: Too big PTO during handshakes
    - MINOR: quic: Add a trace about variable states in qc_prep_fast_retrans()
    - BUG/MINOR: quic: Do not ignore coalesced packets in qc_prep_fast_retrans()
    - MINOR: quic: When probing Handshake packet number space, also probe the Initial one
    - BUG/MAJOR: quic: Possible crash when processing 1-RTT during 0-RTT session
    - MEDIUM: quic: Remove qc_conn_finalize() from the ClientHello TLS callbacks
    - BUG/MINOR: quic: Unchecked source connection ID
    - MEDIUM: listener: move the analysers mask to the bind_conf
    - MINOR: listener: move maxseg and tcp_ut to bind_conf
    - MINOR: listener: move maxaccept from listener to bind_conf
    - MINOR: listener: move the backlog setting from listener to bind_conf
    - MINOR: listener: move the maxconn parameter to the bind_conf
    - MINOR: listener: move the ->accept callback to the bind_conf
    - MINOR: listener: remove the useless ->default_target field
    - MINOR: listener: move the nice field to the bind_conf
    - MINOR: listener: move the NOLINGER option to the bind_conf
    - MINOR: listener: move the NOQUICKACK option to the bind_conf
    - MINOR: listener: move the DEF_ACCEPT option to the bind_conf
    - MINOR: listener: move TCP_FO to bind_conf
    - MINOR: listener: move the ACC_PROXY and ACC_CIP options to bind_conf
    - MINOR: listener: move LI_O_UNLIMITED and LI_O_NOSTOP to bind_conf
    - MINOR: listener: get rid of LI_O_TCP_L4_RULES and LI_O_TCP_L5_RULES
    - CLEANUP: listener: remove the now unused options field
    - MINOR: listener: remove the now useless LI_F_QUIC_LISTENER flag
    - CLEANUP: config: remove test for impossible case regarding bind thread mask
    - MINOR: thread: add a simple thread_set API
    - MEDIUM: listener/config: make the "thread" parser rely on thread_sets
    - CLEANUP: config: stop using bind_tgroup and bind_thread
    - CLEANUP: listener/thread: remove now unused bind_conf's bind_tgroup/bind_thread
    - CLEANUP: listener/config: remove the special case for shards==1
    - MEDIUM: config: restrict shards, not bind_conf to one group each
    - BUG/MEDIUM: quic: do not split STREAM frames if no space
    - BUILD: thread: fix build warnings with older gcc compilers
2023-02-04 10:51:05 +01:00
Willy Tarreau 15c8428060 BUILD: thread: fix build warnings with older gcc compilers
The "{ 0 }" form to initialize an empty structure triggers build warnings
on gcc 4.8, let's use the more common "{ }" instead.
2023-02-04 10:49:01 +01:00
Amaury Denoyelle f2f08f88ef BUG/MEDIUM: quic: do not split STREAM frames if no space
When building STREAM frames in a packet buffer, if a frame is too large
it will be splitted in two. A shorten version will be used and the
original frame will be modified to represent the remaining space.

To ensure there is enough space to store the frame data length encoded
as a QUIC integer, we use the function max_available_room(). This
function can return 0 if there not only a small space left which is
insufficient for the frame header and the shorten data. Prior to this
patch, this wasn't check and an empty unneeded STREAM frame was built
and sent for nothing.

Change this by checking the value return by max_available_room(). If 0,
do not try to split this frame and continue to the next ones in the
packet.

On 2.6, this patch serves as an optimization which will prevent the building
of unneeded empty STREAM frames.

On 2.7, this behavior has the side-effect of triggering a BUG_ON()
statement on quic_build_stream_frame(). This BUG_ON() ensures that we do
not use quic_frame with OFF bit set if its offset is 0. This can happens
if the condition defined above is reproduced for a STREAM frame at
offset 0. An empty unneeded frame is built as descibed. The problem is
that the original frame is modified with its OFF bit set even if the
offset is still 0.

This must be backported up to 2.6.
2023-02-03 19:19:50 +01:00
Willy Tarreau 50440457e3 MEDIUM: config: restrict shards, not bind_conf to one group each
Now that we're using thread_sets there's no need to restrict an entire
bind_conf to 1 group, the real concern being the FD, we can move that
restriction to the shard only. This means that as long as we have enough
shards and that they're properly aligned on group boundaries (i.e. shards
are an integer divider of the number of threads), we can support "bind"
lines spanning more than one group.

The check is still performed for shards to span more than one group,
and an error is emitted when this happens. But at least now it becomes
possible to have this:

    global
       nbthread 256

    frontend foo
       bind :1111 shards 4
       bind :2222 shards by-thread
2023-02-03 18:00:21 +01:00
Willy Tarreau 484093df80 CLEANUP: listener/config: remove the special case for shards==1
In fact this case is already handled by the regular shards code, there
is no need to special-case it.
2023-02-03 18:00:21 +01:00
Willy Tarreau f2988e1447 CLEANUP: listener/thread: remove now unused bind_conf's bind_tgroup/bind_thread
Not needed anymore since last commit, let's get rid of it.
2023-02-03 18:00:21 +01:00
Willy Tarreau e6b88592a8 CLEANUP: config: stop using bind_tgroup and bind_thread
Let's now retrieve the first thread group and its mask from the
thread_set so that we don't need these fields in the bind_conf anymore.
For now we're still limited to the first group (like before) but that
allows to get rid of these fields and to make sure that there's nothing
"special" being done there anymore.
2023-02-03 18:00:21 +01:00
Willy Tarreau f0de8cacc4 MEDIUM: listener/config: make the "thread" parser rely on thread_sets
Instead of reading and storing a single group and a single mask for a
"thread" directive on a bind line, we now store the complete range in
a thread set that's stored in the bind_conf. The bind_parse_thread()
function now just calls parse_thread_set() to complete the current set,
which starts empty, and thread_resolve_group_mask() was updated to
support retrieving thread group numbers or absolute thread numbers
directly from the pre-filled thread_set, and continue to feed bind_tgroup
and bind_thread. The CLI parsers which were pre-initialized to set the
bind_tgroup to 1 cannot do it anymore as it would prevent one from
restricting the thread set. Instead check_config_validity() now detects
the CLI frontend and passes the info down to thread_resolve_group_mask()
that will automatically use only the group 1's threads for these
listeners. The same is done for the peers listeners for now.

At this step it's already possible to start with all previous valid
configs as well as extended ones supporting comma-delimited thread
sets. In addition the parser already accepts large ranges spanning
multiple groups, but since the underlying listeners infrastructure
is not read, for now we're maintaining a specific check against this
at the higher level of the config validity check.

The patch is a bit large because thread resolution is performed in
multiple steps, so we need to adjust all of them at once to preserve
functional and technical consistency.
2023-02-03 18:00:21 +01:00
Willy Tarreau bef43dfa60 MINOR: thread: add a simple thread_set API
The purpose is to be able to store large thread sets, defined by ranges
that may cross group boundaries, as well as define lists of groups and
masks. The thread_set struct implements the storage, and the parser is
in parse_thread_set(), with a focus on "bind" lines, but not only.
2023-02-03 18:00:21 +01:00
Willy Tarreau 53c6c673ac CLEANUP: config: remove test for impossible case regarding bind thread mask
During 2.5 development, a fallback was implemented for bind "thread"
directives that would not map to existing threads, with commit e3f4d7496
("MEDIUM: config: resolve relative threads on bind lines to absolute ones").
The approch consisted in remapping the threads to other ones. But now
that relative threads and not absolute threads are stored in this mask,
this case cannot happen anymore, and this confusing hack is not needed
anymore.
2023-02-03 18:00:20 +01:00
Willy Tarreau 9e2682afed MINOR: listener: remove the now useless LI_F_QUIC_LISTENER flag
This flag is only used to tag a QUIC listener, which we now know by
its bind_conf's xprt as well. It's only used to decide whether or not
to perform an extra initialization step on the listener. Let's drop it
as well as the flags field.

With the various fields and options moved, the listener struct reduced
by 48 bytes total.
2023-02-03 18:00:20 +01:00
Willy Tarreau b25634d23e CLEANUP: listener: remove the now unused options field
All options that made sense were moved to the bind_conf, and remaining
ones were removed. This field isn't used at all anymore. The thr_idx
field was moved there to plug the hole.
2023-02-03 18:00:20 +01:00
Willy Tarreau 4c1d3a953d MINOR: listener: get rid of LI_O_TCP_L4_RULES and LI_O_TCP_L5_RULES
LI_O_TCP_L4_RULES and LI_O_TCP_L5_RULES are only set by from the proxy
based on the presence or absence of tcp_req l4/l5 rules. It's basically
as cheap to check the list as it is to check the flag, except that there
is no need to maintain a copy. Let's get rid of them, and this may ease
addition of more dynamic stuff later.
2023-02-03 18:00:20 +01:00
Willy Tarreau 1714680cec MINOR: listener: move LI_O_UNLIMITED and LI_O_NOSTOP to bind_conf
These two flags are entirely for internal use and are even per proxy
in practice since they're used for peers and CLI to indicate (for the
first one) that the listener(s) are not subject to connection limits,
and for the second that the listener(s) should not be stopped on
soft-stop. No need to keep them in the listeners, let's move them to
the bind_conf under names BC_O_UNLIMITED and BC_O_NOSTOP.
2023-02-03 18:00:20 +01:00
Willy Tarreau f1b4730f7d MINOR: listener: move the ACC_PROXY and ACC_CIP options to bind_conf
These are only set per bind line and used when creating a sessions,
we can move them to the bind_conf under the names BC_O_ACC_PROXY and
BC_O_ACC_CIP respectively.
2023-02-03 18:00:20 +01:00
Willy Tarreau c492f1b17f MINOR: listener: move TCP_FO to bind_conf
It's set per bind line ("tfo") and only used in tcp_bind_listener() so
there's no point keeping the address family tests, let's just store the
flag in the bind_conf under the name BC_O_TCP_FO.
2023-02-03 18:00:20 +01:00