Commit Graph

15753 Commits

Author SHA1 Message Date
Willy Tarreau
4c67bd6a06 [RELEASE] Released version 2.5-dev9
Released version 2.5-dev9 with the following main changes :
    - head-truc
    - REGTESTS: lua: test the httpclient:get() feature
    - Revert "head-truc"
    - BUG/MEDIUM: httpclient: replace ist0 by istptr
    - MINOR: config: use a standard parser for the "nbthread" keyword
    - CLEANUP: init: remove useless test against MAX_THREADS in affinity loop
    - MEDIUM: init: de-uglify the per-thread affinity setting
    - MINOR: init: extract the setup and end of threads to their own functions
    - MINOR: log: Try to get the status code when MUX_EXIT_STATUS is retrieved
    - MINOR: mux-h1: Set error code if possible when MUX_EXIT_STATUS is returned
    - MINOR: mux-h1: Be able to set custom status code on parsing error
    - MEDIUM: mux-h1: Reject HTTP/1.0 GET/HEAD/DELETE requests with a payload
    - MEDIUM: h1: Force close mode for invalid uses of T-E header
    - BUG/MINOR: mux-h1/mux-fcgi: Sanitize TE header to only send "trailers"
    - MINOR: http: Add 422-Unprocessable-Content error message
    - MINOR: h1: Change T-E header parsing to fail if chunked encoding is found twice
    - BUG/MEDIUM: mux-h1/mux-fcgi: Reject messages with unknown transfer encoding
    - REGTESTS: Add script to validate T-E header parsing
    - REORG: pools: move default settings to defaults.h
    - DOC: peers: fix doc "enable" statement on "peers" sections
    - MINOR: Makefile: add MEMORY_POOLS to the list of DEBUG_xxx options
    - MINOR: ssl: Set connection error code in case of SSL read or write fatal failure
    - MINOR: ssl: Rename ssl_bc_hsk_err to ssl_bc_err
    - MINOR: ssl: Store the last SSL error code in case of read or write failure
    - REGTESTS: ssl: enable show_ssl_ocspresponse.vtc again
    - REGTESTS: ssl: enable ssl_crt-list_filters.vtc again
    - BUG/MEDIUM: lua: fix wakeup condition from sleep()
    - BUG/MAJOR: lua: use task_wakeup() to properly run a task once
    - MINOR: arg: Be able to forbid unresolved args when building an argument list
    - BUG/MINOR: tcpcheck: Don't use arg list for default proxies during parsing
    - BUG/MINOR: tcp-rules: Stop content rules eval on read error and end-of-input
    - MINOR: tasks: catch TICK_ETERNITY with BUG_ON() in __task_queue()
    - REGTESTS: ssl: show_ssl_ocspresponse w/ freebsd won't use base64
    - REGTESTS: ssl: wrong feature cmd in show_ssl_ocspresponse.vtc
    - CLEANUP: tasks: remove the long-unused work_lists
    - MINOR: task: provide 3 task_new_* wrappers to simplify the API
    - MINOR: time: uninline report_idle() and move it to task.c
    - REORG: sched: move idle time calculation from time.h to task.h
    - REORG: sched: move the stolen CPU time detection to sched_entering_poll()
    - BUG/MEDIUM: filters: Fix a typo when a filter is attached blocking the release
    - BUG/MEDIUM: http-ana: Clear request analyzers when applying redirect rule
    - MINOR: httpclient: destroy() must free the headers and the ists
    - MINOR: httpclient: set HTTPCLIENT_F_ENDED only in release
    - MINOR: httpclient: stop_and_destroy() ask the applet to autokill
    - MINOR: httpclient: test if started during stop_and_destroy()
    - MINOR: httpclient/lua: implement garbage collection
    - BUG/MEDIUM: httpclient/lua: crash because of b_xfer and get_trash_chunk()
    - MINOR: httpclient: destroy checks if a client was started but not stopped
    - BUG/MINOR: httpclient/lua: does not process headers when failed
    - MINOR: httpclient/lua: supports headers via named arguments
    - CLEANUP: server: always include the storage for SSL settings
    - CLEANUP: sample: rename sample_conv_var2smp() to *_sint
    - CLEANUP: sample: uninline sample_conv_var2smp_str()
    - MINOR: sample: provide a generic var-to-sample conversion function
    - BUG/MEDIUM: sample: properly verify that variables cast to sample
    - BUILD: action: add the relevant structures for function arguments
    - BUILD: extcheck: needs to include stream-t.h
    - BUILD: hlua: needs to include stream-t.h
    - BUILD: stats: define several missing structures in stats.h
    - BUILD: resolvers: define missing types in resolvers.h
    - BUILD: httpclient: include missing ssl_sock-t
    - BUILD: sample: include openssl-compat
    - BUILD: http_ana: need to include proxy-t to get redirect_rule
    - BUILD: http_rules: requires http_ana-t.h for REDIRECT_*
    - BUILD: vars: need to include xxhash
    - BUILD: peers: need to include eb{32/mb/pt}tree.h
    - BUILD: ssl_ckch: include ebpttree.h in ssl_ckch.c
    - BUILD: compiler: add the container_of() and container_of_safe() macros
    - BUILD: idleconns: include missing ebmbtree.h at several places
    - BUILD: connection: connection.h needs list.h and server.h
    - BUILD: tree-wide: add missing http_ana.h from many places
    - BUILD: cfgparse-ssl: add missing errors.h
    - BUILD: tcp_sample: include missing errors.h and session-t.h
    - BUILD: mworker: mworker-prog needs time.h for the 'now' variable
    - BUILD: tree-wide: add several missing activity.h
    - BUILD: compat: fix -Wundef on SO_REUSEADDR
    - CLEANUP: pools: pools-t.h doesn't need to include thread-t.h
    - REORG: pools: uninline the UAF allocator and force-inline the rest
    - REORG: thread: uninline the lock-debugging code
    - MINOR: thread/debug: replace nsec_now() with now_mono_time()
    - CLEANUP: remove some unneeded includes from applet-t.h
    - REORG: listener: move bind_conf_alloc() and listener_state_str() to listener.c
    - CLEANUP: listeners: do not include openssl-compat
    - CLEANUP: servers: do not include openssl-compat
    - REORG: ssl: move ssl_sock_is_ssl() to connection.h and rename it
    - CLEANUP: mux_fcgi: remove dependency on ssl_sock
    - CLEANUP: ssl/server: move ssl_sock_set_srv() to srv_set_ssl() in server.c
    - REORG: ssl-sock: move the sslconns/totalsslconns counters to global
    - REORG: sample: move the crypto samples to ssl_sample.c
    - REORG: sched: moved samp_time and idle_time to task.c as well
    - REORG: time/ticks: move now_ms and global_now_ms definitions to ticks.h
    - CLEANUP: tree-wide: remove unneeded include time.h in ~20 files
    - REORG: activity: uninline activity_count_runtime()
    - REORG: acitvity: uninline sched_activity_entry()
    - CLEANUP: stream: remove many unneeded includes from stream-t.h
    - CLEANUP: stick-table: no need to include socket nor in.h
    - MINOR: connection: use uint64_t for the hashes
    - REORG: connection: move the hash-related stuff to connection.c
    - REORG: connection: uninline conn_notify_mux() and conn_delete_from_tree()
    - REORG: server: uninline the idle conns management functions
    - REORG: ebtree: split structures into their own file ebtree-t.h
    - CLEANUP: tree-wide: only include ebtree-t from type files
    - REORG: connection: move the largest inlines from connection.h to connection.c
    - CLEANUP: connection: do not include http_ana!
    - CLEANUP: connection: remove unneeded tcpcheck-t.h and use only session-t.h
    - REORG: connection: uninline the rest of the alloc/free stuff
    - REORG: task: uninline the loop time measurement code
    - CLEANUP: time: move a few configurable defines to defaults.h
    - CLEANUP: fd: do not include time.h
    - REORG: fd: uninline compute_poll_timeout()
    - CLENAUP: wdt: use ha_tkill() instead of accessing pthread directly
    - REORG: thread: move the thread init/affinity/stop to thread.c
    - REORG: thread: move ha_get_pthread_id() to thread.c
    - MINOR: thread: use a dedicated static pthread_t array in thread.c
    - CLEANUP: thread: uninline ha_tkill/ha_tkillall/ha_cpu_relax()
    - DOC: configuration: add clarification on escaping in keyword arguments
    - BUG/MINOR: task: fix missing include with DEBUG_TASK
    - MINOR: pools: report the amount used by thread caches in "show pools"
    - MINOR: quic: Distinguish packet and SSL read enc. level in traces
    - MINOR: quic: Add a function to dump SSL stack errors
    - MINOR: quic: BUG_ON() SSL errors.
    - MINOR: quic: Fix SSL error issues (do not use ssl_bio_and_sess_init())
    - BUG/MEDIUM: mux-quic: reinsert all streams in by_id tree
    - BUG/MAJOR: xprt-quic: do not queue qc timer if not set
    - MINOR: mux-quic: release connection if no more bidir streams
    - BUG/MAJOR: quic: remove qc from receiver cids tree on free
    - BUG/MEDIUM: mux_h2: Handle others remaining read0 cases on partial frames
    - MINOR: qpack: do not encode invalid http status code
    - MINOR: qpack: support non-indexed http status code encoding
    - MINOR: qpack: fix memory leak on huffman decoding
    - CLEANUP: mux-quic: remove unused code
    - BUG/MINOR: quic: fix includes for compilation
    - BUILD: connection: avoid a build warning on FreeBSD with SO_USER_COOKIE
    - BUILD: init: avoid a build warning on FreeBSD with USE_PROCCTL
    - REORG: time: move time-keeping code and variables to clock.c
    - REORG: clock: move the updates of cpu/mono time to clock.c
    - MINOR: activity: get the run_time from the clock updates
    - CLEANUP: clock: stop exporting before_poll and after_poll
    - REORG: clock: move the clock_id initialization to clock.c
    - REORG: clock/wdt: move wdt timer initialization to clock.c
    - MINOR: clock: move the clock_ids to clock.c
    - MINOR: wdt: move wd_timer to wdt.c
    - CLEANUP: wdt: do not remap SI_TKILL to SI_LWP, test the values directly
    - REORG: thread/sched: move the task_per_thread stuff to thread_ctx
    - REORG: thread/clock: move the clock parts of thread_info to thread_ctx
    - REORG: thread/sched: move the thread_info flags to the thread_ctx
    - REORG: thread/sched: move the last dynamic thread_info to thread_ctx
    - MINOR: thread: make "ti" a const pointer and clean up thread_info a bit
    - MINOR: threads: introduce a minimalistic notion of thread-group
    - MINOR: global: add a new "thread-groups" directive
    - MINOR: global: add a new "thread-group" directive
    - MINOR: threads: make tg point to the current thread's group
    - MEDIUM: threads: automatically assign threads to groups
    - MINOR: threads: set the group ID and its bit in the thread group
    - MINOR: threads: set the tid, ltid and their bit in thread_cfg
    - MEDIUM: threads: replace ha_set_tid() with ha_set_thread()
    - MINOR: threads: add the current group ID in thread-local "tgid" variable
    - MINOR: debug: report the group and thread ID in the thread dumps
    - MEDIUM: listeners: support the definition of thread groups on bind lines
    - MINOR: threads: add a new function to resolve config groups and masks
    - MEDIUM: config: resolve relative threads on bind lines to absolute ones
    - MEDIUM: stick-table: never learn the "conn_cur" value from peers
2021-10-08 18:22:24 +02:00
Willy Tarreau
db2ab8218c MEDIUM: stick-table: never learn the "conn_cur" value from peers
There have been a large number of issues reported with conn_cur
synchronization because the concept is wrong. In an active-passive
setup, pushing the local connections count from the active node to
the passive one will result in the passive node to have a higher
counter than the real number of connections. Due to this, after a
switchover, it will never be able to close enough connections to
go down to zero. The same commonly happens on reloads since the new
process preloads its values from the old process, and if no connection
happens for a key after the value is learned, it is impossible to reset
the previous ones. In active-active setups it's a bit different, as the
number of connections reflects the number on the peer that pushed last.

This patch solves this by marking the "conn_cur" local and preventing
it from being learned from peers. It is still pushed, however, so that
any monitoring system that collects values from the peers will still
see it.

The patch is tiny and trivially backportable. While a change of behavior
in stable branches is never welcome, it remains possible to fix issues
if reports become frequent.
2021-10-08 17:53:12 +02:00
Willy Tarreau
e3f4d7496d MEDIUM: config: resolve relative threads on bind lines to absolute ones
Now threads ranges specified on bind lines will be turned to effective
ones that will lead to a usable thread mask and a group ID.
2021-10-08 17:22:26 +02:00
Willy Tarreau
627def9e50 MINOR: threads: add a new function to resolve config groups and masks
In the configuration sometimes we'll omit a thread group number to designate
a global thread number range, and sometimes we'll mention the group and
designate IDs within that group. The operation is more complex than it
seems due to the need to check for ranges spanning between multiple groups
and determining groups from threads from bit masks and remapping bit masks
between local/global.

This patch adds a function to perform this operation, it takes a group and
mask on input and updates them on output. It's designed to be used by "bind"
lines but will likely be usable at other places if needed.

For situations where specified threads do not exist in the group, we have
the choice in the code between silently fixing the thread set or failing
with a message. For now the better option seems to return an error, but if
it turns out to be an issue we can easily change that in the future. Note
that it should only happen with "x/even" when group x only has one thread.
2021-10-08 17:22:26 +02:00
Willy Tarreau
d57b9ff7af MEDIUM: listeners: support the definition of thread groups on bind lines
This extends the "thread" statement of bind lines to support an optional
thread group number. When unspecified (0) it's an absolute thread range,
and when specified it's one relative to the thread group. Masks are still
used so no more than 64 threads may be specified at once, and a single
group is possible. The directive is not used for now.
2021-10-08 17:22:26 +02:00
Willy Tarreau
a3870b7952 MINOR: debug: report the group and thread ID in the thread dumps
Now thread dumps will report the thread group number and the ID within
this group. Note that this is still quite limited because some masks
are calculated based on the thread in argument while they have to be
performed against a group-level thread ID.
2021-10-08 17:22:26 +02:00
Willy Tarreau
b90935c908 MINOR: threads: add the current group ID in thread-local "tgid" variable
This is the equivalent of "tid" for ease of access. In the future if we
make th_cfg a pure thread-local array (not a pointer), it may make sense
to move it there.
2021-10-08 17:22:26 +02:00
Willy Tarreau
43ab05b3da MEDIUM: threads: replace ha_set_tid() with ha_set_thread()
ha_set_tid() was randomly used either to explicitly set thread 0 or to
set any possibly incomplete thread during boot. Let's replace it with
a pointer to a valid thread or NULL for any thread. This allows us to
check that the designated threads are always valid, and to ignore the
thread 0's mapping when setting it to NULL, and always use group 0 with
it during boot.

The initialization code is also cleaner, as we don't pass ugly casts
of a thread ID to a pointer anymore.
2021-10-08 17:22:26 +02:00
Willy Tarreau
cc7a11ee3b MINOR: threads: set the tid, ltid and their bit in thread_cfg
This will be a convenient way to communicate the thread ID and its
local ID in the group, as well as their respective bits when creating
the threads or when only a pointer is given.
2021-10-08 17:22:26 +02:00
Willy Tarreau
6eee85f887 MINOR: threads: set the group ID and its bit in the thread group
This will ease the reporting of the current thread group ID when coming
from the thread itself, especially since it returns the visible ID,
starting at 1.
2021-10-08 17:22:26 +02:00
Willy Tarreau
e6806ebecc MEDIUM: threads: automatically assign threads to groups
This takes care of unassigned threads groups and places unassigned
threads there, in a more or less balanced way. Too sparse allocations
may still fail though. For now with a maximum group number fixed to 1
nothing can really fail.
2021-10-08 17:22:26 +02:00
Willy Tarreau
fc69e410e6 MINOR: threads: make tg point to the current thread's group
A the "tg" thread-local variable now always points to the current
thread group. It's pre-initializd to the first one during boot and is
set to point to the thread's one by ha_set_tid(). This last one takes
care of checking whether the thread group was assigned or not because
it may be called during boot before threads are initialized.
2021-10-08 17:22:26 +02:00
Willy Tarreau
d04bc3ac21 MINOR: global: add a new "thread-group" directive
This registers a mapping of threads to groups by enumerating for each thread
what group it belongs to, and marking the group as assigned. It takes care of
checking for redefinitions, overlaps, and holes. It supports both individual
numbers and ranges. The thread group is referenced from the thread config.
2021-10-08 17:22:26 +02:00
Willy Tarreau
c33b969e35 MINOR: global: add a new "thread-groups" directive
This is used to configure the number of thread groups. For now it can
only be 1.
2021-10-08 17:22:26 +02:00
Willy Tarreau
f9662848f2 MINOR: threads: introduce a minimalistic notion of thread-group
This creates a struct tgroup_info which knows the thread ID of the first
thread in a group, and the number of threads in it. For now there's only
one thread group supported in the configuration, but it may be forced to
other values for development purposes by defining MAX_TGROUPS, and it's
enabled even when threads are disabled and will need to remain accessible
during boot to keep a simple enough internal API.

For the purpose of easing the configurations which do not specify a thread
group, we're starting group numbering at 1 so that thread group 0 can be
"undefined" (i.e. for "bind" lines or when binding tasks).

The goal will be to later move there some global items that must be
made per-group.
2021-10-08 17:22:26 +02:00
Willy Tarreau
6036342f58 MINOR: thread: make "ti" a const pointer and clean up thread_info a bit
We want to make sure that the current thread_info accessed via "ti" will
remain constant, so that we don't accidentally place new variable parts
there and so that the compiler knows that info retrieved from there is
not expected to have changed between two function calls.

Only a few init locations had to be adjusted to use the array and the
rest is unaffected.
2021-10-08 17:22:26 +02:00
Willy Tarreau
b4e34766a3 REORG: thread/sched: move the last dynamic thread_info to thread_ctx
The last 3 fields were 3 list heads that are per-thread, and which are:
  - the pool's LRU head
  - the buffer_wq
  - the streams list head

Moving them into thread_ctx completes the removal of dynamic elements
from the struct thread_info. Now all these dynamic elements are packed
together at a single place for a thread.
2021-10-08 17:22:26 +02:00
Willy Tarreau
a0b99536c8 REORG: thread/sched: move the thread_info flags to the thread_ctx
The TI_FL_STUCK flag is manipulated by the watchdog and scheduler
and describes the apparent life/death of a thread so it changes
all the time and it makes sense to move it to the thread's context
for an active thread.
2021-10-08 17:22:26 +02:00
Willy Tarreau
45c38e22bf REORG: thread/clock: move the clock parts of thread_info to thread_ctx
The "thread_info" name was initially chosen to store all info about
threads but since we now have a separate per-thread context, there is
no point keeping some of its elements in the thread_info struct.

As such, this patch moves prev_cpu_time, prev_mono_time and idle_pct to
thread_ctx, into the thread context, with the scheduler parts. Instead
of accessing them via "ti->" we now access them via "th_ctx->", which
makes more sense as they're totally dynamic, and will be required for
future evolutions. There's no room problem for now, the structure still
has 84 bytes available at the end.
2021-10-08 17:22:26 +02:00
Willy Tarreau
1a9c922b53 REORG: thread/sched: move the task_per_thread stuff to thread_ctx
The scheduler contains a lot of stuff that is thread-local and not
exclusively tied to the scheduler. Other parts (namely thread_info)
contain similar thread-local context that ought to be merged with
it but that is even less related to the scheduler. However moving
more data into this structure isn't possible since task.h is high
level and cannot be included everywhere (e.g. activity) without
causing include loops.

In the end, it appears that the task_per_thread represents most of
the per-thread context defined with generic types and should simply
move to tinfo.h so that everyone can use them.

The struct was renamed to thread_ctx and the variable "sched" was
renamed to "th_ctx". "sched" used to be initialized manually from
run_thread_poll_loop(), now it's initialized by ha_set_tid() just
like ti, tid, tid_bit.

The memset() in init_task() was removed in favor of a bss initialization
of the array, so that other subsystems can put their stuff in this array.

Since the tasklet array has TL_CLASSES elements, the TL_* definitions
was moved there as well, but it's not a problem.

The vast majority of the change in this patch is caused by the
renaming of the structures.
2021-10-08 17:22:26 +02:00
Willy Tarreau
6414e4423c CLEANUP: wdt: do not remap SI_TKILL to SI_LWP, test the values directly
We used to remap SI_TKILL to SI_LWP when SI_TKILL was not available
(e.g. FreeBSD) but that's ugly and since we need this only in a single
switch/case block in wdt.c it's even simpler and cleaner to perform the
two tests there, so let's do this.
2021-10-08 17:22:26 +02:00
Willy Tarreau
b474f43816 MINOR: wdt: move wd_timer to wdt.c
The watchdog timer had no more reason for being shared with the struct
thread_info since the watchdog is the only user now. Let's remove it
from the struct and move it to a static array in wdt.c. This removes
some ifdefs and the need for the ugly mapping to empty_t that might be
subject to a cast to a long when compared to TIMER_INVALID. Now timer_t
is not known outside of wdt.c and clock.c anymore.
2021-10-08 17:22:26 +02:00
Willy Tarreau
2169498941 MINOR: clock: move the clock_ids to clock.c
This removes the knowledge of clockid_t from anywhere but clock.c, thus
eliminating a source of includes burden. The unused clock_id field was
removed from thread_info, and the definition setting of clockid_t was
removed from compat.h. The most visible change is that the function
now_cpu_time_thread() now takes the thread number instead of a tinfo
pointer.
2021-10-08 17:22:26 +02:00
Willy Tarreau
6cb0c391e7 REORG: clock/wdt: move wdt timer initialization to clock.c
The code that deals with timer creation for the WDT was moved to clock.c
and is called with the few relevant arguments. This removes the need for
awareness of clock_id from wdt.c and as such saves us from having to
share it outside. The timer_t is also known only from both ends but not
from the public API so that we don't have to create a fake timer_t
anymore on systems which do not support it (e.g. macos).
2021-10-08 17:22:26 +02:00
Willy Tarreau
44c58da52f REORG: clock: move the clock_id initialization to clock.c
This was previously open-coded in run_thread_poll_loop(). Now that
we have clock.c dedicated to such stuff, let's move the code there
so that we don't need to keep such ifdefs nor to depend on the
clock_id.
2021-10-08 17:22:26 +02:00
Willy Tarreau
2c6a998727 CLEANUP: clock: stop exporting before_poll and after_poll
We don't need to export them anymore so let's make them static.
2021-10-08 17:22:26 +02:00
Willy Tarreau
20adfde9c8 MINOR: activity: get the run_time from the clock updates
Instead of fiddling with before_poll and after_poll in
activity_count_runtime(), the function is now called by
clock_entering_poll() which passes it the number of microseconds
spent working. This allows to remove all calls to
activity_count_runtime() from the pollers.
2021-10-08 17:22:26 +02:00
Willy Tarreau
f9d5e1079c REORG: clock: move the updates of cpu/mono time to clock.c
The entering_poll/leaving_poll/measure_idle functions that were hard
to classify and used to move to various locations have now been placed
into clock.c since it's precisely about time-keeping. The functions
were renamed to clock_*. The samp_time and idle_time values are now
static since there is no reason for them to be read from outside.
2021-10-08 17:22:26 +02:00
Willy Tarreau
5554264f31 REORG: time: move time-keeping code and variables to clock.c
There is currently a problem related to time keeping. We're mixing
the functions to perform calculations with the os-dependent code
needed to retrieve and adjust the local time.

This patch extracts from time.{c,h} the parts that are solely dedicated
to time keeping. These are the "now" or "before_poll" variables for
example, as well as the various now_*() functions that make use of
gettimeofday() and clock_gettime() to retrieve the current time.

The "tv_*" functions moved there were also more appropriately renamed
to "clock_*".

Other parts used to compute stolen time are in other files, they will
have to be picked next.
2021-10-08 17:22:26 +02:00
Willy Tarreau
28345c6652 BUILD: init: avoid a build warning on FreeBSD with USE_PROCCTL
It was brought by a variable declared after some statements in commit
21185970c ("MINOR: proc: setting the process to produce a core dump on
FreeBSD."). It's worth noting that some versions of clang seem to ignore
-Wdeclaration-after-statement by default. No backport is needed.
2021-10-08 17:21:48 +02:00
Willy Tarreau
de361ad22e BUILD: connection: avoid a build warning on FreeBSD with SO_USER_COOKIE
It was brough by an unneeded addition of a local variable after a test
in commit f7f53afcf ("BUILD/MEDIUM: tcp: set-mark setting support for
FreeBSD."). No backport needed.
2021-10-08 17:21:48 +02:00
Amaury Denoyelle
eb01f597eb BUG/MINOR: quic: fix includes for compilation
Fix missing includes in quic code following the general recent include
reorganization. This fixes the compilation error with QUIC enabled.
2021-10-08 15:59:02 +02:00
Amaury Denoyelle
769e9ffd94 CLEANUP: mux-quic: remove unused code
Remove unused code in mux-quic. This is mostly code related to the
backend side. This code is untested for the moment, its removal will
simplify the code maintenance.
2021-10-08 15:48:00 +02:00
Amaury Denoyelle
9c8c4fa3a2 MINOR: qpack: fix memory leak on huffman decoding
Remove an unneeded strdup invocation during QPACK huffman decoding. A
temporary storage buffer is passed by the function and exists after
decoding so no need to duplicate memory here.
2021-10-08 15:45:57 +02:00
Amaury Denoyelle
3a590c7ff2 MINOR: qpack: support non-indexed http status code encoding
If a HTTP status code is not present in the QPACK static table, encode
it with a literal field line with name reference.
2021-10-08 15:30:18 +02:00
Amaury Denoyelle
fccffe08b3 MINOR: qpack: do not encode invalid http status code
Ensure that the HTTP status code is valid before encoding with QPACK. An
error is return if this is not the case.
2021-10-08 15:28:35 +02:00
Christopher Faulet
485da0b053 BUG/MEDIUM: mux_h2: Handle others remaining read0 cases on partial frames
We've found others places where the read0 is ignored because of an
incomplete frame parsing. This time, it happens during parsing of
CONTINUATION frames.

When frames are parsed, incomplete frames are properly handled and
H2_CF_DEM_SHORT_READ flag is set. It is also true for HEADERS
frames. However, for CONTINUATION frames, there is an exception. Besides
parsing the current frame, we try to peek header of the next one to merge
payload of both frames, the current one and the next one. Idea is to create
a sole HEADERS frame before parsing the payload. However, in this case, it
is possible to have an incomplete frame too, not the current one but the
next one. From the demux point of view, the current frame is complete. We
must go to the internal function h2c_decode_headers() to detect an
incomplete frame. And this case was not identified and fixed when
H2_CF_DEM_SHORT_READ flag was introduced in the commit b5f7b5296
("BUG/MEDIUM: mux-h2: Handle remaining read0 cases on partial frames")

This bug was reported in a comment of the issue #1362. The patch must be
backported as far as 2.0.
2021-10-08 09:17:27 +02:00
Amaury Denoyelle
2af1985af8 BUG/MAJOR: quic: remove qc from receiver cids tree on free
Remove the quic_conn from the receiver connection_ids tree on
quic_conn_free. This fixes a crash due to dangling references in the
tree after a quic connection release.

This operation must be conducted under the listener lock. For this
reason, the quic_conn now contains a reference to its attached listener.
2021-10-07 17:35:25 +02:00
Amaury Denoyelle
d595f108db MINOR: mux-quic: release connection if no more bidir streams
Use the count of bidirectional streams to call qc_release in qc_detach.
We cannot inspect the by_id tree because uni-streams are never removed
from it. This allows the connection to be properly freed.
2021-10-07 17:35:25 +02:00
Amaury Denoyelle
336f6fd964 BUG/MAJOR: xprt-quic: do not queue qc timer if not set
Do not queue the pto/loss-detection timer if set to TICK_ETERNITY. This
usage is invalid with the scheduler and cause a BUG_ON trigger.
2021-10-07 17:35:25 +02:00
Amaury Denoyelle
139814a67a BUG/MEDIUM: mux-quic: reinsert all streams in by_id tree
It is required that all qcs streams are in the by_id tree for the xprt
to function correctly. Without this, some ACKs are not properly emitted
by xprt.

Note that this change breaks the free of the connection because the
condition eb_is_empty in qc_detach is always true. This will be fixed in
a following patch.
2021-10-07 17:35:25 +02:00
Frédéric Lécaille
75dd2b7987 MINOR: quic: Fix SSL error issues (do not use ssl_bio_and_sess_init())
It seems it was a bad idea to use the same function as for TCP ssl sockets
to initialize the SSL session objects for QUIC with ssl_bio_and_sess_init().
Indeed, this had as very bad side effects to generate SSL errors due
to the fact that such BIOs initialized for QUIC could not finally be controlled
via the BIO_ctrl*() API, especially BIO_ctrl() function used by very much other
internal OpenSSL functions (BIO_push(), BIO_pop() etc).
Others OpenSSL base QUIC implementation do not use at all BIOs to configure
QUIC connections. So, we decided to proceed the same way as ngtcp2 for instance:
only initialize an SSL object and call SSL_set_quic_method() to set its
underlying method. Note that calling this function silently disable this option:
SSL_OP_ENABLE_MIDDLEBOX_COMPAT.
We implement qc_ssl_sess_init() to initialize SSL sessions for QUIC connections
to do so with a retry in case of allocation failure as this is done by
ssl_bio_and_sess_init(). We also modify the code part for haproxy servers.
2021-10-07 17:35:25 +02:00
Frédéric Lécaille
7c881bdab8 MINOR: quic: BUG_ON() SSL errors.
As this QUIC implementation is still experimental, let's BUG_ON()
very important SSL handshake errors.
Also dump the SSL errors before BUG_ON().
2021-10-07 17:35:25 +02:00
Frédéric Lécaille
6f0fadb5a7 MINOR: quic: Add a function to dump SSL stack errors
This has been very helpful to fix SSL related issues.
2021-10-07 17:35:25 +02:00
Frédéric Lécaille
57e6e9eef8 MINOR: quic: Distinguish packet and SSL read enc. level in traces
This is only to distinguish the encryption level of packet traces from
the TLS stack current read encryption level.
2021-10-07 17:35:25 +02:00
Willy Tarreau
1b4a714266 MINOR: pools: report the amount used by thread caches in "show pools"
The "show pools" command provides some "allocated" and "used" estimates
on the pools objects, but this applies to the shared pool and the "used"
includes what is currently assigned to thread-local caches. It's possible
to know how much each thread uses, so let's dump the total size allocated
by thread caches as an estimate. It's only done when pools are enabled,
which explains why the patch adds quite a lot of ifdefs.
2021-10-07 17:30:06 +02:00
Amaury Denoyelle
1a9b8a6122 BUG/MINOR: task: fix missing include with DEBUG_TASK
Following include reorganzation, there is some missing include files for
task.h when compiling with DEBUG_TASK :
- activity.h for task_profiling_mask
- time.h for now_mono_time()

This is present since the following commit
  d8b325c74826bdb02759f62c41a00455dbae3431
  REORG: task: uninline the loop time measurement code

No need to backport this.
2021-10-07 16:44:49 +02:00
Thayne McCombs
cd34ad7133 DOC: configuration: add clarification on escaping in keyword arguments
Add a more precise description on how backslash escaping is different
than the top-level parser, and give examples of how to handle single
quotes inside arguments.
2021-10-07 07:51:30 +02:00
Willy Tarreau
aa992761d8 CLEANUP: thread: uninline ha_tkill/ha_tkillall/ha_cpu_relax()
These ones are rarely used or only to waste CPU cycles waiting, and are
the last ones requiring system includes in thread.h. Let's uninline them
and move them to thread.c.
2021-10-07 01:41:15 +02:00
Willy Tarreau
5e03dfaaf6 MINOR: thread: use a dedicated static pthread_t array in thread.c
This removes the thread identifiers from struct thread_info and moves
them only in static array in thread.c since it's now the only file that
needs to touch it. It's also the only file that needs to include
pthread.h, beyond haproxy.c which needs it to start the poll loop. As
a result, much less system includes are needed and the LoC reduced by
around 3%.
2021-10-07 01:41:15 +02:00