Transfer-Encoding header is not supported in HTTP/1.0. However, softwares
dealing with HTTP/1.0 and HTTP/1.1 messages may accept it and transfer
it. When a Content-Length header is also provided, it must be
ignored. Unfortunately, this may lead to vulnerabilities (request smuggling
or response splitting) if an intermediary is only implementing
HTTP/1.0. Because it may ignore Transfer-Encoding header and only handle
Content-Length one.
To avoid any security issues, when Transfer-Encoding and Content-Length
headers are found in a message, the close mode is forced. The same is
performed for HTTP/1.0 message with a Transfer-Encoding header only. This
change is conform to what it is described in the last HTTP/1.1 draft. See
also httpwg/http-core#879.
Note that Content-Length header is also removed from any incoming messages
if a Transfer-Encoding header is found. However it is not true (not yet) for
responses generated by HAProxy.
This kind of requests is now forbidden and rejected with a
413-Payload-Too-Large error.
It is unexpected to have a payload for GET/HEAD/DELETE requests. It is
explicitly allowed in HTTP/1.1 even if some servers may reject such
requests. However, HTTP/1.0 is not clear on this point and some old servers
don't expect any payload and never look for body length (via Content-Length
or Transfer-Encoding headers).
It means that some intermediaries may properly handle the payload for
HTTP/1.0 GET/HEAD/DELETE requests, while some others may totally ignore
it. That may lead to security issues because a request smuggling attack is
possible.
To prevent any issue, those requests are now rejected.
See also httpwg/http-core#904
When a parsing error is triggered, the status code may be customized by
setting H1C .errcode field. By default a 400-Bad-Request is returned. The
function h1_handle_bad_req() has been renamed to h1_handle_parsing_error()
to be more generic.
In h1_ctl(), if output parameter is provided when MUX_EXIT_STATUS is
returned, it is used to set the error code. In addition, any client errors
(4xx), except for 408 ones, are handled as invalid errors
(MUX_ES_INVALID_ERR). This way, it will be possible to customize the parsing
error code for request messages.
The mux .ctl callback can provide some information about the mux to the
caller if the third parameter is provided. Thus, when MUX_EXIT_STATUS is
retrieved, a pointer on the status is now passed. The mux may fill it. It
will be pretty handy to provide custom error code from h1 mux instead of
default ones (400/408/500/501).
The startup code was still ugly with tons of unreadable nested ifdefs.
Let's just have one function to set up the extra threads and another one
to wait for their completion. The ifdefs are isolated into their own
functions now and are more readable, just like the end of main(), which
now uses the same statements to start thread 0 with and without threads.
Till now the threads startup was quite messy:
- we would start all threads but one
- then we would change all threads' CPU affinities
- then we would manually start the poll loop for the current thread
Let's change this by moving the CPU affinity setting code to a function
set_thread_cpu_affinity() that does this job for the current thread only,
and that is called during the thread's initialization in the polling loop.
It takes care of not doing this for the master, and will result in all
threads to be properly bound earlier and with cleaner code. It also
removes some ugly nested ifdefs.
Probably because of some copy-paste from "nbproc", "nbthread" used to
be parsed in cfgparse instead of using a registered parser. Let's fix
this to clean up the code base now.
ASAN reported a buffer overflow in the httpclient. This overflow is the
consequence of ist0() which is incorrect here.
Replace all occurences of ist0() by istptr() which is more appropried
here since all ist in the httpclient were created from strings.
This reg-test is heavily inspired by the lua_socket.vtc one.
It replaces the HTTP/1.1 request made manually with a socket object with
an httpclient object.
Released version 2.5-dev8 with the following main changes :
- BUILD: compiler: fixed a missing test on defined(__GNUC__)
- BUILD: halog: fix a -Wundef warning on non-glibc systems
- BUILD: threads: fix -Wundef for _POSIX_PRIORITY_SCHEDULING on libmusl
- BUG/MINOR: compat: make sure __WORDSIZE is always defined
- BUILD: sample: fix format warning on 32-bit archs in sample_conv_be2dec_check()
- CLEANUP: pools: factor all malloc_trim() calls into trim_all_pools()
- MINOR: pools: automatically disable malloc_trim() with external allocators
- MINOR: pools: report it when malloc_trim() is enabled
- DOC: Add .mailmap
- CLEANUP: tree-wide: fix prototypes for functions taking no arguments.
- CLEANUP: Remove prototype for non-existent thread_get_default_count()
- CLEANUP: acl: Remove unused variable when releasing an acl expression
- BUG/MAJOR: mux-h1: Don't eval input data if an error was reported
- DOC: update Tim's address in .mailmap
- MINOR: pools: use mallinfo2() when available instead of mallinfo()
- BUG/MINOR: tcpcheck: Improve LDAP response parsing to fix LDAP check
- DOC: management: certificate files must be sanitized before injection
- BUG/MINOR: connection: prevent null deref on mux cleanup task allocation
- BUILD: ist: prevent gcc11 maybe-uninitialized warning on istalloc
- BUG/MINOR: cli/payload: do not search for args inside payload
- BUILD: sockpair: do not set unused flag
- BUILD: proto_uxst: do not set unused flag
- BUILD: fd: remove unused variable totlen in fd_write_frag_line()
- MINOR: applet: remove the thread mask from appctx_new()
- REORG: threads: move ha_get_pthread_id() to tinfo.h
- CLEANUP: Apply ist.cocci
- DEV: coccinelle: Add ist.cocci
- CLEANUP: Apply bug_on.cocci
- DEV: coccinelle: Add xalloc_size.cocci
- DEV: coccinelle: Add bug_on.cocci
- CLEANUP: Apply xalloc_size.cocci
- DEV: coccinelle: Add xalloc_cast.cocci
- BUG/MINOR: flt-trace: fix an infinite loop when random-parsing is set
- MINOR: httpclient: add the EOH when no headers where provided
- CLEANUP: Include check.h in flt_spoe.c
- CLEANUP: Remove unreachable `break` from parse_time_err()
- BUG/MINOR: server: allow 'enable health' only if check configured
- BUG/MINOR: server: alloc dynamic srv ssl ctx if proxy uses ssl chk rule
- MINOR: server: enable more keywords for ssl checks for dynamic servers
- MINOR: server: enable more check related keywords for dynamic servers
- REORG: server: move slowstart init outside of checks
- MINOR: server: enable slowstart for dynamic server
- MEDIUM: listener: deprecate "process" in favor of "thread" on bind lines
- BUG/MEDIUM: leastconn: fix rare possibility of divide by zero
- BUG/MINOR: quic: Possible NULL pointer dereferencing when dumping streams.
- MINOR: quic: Move transport parmaters to anynomous struct.
- MINOR: mux_quic: Add QUIC mux layer.
- MINOR: connection: Add callbacks definitions for QUIC.
- MINOR: quic: Attach QUIC mux connection objet to QUIC connection.
- MINOR: quic: Add a new definition to store STREAM frames.
- MINOR: h3: Add HTTP/3 definitions.
- MINOR: qpack: Add QPACK compression.
- MINOR: quic_sock: Finalize the QUIC connections.
- MINOR: quic: Disable the action of ->rcv_buf() xprt callback
- MINOR: quic: Add callbacks for (un)scribing to QUIC xprt.
- MINOR: quic: Variable-length integer encoding/decoding into/from buffer struct.
- BUG/MINOR: quic: Wrong ->accept() error handling
- MINOR: quic: Add a wrapper function to update transport parameters.
- MINOR: quic: Update the streams transport parameters.
- MINOR: quic: Avoid header collisions
- MINOR: quic: Replace max_packet_size by max_udp_payload size.
- MINOR: quic: Enable some quic, h3 and qpack modules compilation.
- MINOR: quic: Move an SSL func call from QUIC I/O handler to the xprt init.
- MINOR: quic: Initialize the session before starting the xprt.
- BUG/MINOR: quic: Do not check the acception of a new conn from I/O handler.
- MINOR: quic: QUIC conn initialization from I/O handler
- MINOR: quic: Remove header protection for conn with context
- MINOR: quic: Derive the initial secrets asap
- MINOR: quic: Remove header protection also for Initial packets
- BUG/MINOR: quic: Wrong memory free in quic_update_ack_ranges_list()
- MINOR: quic: quic_update_ack_ranges_list() code factorization
- MINOR: quic: Useless test in quic_update_ack_ranges_list()
- MINOR: quic: Remove a useless variable in quic_update_ack_ranges_list()
- BUG/MINOR: quic: Missing cases treatement when updating ACK ranges
- CLEAUNUP: quic: Usage of a useless variable in qc_treat_rx_pkts()
- BUG/MINOR: quic: Wrong RX packet reference counter usage
- MINOR: quic: Do not stop the packet parsing too early in qc_treat_rx_packets()
- MINOR: quic: Add a lock for RX packets
- MINOR: quic: Move the connection state
- MINOR: quic: Replace quic_conn_ctx struct by ssl_sock_ctx struct
- MINOR: quic: Replace the RX list of packet by a thread safety one.
- MINOR: quic: Replace the RX unprotected packet list by a thread safety one.
- MINOR: quic: Add useful traces for I/O dgram handler
- MINOR: quic: Do not wakeup the xprt task on ACK receipt
- MINOR: quic: Connection allocations rework
- MINOR: quic: Move conn_prepare() to ->accept_conn() callback
- MINOR: quic: Make qc_lstnr_pkt_rcv() be thread safe.
- MINOR: quic: Add a ring buffer implementation for QUIC
- MINOR: quic: Prefer x25519 as ECDH preferred parametes.
- MINOR: quic: Add the QUIC v1 initial salt.
- BUG/MINOR: quic: Too much reduced computed space to build handshake packets
- MINOR: net_helper: add functions for pointers
- MINOR: quic: Add ring buffer definition (struct qring) for QUIC
- MINOR: proto_quic: Allocate TX ring buffers for listeners
- MINOR: quic: Initialize pointers to TX ring buffer list
- MINOR: quic: Make use of TX ring buffers to send QUIC packets
- MINOR: quic_tls: Make use of the QUIC V1 salt.
- MINOR: quic: Remove old TX buffer implementation
- MINOR: Add function for TX packets reference counting
- MINOR: quic: Add TX packets at the very last time to their tree.
- MINOR: quic: Unitialized mux context upon Client Hello message receipt.
- MINOR: quic: Missing encryption level rx.crypto member initialization and lock.
- MINOR: quic: Rename ->rx.rwlock of quic_enc_level struct to ->rx.pkts_rwlock
- MINOR: quic: Make qc_treat_rx_pkts() be thread safe.
- MINOR: quic: Make ->tx.frms quic_pktns struct member be thread safe
- MINOR: quic: Replace quic_tx_frm struct by quic_frame struct
- MINOR: quic: Add a mask for TX frame builders and their authorized packet types
- MINOR: quic: Add a useful function to compute any frame length.
- MINOR: quic: Add the QUIC connection state to traces
- MINOR: quic: Store post handshake frame in ->pktns.tx.frms MT_LIST
- MINOR: quic: Add the packet type to quic_tx_packet struct
- MINOR: quic: Modify qc_do_build_hdshk_pkt() to accept any packet type
- MINOR: quic: Atomically handle packet number space ->largest_acked_pn variable
- MINOR: quic: Modify qc_build_cfrms() to support any frame
- MINOR: quic: quic_conn_io_cb() task rework
- MINOR: quic: Make qc_build_hdshk_pkt() atomically consume a packet number
- MINOR: quic: qc_do_build_hdshk_pkt() does not need to pass a copy of CRYPTO frame
- MINOR: quic: Remove Application level related functions
- MINOR: quic: Rename functions which do not build only Handshake packets
- MINOR: quic: Make circular buffer internal buffers be variable-sized.
- MINOR: quic: Add a pool for TX ring buffer internal buffer
- MINOR: quic: Make use of the last cbuf API when initializing TX ring buffers
- MINOR: quic: Missing acks encoded size updates.
- MINOR: quic: Evaluate the packet lengths in advance
- MINOR: quic: Update the TLS extension for QUIC transport parameters
- MINOR: quic: Fix handshake state debug strings
- MINOR: quic: Atomically get/set the connection state
- MINOR: quic: Missing QUIC encryption level for qc_build_pkt()
- MINOR: quic: Coalesce Application level packets with Handshake packets.
- MINOR: quic: Wrong flags handling for acks
- MINOR: quic: Missing case when discarding HANDSHAKE secrets
- MINOR: quic: Post handshake packet building improvements
- MINOR: quic: Prepare Application level packet asap.
- MINOR: h3: Send h3 settings asap
- MINOR: quic: Wrong STREAM frame length computing
- MINOR: quic: Wrong short packet minimum length
- MINOR: quic: Prepare STREAM frames to fill QUIC packets
- MINOR: h3: change default settings
- MINOR: quic-enc: fix varint encoding
- MINOR: qpack: fix wrong comment
- MINOR: qpack: generate headers list on decoder
- MINOR: h3: parse headers to htx
- MINOR: h3: allocate stream on headers
- MEDIUM: mux-quic: implement ring buffer on stream tx
- MINOR: mux-quic: send SETTINGS on uni stream
- MINOR: h3: define snd_buf callback and divert mux ops
- MINOR: mux-quic: define FIN stream flag
- MINOR: qpack: create qpack-enc module
- MINOR: qpack: encode headers functions
- MINOR: h3: encode htx headers to QPACK
- MINOR: h3: send htx data
- MINOR: h3/mux: detect fin on last h3 frame of the stream
- MINOR: quic: Shorten some handshakes
- MINOR: quic: Make QUIC-TLS support at least two initial salts
- MINOR: quic: Attach the QUIC connection to a thread.
- MINOR: quic: Missing active_connection_id_limit default value
- MINOR: quic_sock: Do not flag QUIC connections as being set
- MINOR: buf: Add b_force_xfer() function
- MINOR: quic: Make use of buffer structs to handle STREAM frames
- MINOR: mux_quic: move qc_process() code to qc_send()
- MINOR: quic: Add a typedef for unsigned long long
- MINOR: quic: Confusion between TX/RX for the frame builders
- MINOR: quic: Wrong packet flags settings during frame building
- MINOR: quic: Constantness fixes for frame builders/parsers.
- MINOR: quic_tls: Client/serveur state reordering
- MINOR: quic: Wrong packet loss detection due to wrong pktns order
- MINOR: quic: Wrong packet number space selection in quic_loss_pktns()
- MINOR: quic: Initial packet number spaced not discarded
- MINOR: quic: Add useful trace about pktns discarding
- MINOR: mux_quic: Export the mux related flags
- MINOR: quic: Implement quic_conn_subscribe()
- MINOR: quic: Wake up the mux upon ACK receipt
- MINOR: quic: Stream FIN bit fix in qcs_push_frame()
- MINOR: quic: Implement qc_process_mux()
- MINOR: quic: Wake up the xprt from mux
- CLEANUP: quic: Remove useless inline functions
- MINOR: quic: RX packets memory leak
- MINOR: quic: Possible endless loop in qc_treat_rx_pkts()
- MINOR: quic: Crash upon too big packets receipt
- MINOR: quic: define close handler
- MEDIUM: quic: implement mux release/conn free
- MINOR: quic: fix qcc subs initialization
- BUG/MINOR: h1-htx: Fix a typo when request parser is reset
- BUG/MEDIUM: mux-h1: Adjust conditions to ask more space in the channel buffer
- BUG/MEDIUM: stream-int: Notify stream that the mux wants more room to xfer data
- BUG/MEDIUM: stream: Stop waiting for more data if SI is blocked on RXBLK_ROOM
- MINOR: stream-int: Set CO_RFL transient/persistent flags apart in si_cs_rcv()
- MINOR: htx: Add an HTX flag to know when a message is fragmented
- MINOR: htx: Add a function to know if the free space wraps
- BUG/MEDIUM: stream-int: Defrag HTX message in si_cs_recv() if necessary
- MINOR: stream-int: Notify mux when the buffer is not stuck when calling rcv_buf
- BUG/MINOR: http-ana: increment internal_errors counter on response error
- MINOR: stats: Enable dark mode on stat web page
- CLEANUP: stats: Fix some alignment mistakes
- MINOR: httpclient: httpclient_data() returns the available data
- MINOR: httpclient: httpclient_ended() returns 1 if the client ended
- MINOR: httpclient/lua: httpclient:get() API in lua
- MINOR: httpclient/lua: implement the headers in the response object
- BUG/MINOR: httpclient/lua: return an error on argument check
- CLEANUP: slz: Mark `reset_refs` as static
src/hlua.c:7074:6: error: variable 'url_str' is used uninitialized whenever 'if' condition is false [-Werror,-Wsometimes-uninitialized]
if (lua_type(L, -1) == LUA_TSTRING)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/hlua.c:7079:36: note: uninitialized use occurs here
hlua_hc->hc->req.url = istdup(ist(url_str));
^~~~~~~
Return an error on the stack if the argument is not a string.
Provide a new field "headers" in the response of the HTTPClient, which
contains all headers of the response.
This field is a multi-dimensionnal table which could be represented this
way in lua:
headers = {
["content-type"] = { "text/html" },
["cache-control"] = { "no-cache" }
}
This commit provides an hlua_httpclient object which is a bridge between
the httpclient and the lua API.
The HTTPClient is callable in lua this way:
local httpclient = core.httpclient()
local response = httpclient:get("http://127.0.0.1:9000/?s=9999")
core.Debug("Status: ".. res.status .. ", Reason : " .. res.reason .. ", Len:" .. string.len(res.body) .. "\n")
The resulting response object will provide a "status" field which
contains the status code, a "reason" string which contains the reason
string, and a "body" field which contains the response body.
The implementation uses the httpclient callback to wake up the lua task
which yield each time it pushes some data. The httpclient works in the
same thread as the lua task.
According with the W3 CSS specification, media queries 5 allow
the browser to enable some CSS when dark mode is enabled. This
patch defines dark mode CSS for the stats page.
https://www.w3.org/TR/mediaqueries-5/#prefers-color-scheme
A bug was introduced in the commit cff0f739e5 ("MINOR: counters: Review
conditions to increment counters from analysers"). The internal_errors
counter for the target server was incremented twice. The counter for the
session listener needs to be incremented instead.
This must be backported everywhere the commit cff0f739e5 is.
The transient flag CO_RFL_BUF_NOT_STUCK should now be set when the mux's
rcv_buf() function is called, in si_cs_recv(), to be sure the mux is able to
perform some optimisation during data copy. This flag is set when we are
sure the channel buffer is not stuck. Concretely, it happens when there are
data scheduled to be sent.
It is not a fix and this flag is not used for now. But it makes sense to have
this info to be sure to be able to do some optimisations if necessary.
This patch is related to the issue #1362. It may be backported to 2.4 to
ease future backports.
The stream interface is now responsible for defragmenting the HTX message of
the input channel if necessary, before calling the mux's .rcv_buf()
function. The defrag is performed if the underlying buffer contains only
input data while the HTX message free space is not contiguous.
The defrag is important here to be sure the mux and the app layer have the
same criteria to decide if a buffer is full or not. Otherwise, the app layer
may wait for more data because the buffer is not full while the mux is
blocked because it needs more space to proceed.
This patch depends on following commits:
* MINOR: htx: Add an HTX flag to know when a message is fragmented
* MINOR: htx: Add a function to know if the free space wraps
This patch is related to the issue #1362. It may be backported as far as 2.0
after some observation period (not sure it is required or not).
HTX_FL_FRAGMENTED flag is now set on an HTX message when it is
fragmented. It happens when an HTX block is removed in the middle of the
message and flagged as unused. HTX_FL_FRAGMENTED flag is removed when all
data are removed from the message or when the message is defragmented.
Note that some optimisations are still possible because the flag can be
avoided in other situations. For instance when the last header of a bodyless
message is removed.
In si_cs_recv(), some CO_RFL flags are set when the mux's .rcv_buf()
function is called. Some are persitent inside si_cs_recv() scope, some
others must be computed at each call to rcv_buf(). This patch takes care of
distinguishing them.
Among others, CO_RFL_KEEP_RECV is a persistent flag while CO_RFL_BUF_WET is
transient.
If the stream-interface is waiting for more buffer room to store incoming
data, it is important at the stream level to stop to wait for more data to
continue. Thanks to the previous patch ("BUG/MEDIUM: stream-int: Notify
stream that the mux wants more room to xfer data"), the stream is woken up
when this happens. In this patch, we take care to interrupt the
corresponding tcp-content ruleset or to stop waiting for the HTTP message
payload.
To ease detection of the state, si_rx_blocked_room() helper function has
been added. It returns non-zero if the stream interface's Rx path is blocked
because of lack of room in the input buffer.
This patch is part of a series related to the issue #1362. It should be
backported as ar as 2.0, probably with some adaptations. So be careful
during backports.
When the mux failed to transfer data to the upper layer because of a lack of
room, it is important to wake the stream up to let it handle this
event. Otherwise, if the stream is waiting for more data, both the stream
and the mux reamin blocked waiting for each other.
When this happens, the mux set the CS_FL_WANT_ROOM flag on the
conn-stream. Thus, in si_cs_recv() we are able to detect this event. Today,
the stream-interface is blocked. But, it is not enough to wake the stream
up. To fix the bug, CF_READ_PARTIAL flag is extended to also handle cases
where a read exception occurred. This flag should idealy be renamed. But for
now, it is good enough. By setting this flag, we are sure the stream will be
woken up.
This patch is part of a series related to the issue #1362. It should be
backported as far as 2.0, probably with some adaptations. So be careful
during backports.
When a message is parsed and copied into the channel buffer, in
h1_process_demux(), more space is requested if some pending data remain
after the parsing while the channel buffer is not empty. To do so,
CS_FL_WANT_ROOM flag is set. It means the H1 parser needs more space in the
channel buffer to continue. In the stream-interface, when this flag is set,
the SI is considered as blocked on the RX path. It is only unblocked when
some data are sent.
However, it is not accurrate because the parsing may be stopped because
there is not enough data to continue. For instance in the middle of a chunk
size. In this case, some data may have been already copied but the parser is
blocked because it must receive more data to continue. If the calling SI is
blocked on RX at this stage when the stream is waiting for the payload
(because http-buffer-request is set for instance), the stream remains stuck
infinitely.
To fix the bug, we must request more space to the app layer only when it is
not possible to copied more data. Actually, this happens when data remain in
the input buffer while the H1 parser is in states MSG_DATA or MSG_TUNNEL, or
when we are unable to copy headers or trailers into a non-empty buffer.
The first condition is quite easy to handle. The second one requires an API
refactoring. h1_parse_msg_hdrs() and h1_parse_msg_tlrs() fnuctions have been
updated. Now it is possible to know when we need more space in the buffer to
copy headers or trailers (-2 is returned). In the H1 mux, a new H1S flag
(H1S_F_RX_CONGESTED) is used to track this state inside h1_process_demux().
This patch is part of a series related to the issue #1362. It should be
backported as far as 2.0, probably with some adaptations. So be careful
during backports.
In h1_postparse_req_hdrs(), if we need more space to copy headers, the request
parser is reset. However, because of a typo, it was reset as a response parser
instead of a request one. h1m_init_req() must be called.
This patch must be backported as far as 2.2.
We wake up the xprt as soon as STREAM frames have been pushed to
the TX mux buffer (->tx.buf).
We also make the mux subscribe() to the xprt layer if some data
remain in its ring buffer after having try to transfer them to the
xprt layer (TX mux buffer for the stream full).
Also do not consider a buffer in the ring if not allocated (see b_size(buf))
condition in the for(;;) loop.
Make a call to qc_process_mux() if possible when entering qc_send() to
fill the mux with data from streams in the send or flow control lists.
The FIN of a STREAM frame to be built must be set if there is no more
at all data in the ring buffer.
Do not do anything if there is nothing to transfer the ->tx.buf mux
buffer via b_force_xfer() (without zero copy)
When ACK have been received by the xprt, it must wake up the
mux if this latter has subscribed to SEND events. This is the
role of qcs_try_to_consume() to detect such a situation. This
is the function which consumes the buffer filled by the mux.
It is important to know if the packet number spaces used during the
handshakes have really been discarding. If not, this may have a
significant impact on the packet loss detection.
There were cases where the Initial packet number space was not discarded.
This leaded the packet loss detection to continue to take it into
considuration during the connection lifetime. Some Application level
packets could not be retransmitted.
During the packet loss detection we must treat the paquet number
in this order Initial -> Handshake -> O1RTT. This was not the case
due to the chosen order to implement the array of packet number space
which was there before the packet loss detection implementation.
