Generate an SNI expression which uses the Host header of the request.
This is mandatory for most of the SSL servers nowadays.
Must be backported in 2.5 with the previous patch which export
server_parse_sni_expr().
-c sets a corruption rate in % of the packets.
-o sets the start offset of the area to be corrupted.
-w sets the length of the area to be corrupted.
A single byte within that area will then be randomly XORed with a
random value.
In order to ease addition of new types of perturbations to udp-perturb,
let's first switch to getopt() and get rid of the positional arguments.
The random seed was already a conditional option of the rate which was
a conditional option as well. We add -r and -s for the rate and the
seed, and new options will follow.
The appctx owner is not a stream-interface anymore. It is now a
conn-stream. However, sink code was not updated accordingly. It is now
fixed.
It is 2.6-specific, no backport is needed.
The appctx owner is not a stream-interface anymore. It is now a conn-stream.
In the cli I/O handler for the command "debug dev fd", we still handle it as
a stream-interface. It is now fixed.
It is 2.6-specific, no backport is needed.
Since the CS/SI refactoring, the .release callback function may be called
twice. The first call when a shutdown for read or for write is performed.
The second one when the applet is detached from its conn-stream. The second
call must be guarded, just like the first one, to only be performed is the
stream-interface is not the in disconnected (SI_ST_DIS) or closed
(SI_ST_CLO) state.
To simplify the fix, we now always rely on si_applet_release() function.
It is 2.6-specific, no backport is needed.
The httpclient lua code is lacking the end callback, which means it
won't be able to wake up the lua code after a longjmp if the connection
was closed without any data.
Must be backported to 2.5.
This commit reverts this one:
"d5066dd9d BUG/MEDIUM: quic: qc_prep_app_pkts() retries on qc_build_pkt() failures"
After having filled the congestion control window, qc_build_pkt() always fails.
Then depending on the relative position of the writer and reader indexes for the
TX buffer, this could lead this function to try to reuse the buffer even if not full.
In such case, we do not always mark the end of the data in this TX buffer. This
is something the reader cannot understand: it reads a false datagram length,
then a wrong packet address from the TX buffer, leading to an invalid pointer
dereferencing.
STREAM frames which are not acknowledged in order are inserted in ->tx.acked_frms
tree ordered by the STREAM frame offset values. Then, they are consumed in order
by qcs_try_to_consume(). But, when we retransmit frames, we possibly have to
insert the same STREAM frame node (with the same offset) in this tree.
The problem is when they have different lengths. Unfortunately the restransmitted
frames are not inserted because of the tree nature (EB_ROOT_UNIQUE). If the STREAM
frame which has been successfully inserted has a smaller length than the
retransmitted ones, when it is consumed they are tailing bytes in the STREAM
(retransmitted ones) which indefinitively remains in the STREAM TX buffer
which will never properly be consumed, leading to a blocking state.
At this time this may happen because we sometimes build STREAM frames
with null lengths. But this is another issue.
The solution is to use an EB_ROOT tree to support the insertion of STREAM frames
with the same offset but with different lengths. As qcs_try_to_consume() support
the STREAM frames retransmission this modification should not have any impact.
In the same way than for be2hex.vtc, a "Connection: close" header is added
to all responses to avoid any connection reuse. This should avoid any "HTTP
header incomplete" errors.
The httpclient mistakenly use the htx_get_first{_blk}() functions instead
of the htx_get_head{_blk}() functions. Which could stop the httpclient
because it will be without the start line, waiting for data that won't never
come.
Must be backported in 2.5.
Remove the UNUSED blocks when iterating on headers, we should not stop
when encountering one. We should only stop iterating once we found the
EOH block. It doesn't provoke a problem, since we don't manipulates
the headers before treating them, but it could evolve in the future.
Must be backported to 2.5.
Consume partly the blocks in the httpclient I/O handler when there is
not enough room in the destination buffer for the whole block or when
the block is not contained entirely in the channel's output.
It prevents the I/O handler to be stuck in cases when we need to modify
the buffer with a filter for exemple.
Must be backported in 2.5.
In httpclient_applet_io_handler(), on the response path, we don't check
if the data are in the output part of the channel, and could consume
them before they were analyzed.
To fix this issue, this patch checks for the stline and the headers if
the msg_state is >= HTTP_MSG_DATA which means the stline and headers
were analyzed. For the data part, it checks if each htx blocks is in the
output before copying it.
Must be backported in 2.5.
Released version 2.6-dev3 with the following main changes :
- DEBUG: rename WARN_ON_ONCE() to CHECK_IF()
- DEBUG: improve BUG_ON output message accuracy
- DEBUG: implement 4 levels of choices between warn and crash.
- DEBUG: add two new macros to enable debugging in hot paths
- DEBUG: buf: replace some sensitive BUG_ON() with BUG_ON_HOT()
- DEBUG: buf: add BUG_ON_HOT() to most buffer management functions
- MINOR: channel: don't use co_set_data() to decrement output
- DEBUG: channel: add consistency checks using BUG_ON_HOT() in some key functions
- MINOR: conn-stream: Improve API to have safe/unsafe accessors
- MEDIUM: tree-wide: Use unsafe conn-stream API when it is relevant
- CLEANUP: stream-int: Make si_cs_send() function static
- REORG: stream-int: Uninline si_sync_recv() and make si_cs_recv() private
- BUG/MEDIUM: mux-fcgi: Don't rely on SI src/dst addresses for FCGI health-checks
- BUG/MEDIUM: htx: Fix a possible null derefs in htx_xfer_blks()
- REGTESTS: fix the race conditions in normalize_uri.vtc
- DEBUG: stream-int: Fix BUG_ON used to test appctx in si_applet_ops callbacks
- BUILD: debug: fix build warning on older compilers around DEBUG_STRICT_ACTION
- CLEANUP: connection: Indicate unreachability to the compiler in conn_recv_proxy
- MINOR: connection: Transform safety check in PROXYv2 parsing into BUG_ON()
- DOC: install: it's DEBUG_CFLAGS, not DEBUG, which is set to -g
- DOC: install: describe the DEP variable
- DOC: install: describe how to choose options used in the DEBUG variable
- MINOR: queue: Replace if() + abort() with BUG_ON()
- CLEANUP: adjust indentation in bidir STREAM handling function
- MINOR: quic: simplify copy of STREAM frames to RX buffer
- MINOR: quic: handle partially received buffered stream frame
- MINOR: mux-quic: define flag for last received frame
- BUG/MINOR: quic: support FIN on Rx-buffered STREAM frames
- MEDIUM: quic: rearchitecture Rx path for bidirectional STREAM frames
- REGTESTS: fix the race conditions in secure_memcmp.vtc
- CLEANUP: stream: Remove useless tests on conn-stream in stream_dump()
- BUILD: ssl: another build warning on LIBRESSL_VERSION_NUMBER
- MINOR: quic: Ensure PTO timer is not set in the past
- MINOR: quic: Post handshake I/O callback switching
- MINOR: quic: Drop the packets of discarded packet number spaces
- CLEANUP: quic: Useless tests in qc_try_rm_hp()
- CLEANUP: quic: Indentation fix in qc_prep_pkts()
- MINOR: quic: Assemble QUIC TLS flags at the same level
- BUILD: conn_stream: avoid null-deref warnings on gcc 6
- BUILD: connection: do not declare register_mux_proto() inline
- BUILD: http_rules: do not declare http_*_keywords_registre() inline
- BUILD: trace: do not declare trace_registre_source() inline
- BUILD: tcpcheck: do not declare tcp_check_keywords_register() inline
- DEBUG: reduce the footprint of BUG_ON() calls
- BUG/MEDIUM: httpclient/lua: infinite appctx loop with POST
- BUG/MINOR: pool: always align pool_heads to 64 bytes
- DEV: udp: add a tiny UDP proxy for testing
- DEV: udp: implement pseudo-random reordering/loss
- DEV: udp: add an optional argument to set the prng seed
- BUG/MINOR: quic: fix segfault on CC if mux uninitialized
- BUG/MEDIUM: pools: fix ha_free() on area in the process of being freed
- CLEANUP: tree-wide: remove a few rare non-ASCII chars
- CI: coverity: simplify debugging options
- CLEANUP: quic: complete ABORT_NOW with a TODO comment
- MINOR: quic: qc_prep_app_pkts() implementation
- MINOR: quic: Send short packet from a frame list
- MINOR: quic: Make qc_build_frms() build ack-eliciting frames from a list
- MINOR: quic: Export qc_send_app_pkts()
- MINOR: mux-quic: refactor transport parameters init
- MINOR: mux-quic: complete functions to detect stream type
- MINOR: mux-quic: define new unions for flow-control fields
- MEDIUM: mux-quic: use direct send transport API for STREAMs
- MINOR: mux-quic: retry send opportunistically for remaining frames
- MEDIUM: mux-quic: implement MAX_STREAMS emission for bidir streams
- BUILD: fix kFreeBSD build.
- MINOR: quic: Retry on qc_build_pkt() failures
- BUG/MINOR: quic: Missing recovery start timer reset
- CLEANUP: quic: Remove QUIC path manipulations out of the congestion controller
- MINOR: quic: Add a "slow start" callback to congestion controller
- MINOR: quic: Persistent congestion detection outside of controllers
- CLEANUP: quic: Remove useless definitions from quic_cc_event struct
- BUG/MINOR: quic: Confusion betwen "in_flight" and "prep_in_flight" in quic_path_prep_data()
- MINOR: quic: More precise window update calculation
- CLEANUP: quic: Remove window redundant variable from NewReno algorithm state struct
- MINOR: quic: Add quic_max_int_by_size() function
- BUG/MAJOR: quic: Wrong quic_max_available_room() returned value
- MINOR: pools: add a new global option "no-memory-trimming"
- BUG/MINOR: add missing modes in proxy_mode_str()
- BUG/MINOR: cli: shows correct mode in "show sess"
- BUG/MEDIUM: quic: do not drop packet on duplicate stream/decoding error
- MINOR: stats: Add dark mode support for socket rows
- BUILD: fix recent build breakage of freebsd caused by kFreeBSD build fix
- BUG/MINOR: httpclient: Set conn-stream/channel EOI flags at the end of request
- BUG/MINOR: hlua: Set conn-stream/channel EOI flags at the end of request
- BUG/MINOR: stats: Set conn-stream/channel EOI flags at the end of request
- BUG/MINOR: cache: Set conn-stream/channel EOI flags at the end of request
- BUG/MINOR: promex: Set conn-stream/channel EOI flags at the end of request
- BUG/MEDIUM: stream: Use the front analyzers for new listener-less streams
- DEBUG: cache: Update underlying buffer when loading HTX message in cache applet
- BUG/MEDIUM: mcli: Properly handle errors and timeouts during reponse processing
- DEBUG: stream: Add the missing descriptions for stream trace events
- DEBUG: stream: Fix stream trace message to print response buffer state
- MINOR: proxy: Store monitor_uri as a `struct ist`
- MINOR: proxy: Store fwdfor_hdr_name as a `struct ist`
- MINOR: proxy: Store orgto_hdr_name as a `struct ist`
- MEDIUM: proxy: Store server_id_hdr_name as a `struct ist`
- CLEANUP: fcgi: Replace memcpy() on ist by istcat()
- CLEANUP: fcgi: Use `istadv()` in `fcgi_strm_send_params`
- BUG/MAJOR: mux-pt: Always destroy the backend connection on detach
- DOC: sample fetch methods: move distcc_* to the right locations
- MINOR: rules: record the last http/tcp rule that gave a final verdict
- MINOR: stream: add "last_rule_file" and "last_rule_line" samples
- BUG/MINOR: session: fix theoretical risk of memleak in session_accept_fd()
- MINOR: quic: Add max_idle_timeout advertisement handling
- MEDIUM: quic: Remove the QUIC connection reference counter
- BUG/MINOR: quic: ACK_REQUIRED and ACK_RECEIVED flag collision
- BUG/MINOR: quic: Missing check when setting the anti-amplification limit as reached
- MINOR: quic: Add a function to compute the current PTO
- MEDIUM: quic: Implement the idle timeout feature
- BUG/MEDIUM: quic: qc_prep_app_pkts() retries on qc_build_pkt() failures
- CLEANUP: quic: Comments fix for qc_prep_(app)pkts() functions
- MINOR: mux-quic: prevent push frame for unidir streams
- MINOR: mux-quic: improve opportunistic retry sending for STREAM frames
- MINOR: quic: implement sending confirmation
- MEDIUM: mux-quic: improve bidir STREAM frames sending
- MEDIUM: check: do not auto configure SSL/PROXY for dynamic servers
- REGTESTS: server: test SSL/PROXY with checks for dynamic servers
- MEDIUM: server: remove experimental-mode for dynamic servers
- BUG/MINOR: buffer: fix debugging condition in b_peek_varint()
The BUG_ON_HOT() test condition added to b_peek_varint() by commit
8873b85bd ("DEBUG: buf: add BUG_ON_HOT() to most buffer management
functions") was wrong as <data> in this function is not b->data,
so that was triggering during live dumps of H2 traces on the CLI
when built with -DDEBUG_STRICT=2. No backport is needed.
Dynamic servers feature is now judged to be stable enough. Remove the
experimental-mode requirement for "add/del server" commands. This should
facilitate dynamic servers adoption.
For server checks, SSL and PROXY is automatically inherited from the
server settings if no specific check port is specified. Change this
behavior for dynamic servers : explicit "check-ssl"/"check-send-proxy"
are required for them.
Without this change, it is impossible to add a dynamic server with
SSL/PROXY settings and checks without, if the check port is not
explicit. This is because "no-check-ssl"/"no-check-send-proxy" keywords
are not available for dynamic servers.
This change respects the principle that dynamic servers on the CLI
should not reuse the same shortcuts used during the config file parsing.
Mostly because we expect this feature to be manipulated by automated
tools, contrary to the config file which should aim to be the shortest
possible for human readability.
Update the documentation of the "check" keyword to reflect this change.
The current implementation of STREAM frames emission has some
limitation. Most notably when we cannot sent all frames in a single
qc_send run.
In this case, frames are left in front of the MUX list. It will be
re-send individually before other frames, possibly another frame from
the same STREAM with new data. An opportunity to merge the frames is
lost here.
This method is now improved. If a frame cannot be send entirely, it is
discarded. On the next qc_send run, we retry to send to this position. A
new field qcs.sent_offset is used to remember this. A new frame list is
used for each qc_send.
The impact of this change is not precisely known. The most notable point
is that it is a more logical method of emission. It might also improve
performance as we do not keep old STREAM frames which might delay other
streams.
Implement a new MUX function qcc_notify_send. This function must be
called by the transport layer to confirm the sending of STREAM data to
the MUX.
For the moment, the function has no real purpose. However, it will be
useful to solve limitations on push frame and implement the flow
control.
For the moment, the transport layer function qc_send_app_pkts lacks
features. Most notably, it only send up to a single Tx buffer and won't
retry even if there is frames left and its Tx buffer is now empty.
To overcome this limitation, the MUX implements an opportunistic retry
sending mechanism. qc_send_app_pkts is repeatedly called until the
transport layer is blocked on an external condition (such as congestion
control or a sendto syscall error).
The blocking was detected by inspecting the frame list before and after
qc_send_app_pkts. If no frame has been poped by the function, we
considered the transport layer to be blocked and we stop to send. The
MUX is subscribed on the lower layer to send the frames left.
However, in case of STREAM frames, qc_send_app_pkts might use only a
portion of the data and update the frame offset. So, for STREAM frames,
a new mechanism is implemented : if the offset field of the first frame
has not been incremented, it means the transport layer is blocked.
This should improve transfers execution. Before this change, there is a
possibility of interrupted transfer if the mux has not sent everything
possible and is waiting on a transport signaling which will never
happen.
In the future, qc_send_app_pkts should be extended to retry sending by
itself. All this code burden will be removed from the MUX.
For the moment, unidirectional streams handling is not identical to
bidirectional ones in MUX/H3 layer, both in Rx and Tx path. As a safety,
skip over uni streams in qc_send.
In fact, this change has no impact because qcs.tx.buf is emptied before
we start using qcs_push_frame, which prevents the call to
qcs_push_frame. However, this condition will soon change to improve
bidir streams emission, so an explicit check on stream type must be
done.
It is planified to unify uni and bidir streams handling in a future
stage. When implemented, the check will be removed.
The aim of the idle timeout is to silently closed the connection after a period
of inactivity depending on the "max_idle_timeout" transport parameters advertised
by the endpoints. We add a new task to implement this timer. Its expiry is
updated each time we received an ack-eliciting packet, and each time we send
an ack-eliciting packet if no other such packet was sent since we received
the last ack-eliciting packet. Such conditions may be implemented thanks
to QUIC_FL_CONN_IDLE_TIMER_RESTARTED_AFTER_READ new flag.
This packet number space flags were defined with the same value because
defined at different places in the file. Assemble them at the same location
with different values.
This bug could unvalidate the peer address after it was validated
during the handshake leading to the anti-amplication limit to be
enabled again after having been disabled. The situation could not
be unblocked (deadlock).
There is no need to use such a reference counter anymore since the QUIC
connections are always handled by the same thread.
quic_conn_drop() is removed. Its code is merged into quic_conn_release().
When we store the remote transport parameters, we compute the maximum idle
timeout for the connection which is the minimum of the two advertised
max_idle_timeout transport parameter values if both have non-null values, or the
maximum if one of the value is set and non-null.
Andrew Suffield reported in issue #1596 that we've had a bug in
session_accept_fd() since 2.4 with commit 1b3c931bf ("MEDIUM:
connections: Introduce a new XPRT method, start().") where an error
label is wrong and may cause the leak of the freshly allocated session
in case conn_xprt_start() returns < 0.
The code was checked there and the only two transport layers available
at this point are raw_sock and ssl_sock. The former doesn't provide a
->start() method hence conn_xprt_start() will always return zero. The
second does provide such a function, but it may only return <0 if the
underlying transport (raw_sock) has such a method and fails, which is
thus not the case.
So fortunately it is not possible to trigger this leak.
The patch above also touched the accept code in quic_sock() which was
mostly a plain copy of the session code, but there the move didn't
have this impact, and since then it was simplified and the next change
moved it to its final destination with the proper error label.
This should be backported as far as 2.4 as a long-term safety measure
(e.g. if in the future we have a reason for making conn_xprt_start()
to start failing), but will not have any positive nor negative effect
in the short term.
These two sample fetch methods report respectively the file name and the
line number where was located the last rule that was final. This is aimed
at being used on log-format lines to help admins figure what rule in the
configuration gave a final verdict, and help understand the condition
that led to the action.
For example, it's now possible to log the last matched rule by adding
this to the log-format:
... lr=%[last_rule_file]:%[last_rule_line]
A regtest is provided to test various combinations of final rules, some
even on top of each other from different rulesets.
When a tcp-{request,response} content or http-request/http-response
rule delivers a final verdict (deny, accept, redirect etc), the last
evaluated one will now be recorded in the stream. The purpose is to
permit to log the last one that performed a final action. For now
the log is not produced.
The distcc* sample fetch methods were surprisingly located within the
"internal state" section, while they in fact depend on L6 contents.
This can be backported to all versions where they appear.
In TCP, when a conn-stream is detached from a backend connection, the
connection must be always closed. It was only performed if an error or a
shutdown occurred or if there was no connection owner. But it is a problem,
because, since the 2.3, backend connections are always owned by a
session. This way it is possible to have idle connections attached to a
session instead of a server. But there is no idle connections in TCP. In
addition, when a session owns a connection it is responsible to close it
when it is released. But it only works for idle connections. And it only
works if the session is released.
Thus there is the place for bugs here. And indeed, a connection leak may
occur if a connection retry is performed because of a timeout. In this case,
the underlying connection is still alive and is waiting to be fully
established. Thus, when the conn-stream is detached from the connection, the
connection is not closed. Because the PT multiplexer is quite simple, there
is no timeout at this stage. We depend on the kenerl to be notified and
finally close the connection. With an unreachable server, orphan backend
connections may be accumulated for a while. It may be perceived as a leak.
Because there is no reason to keep such backend connections, we just close
it now. Frontend connections are still closed by the session or when an
error or a shutdown occurs.
This patch should fix the issue #1522. It must be backported as far as
2.0. Note that the 2.2 and 2.0 are not affected by this bug because there is
no owner for backend TCP connections. But it is probably a good idea to
backport the patch on these versions to avoid any future bugs.
Found manually, while creating the previous commits to turn `struct proxy`
members into ists.
There is an existing Coccinelle rule to replace this pattern by `istadv()` in
`ist.cocci`:
@@
struct ist i;
expression e;
@@
- i.ptr += e;
- i.len -= e;
+ i = istadv(i, e);
But apparently it is not smart enough to match ists that are stored in another
struct. It would be useful to make the existing rule more generic, so that it
might catch similar cases in the future.
The server_id_hdr_name is already processed as an ist in various locations lets
also just store it as such.
see 0643b0e7e ("MINOR: proxy: Make `header_unique_id` a `struct ist`") for a
very similar past commit.
The orgto_hdr_name is already processed as an ist in `http_process_request`,
lets also just store it as such.
see 0643b0e7e ("MINOR: proxy: Make `header_unique_id` a `struct ist`") for a
very similar past commit.
The fwdfor_hdr_name is already processed as an ist in `http_process_request`,
lets also just store it as such.
see 0643b0e7e ("MINOR: proxy: Make `header_unique_id` a `struct ist`") for a
very similar past commit.
The monitor_uri is already processed as an ist in `http_wait_for_request`, lets
also just store it as such.
see 0643b0e7e ("MINOR: proxy: Make `header_unique_id` a `struct ist`") for a
very similar past commit.
Channels buffer state is displayed in the strem trace messages. However,
because of a typo, the request buffer was used instead of the response one.
This patch should be backported as far as 2.2.
The response analyzer of the master CLI only handles read errors. So if
there is a write error, the session remains stuck because some outgoing data
are blocked in the channel and the response analyzer waits everything to be
sent. Because the maxconn is set to 10 for the master CLI, it may be
unresponsive if this happens to many times.
Now read and write errors, timeouts and client aborts are handled.
This patch should solve the issue #1512. It must be backported as far as
2.0.
In the I/O handler of the cache applet, we must update the underlying buffer
when the HTX message is loaded, using htx_from_buf() function instead of
htxbuf(). It is important because the applet will update the message by
adding new HTX blocks. This way, the state of the underlying buffer remains
consistant with the state of the HTX message.
It is especially important if HAProxy is compiled with "DEBUG_STRICT=2"
mode. Without this patch, channel_add_input() call crashed if the channel
was empty at the begining of the I/O handler.
Note that it is more a build/debug issue than a bug. But this patch may
prevent future bugs. For now it is safe because htx_to_buf() function is
systematically called, updating accordingly the underlying buffer.
This patch may be backported as far as 2.0.
For now, for a stream, request analyzers are set at 2 stages. The first one
is when the stream is created. The session's listener analyzers, if any, are
set on the request channel. In addition, some HTTP analyzers are set for HTX
streams (AN_REQ_WAIT_HTTP and AN_REQ_HTTP_PROCESS_FE). The second one is
when the backend is set on the stream. At the stage, request analyzers are
updated using the backend settings.
It is an issue for client applets because there is no listener attached to
the stream. In addtion, it may have no specific/dedicated backend. Thus,
several request analyzers are missing. Among others, the HTTP analyzers for
HTTP applets. The HTTP client is the only one affected for now.
To fix the bug, when a stream is created without a listener, we use the
frontend to set the request analyzers. Note that there is no issue with the
response channel because its analyzers are set when the server connection is
established.
This patch may be backported to all stable versions. Because only the HTTP
client is affected, it must at least be backported to 2.5. It is related to
the issue #1593.
This bug is the same than for the HTTP client. See "BUG/MINOR: httpclient:
Set conn-stream/channel EOI flags at the end of request" for details.
This patch must be backported as far as 2.0. But only CF_EOI must be set
because applets are not attached to a conn-stream on older versions.
