The CFLAGS appended by USE_ENGINE can now be overridden using
ENGINE_CFLAGS. These would have been better located inside the
OPENSSL stuff but it's a bit too late now.
There are multiple options for 51DEGREES, v3/v4, threading or not,
pattern/trie for v3, vhash for v4, use of libatomic, etc. While the
current rules deal with all of that correctly, it's too difficult to
focus on one version because the two are interleaved for every single
option. Let's just split them into two independent blocks. This removes
some if/endif, and makes the lecture much more straightforward.
Some conditional blocks have become out of control over time and are
totally unreadble. It took 15 minutes to figure what "endif" matched
what "if" in the PCRE one for example, and DA and 51D use multiple
levels as well that are not easy to sort out.
Let's reindent the whole thing. Most places that were already indented
used 2 spaces per level, so here we're keeping that principle. It was
just not done on the two last ones that are used to define some rules
because we don't want spaces before rule names. A few had the opening
condition indicated on the endif line.
It would be desirable that over time this more maintainable layout is
preserved.
The PCRE/PCRE2 CFLAGS forcefully add -DUSE_PCRE or -DUSE_PCRE2 because
we want that USE_STATIC_PCRE or USE_PCRE_JIT implicitly enables them.
However, doing it this way is incorrect because the option is not visible
in BUILD_FEATURES, and for example, some regtests depending on such
features (such as map_redirect.vtc) would be skipped if only the static
or jit versions are enabled.
The correct way to do this is to always set USE_PCRE feature for such
variants instead of adding the define.
This could almost be backported but would require to backport other
makefile patches and likely only has effects on the reg-tests at the
moment, so it's probably not worth the hassle.
Lua and 51d make use of -lm, which would be better served by having its
own option than being passed in the LDFLAGS. It also simplifies linking
against a static version of libm. The option uses its own LDFLAGS which
are automatically collected into OPTIONS_LDFLAGS.
Two places, 51Dv4 and AIX7.2, used to forcefully add -latomic to the
ldflags (and via different variables). This must not be done because
it depends on compiler, arch, etc. USE_LIBATOMIC=implicit is much
better: it allows the user to forcefully disable it if undesired.
The LIBATOMIC_LDFLAGS are set to -latomic and automatically added
to OPTIONS_LDFLAGS.
It will make this dependency appear in haproxy -vv but that's not
and issue and it may even sometimes help when troubleshooting.
The HLUA_PREPEND_PATH and HLUA_PREPEND_CPATH settings were only applied
when LUA_LIB_NAME was empty, otherwise they were silently ignored. Let's
take them out of that conditional block as this makes no sense to enforce
such a restriction (the main reason in fact is that this whole block is
unreadable).
Also take this opportunity to unfold the last two imbricated tests of
LUA_LIB_NAME and put comments around certain blocks to know what "endif"
matches what "if".
While LUA_INC is sometimes set in the makefile (only when LUA_LIB_NAME
is not set), LUA_LIB is never pre-initialized and faces the risk of
being accidently inherited from the environment. Let's make sure both
are properly reset first when not explicitly set. For this we always
set LUA_INC based on the autodetection if it's not set, and always
pre-initialize LUA_LIB to empty. This also helps make that block
slightly less difficult to understand.
There used to be special cases where USE_DL was only for the SSL library,
then for Lua, then was used globally, but each of them kept their own copy
of -ldl. When building on a system supporting libdl, with SSL and Lua
enabled, no less than 3 -ldl are found on the linker's command line.
What matters is only that it's close to the end, so let's remove the old
specific ones and move the global one to the end. The option now uses its
own DL_LDFLAGS that is automatically collected into OPTIONS_LDFLAGS.
I got a build error when adding USE_OPENSSL_WOLFSSL to my make command
line because SSL_INC was still set and caused some conflicting headers
to be included first. There's already an exclusion test for the wolfssl
variant used for SSL_LIB, make it also cover SSL_INC to avoid this.
This may be backported to 2.7 to ease testing of wolfssl.
The default include paths for wolfssl didn't match the explicit pattern
one. This was causing some confusion about what to look for, complexifying
the rules and making /usr/local/include to be automatically included if a
path was not set.
Let's just proceed as we usually do, i.e. pass -I only when a path is
specified, so that it works similarly to openssl. Let's also simplify
the LDFLAG rule at the same time.
This may be backported to 2.7 to ease testing of wolfssl.
It happens that a few "if USE_foo" were placed too low in the makefile,
and would mostly work by luck thanks to not using variables that were
already referenced before. The opentracing include is even trickier
because it extends OPTIONS_CFLAGS that was last read a few lines before
being included, but it only works because COPTS is defined as a macro and
not a variable, so it will be evaluated later. At least now it doesn't
touch OPTIONS_* anymore and since it's cleanly arranged, it will work by
default via the flags collector.
Let's just move these late USE_* handlers upper and place a visible
delimiter after them reminding not to add any after.
Now OPTIONS_CFLAGS and OPTIONS_LDFLAGS don't need to be set anymore
for options USE_xxx that set xxx_CFLAGS or xxx_LDFLAGS. These ones
will be automatically connected.
The only entry for now that was ready for this was PCRE2, so it was
adjusted so as not to append to OPTIONS_LDFLAGS anymore. More will
come later.
The new function collect_opts_flags now scans all USE_* options defined
in use_opts and appends the corresponding *_CFLAGS and *_LDFLAGS to
OPTIONS_{C,LD}FLAGS respectively. This will be useful to get rid of all
the individual concatenations to these variables.
A lot of _SRC, _INC, _LIB etc variables are set and expected to be
initialized to an empty string by default. However, an in-depth
review of all of them showed that WOLFSSL_{INC,LIB}, SSL_{INC,LIB},
LUA_{INC,LIB}, and maybe others were not always initialized and could
sometimes leak from the environment and as such cause strange build
issues when running from cascaded scripts that had exported them.
The approach taken here consists in iterating over all USE_* options
and unsetting any _SRC, _INC, _LIB, _CFLAGS and _LDFLAGS that follows
the same name. For the few variable names options that don't exactly
match the build option (SSL & WOLFSSL), these ones are specifically
added to the list. The few that were explicitly cleared in their own
sections were just removed since not needed anymore. Note that an
"undefine" command appeared in GNU make 3.82 but since we support
older ones we can only initialize the variables to an empty string
here. It's not a problem in practice.
We're now certain that these variables are empty wherever they are
used, and that it is possible to just append to them, or use them
as-is.
Some macros and functions are barely understandable and are only used
to iterate over known options from the use_opts list. Better assign
them a name and move them into a dedicated file to clean the makefile
a little bit. Now at least "use_opts" only appears once, where it is
defined. This also allowed to completely remove the BUILD_FEATURES
macro that caused some confusion until previous commit.
The BUILD_FEATURES string was created too early to inherit implicit
additions. This could make the features list report that some features
were disabled while they had later been enabled. Better make it a macro
that is interpreted where needed based on the current state of each
option.
Remove ABORT_NOW() on remote unidirectional stream closure. This is
required to ensure our implementation is evolutive enough to not fail on
unknown stream type.
Note that for the moment MAX_STREAMS_UNI flow-control frame is never
emitted. This should be unnecessary for HTTP/3 which have a limited
usage of unidirectional streams but may be required if other application
protocols are supported in the future.
ABORT_NOW() was triggered by s2n-quic which opens an unknown
unidirectional stream with greasing. This was detected by QUIC interop
runner for http3 testcase.
This must be backported up to 2.6.
Use a stream error when possible instead of always closing the whole
connection. This requires a new field <err> in h3s structure.
Change slightly the decoding loop to facilitate error propagation. It
will be interrupted as soon as <h3s.err> or <h3c.err> is non null. In
the later case, a CONNECTION_CLOSE is requested through
qcc_emit_cc_app().
For stream error, H3 layer uses qcc_abort_stream_read() coupled with
qcc_reset_stream(). This is in conformance with RFC 9114 which
recommends to use STOP_SENDING + RESET_STREAM emission on stream error.
This commit is part of implementing H3 errors at the stream level.
This should be backported up to 2.7.
Implement STOP_SENDING. This is divided in two main functions :
* qcc_abort_stream_read() which can be used by application protocol to
request for a STOP_SENDING. This set the flag QC_SF_READ_ABORTED.
* qcs_send_reset() is a static function called after the preceding one.
It will send a STOP_SENDING via qcc_send().
QC_SF_READ_ABORTED flag is now properly used : if activated on a stream
during qcc_recv(), <qcc.app_ops.decode_qcs> callback is skipped. Also,
abort reading on unknown unidirection remote stream is now fully
supported with the emission of a STOP_SENDING as specified by RFC 9000.
This commit is part of implementing H3 errors at the stream level. This
will allows the H3 layer to request the peer to close its endpoint for
an error on a stream.
This should be backported up to 2.7.
Implement RESET_STREAM reception by mux-quic. On reception, qcs instance
will be mark as remotely closed and its Rx buffer released. The stream
layer will be flagged on error if still attached.
This commit is part of implementing H3 errors at the stream level.
Indeed, on H3 stream errors, STOP_SENDING + RESET_STREAM should be
emitted. The STOP_SENDING will in turn generate a RESET_STREAM by the
remote peer which will be handled thanks to this patch.
This should be backported up to 2.7.
It is unnecessary to increase stream credit once its size is known.
Indeed, a peer cannot sent a greater offset than the value advertized.
Else, connection will be closed on STREAM reception with
FINAL_SIZE_ERROR.
This commit is a small optimization and may prevent the emission of
unneeded MAX_STREAM_DATA frames on some occasions.
It should be backported up to 2.7.
Implement mux_ops shutw operation for QUIC mux. A RESET_STREAM is
emitted unless the stream is already closed due to all data or
RESET_STREAM already transmitted.
This operation is notably useful when upper stream layer wants to close
the connection early due to an error.
This was tested by using a HTTP server which listens with PROXY protocol
support. The corresponding server line on haproxy configuration
deliberately not specify send-proxy. This causes the server to close
abruptly the connection. Without this patch, nothing was done on the QUIC
stream which was kept open until the whole connection is closed. Now, a
proper RESET_STREAM is emitted to report the error.
This should be backported up to 2.7.
The httpclient emits logs in the httplog format, however it still
display the frontend, the backend and the server.
In the case of the httpclient we only need to know that we are using the
httpclient, so the backend and server information are irelevant.
In the case of extra code the name of the proxy can be long and will be
displayed twice which is not useful.
This is the same log-format as the httplog but the %b/%s is now -/- so
the format is still compatible with an httplog parser.
Before:
<134>Dec 22 15:19:27 haproxy[1013520]: -:- [22/Dec/2022:15:19:27.482] <HTTPCLIENT> <HTTPCLIENT>/<HTTPCLIENT> 2/0/4/6/10 200 848 - - ---- 0/0/0/0/0 0/0 {92.123.236.161} "GET http://r3.o.lencr.org/1234 HTTP/1.1"
After:
<134>Dec 22 15:19:27 haproxy[1013520]: -:- [22/Dec/2022:15:19:27.482] <HTTPCLIENT> -/- 2/0/4/6/10 200 848 - - ---- 0/0/0/0/0 0/0 {92.123.236.161} "GET http://r3.o.lencr.org/1234 HTTP/1.1"
Don't try to create a request with a body in httpclient_req_gen() if the
payload ist has a ptr but no len.
Sometimes people have their httpclient stuck because they use an ist
with a data ptr but no len. Check the len so this mistake doesn't block
the client.
The same change was already performed for the cli. The stats applet and the
prometheus exporter are also concerned. Both use the stats API and rely on
pool functions to get total pool usage in bytes. pool_total_allocated() and
pool_total_used() must return 64 bits unsigned integer to avoid any wrapping
around 4G.
This may be backported to all versions.
As state in RFC9113#8.1, HEADERS frame with the ES flag set that carries an
informational status code is malformed. However, there is no test on this
condition.
On 2.4 and higher, it is hard to predict consequences of this bug because
end of the message is only reported with a flag. But on 2.2 and lower, it
leads to a crash because there is an unexpected extra EOM block at the end
of an interim response.
Now, when a ES flag is detected on a HEADERS frame for an interim message, a
stream error is sent (RST_STREAM/PROTOCOL_ERROR).
This patch should solve the issue #1972. It should be backported as far as
2.0.
Tests a subpart of the ocsp auto update feature. It will mainly focus on
the 'auto' mode since the 'on' one relies strongly on timers way too
long to be used in a regtest context.
This patch effectively enables the ocsp auto update mechanism. If a
least one ocsp-update option is enabled in a crt-list, then the ocsp
auto update task is created. It will look into the dedicated ocsp update
tree for the next update to be updated, use the http_client to send the
ocsp request to the proper responder, validate the received ocsp
response and update the ocsp response tree before finally reinserting
the entry in the ocsp update tree (with a next update time set to
now+1H).
The main task will then sleep until another entry needs to be updated.
The task gets scheduled after config check in order to avoid trying to
update ocsp responses while configuration is still being parsed (and
certificates and actual ocsp responses are loaded).
This patch contains the main function of the ocsp auto update mechanism
as well as an init and destroy function of the task used for this.
The task is not created in this patch but in a later one.
The function has two distinct parts and the branching to one or the
other is completely based on the fact that the cur_ocsp pointer of the
ssl_ocsp_task_ctx member is set.
If the pointer is not set, we need to look at the first item of the
update tree and see if it needs to be updated. If it does not we simply
wait until the time is right and let the task asleep. If it does need to
be updated, we simply build and send the corresponding ocsp request
thanks to the http_client. The task is then sent to sleep with an expire
time set to infinity. The http_client will wake it back up once the
response is received (or a timeout occurs). Just note that during this
whole process the cetificate_ocsp object corresponding to the entry
being updated is taken out of the update tree and only stored in the
ssl_ocsp_task_ctx context.
Once the task is waken up by the http_client, it branches on the
response processing part of the function which basically checks that the
response is valid and inserts it into the ocsp_response tree. The task
then goes back to sleep until another entry needs to be updated.
When 'ocsp-update' is enabled for a given certificate, we need to insert
the certificate_ocsp member of this certificate in the OCSP update tree
as well as the already existing OCSP response tree. For such an entry to
be created, the certificate needs to contain an "OCSP URI" field, and we
also need to know the certificate's issuer, that is used to build the
OCSP_CERTID. When no OCSP response is known for a given certificate, an
empty certificate_ocsp object gets created so that it can be inserted in
the ocsp update tree.
The entry is inserted on the first spot of the update tree since its
expire time is 0. Then whenever the update task is started, it will try
to get responses for those certificates first.
In order for the update process to work, we also need to store some
information relative to the main certificate into the certificate_ocsp
structure. This avoids having to keep a reference to a ckch in an ocsp
tree entry.
This patch adds a reference to the certificate chain as well as the ocsp
issuer that might have been filled during init into the certificate_ocsp
object. It also gets the ocsp uri at this time since it is contained in
the server's certificate. We only take the first uri that might be
contained in the certificate though.
Those fields are only filled when ocsp auto update is enabled for the
concerned certificate.
The 'ocsp-update' option is parsed at the same time as all the other
bind line options but it does not actually have anything to do with the
bind line since it concerns the frontend certificate instead. For that
reason, we should have a mean to identify inconsistencies in the
configuration and raise an error when a given certificate has two
different ocsp-update modes specified in one or more crt-lists.
The simplest way to do it is to store the ocsp update mode directly in
the ckch and not only in the ssl_bind_conf.
This option will define how the ocsp update mechanism behaves. The
option can either be set to 'on' or 'off' and can only be specified in a
crt-list entry so that we ensure that it concerns a single certificate.
The 'off' mode is the default one and corresponds to the old behavior
(no automatic update).
When the option is set to 'on', we will try to get an ocsp response
whenever an ocsp uri can be found in the frontend's certificate. The
only limitation of this mode is that the certificate's issuer will have
to be known in order for the OCSP certid to be built.
This patch only adds the parsing of the option. The full functionality
will come in a later commit.
The OCSP update tree holds ocsp responses that will need to be updated
automatically. The entries are inserted in an eb64_tree where the keys
are the absolute time after which the entry will need to be updated.
