When an H2 request is converted into an HTX message, All cookie headers are
grouped into one, each value separated by a semicolon (;). To do so, we add the
header "cookie" with the first value and then we update the value by appending
other cookies. But during this operation, only the size of the HTX block is
updated. And not the data length of the whole HTX message.
It is an old bug and it seems to work by chance till now. But it may lead to
undefined behaviour by time to time.
This patch must be backported to 2.0 and 1.9
Released version 2.0.0 with the following main changes :
- MINOR: fd: Don't use atomic operations when it's not needed.
- DOC: mworker-prog: documentation for the program section
- MINOR: http: add a new "http-request replace-uri" action
- BUG/MINOR: 51d/htx: The _51d_fetch method, and the methods it calls are now HTX aware.
- MINOR: 51d: Added dummy libraries for the 51Degrees module for testing.
- MINOR: mworker: change formatting in uptime field of "show proc"
- MINOR: mworker: add the HAProxy version in "show proc"
- MINOR: doc: Remove -Ds option in man page
- MINOR: doc: add master-worker in the man page
- MINOR: doc: mention HAPROXY_LOCALPEER in the man
- BUILD: Silence gcc warning about unused return value
- CLEANUP: 51d: move the 51d dummy lib to contrib/51d/src to match the real lib
- BUILD: travis-ci: add 51Degree device detection, update openssl to 1.1.1c
- MINOR: doc: update the manpage and usage message about -S
- BUILD/MINOR: 51d: Updated build registration output to indicate thatif the library is a dummy one or not.
- BUG/MEDIUM: h1: Don't wait for handshake if we had an error.
- BUG/MEDIUM: h1: Wait for the connection if the handshake didn't complete.
- BUG/MINOR: task: prevent schedulable tasks from starving under high I/O activity
- BUG/MINOR: fl_trace/htx: Be sure to always forward trailers and EOM
- BUG/MINOR: channel/htx: Call channel_htx_full() from channel_full()
- BUG/MINOR: http: Use the global value to limit the number of parsed headers
- BUG/MINOR: htx: Detect when tail_addr meet end_addr to maximize free rooms
- BUG/MEDIUM: htx: Don't change position of the first block during HTX analysis
- CLEANUP: channel: Remove channel_htx_fwd_payload() and channel_htx_fwd_all()
- BUG/MEDIUM: proto_htx: Introduce the state ENDING during forwarding
- MINOR: htx: Add 3 flags on the start-line to deal with the request schemes
- MINOR: h2: Set flags about the request's scheme on the start-line
- MINOR: mux-h1: Set flags about the request's scheme on the start-line
- MINOR: mux-h2: Forward clients scheme to servers checking start-line flags
- MEDIUM: server: server-state only rely on server name
- CLEANUP: connection: rename the wait_event.task field to .tasklet
- CLEANUP: tasks: rename task_remove_from_tasklet_list() to tasklet_remove_*
- BUG/MEDIUM: connections: Don't call shutdown() if we want to disable linger.
- DOC: add some environment variables in section 2.3
- BUILD: makefile: clarify the "help" output and list options
- BUG/MINOR: mux-h1: Wake busy mux for I/O when message is fully sent
- BUG: tasks: fix bug introduced by latest scheduler cleanup
- BUG/MEDIUM: mux-h2: fix early close with option abortonclose
- BUG/MEDIUM: connections: Don't use ALPN to pick mux when in mode TCP.
- BUG/MEDIUM: connections: Don't try to send early data if we have no mux.
- BUG/MEDIUM: mux-h2: properly account for the appended data in HTX
- BUILD: makefile: further clarify the "help" output and list targets
- BUILD: makefile: rename "linux2628" to "linux-glibc" and remove older targets
- BUILD: travis-ci: switch to linux-glibc instead of linux2628
- DOC: update few references to the linux* targets and change them to linux-glibc
- BUILD: makefile: detect and reject recently removed linux targets
- BUILD: makefile: enable linux namespaces by default on linux
- BUILD: makefile: enable TFO on linux platforms
- BUILD: makefile: enable getaddrinfo on the linux-glibc target
- DOC: small updates to the CONTRIBUTING file
- BUG/MEDIUM: ssl: Make sure we initiate the handshake after using early data.
- CLEANUP: removed obsolete examples an move a few to better places
- DOC: Fix typos in CONTRIBUTING
- DOC: update the outdated ROADMAP file
- DOC: create a BRANCHES file to explain the life cycle
- DOC: mention in INSTALL haproxy 2.0 is a long-term supported stable version
- BUILD: travis-ci: TFO and GETADDRINFO are now enabled by default
- BUILD: makefile: make the obsolete target detection compatible with make-3.80
- BUILD: tools: work around an internal compiler bug in gcc-3.4
- BUILD: pattern: work around an internal compiler bug in gcc-3.4
- BUILD: makefile: enable USE_RT on Solaris
- BUILD: makefile: do not use echo -n
- DOC: mention a few common build errors in the INSTALL file
These are some errors met when trying to build with gcc 3.4 on an
old (13 years-old) Solaris 10 and on an even older Linux 2.4 with
glibc 2.2.5. A few options were enough to fix the build there.
gcc-3.4 fails to compile pattern.c :
src/pattern.c: In function `pat_match_ip':
src/pattern.c:1092: error: unrecognizable insn:
(insn 186 185 187 9 src/pattern.c:970 (set (reg/f:SI 179)
(high:SI (const:SI (plus:SI (symbol_ref:SI ("static_pattern") [flags 0x22] <var_decl fe5bae80 static_pattern>)
(const_int 8 [0x8]))))) -1 (nil)
(nil))
src/pattern.c:1092: internal compiler error: in extract_insn, at recog.c:2083
This happens when performing the memcpy() on the union, and in this
case the workaround is trivial (and even cleaner) using a cast instead.
gcc-3.4 fails to compile standard.c :
src/standard.c: In function `str2sa_range':
src/standard.c:1034: error: unrecognizable insn:
(insn 582 581 583 37 src/standard.c:949 (set (reg/f:SI 262)
(high:SI (const:SI (plus:SI (symbol_ref:SI ("*ss.4") [flags 0x22] <var_decl fe782e80 ss>)
(const_int 2 [0x2]))))) -1 (nil)
(nil))
src/standard.c:1034: internal compiler error: in extract_insn, at recog.c:2083
The workaround is explained here :
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=21613
It only requires creating a local variable containing the result of the
cast, which is totally harmless, so let's do it.
At least the load load balancing was done. Other points are being carried
since 1.5 or so, they should go into the issue tracker with no version
indication.
The following example files awere removed as irrelevant by this
time :
auth.cfg check.conf ssl.cfg haproxy.spec
The following scripts were removed as having been unused for more
than a decade :
debug2ansi debug2html debugfind check init.haproxy stats_haproxy.sh
seemless_reload.txt was moved to doc/ where it's more suitable.
haproxy.vim was moved to contrib/syntax-highlight/
scripts/create-release was updated not to try to update haproxy.spec
anymore.
When we're done sending/receiving early data, and we add the handshake
flags on the connection, make sure we wake the associated tasklet up, so that
the handshake will be initiated.
There's an abstract explaining what is discussed in the file, a small
explanation of how the project works, which justifies the measures
taken here, and instructions about what to do when a patch is ignored,
or how to annoy everyone.
getaddrinfo() has been available since glibc 2.3.3 or so and is generally
enabled by distro packagers. The main reason for not enabling it on Linux
in the past is that it was known broken on some libc alternatives. It's
the right moment to enable it by default with glibc.
TCP Fast Open is supported on all supported Linux kernels and on all
kernels shipped in supported distros, except the older 2.6.32 that
comes with RHEL6. However the option is harmless, will not prevent
from building and smoothly falls back even if forcefully enabled, so
it makes sense to enable it by default. It's still possible to pass
"USE_TFO=" to force it disabled if really desired.
Oldest kernel found on a supported Linux distro (2.6.32 + backports on
RHEL6) supports network namespaces, so we have no reason not to enable
them by default on the linux-glibc target.
We've just removed old linux targets "linux22", "linux24", "linux24e",
"linux26" and "linux2628" and it's likely that many build scripts and
packages will still reference these. So let's have the makefile detect
these and reject with instructions instead of silently building with
incorrect options.
The INSTALL guide, the Lua doc and the Prometheus exporter's README all
used to reference "linux2628", "linux26" or even "linux". These were all
updated to consistently reflect "linux-glibc" instead. The default options
were updated there as well so that it should build cleanly on most distros.
The linux targets have become more than confusing over time. We used to
have "linux2628" to match the features available in kernels 2.6.28 and
above, without consideration for the libc, and due to many new features
appearing later in kernels, some other options were added that are not
enabled by default in linux2628, so this target doesn't make any sense
anymore. The older ones (linux 2.2, linux 2.4, ...) do not make sense
either since these versions are not supported anymore. Let's clean things
up by creating a new "linux-glibc" target that matches what is available
by default on Linux kernels and glibc present on supported distros at the
time of release. Other libc implementation may use a custom or generic
target or be added later if needed.
All the older linux targets were removed.
When commit 0350b90e3 ("MEDIUM: htx: make htx_add_data() never defragment
the buffer") was introduced, it made htx_add_data() actually be able to
add less data than it was asked for, and the callers must use the returned
value to know how much was added. The H2 code used to rely on the frame
length instead of the return value. A version of the code doing this was
written but is obviously not the one that got merged, resulting in breaking
large uploads or downloads when HTX would have instead defragmented the
buffer because the HTX side sees less contents than what the H2 side sees.
This patch fixes this again. No backport is needed.
In connect_server(), if we don't yet have a mux, because we're choosing
one depending on the ALPN, don't attempt to send early data. We can't do
it because those data would depend on the mux, that will only be determined
by the handshake.
This should be backported to 1.9.
In connect_server(), don't wait until we negociate the ALPN to choose the
mux, the only mux we want to use is the mux_pt anyway.
This should be backported to 1.9.
Olivier found that commit 99ad1b3e8 ("MINOR: mux-h2: stop relying on
CS_FL_REOS") managed to break abortonclose again with H2. What happens
is that while the CS_FL_REOS flag was set on some transitions to the
HREM state, it's not set on all and is in fact only set when the low
level connection is closed. So making the replacement condition match
the HREM and ERROR states is not correct and causes completely correct
requests to send advertise an early close of the connection layer while
only the stream's input is closed.
In order to avoid this, we now properly split the checks for the CLOSED
state and for the closed connection. This way there is no risk to set
the EOS flag too early on the connection.
No backport is needed.
In commit 86eded6c6 ("CLEANUP: tasks: rename task_remove_from_tasklet_list()
to tasklet_remove_*") which consisted in removing the casts between tasks
and tasklet, I was a bit too fast to believe that we only saw tasklets in
this function since process_runnable_tasks() also uses it with tasks under
a cast. So removing the bookkeeping on task_list_size was not appropriate.
Bah, the joy of casts which hide the real thing...
This patch does two things at once to address this mess once for all:
- it restores the decrement of task_list_size when it's a real task,
but moves it to process_runnable_task() since it's the only place
where it's allowed to call it with a task
- it moves the increment there as well and renames
task_insert_into_tasklet_list() to tasklet_insert_into_tasklet_list()
of obvious consistency reasons.
This way the increment/decrement of task_list_size is made at the only
places where the cast is enforced, so it has less risks to be missed.
The comments on top of these functions were updated to reflect that they
are only supposed to be used with tasklets and that the caller is responsible
for keeping task_list_size up to date if it decides to enforce a task there.
Now we don't have to worry anymore about how these functions work outside
of the scheduler, which is better longterm-wise. Thanks to Christopher for
spotting this mistake.
No backport is needed.
If a mux is in busy mode when the outgoing EOM is consummed, it is important to
wake it up for I/O. Because in busy mode, the mux is not subscribed for
receive. Otherwise, it depends on the applicative layer to shutdown the H1
stream. Wake it up allows the mux to catch the read0 as soon as possible.
This patch must be backported to 1.9.
The list of enable and disabled build options now appears separately
at the end of "make help". This is convenient to know what is enabled
by default on a given target. For example :
$ make help TARGET=linux2628
Enabled features for TARGET 'linux2628' (disable with 'USE_xxx=') :
EPOLL NETFILTER POLL THREAD TPROXY LINUX_TPROXY LINUX_SPLICE LIBCRYPT
CRYPT_H FUTEX ACCEPT4 CPU_AFFINITY DL RT PRCTL THREAD_DUMP
Disabled features for TARGET 'linux2628' (enable with 'USE_xxx=1') :
KQUEUE MY_EPOLL MY_SPLICE PCRE PCRE_JIT PCRE2 PCRE2_JIT PRIVATE_CACHE
PTHREAD_PSHARED REGPARM STATIC_PCRE STATIC_PCRE2 VSYSCALL GETADDRINFO
OPENSSL LUA MY_ACCEPT4 ZLIB SLZ TFO NS DEVICEATLAS 51DEGREES WURFL
SYSTEMD OBSOLETE_LINKER EVPORTS
In conn_sock_shutw(), avoid calling shutdown() if linger_risk is set. Not
doing so will result in getting sockets in TIME_WAIT for some time.
This is particularly observable with health checks.
This should be backported to 1.9.
The function really only operates on tasklets, its arguments are always
tasklets cast as tasks to match the function's type, to be cast back to
a struct tasklet. Let's rename it to tasklet_remove_from_tasklet_list(),
take a struct tasklet, and get rid of the undesired task casts.
It's really confusing to call it a task because it's a tasklet and used
in places where tasks and tasklets are used together. Let's rename it
to tasklet to remove this confusion.
Since h7da71293e431b5ebb3d6289a55b0102331788ee6as has been added, the
server name (srv->id in the code) is now unique per backend, which
means it can reliabely be used to identify a server recovered from the
server-state file.
This patch cleans up the parsing of server-state file and ensure we use
only the server name as a reliable key.
By default, the scheme "https" is always used. But when an explicit scheme was
defined and when this scheme is "http", we use it in the request sent to the
server. This is done by checking flags of the start-line. If the flag
HTX_SL_F_HAS_SCHM is set, it means an explicit scheme was defined on the client
side. And if the flag HTX_SL_F_SCHM_HTTP is set, it means the scheme "http" was
used.
We first try to figure out if the URI of the start-line is absolute or not. So,
if it does not start by a slash ("/"), it means the URI is an absolute one and
the flag HTX_SL_F_HAS_SCHM is set. Then checks are performed to know if the
scheme is "http" or "https" and the corresponding flag is set,
HTX_SL_F_SCHM_HTTP or HTX_SL_F_SCHM_HTTPS. Other schemes, for instance ftp, are
ignored.
The flag HTX_SL_F_HAS_SCHM is always set because H2 requests have always an
explicit scheme. Then, the pseudo-header ":scheme" is tested. If it is set to
"http", the flag HTX_SL_F_SCHM_HTTP is set. Otherwise, for all other cases, the
flag HTX_SL_F_SCHM_HTTPS is set. For now, it seems reasonable to have a fallback
on the scheme "https".
The first one, HTX_SL_F_HAS_SCHM, will be used to know the request has an
explicit scheme. So, in H2, it is always true because the pseudo-header
":scheme" is mandatory. In H1, it is only true when an absolute URI is found on
the start-line. The other flags, HTX_SL_F_SCHM_HTTP and HTX_SL_F_SCHM_HTTPS,
will be used to know which scheme the request have. For now, other protocols are
not handled.
The aim of these flags is to pass this information to the backend side in
general, and to the H2 mux in particular. So the multiplexer will have a chance
to use this information to send the right scheme to the server.
This state is used in the legacy HTTP when everything was received from an
endpoint but a filter doesn't forward all the data. It is used to not report a
client or a server abort, depending on channels flags.
The same must be done on HTX streams. Otherwise, the message may be
truncated. For instance, it may happen with the filter trace with the random
forwarding enabled on the response channel.
This patch must be backported to 1.9.
In the HTX structure, the field <first> is used to know where to (re)start the
analysis. It may differ from the message's head. It is especially important to
update it to handle 1xx messages, to be sure to restart the analysis on the next
message (another 1xx message or the final one). It is also updated when some
data are forwarded (the headers or part of the body). But this update is an
error and must never be done at the analysis level. It is a bug, because some
sample fetches may be used after the data forwarding (but before the first send
of course). At this stage, if the first block position does not point on the
start-line, most of HTTP sample fetches fail.
So now, when something is forwarding by HTX analyzers, the first block position
is not update anymore.
This issue was reported on Github. See #119. No backport needed.
When a block's payload is moved during an expansion or when the whole block is
removed, the addresses of free spaces are updated accordingly. We must be
careful to reset them when <tail_addr> becomes equal to <end_addr>. In this
situation, we can maximize the free space between the blocks and their payload
and set the other one to 0. It is also important to be sure to never have
<end_addr> greater than <tail_addr>.
Instead of using the macro MAX_HTTP_HDR to limit the number of headers parsed
before throwing an error, we now use the custom global variable
global.tune.max_http_hdr.
This patch must be backported to 1.9.
