Callers of si_appctx() always use the result without checking it because
they know by construction that it's valid. This results in unchecked null
pointer warnings at -Wextra, so let's remove this test and make it clear
that it's up to the caller to check validity first.
stktable_data_ptr() currently performs null pointer checks but most
callers don't check the result since they know by construction that
it cannot be null. This causes valid warnings when building with
-Wextra which are worth addressing since it will result in better
code. Let's provide an unguarded version of this function for use
where the check is known to be useless and untested.
Theorically nothing would prevent a front applet form connecting to a stats
socket, and if a "getsock" command was issued, it would cause a crash. Right
now nothing in the code does this so in its current form there is no impact.
It may or may not be backported to 1.8.
These two functions were apparently written on the same model as their
parents when added by commit 11bcb6c4f ("[MEDIUM] IPv6 support for syslog")
except that they perform an assignment instead of a return, and as a
result fall through the next case where the assigned value may possibly
be partially overwritten. At least under Linux the port offset is the
same in both sockaddr_in and sockaddr_in6 so the value is written twice
without side effects.
This needs to be backported as far as 1.5.
In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates
must be skipped until the end of the message is found. Then the parsing is
restarted from the beginning. But not all updates were skipped, leading to
invalid rewritting or segfault.
No backport is needed.
A null pointer assignment was missing after free() in function
pat_ref_reload() which can lead to segfault.
This bug was introduced in commit b5997f7 ("MAJOR: threads/map: Make
acls/maps thread safe").
Must be backported to 1.8.
Just like we used to do in proto_http, we now check that each and every
occurrence of the content-length header field and each of its values are
exactly identical, and we normalize the header to return the last value
of the first header with spaces trimmed.
The transfer-encoding header processing was a bit lenient in this part
because it was made to read messages already validated by haproxy. We
absolutely need to reinstate the strict processing defined in RFC7230
as is currently being done in proto_http.c. That is, transfer-encoding
presence alone is enough to cancel content-length, and must be
terminated by the "chunked" token, except in the response where we
can fall back to the close mode if it's not last.
For this we now use a specific parsing function which updates the
flags and we introduce a new flag H1_MF_XFER_ENC indicating that the
transfer-encoding header is present.
Last, if such a header is found, we delete all content-length header
fields found in the message.
This flag is usefull to handle cases where there is no body, regardless of CL or
TE headers (for instance, responses to HEAD requests). It will not be set by the
parser itself.
The new function h1_parse_connection_header() is called when facing a
connection header in the generic parser, and it will set up to 3 bits
in h1m->flags indicating if at least one "close", "keep-alive" or "upgrade"
tokens was seen.
This will be needed for the mux to know how to process the Connection
header, and will save it from having to re-parse the request line since
it's captured on the fly.
While it was possible to consider the status before parsing response
headers, it's wrong to do it for request headers and could lead to
random behaviours due to this status matching other fields instead.
Additionnally there is little to no value in doing this for each and
every new header field. It's much better to reset the content-length
at once in the callerwhen seeing such statuses (which currently is only
the H2 mux).
No backport is needed, this is purely 1.9.
Released version 1.9-dev2 with the following main changes :
- BUG/MINOR: buffers: Fix b_slow_realign when a buffer is realign without output
- BUG/MEDIUM: threads: fix the no-thread case after the change to the sync point
- BUG/MEDIUM: servers: check the queues once enabling a server
- BUG/MEDIUM: queue: prevent a backup server from draining the proxy's connections
- MEDIUM: mux: Remove const on the buffer in mux->snd_buf()
- CLEANUP: backend: Move mux install to call it at only one place
- MINOR: conn_stream: add an tx buffer to the conn_stream
- MINOR: conn_stream: add cs_send() as a default snd_buf() function
- MINOR: backend: Try to find the best mux for outgoing connections
- MEDIUM: backend: don't rely on mux_pt_ops in connect_server()
- MINOR: mux: Add info about the supported side in alpn_mux_list structure
- MINOR: mux: Unlink ALPN and multiplexers to rather speak of mux protocols
- MINOR: mux: Print the list of existing mux protocols during HA startup
- MEDIUM: checks: use the new rendez-vous point to spread check result
- MEDIUM: haproxy: don't use sync_poll_loop() anymore in the main loop
- MINOR: threads: remove the previous synchronization point
- MAJOR: server: make server state changes synchronous again
- CLEANUP: server: remove the update list and the update lock
- BUG/MINOR: threads: Remove the unexisting lock label "UPDATED_SERVERS_LOCK"
- BUG/MEDIUM: stream_int: Don't check CO_FL_SOCK_RD_SH flag to trigger cs receive
- MINOR: mux: Change get_mux_proto to get an ist as parameter
- MINOR: mux: Improve the message with the list of existing mux protocols
- MINOR: mux/frontend: Add 'proto' keyword to force the mux protocol
- MINOR: mux/server: Add 'proto' keyword to force the multiplexer's protocol
- MEDIUM: mux: Use the mux protocol specified on bind/server lines
- BUG/MEDIUM: connection/mux: take care of serverless proxies
- MINOR: queue: make sure the pendconn is released before logging
- MINOR: stream: rename {srv,prx}_queue_size to *_queue_pos
- MINOR: queue: store the queue index in the stream when enqueuing
- MINOR: queue: replace the linked list with a tree
- MEDIUM: add set-priority-class and set-priority-offset
- MEDIUM: queue: adjust position based on priority-class and priority-offset
- DOC: update the roadmap about priority queues
- BUG/MINOR: ssl: empty connections reported as errors.
- MINOR: connections: Make rcv_buf mandatory and nuke cs_recv().
- MINOR: connections: Move rxbuf from the conn_stream to the h2s.
- MINOR: connections: Get rid of txbuf.
- MINOR: tasks: Allow tasklet_wakeup() to wakeup a task.
- MINOR: connections/mux: Add the wait reason(s) to wait_list.
- MINOR: stream_interface: Don't use si_cs_send() as a task handler.
- MINOR: stream_interface: Give stream_interface its own wait_list.
- MINOR: mux_h2: Don't use h2_send() as a callback.
- MINOR: checks: Add event_srv_chk_io().
- BUG/MEDIUM: tasks: Don't insert in the global rqueue if nbthread == 1
- BUG/MEDIUM: sessions: Don't use t->state.
- BUG/MEDIUM: ssl: fix missing error loading a keytype cert from a bundle.
- BUG/MEDIUM: ssl: loading dh param from certifile causes unpredictable error.
- BUG/MINOR: map: fix map_regm with backref
- DOC: dns: explain set server ... fqdn requires resolver
- DOC: add documentation for prio_class and prio_offset sample fetches.
- DOC: ssl: Use consistent naming for TLS protocols
- DOC: update the layering design notes
- MINOR: tasks: Don't special-case when nbthreads == 1
- MINOR: fd cache: And the thread_mask with all_threads_mask.
- BUG/MEDIUM: lua: socket timeouts are not applied
- BUG/MINOR: lua: fix extra 500ms added to socket timeouts
- BUG/MEDIUM: server: update our local state before propagating changes
- BUG/MEDIUM: cli/threads: protect all "proxy" commands against concurrent updates
- DOC: server/threads: document which functions need to be called with/without locks
- BUG/MEDIUM: cli/threads: protect some server commands against concurrent operations
- BUG/MEDIUM: streams: Don't forget to remove the si from the wait list.
- BUG/MEDIUM: tasklets: Add the thread as active when waking a tasklet.
- BUG/MEDIUM: stream-int: Check if the conn_stream exist in si_cs_io_cb.
- BUG/MEDIUM: H2: Activate polling after successful h2_snd_buf().
- BUG/MEDIUM: stream_interface: Call the wake callback after sending.
- BUG/MAJOR: queue/threads: make pendconn_redistribute not lock the server
- BUG/MEDIUM: connection: don't forget to always delete the list's head
- BUG/MEDIUM: lb/threads: always properly lock LB algorithms on maintenance operations
- BUG/MEDIUM: check/threads: do not involve the rendez-vous point for status updates
- BUG/MINOR: chunks: do not store -1 into chunk_printf() in case of error
- BUG/MEDIUM: http: don't store exp_replace() result in the trash's length
- BUG/MEDIUM: http: don't store url_decode() result in the samples's length
- BUG/MEDIUM: dns: don't store dns_build_query() result in the trash's length
- BUG/MEDIUM: map: don't store exp_replace() result in the trash's length
- BUG/MEDIUM: connection: don't store recv() result into trash.data
- BUG/MEDIUM: cli/ssl: don't store base64dec() result in the trash's length
- MINOR: chunk: remove impossible tests on negative chunk->data
- MINOR: sample: remove impossible tests on negative smp->data.u.str.data
- DOC: Fix spelling error in configuration doc
- REGTEST/MINOR: Missing mandatory "ignore_unknown_macro".
- REGTEST/MINOR: Add a new class of regression testing files.
- BUG/MEDIUM: unix: provide a ->drain() function
- MINOR: connection: make conn_sock_drain() work for all socket families
- BUG/MINOR: lua: Bad HTTP client request duration.
- REGEST/MINOR: Add reg testing files.
- BUG/MEDIUM: mux_pt: dereference the connection with care in mux_pt_wake()
- REGTEST/MINOR: Add a reg testing file for b406b87 commit.
- BUG/MEDIUM: lua: reset lua transaction between http requests
- MINOR: add be_conn_free sample fetch
- MINOR: Add srv_conn_free sample fetch
- BUG/MEDIUM: hlua: Make sure we drain the output buffer when done.
- MINOR: checks: Call wake_srv_chk() when we can finally send data.
- BUG/MEDIUM: stream_interface: try to call si_cs_send() earlier.
- BUG/MAJOR: thread: lua: Wrong SSL context initialization.
- REGTEST/MINOR: Add a reg testing file for 3e60b11.
- BUG/MEDIUM: hlua: Don't call RESET_SAFE_LJMP if SET_SAFE_LJMP returns 0.
- REGTEST/MINOR: lua: Add reg testing files for 70d318c.
- BUG/MEDIUM: dns/server: fix incomatibility between SRV resolution and server state file
- BUG/MEDIUM: ECC cert should work with TLS < v1.2 and openssl >= 1.1.1
- MINOR: tools: make date2str_log() take some consts
- MINOR: thread: implement HA_ATOMIC_XADD()
- BUG/MINOR: stream: use atomic increments for the request counter
- BUG/MEDIUM: session: fix reporting of handshake processing time in the logs
- BUG/MEDIUM: h2: fix risk of memory leak on malformated wrapped frames
- BUG/MAJOR: buffer: fix incorrect check in __b_putblk()
- MINOR: log: move the log code to sess_build_logline() to add extra arguments
- MINOR: log: make the backend fall back to the frontend when there's no stream
- MINOR: log: make sess_build_logline() not dereference a NULL stream for txn
- MINOR: log: don't unconditionally pick log info from s->logs
- CLEANUP: log: make the low_level lf_{ip,port,text,text_len} functions take consts
- MINOR: log: keep a copy of the backend connection early in sess_build_logline()
- MINOR: log: do not dereference a null stream to access captures
- MINOR: log: be sure not to dereference a null stream for a target
- MINOR: log: don't check the stream-int's conn_retries if the stream is NULL
- MINOR: log: use NULL for the unique_id if there is no stream
- MINOR: log: keep a copy of s->flags early to avoid a dereference
- MINOR: log: use zero as the request counter if there is no stream
- MEDIUM: log: make sess_build_logline() support being called with no stream
- MINOR: log: provide a function to emit a log for a session
- MEDIUM: h2: produce some logs on early errors that prevent streams from being created
- BUG/MINOR: h1: fix buffer shift after realignment
- MINOR: connection: make the initialization more consistent
- MINOR: connection: add new function conn_get_proxy()
- MINOR: connection: add new function conn_is_back()
- MINOR: log: One const should be enough.
- BUG/MINOR: dns: check and link servers' resolvers right after config parsing
- BUG/MINOR: http/threads: atomically increment the error snapshot ID
- MINOR: snapshot: restart on the event ID and not the stream ID
- MINOR: snapshot: split the error snapshots into common and proto-specific parts
- MEDIUM: snapshot: start to reorder the HTTP snapshot output a little bit
- MEDIUM: snapshot: implement a show() callback and use it for HTTP
- MINOR: proxy: add a new generic proxy_capture_error()
- MINOR: http: make the HTTP error capture rely on the generic proxy code
- MINOR: http: remove the pointer to the error snapshot in http_capture_bad_message()
- REORG: cli: move the "show errors" handler from http to proxy
- BUG/MEDIUM: snapshot: take the proxy's lock while dumping errors
- MEDIUM: snapshots: dynamically allocate the snapshots
- MEDIUM: snapshot: merge the captured data after the descriptor
- MEDIUM: mworker: remove register/unregister signal functions
- MEDIUM: mworker: use the haproxy poll loop
- BUG/MINOR: mworker: no need to stop peers for each proxy
- MINOR: mworker: mworker_cleanlisteners() delete the listeners
- MEDIUM: mworker: block SIGCHLD until the master is ready
- MEDIUM: mworker: never block SIG{TERM,INT} during reload
- MEDIUM: startup: unify signal init between daemon and mworker mode
- MINOR: mworker: don't deinit the poller fd when in wait mode
- MEDIUM: mworker: master wait mode use its own initialization
- MEDIUM: mworker: replace the master pipe by socketpairs
- MINOR: mworker: keep and clean the listeners
- MEDIUM: threads: close the thread-waker pipe during deinit
- MEDIUM: mworker: call per_thread deinit in mworker_reload()
- REORG: http: move the HTTP semantics definitions to http.h/http.c
- REORG: http: move http_get_path() to http.c
- REORG: http: move error codes production and processing to http.c
- REORG: http: move the log encoding tables to log.c
- REORG: http: move some header value processing functions to http.c
- BUG/MAJOR: kqueue: Don't reset the changes number by accident.
- MEDIUM: protocol: use a custom AF_MAX to help protocol parser
- MEDIUM: protocol: sockpair protocol
- TESTS: add a python wrapper for sockpair@
- BUG/MINOR: server: Crash when setting FQDN via CLI.
- BUG/MINOR: h2: report asynchronous end of stream on closed connections
- BUILD: fix build without thread
- BUG/MEDIUM: tasks: Don't forget to decrement task_list_size in tasklet_free().
- MEDIUM: connections: Don't reset the polling flags in conn_fd_handler().
- MEDIUM: connections/mux: Add a recv and a send+recv wait list.
- MEDIUM: connections: Get rid of the recv() method.
- MINOR: h2: Let user of h2_recv() and h2_send() know xfer has been done.
- MEDIUM: h2: always subscribe to receive if allowed.
- MEDIUM: h2: Don't use a wake() method anymore.
- MEDIUM: stream_interface: Make recv() subscribe when more data is needed.
- MINOR: connections: Add a "handle" field to wait_list.
- MEDIUM: mux_h2: Revamp the send path when blocking.
- MEDIUM: stream_interfaces: Starts receiving from the upper layers.
- MINOR: checks: Give checks their own wait_list.
- MINOR: conn_streams: Remove wait_list from conn_streams.
- REORG: h1: create a new h1m_state
- MINOR: h1: add the restart offsets into struct h1m
- MINOR: h1: remove the unused states from h1m_state
- MINOR: h1: provide a distinct init() function for request and response
- MINOR: h1: add a message flag to indicate that a message carries a response
- MINOR: h2: make sure h1m->err_pos field is correct on chunk error
- MINOR: h1: properly pre-initialize err_pos to -2
- MINOR: mux_h2: replace the req,res h1 messages with a single h1 message
- MINOR: h2: pre-initialize h1m->err_pos to -1 on the output path
- MEDIUM: h1: consider err_pos before deciding to accept a header name or not
- MEDIUM: h1: make the parser support a pointer to a start line
- MEDIUM: h1: let the caller pass the initial parser's state
- MINOR: h1: make the message parser support a null <hdr> argument
- MEDIUM: h1: support partial message parsing
- MEDIUM: h1: remove the useless H1_MSG_BODY state
- MINOR: h2: store the HTTP status into the H2S, not the H1M
- MINOR: h1: remove the HTTP status from the H1M struct
- MEDIUM: h1: implement the request parser as well
- MINOR: h1: add H1_MF_TOLOWER to decide when to turn header names to lower case
- MINOR: connection: pass the proxy when creating a connection
- BUG/MEDIUM: h2: Don't forget to empty the wait lists on destroy.
- BUG/MEDIUM: h2: Don't forget to set recv_wait_list to NULL in h2_detach.
- BUG/MAJOR: h2: reset the parser's state on mux buffer full
The h2 parser has this specificity that if it cannot send the headers
frame resulting from the headers it just parsed, it needs to drop it
and parse it again later. Since commit 8852850 ("MEDIUM: h1: let the
caller pass the initial parser's state"), when this happens the parser
remains in the data state and the headers are not parsed again next
time, resulting in a parse error. Let's reset the parser on exit there.
No backport is needed.
If we're detaching the conn_stream, and it was subscribed to be waken up
when more data was available to receive, unsubscribe it.
No backport is needed.
Empty both send_list and fctl_list when destroying the h2 context, so that
if we're freeing the stream after, it doesn't try to remove itself from the
now-deleted list.
No backport is needed.
Till now it was very difficult for a mux to know what proxy it was
working for. Let's pass the proxy when the mux is instanciated at
init() time. It's not yet used but the H1 mux will definitely need
it, just like the H2 mux when dealing with backend connections.
The h1 parser used to systematically turn header field names to lower
case because it was designed for H2. Let's add a flag which is off by
default to condition this behaviour so that when using it from an H1
parser it will not affect the message.
The original H1 request parsing code was reintroduced into the generic
H1 parser so that it can be used regardless of the direction. If the
parser is interrupted and restarts, it makes use of the H1_MF_RESP
flag to decide whether to re-parse a request or a response. While
parsing the request, the method is decoded and set into the start line
structure.
The HTTP status is not relevant to the H1 message but to the H2 stream
itself. It used to be placed there by pure convenience but better move
it before it's too hard to remove.
This state was only a delimiter between headers and body but it now
causes more harm than good because it requires someone to change it.
Since the H1 parser knows if we're in DATA or CHUNK_SIZE, simply let
it set the right next state so that h1m->state constantly matches
what is expected afterwards.
While it was not needed in the H2 mux which was reading full H1 messages
from the channel, it is mandatory for the H1 mux reading contents from
outside to be able to restart on a message. The problem is that the
headers are indexed on the fly, and it's not fun to have to store
everything between calls.
The solution here is to complete the first pass doing a partial restart,
and only once the end of message was found, to start over it again at
once, filling entries. This way there is a bounded number of passes on
the contents and no need to store an intermediary result anymore. Later
this principle could even be used to decide to completely drop an output
buffer to save memory.
This will allow the parser to fill some extra fields like the method or
status without having to store them permanently in the HTTP message. At
this point however the parser cannot restart from an interrupted read.
Till now the H1 parser made for H2 used to be lenient on invalid header
field names because they were supposed to be produced by haproxy. Now
instead we'll rely on err_pos to know how to act (ie: -2 == must block).
There's no reason to have the two sides in H1 format since we only use
one at a time (the response at the moment). While completely removing
the request declaration, let's rename the response to "h1m" to clarify
that it's the unique h1 message there.
This way we maintain the old mechanism stating that -2 means we block
on errors, -1 means we only capture them, and a positive value indicates
the position of the first error.
Currently the only user of struct h1m is the h2 mux when it has to parse
an H1 message coming from the channel. Unfortunately this is not enough
to efficiently parse HTTP/1 messages like those coming from the network
as we don't want to restart from scratch at every byte received.
This patch reintroduces the "next" offset into the H1 message so that any
H1 parser can use it to restart when called with a state that is not the
initial state.
This is the *parsing* state of an HTTP/1 message. Currently the h1_state
is composite as it's made both of parsing and control (100SENT, BODY,
DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state
that can be used by H1 parsers. For now it's equivalent to h1_state.
Instead of waiting for the connection layer to let us know we can read,
attempt to receive as soon as process_stream() is called, and subscribe
to receive events if we can't receive yet.
Now, except for idle connections, the recv(), send() and wake() methods are
no more, all the lower layers do is waking tasklet for anybody waiting
for I/O events.
Change fctl_list and send_list to be lists of struct wait_list, and nuke
send_wait_list, as it's now redundant.
Make the code responsible for shutr/shutw subscribe to those lists.
Instead of having our wake() method called each time a fd event happens,
just subscribe to recv/send events, and get our tasklet called when that
happens. If any recv/send was possible, the equivalent of what h2_wake_cb()
will be done.
Let the connection layer know we're always interested in getting more data,
so that we get scheduled as soon as data is available, instead of relying
on the wake() method.
Make h2_recv() and h2_send() return 1 if data has been sent/received, or 0
if it did not. That way the caller will be able to know if more work may
have to be done.
Remove the recv() method from mux and conn_stream.
The goal is to always receive from the upper layers, instead of waiting
for the connection later. For now, recv() is still called from the wake()
method, but that should change soon.
