Commit Graph

8666 Commits

Author SHA1 Message Date
Willy Tarreau
3906e22f6f MINOR: htx: add buf_room_for_htx_data() to help optimize buffer transfers
The small HTX overhead is enough to make the system perform multiple
reads and unaligned memory copies. Here we provide a function whose
purpose is to reduce the apparent room in a buffer by the size of the
overhead for DATA blocks, which is the struct htx plus 2 blocks (one
for DATA, one for the end of message so that small blocks can fit at
once). The muxes using HTX will be encouraged to use this one instead
of b_room() to compute the available buffer room and avoid filling
their demux buf with more data than can fit at once into the HTX
buffer.
2018-12-05 10:57:42 +01:00
Willy Tarreau
8ae4235f94 MINOR: htx: make htx_from_buf() adjust the size only on new buffers
This one is used a lot during transfers, let's avoid resetting its
size when there are already data in the buffer since it implies the
size is correct.
2018-12-05 10:57:42 +01:00
Olivier Houchard
8706c81316 BUG/MEDIUM: mux_pt: Always set CS_FL_RCV_MORE.
When using the mux_pt, as we can't know if there's more data to be read,
always set CS_FL_RCV_MORE, and only remove it if we got an error or a shutr
and rcv_buf() returned 0.
2018-12-04 19:23:56 +01:00
Olivier Houchard
29a22bc0a7 MEDIUM: h1: Realign the ibuf before calling rcv_buf if needed.
If the ibuf only contains a small amount of data, realign it
before calling rcv_buf(), as it's probably going to be cheaper
to do so than to do 2 calls to recv().
2018-12-04 18:42:33 +01:00
Olivier Houchard
cf42d5afa0 BUG/MEDIUM: h1: Correctly report used data with no len.
When we have no content-length, and not in chunk mode, correctly
report the used data. We really used "ret", and not "max".
2018-12-04 18:32:39 +01:00
Willy Tarreau
2fb1d4caaa MINOR: mux-h2: stop on non-DATA and non-EOM HTX blocks
We don't want to send such blocks as DATA frames if they were ever to
appear, let's quit when meeting them.
2018-12-04 18:32:39 +01:00
Willy Tarreau
ee57376ffb BUG/MEDIUM: mux-h2: don't send more HTX data than requested
It's incorrect to send more bytes than requested, because some filters
(e.g. compression) might intentionally hold on some blocks, so DATA
blocks must not be processed past the advertised byte count. It is not
the case for headers however.

No backport is needed.
2018-12-04 18:32:39 +01:00
Willy Tarreau
b08d91fbc5 BUG/MEDIUM: mux-h2: stop sending HTX once the mux is blocked
If we're blocking on mux full, mux busy or whatever, we must get out of
the loop. In legacy mode this problem doesn't exist as we can normally
return 0 but here it's not a sufficient condition to stop sending, so
we must inspect the blocking flags as well.

No backport is needed.
2018-12-04 18:32:39 +01:00
Willy Tarreau
0c22fa7d6f BUG/MEDIUM: mux-h2: make sure to always report HTX EOM when consumed by headers
The way htx_xfer_blks() was used is wrong, if we receive data, we must
report everything we found, not just the headers blocks. This ways causing
the EOM to be postponed and some fast responses (or errors) to be incorrectly
delayed.

No backport is needed.
2018-12-04 18:32:39 +01:00
Willy Tarreau
0f799ca4df BUG/MEDIUM: mux-h2: properly update the window size in HTX mode
When sending data in HTX mode, we forgot to update the window size, it
was the cause of the limitation to 1 GB in testing.

No backport is needed.
2018-12-04 18:32:39 +01:00
Olivier Houchard
8122a8d681 BUG/MEDIUM: h2: When sending in HTX, make sure the caller knows we sent all.
In h2_snd_buf(), when running with htx, make sure we return the amount of
data the caller specified, if we emptied the buffer, as it is what the
caller expects, and will lead to him properly consider the buffer to be
empty.
2018-12-04 18:32:39 +01:00
Christopher Faulet
f3d480517f BUG/MINOR: proto_htx: Truncate the request when an error is detected
When HTTP_MSGF_ERROR is set on a channel (the request or the response), the
request must be truncated, not the response.
2018-12-04 16:43:30 +01:00
Christopher Faulet
1a7ad7ad18 BUG/MEDIUM: mux-h1: Release the mux H1 in h1_process() if there is no h1s
With the current design, there is always an H1 stream attached to the mux. So
after the conn_stream is detached, if we don't create a new H1 stream in
h1_process(), it is important to release the mux.
2018-12-04 16:43:30 +01:00
Christopher Faulet
c386a8851e MINOR: mux-h1: Make sure to return 1 in h1_recv() when needed
In h1_recv(), return 1 if we have data available, or if h1_recv_allowed()
failed, to be sure h1_process() is called. Also don't subscribe if our buffer
is full.
2018-12-04 16:43:30 +01:00
Christopher Faulet
37e3607e37 BUG/MEDIUM: mux-h1: Always set CS_FL_RCV_MORE when data are received in h1_recv()
It is a warranty that the data will be handled by the stream, even if an error
is reported on the connection or on the conn_stream.
2018-12-04 16:43:30 +01:00
Olivier Houchard
75159a96de MEDIUM: mux-h1: Revamp the way subscriptions are handled.
Don't always wake the tasklets subscribed to recv or send events as soon as
we had any I/O event, and don't call the wake() method if there were no
subscription, instead, wake the recv tasklet if we received data in h2_recv(),
and wake the send tasklet if we were able to send data in h2_send(), and the
buffer is not full anymore.
Only call the data_cb->wake() method if we get an error/a read 0, just in
case the stream was not subscribed to receive events.
2018-12-04 16:43:30 +01:00
Olivier Houchard
c490efd625 BUG/MEDIUM: stream_interface: Make REALLY sure we read all the data.
In si_cs_recv(), try inconditionally to recv as long as the CS_FL_RCV_MORE is
set on the conn_stream, or we will miss some data.
2018-12-04 16:43:30 +01:00
Christopher Faulet
6e54095d0a BUG/MINOR: flt_trace/compression: Use the right flag to add the HTX support
Of course, the flag FLT_CFG_FL_HTX must be used and not
STRM_FLT_FL_HAS_FILTERS. "Fortunately", these 2 flags have the same value, so
everything worked as expected.
2018-12-04 16:43:30 +01:00
Olivier Houchard
435ce2d71d BUG/MEDIUM: h2: Don't forget to wake the tasklet after shutr/shutw.
When reaching h2_shutr/h2_shutw, as we may have generated an empty frame,
a goaway or a rst, make sure we wake the I/O tasklet, or we may not send
what we just generated.
Also in h2_shutw(), don't forget to return if all went well, we don't want
to subscribe the h2s to wait events.
2018-12-04 05:57:34 +01:00
Olivier Houchard
7ccff1a3d3 BUG/MEDIUM: h1: Destroy a connection after detach if it has no owner.
Destroy the connection while detaching, even if it has keep alive, if it has
no owner, or nobody else will be able to do so.
2018-12-04 05:57:31 +01:00
William Lallemand
2fd45fae46 BUG/MEDIUM: mworker: stop proxies which have no listener in the master
The previous code was only stopping the listeners in the master, not the
entire proxy.

Since we now have a polling loop in the master, there might be some side
effects, indeed some things that are still initialized. For example the
checks were still running.
2018-12-04 05:54:33 +01:00
Jrme Magnin
e064a80fa3 BUG/MINOR: fix ssl_fc_alpn and actually add ssl_bc_alpn
When ssl_bc_alpn was meant to be added, a typo slipped in and as a result ssl_fc_alpn behaved as ssl_bc_alpn,
and ssl_bc_alpn was not a valid keyword. this patch aims at fixing this.
2018-12-04 05:53:45 +01:00
Christopher Faulet
1e7af46aae BUG/MINOR: htx: Force HTTP/1.1 on H1 formatting when version is 1.1 or above
This only happens for connections using the h1 mux. We must be sure to force the
version to HTTP/1.1 when the version of the message is 1.1 or above. It is
important for H2 messages to not send an invalid version string (HTTP/2.0) to
peers.
2018-12-04 05:51:39 +01:00
Christopher Faulet
c59ff23804 MINOR: htx: Rename functions htx_*_to_str() to be H1 specific
"_to_h1" suffix is now used because these function produce H1 strings. It avoids
any ambiguity on the output format.
2018-12-04 05:51:37 +01:00
Willy Tarreau
da7e3be36f [RELEASE] Released version 1.9-dev9
Released version 1.9-dev9 with the following main changes :
    - BUILD/MINOR: ssl: fix build with non-alpn/non-npn libssl
    - BUG/MINOR: mworker: Do not attempt to close(2) fd -1
    - BUILD: compression: fix build error with DEFAULT_MAXZLIBMEM
    - MINOR: compression: always create the compression pool
    - BUG/MEDIUM: mworker: fix FD leak upon reload
    - BUILD: htx: fix fprintf format inconsistency on 32-bit platforms
    - BUILD: buffers: buf.h requires unistd to get ssize_t on libmusl
    - MINOR: initcall: introduce a way to register init functions to call at boot
    - MINOR: init: process all initcalls in order at boot time
    - MEDIUM: init: convert all trivial registration calls to initcalls
    - MINOR: thread: provide a set of lock initialisers
    - MINOR: threads: add new macros to declare self-initializing locks
    - MEDIUM: init: use self-initializing spinlocks and rwlocks
    - MINOR: initcall: apply initcall to all register_build_opts() calls
    - MINOR: initcall: use initcalls for most post_{check,deinit} and per_thread*
    - MINOR: initcall: use initcalls for section parsers
    - MINOR: memory: add a callback function to create a pool
    - MEDIUM: init: use initcall for all fixed size pool creations
    - MEDIUM: memory: use pool_destroy_all() to destroy all pools on deinit()
    - MEDIUM: initcall: use initcalls for a few initialization functions
    - MEDIUM: memory: make the pool cache an array and not a thread_local
    - MINOR: ssl: free ctx when libssl doesn't support NPN
    - BUG/MINOR: proto_htx: only mark connections private if NTLM is detected
    - MINOR: h2: make struct h2_ops static
    - BUG/MEDIUM: mworker: avoid leak of client socket
    - REORG: mworker: declare master variable in global.h
    - BUG/MEDIUM: listeners: CLOEXEC flag is not correctly set
    - CLEANUP: http: Fix typo in init_http's comment
    - BUILD: Makefile: Disable -Wcast-function-type if it exists.
    - BUG/MEDIUM: h2: Don't bogusly error if the previous stream was closed.
    - REGTEST/MINOR: script: add run-regtests.sh script
    - REGTEST: Add a basic test for the cache.
    - BUG/MEDIUM: mux_pt: Don't forget to unsubscribe() on attach.
    - BUG/MINOR: ssl: ssl_sock_parse_clienthello ignores session id
    - BUG/MEDIUM: connections: Wake the stream once the mux is chosen.
    - BUG/MEDIUM: connections: Don't forget to detach the connection from the SI.
    - BUG/MEDIUM: stream_interface: Don't check if the handshake is done.
    - BUG/MEDIUM: stream_interface: Make sure we read all the data available.
    - BUG/MEDIUM: h2: Call h2_process() if there's an error on the connection.
    - REGTEST: Fix several issues.
    - REGTEST: lua: check socket functionality from a lua-task
    - BUG/MEDIUM: session: Remove the session from the session_list in session_free.
    - BUG/MEDIUM: streams: Don't assume we have a CS in sess_update_st_con_tcp.
    - BUG/MEDIUM: connections: Don't assume we have a mux in connect_server().
    - BUG/MEDIUM: connections: Remove the connection from the idle list before destroy.
    - BUG/MEDIUM: session: properly clean the outgoing connection before freeing.
    - BUG/MEDIUM: mux_pt: Don't try to send if handshake is not done.
    - MEDIUM: connections: Put H2 connections in the idle list if http-reuse always.
    - MEDIUM: h2: Destroy a connection with no stream if it has no owner.
    - MAJOR: sessions: Store multiple outgoing connections in the session.
    - MEDIUM: session: Steal owner-less connections on end of transaction.
    - MEDIUM: server: Be smarter about deciding to reuse the last server.
    - BUG/MEDIUM: Special-case http_proxy when dealing with outgoing connections.
    - BUG/MINOR: cfgparse: Fix transition between 2 sections with the same name
    - BUG/MINOR: http: Use out buffer instead of trash to display error snapshot
    - BUG/MINOR: htx: Fix block size calculation when a start-line is added/replaced
    - BUG/MINOR: mux-h1: Fix processing of "Connection: " header on outgoing messages
    - BUG/MEDIUM: mux-h1: Reset the H1 parser when an outgoing message is processed
    - BUG/MINOR: proto_htx: Send outgoing data to client to start response processing
    - BUG/MINOR: htx: Stop a header or a start line lookup on the first EOH or EOM
    - BUG/MINOR: connection: report mux modes when HTX is supported
    - MINOR: htx: add a function to cut the beginning of a DATA block
    - MEDIUM: conn_stream: Add a way to get mux's info on a CS from the upper layer
    - MINOR: mux-h1: Implement get_cs_info() callback
    - MINOR: stream: Rely on CS's info if it exists and fallback on session's ones
    - MINOR: proto_htx: Use conn_stream's info to set t_idle duration when possible
    - MINOR: mux-h1: Don't rely on the stream anymore in h1_set_srv_conn_mode()
    - MINOR: mux-h1: Write last chunk and trailers if not found in the HTX message
    - MINOR: mux-h1: Be prepare to fail when EOM is added during trailers parsing
    - MINOR: mux-h1: Subscribe to send in h1_snd_buf() when not all data have been sent
    - MINOR: mux-h1: Consume channel's data in a loop in h1_snd_buf()
    - MEDIUM: mux-h1: Add keep-alive outgoing connections in connections list
    - MINOR: htx: Add function to add an HTX block just before another one
    - MINOR: htx: Add function to iterate on an HTX message using HTX blocks
    - MINOR: htx: Add a function to find the HTX block corresponding to a data offset
    - MINOR: stats: Don't add end-of-data marker and trailers in the HTX response
    - MEDIUM: htx: Change htx_sl to be a struct instead of an union
    - MINOR: htx: Add the start-line offset for the HTX message in the HTX structure
    - MEDIUM: htx: Don't rely on h1_sl anymore except during H1 header parsing
    - MINOR: proto-htx: Use the start-line flags to set the HTTP messsage ones
    - MINOR: htx: Add BODYLESS flags on the HTX start-line and the HTTP message
    - MINOR: proto_htx: Use full HTX messages to send 100-Continue responses
    - MINOR: proto_htx: Use full HTX messages to send 103-Early-Hints responses
    - MINOR: proto_htx: Use full HTX messages to send 401 and 407 responses
    - MINOR: proto_htx: Send valid HTX message when redir mode is enabled on a server
    - MINOR: proto_htx: Send valid HTX message to send 30x responses
    - MEDIUM: proto_htx: Convert all HTTP error messages into HTX
    - MINOR: mux-h1: Process conn_mode on the EOH when no connection header is found
    - MINOR: mux-h1: Change client conn_mode on an explicit close for the response
    - MINOR: mux-h1: Capture bad H1 messages
    - MAJOR: filters: Adapt filters API to be compatible with the HTX represenation
    - MEDIUM: proto_htx/filters: Add data filtering during the forwarding
    - MINOR: flt_trace: Adapt to be compatible with the HTX representation
    - MEDIUM: compression: Adapt to be compatible with the HTX representation
    - MINOR: h2: implement H2->HTX request header frame transcoding
    - MEDIUM: mux-h2: register mux for both HTTP and HTX modes
    - MEDIUM: mux-h2: make h2_rcv_buf() support HTX transfers
    - MEDIUM: mux-h2: make h2_snd_buf() HTX-aware
    - MEDIUM: mux-h2: add basic H2->HTX transcoding support for headers
    - MEDIUM: mux-h2: implement emission of H2 headers frames from HTX blocks
    - MEDIUM: mux-h2: implement the emission of DATA frames from HTX DATA blocks
    - MEDIUM: mux-h2: support passing H2 DATA frames to HTX blocks
    - BUG/MINOR: cfgparse: Fix the call to post parser of the last sections parsed
    - BUG/MEDIUM: mux-h2: don't lose the first response header in HTX mode
    - BUG/MEDIUM: mux-h2: remove the HTX EOM block on H2 response headers
    - MINOR: listener: the mux_proto entry in the bind_conf is const
    - MINOR: connection: create conn_get_best_mux_entry()
    - MINOR: server: the mux_proto entry in the server is const
    - MINOR: config: make sure to associate the proper mux to bind and servers
    - MINOR: hpack: add ":path" to the list of common header fields
    - MINOR: h2: add new functions to produce an HTX message from an H2 response
    - MINOR: mux-h2: mention that the mux is compatible with both sides
    - MINOR: mux-h2: implement an outgoing stream allocator : h2c_bck_stream_new()
    - MEDIUM: mux-h2: start to create the outgoing mux
    - MEDIUM: mux-h2: implement encoding of H2 request on the backend side
    - MEDIUM: mux-h2: make h2_frt_decode_headers() direction-agnostic
    - MEDIUM: mux-h2: make h2_process_demux() capable of processing responses as well
    - MEDIUM: mux-h2: Implement h2_attach().
    - MEDIUM: mux-h2: Don't bother flagging outgoing connections as TOOMANY.
    - REGTEST: Fix LEVEL 4 script 0 of "connection" module.
    - MINOR: connection: Fix a comment.
    - MINOR: mux: add a "max_streams" method.
    - MEDIUM: servers: Add a way to keep idle connections alive.
    - CLEANUP: fix typos in the htx subsystem
    - CLEANUP: Fix typo in the chunk headers file
    - CLEANUP: Fix typos in the h1 subsystem
    - CLEANUP: Fix typos in the h2 subsystem
    - CLEANUP: Fix a typo in the mini-clist header
    - CLEANUP: Fix a typo in the proto_htx subsystem
    - CLEANUP: Fix typos in the proto_tcp subsystem
    - CLEANUP: Fix a typo in the signal subsystem
    - CLEANUP: Fix a typo in the session subsystem
    - CLEANUP: Fix a typo in the queue subsystem
    - CLEANUP: Fix typos in the shctx subsystem
    - CLEANUP: Fix typos in the socket pair protocol subsystem
    - CLEANUP: Fix typos in the map management functions
    - CLEANUP: Fix typo in the fwrr subsystem
    - CLEANUP: Fix typos in the cli subsystem
    - CLEANUP: Fix typo in the 51d subsystem
    - CLEANUP: Fix a typo in the base64 subsystem
    - CLEANUP: Fix a typo in the connection subsystem
    - CLEANUP: Fix a typo in the protocol header file
    - CLEANUP: Fix a typo in the checks header file
    - CLEANUP: Fix typos in the file descriptor subsystem
    - CLEANUP: Fix a typo in the listener subsystem
    - BUG/MINOR: lb-map: fix unprotected update to server's score
    - BUILD: threads: fix minor build warnings when threads are disabled
2018-12-02 19:31:37 +01:00
Willy Tarreau
b54c40ac0b BUILD: threads: fix minor build warnings when threads are disabled
These potential null-deref warnings are emitted on gcc 7 and above
when threads are disabled due to the use of objt_server() after an
existing validity test. Let's switch to __objt_server() since we
know the pointer is valid, it will not confuse the compiler.

Some of these may be backported to 1.8.
2018-12-02 19:28:41 +01:00
Willy Tarreau
c8b476d952 BUG/MINOR: lb-map: fix unprotected update to server's score
The loop trying to figure the best server is theorically capable of
finishing the loop with best == NULL, causing the HA_ATOMIC_SUB()
to fail there. However for this to happen the list should be empty,
which is avoided at the beginning of the function. As it is, the
function still remains at risk so better address this now.

This patch should be backported to 1.8.
2018-12-02 19:22:55 +01:00
Joseph Herlant
75a323f04e CLEANUP: Fix a typo in the listener subsystem
Fixes a typo in the code comment of the listener subsystem.
2018-12-02 18:43:28 +01:00
Joseph Herlant
f69b807fa4 CLEANUP: Fix typos in the file descriptor subsystem
Fixes 2 typos in the code comment of the file descriptor subsystem.
2018-12-02 18:43:25 +01:00
Joseph Herlant
0b75e63dc5 CLEANUP: Fix a typo in the checks header file
Fixes a typo in the code comments of the checks header file.
2018-12-02 18:43:21 +01:00
Joseph Herlant
eeac3c722f CLEANUP: Fix a typo in the protocol header file
Fixes a typo in the code comments of the header file holding the general
protocol primitives.
2018-12-02 18:42:49 +01:00
Joseph Herlant
8a95a6e5ed CLEANUP: Fix a typo in the connection subsystem
Fixes a typo in the code comments of the connection subsystem.
2018-12-02 18:42:12 +01:00
Joseph Herlant
3b4e8e118f CLEANUP: Fix a typo in the base64 subsystem
Fixes a typo in the code comments of the base64 subsystem.
2018-12-02 18:42:08 +01:00
Joseph Herlant
31019eebe9 CLEANUP: Fix typo in the 51d subsystem
Fixes a typo in the code comments of the 51d subsystem.
2018-12-02 18:41:54 +01:00
Joseph Herlant
008b3cefa1 CLEANUP: Fix typos in the cli subsystem
Fixes typos in the code comments of the cli subsystem.
2018-12-02 18:41:44 +01:00
Joseph Herlant
40650960fd CLEANUP: Fix typo in the fwrr subsystem
Fixes a typo in the code comment of the fwrr subsystem.
2018-12-02 18:40:53 +01:00
Joseph Herlant
f43b88bc09 CLEANUP: Fix typos in the map management functions
Fixes typos in the code comments of the MAP management functions.
2018-12-02 18:40:38 +01:00
Joseph Herlant
8bb32ae8c6 CLEANUP: Fix typos in the socket pair protocol subsystem
Fixes typos in the code comments of the socket pair protocol subsystem.
2018-12-02 18:40:33 +01:00
Joseph Herlant
3952643b35 CLEANUP: Fix typos in the shctx subsystem
Fixes typos in the code comments of the shctx subsystem.
2018-12-02 18:40:29 +01:00
Joseph Herlant
d8499ecb6e CLEANUP: Fix a typo in the queue subsystem
Fixes a typo in the code comments of the queue subsystem.
2018-12-02 18:40:11 +01:00
Joseph Herlant
d091bfbc6f CLEANUP: Fix a typo in the session subsystem
Fixes a typo in the code comments of the session subsystem.
2018-12-02 18:39:57 +01:00
Joseph Herlant
f6989ca056 CLEANUP: Fix a typo in the signal subsystem
Fixes a typo in the code comments of the signal subsystem.
2018-12-02 18:39:52 +01:00
Joseph Herlant
a6331475e0 CLEANUP: Fix typos in the proto_tcp subsystem
Fixes typos in the code comments of the proto_tcp subsystem.
2018-12-02 18:39:05 +01:00
Joseph Herlant
e9d5c727c1 CLEANUP: Fix a typo in the proto_htx subsystem
Fixes a typo in the code comments of the proto_htx subsystem.
2018-12-02 18:38:48 +01:00
Joseph Herlant
41abef77cb CLEANUP: Fix a typo in the mini-clist header
Fixes a typo in the code comments of the mini-clist header.
2018-12-02 18:38:15 +01:00
Joseph Herlant
d77575d03e CLEANUP: Fix typos in the h2 subsystem
Fixes typos in the code comments of the h2 subsystem.
2018-12-02 18:38:08 +01:00
Joseph Herlant
30bc509c40 CLEANUP: Fix typos in the h1 subsystem
Fixes typos in the code comments of the h1 subsystem.
2018-12-02 18:38:02 +01:00
Joseph Herlant
be7619aaca CLEANUP: Fix typo in the chunk headers file
Fix a typo detected in the chunk.h header file's code comments.
2018-12-02 18:37:56 +01:00
Joseph Herlant
c42c0e9969 CLEANUP: fix typos in the htx subsystem
Fix typos detected in the code comments of the htx subsystem.
2018-12-02 18:37:50 +01:00
Olivier Houchard
0c18a6fe34 MEDIUM: servers: Add a way to keep idle connections alive.
Add a new keyword for servers, "idle-timeout". If set, unused connections are
kept alive until the timeout happens, and will be picked for reuse if no
other connection is available.
2018-12-02 18:16:53 +01:00