Commit Graph

6986 Commits

Author SHA1 Message Date
David Carlier
7567c4002f BUILD: enable USE_THREAD for OpenBSD build. 2017-11-03 15:41:06 +01:00
Emeric Brun
8c4954c5c2 BUG/MINOR: lua: fix missing lock protection on server.
To avoid inconsistencies server's attributes must be read
or updated under lock.
2017-11-03 15:17:59 +01:00
Emeric Brun
e9fd6b5916 BUG/MINOR: dns: fix missing lock protection on server.
To avoid inconsistencies server's attributes must be read
or updated under lock.
2017-11-03 15:17:55 +01:00
David Carlier
5222d8eb25 BUG/MINOR: stdarg.h inclusion
Needed for the memvprintf part, the va_list type.
Spotted during OpenBSD build.
2017-11-03 15:04:09 +01:00
Emeric Brun
f2fc1fda80 BUG/MINOR: freq: fix infinite loop on freq_ctr_period.
Using peers or stick table we could update an freq_ctr
using a tick value with the first bit set but this
bit is reserved for lock since multithreading support.
2017-11-02 18:09:58 +01:00
Willy Tarreau
4b75fffa2b BUG/MAJOR: buffers: fix get_buffer_nc() for data at end of buffer
This function incorrectly dealt with the case where data doesn't
wrap but lies at the end of the buffer, resulting in Lukas' reported
data corruption with HTTP/2. No backport is needed, it was introduced
for HTTP/2 in 1.8-dev.
2017-11-02 17:16:07 +01:00
William Lallemand
9c54c53f2f BUG/MEDIUM: cache: don't try to resolve wrong filters
Don't try to resolve wrong filters which are not cache filters during
the post configuration callback.
2017-11-02 16:58:25 +01:00
William Lallemand
b620e987d0 BUILD: shctx: allow to be built without openssl
The shctx functions does not depend of openssl anymore, allows to build
them without openssl.
2017-11-02 16:58:25 +01:00
Willy Tarreau
7c2a2ad65c BUG/MINOR: thread: fix a typo in the debug code
__spin_unlock() used to call RWLOCK_WRUNLOCK() to unlock in the
debug code. It's harmless as they happen to be identical.
2017-11-02 16:26:02 +01:00
Emeric Brun
f6ba17da20 BUG/MAJOR: fix deadlock on healthchecks.
Fix bugs due to missing unlock and recursive lock performing
http health check.

The server's lock scope was enlarged to protect all callers
of 'set_server_check_status' and 'chk_report_conn_err'.

This fix also protects tcpcheck against concurrency.
2017-11-02 16:24:37 +01:00
Willy Tarreau
16257f648f BUG/MEDIUM: checks/mux: always enable send-polling after connecting
Before introducing the mux layer, tcp_connect() would poll for sending
to detect the connection establishment. It happens that the health
checks have apparently never explicitly enabled this polling and have
been relying on this implicit one.

Now that there's the mux layer, the conn_stream needs to be enabled
for polling as well and since it's not done in the checks, it's never
done and the check's request doesn't leave the machine, as can be
noticed with http checks.

The solution simply consists in going back to the well-known case
where we enable polling after connecting using cs_want_send() if we
have anything but just a plain connect(). The regular data path is
not affected because the stream interface code automatically computes
the polling needs based on buffer contents.
2017-11-02 15:53:04 +01:00
Willy Tarreau
f13ef96e70 BUG/MEDIUM: h2: don't try to parse incomplete H1 responses
This situation which must not happen does in fact happen when feeding
artificial responses using errorfiles, Lua or an applet. For now it
causes the H1 response parser to loop forever trying to get a more
complete response. Since it cannot progress, let's return an error.
2017-11-02 15:53:04 +01:00
Olivier Houchard
fccf840cdf MINOR: cache: Don't confuse act_return and act_parse_ret. 2017-11-01 15:10:51 +01:00
Olivier Houchard
cd2867a012 MINOR: cache: Remove useless test for nonzero.
Don't bother testing if len is nonzero, we know it is, as we're in the
"else" part of a if (!len), and testing it confuses clang into thinking
ret may be left uninitialized.
2017-11-01 15:10:51 +01:00
Olivier Houchard
7da120bb0e MINOR: mux: Only define pipe functions on linux.
Only define mux_pt_snd_pipe() and mux_pt_rcv_pipe() if splicing is
available.
2017-11-01 15:10:51 +01:00
Emmanuel Hocdet
82913e4f79 BUG/MINOR: send-proxy-v2: string size must include ('\0')
strlen() exclude the terminating null byte ('\0'), add it.
2017-11-01 07:58:20 +01:00
Emmanuel Hocdet
571c7ac0a5 BUG/MINOR: send-proxy-v2: fix dest_len in make_tlv call
Subtract already allocated size from buf_len.
2017-11-01 07:57:42 +01:00
Willy Tarreau
901f75c4a6 [RELEASE] Released version 1.8-rc1
Released version 1.8-rc1 with the following main changes :
    - BUG/MEDIUM: server: Allocate tmptrash before using it.
    - CONTRIB: trace: add the possibility to place trace calls in the code
    - CONTRIB: trace: try to display the function's return value on exit
    - CONTRIB: trace: report the base name only for file names
    - BUILD: ssl: support OPENSSL_NO_ASYNC #define
    - MINOR: ssl: build with recent BoringSSL library
    - BUG/MINOR: ssl: OCSP_single_get0_status can return -1
    - BUG/MINOR: cli: restore "set ssl tls-key" command
    - CLEANUP: cli: remove undocumented "set ssl tls-keys" command
    - IMPORT: sha1: import SHA1 functions
    - MINOR: sample: add the sha1 converter
    - MINOR: sample: add the hex2i converter
    - MINOR: stream-int: stop checking for useless connection flags in chk_snd_conn
    - MINOR: ssl: don't abort after sending 16kB
    - MINOR: connection: move the cleanup of flag CO_FL_WAIT_ROOM
    - MINOR: connection: add flag CO_FL_WILL_UPDATE to indicate when updates are granted
    - MEDIUM: connection: make use of CO_FL_WILL_UPDATE in conn_sock_shutw()
    - MINOR: raw_sock: make use of CO_FL_WILL_UPDATE
    - MINOR: ssl_sock: make use of CO_FL_WILL_UPDATE
    - BUG/MINOR: checks: Don't forget to release the connection on error case.
    - MINOR: buffer: add the buffer input manipulation functions
    - BUG/MEDIUM: prevent buffers being overwritten during build_logline() execution
    - MEDIUM: cfgparse: post section callback
    - MEDIUM: cfgparse: post parsing registration
    - MINOR: lua: add uuid to the Class Proxy
    - MINOR: hlua: Add regex class
    - MINOR: http: Mark the 425 code as "Too Early".
    - MEDIUM: ssl: convert CBS (BoringSSL api) usage to neutral code
    - MINOR: ssl: support Openssl 1.1.1 early callback for switchctx
    - MINOR: ssl: generated certificate is missing in switchctx early callback
    - MEDIUM: ssl: Handle early data with OpenSSL 1.1.1
    - BUILD: Makefile: disable -Wunused-label
    - MINOR: ssl/proto_http: Add keywords to take care of early data.
    - BUG/MINOR: lua: const attribute of a string is overridden
    - MINOR: ssl: Don't abuse ssl_options.
    - MINOR: update proxy-protocol-v2 #define
    - MINOR: merge ssl_sock_get calls for log and ppv2
    - MINOR: add ALPN information to send-proxy-v2
    - MEDIUM: h1: ensure that 1xx, 204 and 304 don't have a payload body
    - CLEANUP: shctx: get ride of the shsess_packet{_hdr} structures
    - MEDIUM: lists: list_for_each_entry{_safe}_from functions
    - REORG: shctx: move lock functions and struct
    - MEDIUM: shctx: allow the use of multiple shctx
    - REORG: shctx: move ssl functions to ssl_sock.c
    - MEDIUM: shctx: separate ssl and shctx
    - MINOR: shctx: rename lock functions
    - MINOR: h1: store the status code in the H1 message
    - BUG/MINOR: spoe: Don't compare engine name and SPOE scope when both are NULL
    - BUG/MINOR: spoa: Update pointer on the end of the frame when a reply is encoded
    - MINOR: action: Add trk_idx inline function
    - MINOR: action: Use trk_idx instead of tcp/http_trk_idx
    - MINOR: action: Add a function pointer in act_rule struct to check its validity
    - MINOR: action: Add function to check rules using an action ACT_ACTION_TRK_*
    - MINOR: action: Add a functions to check http capture rules
    - MINOR: action: Factorize checks on rules calling check_ptr if defined
    - MINOR: acl: Pass the ACLs as an explicit parameter of build_acl_cond
    - MEDIUM: spoe: Add support of ACLS to enable or disable sending of SPOE messages
    - MINOR: spoe: Check uniqness of SPOE engine names during config parsing
    - MEDIUM: spoe: Parse new "spoe-group" section in SPOE config file
    - MEDIUM: spoe/rules: Add "send-spoe-group" action for tcp/http rules
    - MINOR: spoe: Move message encoding in its own function
    - MINOR: spoe: Add a type to qualify the message list during encoding
    - MINOR: spoe: Add a generic function to encode a list of SPOE message
    - MEDIUM: spoe/rules: Process "send-spoe-group" action
    - BUG/MINOR: dns: Fix CLI keyword declaration
    - MAJOR: dns: Refactor the DNS code
    - BUG/MINOR: mailers: Fix a memory leak when email alerts are released
    - MEDIUM: mailers: Init alerts during conf parsing and refactor their processing
    - MINOR: mailers: Use pools to allocate email alerts and its tcpcheck_rules
    - MINOR: standard: Add memvprintf function
    - MINOR: log: Save alerts and warnings emitted during HAProxy startup
    - MINOR: cli: Add "show startup-logs" command
    - MINOR: startup: Extend the scope the MODE_STARTING flag
    - MINOR: threads: Prepare makefile to link with pthread
    - MINOR: threads: Add THREAD_LOCAL macro
    - MINOR: threads: Add atomic-ops and plock includes in import dir
    - MEDIUM: threads: Add hathreads header file
    - MINOR: threads: Add mechanism to register per-thread init/deinit functions
    - MINOR: threads: Add nbthread parameter
    - MEDIUM: threads: Adds a set of functions to handle sync-point
    - MAJOR: threads: Start threads to experiment multithreading
    - MINOR: threads: Define the sync-point inside run_poll_loop
    - MEDIUM: threads/buffers: Define and register per-thread init/deinit functions
    - MEDIUM: threads/chunks: Transform trash chunks in thread-local variables
    - MEDIUM: threads/time: Many global variables from time.h are now thread-local
    - MEDIUM: threads/logs: Make logs thread-safe
    - MEDIUM: threads/pool: Make pool thread-safe by locking all access to a pool
    - MAJOR: threads/fd: Make fd stuffs thread-safe
    - MINOR: threads/fd: Add a mask of threads allowed to process on each fd in fdtab array
    - MEDIUM: threads/fd: Initialize the process mask during the call to fd_insert
    - MINOR: threads/fd: Process cached events of FDs depending on the process mask
    - MINOR: threads/polling: pollers now handle FDs depending on the process mask
    - WIP: SQUASH WITH SYNC POINT
    - MAJOR: threads/task: handle multithread on task scheduler
    - MEDIUM: threads/signal: Add a lock to make signals thread-safe
    - MEDIUM: threads/listeners: Make listeners thread-safe
    - MEDIUM: threads/proxy: Add a lock per proxy and atomically update proxy vars
    - MEDIUM: threads/server: Make connection list (priv/idle/safe) thread-safe
    - MEDIUM: threads/server: Add a lock per server and atomically update server vars
    - MINOR: threads/server: Add a lock to deal with insert in updates_servers list
    - MEDIUM: threads/lb: Make LB algorithms (lb_*.c) thread-safe
    - MEDIUM: threads/stick-tables: handle multithreads on stick tables
    - MINOR: threads/sample: Change temp_smp into a thread local variable
    - MEDIUM: threads/http: Make http_capture_bad_message thread-safe
    - MINOR: threads/regex: Change Regex trash buffer into a thread local variable
    - MAJOR: threads/applet: Handle multithreading for applets
    - MAJOR: threads/peers: Make peers thread safe
    - MAJOR: threads/buffer: Make buffer wait queue thread safe
    - MEDIUM: threads/stream: Make streams list thread safe
    - MAJOR: threads/ssl: Make SSL part thread-safe
    - MEDIUM: threads/queue: Make queues thread-safe
    - MAJOR: threads/map: Make acls/maps thread safe
    - MEDIUM: threads/freq_ctr: Make the frequency counters thread-safe
    - MEDIUM: thread/vars: Make vars thread-safe
    - MEDIUM: threads/filters: Add init/deinit callback per thread
    - MINOR: threads/filters: Update trace filter to add _per_thread callbacks
    - MEDIUM: threads/compression: Make HTTP compression thread-safe
    - MEDIUM: threads/lua: Makes the jmpbuf and some other buffers local to the current thread.
    - MEDIUM: threads/lua: Add locks around the Lua execution parts.
    - MEDIUM: threads/lua: Ensure that the launched tasks runs on the same threads than me
    - MEDIUM: threads/lua: Cannot acces to the socket if we try to access from another thread.
    - MEDIUM: threads/xref: Convert xref function to a thread safe model
    - MEDIUM: threads/tasks: Add lock around notifications
    - MEDIUM: thread/spoe: Make the SPOE thread-safe
    - MEDIUM: thread/dns: Make DNS thread-safe
    - MINOR: threads: Add thread-map config parameter in the global section
    - MINOR: threads/checks: Add a lock to protect the pid list used by external checks
    - MINOR: threads/checks: Set the task process_mask when a check is executed
    - MINOR: threads/mailers: Add a lock to protect queues of email alerts
    - MEDIUM: threads/server: Use the server lock to protect health check and cli concurrency
    - MINOR: threads: Don't start when device a detection module is used
    - BUG/MEDIUM: threads: Run the poll loop on the main thread too
    - BUG/MINOR: threads: Add missing THREAD_LOCAL on static here and there
    - MAJOR: threads: Offically enable the threads support in HAProxy
    - BUG/MAJOR: threads/freq_ctr: fix lock on freq counters.
    - BUG/MAJOR: threads/time: Store the time deviation in an 64-bits integer
    - BUILD: stick-tables: silence an uninitialized variable warning
    - BUG/MINOR: dns: Fix SRV records with the new thread code.
    - MINOR: ssl: Remove the global allow-0rtt option.
    - CLEANUP: threads: replace the last few 1UL<<tid with tid_bit
    - CLEANUP: threads: rename process_mask to thread_mask
    - MINOR: h1: add a function to measure the trailers length
    - MINOR: threads: add a portable barrier for threads and non-threads
    - BUG/MAJOR: threads/freq_ctr: use a memory barrier to detect changes
    - BUG/MEDIUM: threads: Initialize the sync-point
    - MEDIUM: connection: start to introduce a mux layer between xprt and data
    - MINOR: connection: implement alpn registration of muxes
    - MINOR: mux: register the pass-through mux for any ALPN string
    - MEDIUM: session: use the ALPN token and proxy mode to select the mux
    - MINOR: connection: report the major HTTP version from the MUX for logging (fc_http_major)
    - MINOR: connection: introduce conn_stream
    - MINOR: mux: add more methods to mux_ops
    - MINOR: connection: introduce the conn_stream manipulation functions
    - MINOR: mux_pt: implement remaining mux_ops methods
    - MAJOR: connection : Split struct connection into struct connection and struct conn_stream.
    - MINOR: connection: make conn_stream users also check for per-stream error flag
    - MINOR: conn_stream: new shutr/w status flags
    - MINOR: conn_stream: modify cs_shut{r,w} API to pass the desired mode
    - MEDIUM: connection: make conn_sock_shutw() aware of lingering
    - MINOR: connection: add cs_close() to close a conn_stream
    - MEDIUM: mux_pt: make cs_shutr() / cs_shutw() properly close the connection
    - MEDIUM: connection: replace conn_full_close() with cs_close()
    - MEDIUM: connection: make mux->detach() release the connection
    - MEDIUM: stream: do not forcefully close the client connection anymore
    - MEDIUM: checks: exclusively use cs_destroy() to release a connection
    - MEDIUM: connection: add a destroy callback
    - MINOR: session: release the listener with the session, not the stream
    - MEDIUM: session: make use of the connection's destroy callback
    - CONTRIB: hpack: implement a reverse huffman table generator for hpack
    - MINOR: hpack: implement the HPACK Huffman table decoder
    - MINOR: hpack: implement the header tables management
    - MINOR: hpack: implement the decoder
    - MEDIUM: hpack: implement basic hpack encoding
    - MINOR: h2: centralize all HTTP/2 protocol elements and constants
    - MINOR: h2: create a very minimalistic h2 mux
    - MINOR: h2: expose tune.h2.header-table-size to configure the table size
    - MINOR: h2: expose tune.h2.initial-window-size to configure the window size
    - MINOR: h2: expose tune.h2.max-concurrent-streams to limit the number of streams
    - MINOR: h2: create the h2c struct and allocate its pool
    - MINOR: h2: create the h2s struct and the associated pool
    - MINOR: h2: handle two extra stream states for errors
    - MINOR: h2: add a frame header descriptor for incoming frames
    - MEDIUM: h2: allocate and release the h2c context on connection init/end
    - MEDIUM: h2: implement basic recv/send/wake functions
    - MEDIUM: h2: dynamically allocate the demux buffer on Rx
    - MEDIUM: h2: implement the mux buffer allocator
    - MINOR: h2: add the connection and stream flags listing the causes for blocking
    - MINOR: h2: add function h2s_id() to report a stream's ID
    - MINOR: h2: small function to know when the mux is busy
    - MINOR: h2: new function h2c_error to mark an error on the connection
    - MINOR: h2: new function h2s_error() to mark an error on a stream
    - MINOR: h2: add h2_set_frame_size() to update the size in a binary frame
    - MINOR: h2: new function h2_peek_frame_hdr() to retrieve a new frame header
    - MINOR: h2: add a few functions to retrieve contents from a wrapping buffer
    - MINOR: h2: add stream lookup function based on the stream ID
    - MINOR: h2: create dummy idle and closed streams
    - MINOR: h2: add the function to create a new stream
    - MINOR: h2: update the {MUX,DEM}_{M,D}ALLOC flags on buffer availability
    - MEDIUM: h2: start to consider the H2_CF_{MUX,DEM}_* flags for polling
    - MINOR: h2: also terminate the connection on shutr
    - MEDIUM: h2: properly consider all conditions for end of connection
    - MEDIUM: h2: wake the connection up for send on pending streams
    - MEDIUM: h2: start to implement the frames processing loop
    - MINOR: h2: add a function to send a GOAWAY error frame
    - MINOR: h2: match the H2 connection preface on init
    - MEDIUM: h2: enable connection polling for send when a cs wants to emit
    - MEDIUM: h2: enable reading again on the connection if it was blocked on stream buffer full
    - MEDIUM: h2: process streams pending for sending
    - MINOR: h2: send a real SETTINGS frame based on the configuration
    - MEDIUM: h2: detect the presence of the first settings frame
    - MINOR: h2: create a stream parser for the demuxer
    - MINOR: h2: implement PING frames
    - MEDIUM: h2: decode SETTINGS frames and extract relevant settings
    - MINOR: h2: lookup the stream during demuxing
    - MEDIUM: h2: honor WINDOW_UPDATE frames
    - MINOR: h2: implement h2_send_rst_stream() to send RST_STREAM frames
    - MINOR: h2: handle CONTINUATION frames
    - MEDIUM: h2: partial implementation of h2_detach()
    - MEDIUM: h2: unblock a connection when its current stream detaches
    - MEDIUM: h2: basic processing of HEADERS frame
    - MEDIUM: h2: don't use trash to decode headers!
    - MEDIUM: h2: implement the response HEADERS frame to encode the H1 response
    - MEDIUM: h2: send the H1 response body as DATA frames
    - MEDIUM: h2: skip the response trailers if any
    - MEDIUM: h2: properly continue to parse header block when facing a 1xx response
    - MEDIUM: h2: send WINDOW_UPDATE frames for connection
    - MEDIUM: h2: handle request body in DATA frames
    - MINOR: h2: handle RST_STREAM frames
    - MEDIUM: h2: send DATA+ES or RST_STREAM on shutw/shutr
    - MINOR: h2: use a common function to signal some and all streams.
    - MEDIUM: h2: handle GOAWAY frames
    - MINOR: h2: centralize the check for the idle streams
    - MINOR: h2: centralize the check for the half-closed(remote) streams
    - MEDIUM: h2: silently ignore frames higher than last_id after GOAWAY
    - MINOR: h2: properly reject PUSH_PROMISE frames coming from the client
    - MEDIUM: h2: perform a graceful shutdown on "Connection: close"
    - MEDIUM: h2: send a GOAWAY frame when dealing with an empty response
    - MEDIUM: h2: apply a timeout to h2 connections
    - BUG/MEDIUM: h2: fix incorrect timeout handling on the connection
    - MEDIUM: shctx: forbid shctx to read more than expected
    - MEDIUM: cache: configuration parsing and initialization
    - MEDIUM: cache: store objects in cache
    - MEDIUM: cache: deliver objects from cache
2017-10-31 23:18:29 +01:00
William Lallemand
77c1197bfb MEDIUM: cache: deliver objects from cache
Lookup objects in the cache and deliver them using the http-request
action "cache-use".
2017-10-31 21:17:19 +01:00
William Lallemand
4da3f8a1f2 MEDIUM: cache: store objects in cache
Store object in the cache. The cache use an shctx for storage.

It uses an http-response action to store the headers and a filter to
store the body. The http-response action is used in order to allow
modifications by other actions before caching.
2017-10-31 21:17:19 +01:00
William Lallemand
41db46035e MEDIUM: cache: configuration parsing and initialization
Parse a configuration section "cache" and a http-{response,request}
actions.

Example:

    listen frt
        mode http
        http-response cache-store foobar
        http-request cache-use foobar

    cache foobar
        total-max-size 4   # size in megabytes
2017-10-31 21:17:19 +01:00
William Lallemand
7217c46dfe MEDIUM: shctx: forbid shctx to read more than expected
Forbid shctx to read more than expected, it allows you to use a greater
value as a len with shctx_row_data_get(), the size of the destination
buffer for example.
2017-10-31 21:17:19 +01:00
Willy Tarreau
3f133570b8 BUG/MEDIUM: h2: fix incorrect timeout handling on the connection
Previous commit ea3928 (MEDIUM: h2: apply a timeout to h2 connections)
was wrong for two reasons. The first one is that if the client timeout
is not set, it's used as zero, preventing connections from establishing.
The second reason is that if the timeout triggers with active streams
(normally it should not since the task is supposed to be disabled), the
task is removed (h2c->task=NULL), and the last quitting stream might
try to dereference it.

Instead of doing this, we simply not register the task if there's no
timeout (it's useless) and we always control its presence in the streams.
2017-10-31 19:21:06 +01:00
Willy Tarreau
ea39282e85 MEDIUM: h2: apply a timeout to h2 connections
Till now there was no way to deal with a dead H2 connection. Now each
connection creates a task that wakes up to kill the connection. Its
timeout is constantly refreshed when there's some activity. In case
the timeout triggers, the best effort attempts are made at sending a
clean GOAWAY message before closing and signaling the streams.

The timeout is automatically disabled when there's an active stream on
the connection, and restarted when the last stream finishes. This way
it should not affect long sessions.
2017-10-31 18:16:19 +01:00
Willy Tarreau
a1349f0207 MEDIUM: h2: send a GOAWAY frame when dealing with an empty response
Given that we're processing data produced by haproxy, we know that the
situations where haproxy doesn't return anything are :
  - request timeout with option http-ignore-probes : there's no reason to
    hit this since we're creating the stream with the request into it ;

  - tcp-request content reject : this definitely means we want to kill the
    connection and abort keep-alive and any further processing ;

  - using /dev/null as the error file to hide an error

In practice it appears that using the abort on empty response as a hint to
trigger a connection close is very appropriate to continue to give the
control over the connection management. This patch thus tries to send a
GOAWAY frame with the max_id presented as the last stream ID, then sends
an RST_STREAM for the current stream. For the client, this means that the
connection must be shut down immediately after processing the last pending
streams and that the current stream is aborted. This way it's still possible
to force connections to be closed using tcp-request rules.
2017-10-31 18:16:19 +01:00
Willy Tarreau
af1e4f5167 MEDIUM: h2: perform a graceful shutdown on "Connection: close"
After some long brainstorming sessions, it appears that "Connection: close"
seems to be the best signal from the L7 layer to indicate the need to close
the connection. Indeed, in H1 it is only present in very rare cases (eg:
certain unrecoverable errors, some of which could remove it now by the way).
It will also be added when the L7 layer wants to force the connection to
terminate. By default when running in keep-alive mode it is not present.
It's worth mentionning that in H1 with persistent connections, we have sort
of a concurrency-1 mux and this header field is used the same way.

Thus here this patch detects "Connection: close" in response headers and
if seen, sends a GOAWAY frame with the highest possible ID so that the
client knows that it can quit whenever it wants to. If more aggressive
closures are needed in the future, we may decide to advertise the max_id
to abort after the current requests and better honor "http-request deny".
2017-10-31 18:16:19 +01:00
Willy Tarreau
1c661986a8 MINOR: h2: properly reject PUSH_PROMISE frames coming from the client
These ones deserve a connection error as per 5.1.
2017-10-31 18:16:19 +01:00
Willy Tarreau
c0da1964ba MEDIUM: h2: silently ignore frames higher than last_id after GOAWAY
For a graceful shutdown, the specs requries to discard frames with a
stream ID higher than the advertised last_id. (RFC7540#6.8). Well,
finally for now the code is disabled (see last page of #6.8). Some
frames need to be processed anyway to maintain the compression state
and the flow control window state, but we don't have any trivial way
to do this and ignore them at the same time. For the headers it's
the worst case where we can't parse headers frames without coming
from the streams, and we don't want to create such streams as we'd
have to abort them, and aborting would cause errors to flow back.

Possibly that a longterm solution might involve using some dummy
streams and dummy buffers for this and calling the parsers directly.
2017-10-31 18:16:19 +01:00
Willy Tarreau
f182a9a8b4 MINOR: h2: centralize the check for the half-closed(remote) streams
RFC7540#5.1 is pretty clear : "any frame other than WINDOW_UPDATE,
PRIORITY, or RST_STREAM in this state MUST be treated as a connection
error of type STREAM_CLOSED". Instead of dealing with this for each
and every frame type, let's do it once for all in the main demux loop.
2017-10-31 18:16:19 +01:00
Willy Tarreau
f65b80dd47 MINOR: h2: centralize the check for the idle streams
RFC7540#5.1 is pretty clear : "any frame other than HEADERS or PRIORITY
in this state MUST be treated as a connection error". Instead of dealing
with this for each and every frame type, let's do it once for all in the
main demux loop.
2017-10-31 18:16:19 +01:00
Willy Tarreau
e96b0922e9 MEDIUM: h2: handle GOAWAY frames
The ID is respected, and only IDs greater than the advertised last_id
are woken up, with a CS_FL_ERROR flag to signal that the stream is
aborted. This is necessary for a browser to abort a download or to
reject a bad response that affects the connection's state.
2017-10-31 18:16:19 +01:00
Willy Tarreau
23b92aa2bb MINOR: h2: use a common function to signal some and all streams.
Let's replace h2_wake_all_streams() with h2_wake_some_streams(), to
support signaling only streams by their ID (for GOAWAY frames) and
to pass the flags to add on the conn_stream.
2017-10-31 18:16:19 +01:00
Willy Tarreau
c7576eac46 MEDIUM: h2: send DATA+ES or RST_STREAM on shutw/shutr
When a stream sends a shutw, we send an empty DATA frame with the ES
flag set, except if no HEADERS were sent, in which case we rather send
RST_STREAM. On shutr(1) to abort a request, an RST_STREAM frame is sent
if the stream is OPEN and the stream is closed. Care is taken to switch
the stream's state accordingly and to avoid sending an ES bit again or
another RST once already done.
2017-10-31 18:16:19 +01:00
Willy Tarreau
cd234e9fb0 MINOR: h2: handle RST_STREAM frames
These ones are received when the browser aborts a page load, it's the
only moment we can abort the stream.
2017-10-31 18:16:19 +01:00
Willy Tarreau
454f905084 MEDIUM: h2: handle request body in DATA frames
Data frames are received and transmitted. The per-connection and
per-stream amount of data to ACK is automatically updated. Each
DATA frame is ACKed because usually the downstream link is large
and the upstream one is small, so it seems better to waste a few
bytes every few kilobytes to maintain a low ACK latency and help
the sender keep the link busy. The connection's ACK however is
sent at the end of the demux loop and at the beginning of the mux
loop so that a single aggregated one is emitted (connection
windows tend to be much larger than stream windows).

A future improvement would consist in sending a single ACK for
multiple subsequent DATA frames of the same stream (possibly
interleaved with window updates frames), but this is much trickier
as it also requires to remember the ID of the stream for which
DATA frames have to be sent.

Ideally in the near future we should chunk-encode the body sent
to HTTP/1 when there's no content length and when the request is
not a CONNECT. It's just uncertain whether it's the best option
or not for now.
2017-10-31 18:16:19 +01:00
Willy Tarreau
cc0b8c34a6 MEDIUM: h2: send WINDOW_UPDATE frames for connection
When it is detected that the number of received bytes is > 0 on the
connection at the end of the demux call or before starting to process
pending output data, an attempt is made at sending a WINDOW UPDATE on
the connection. In case of failure, it's attempted later.
2017-10-31 18:16:19 +01:00
Willy Tarreau
c199faf5bd MEDIUM: h2: properly continue to parse header block when facing a 1xx response
We still didn't handle the 1xx responses properly.
2017-10-31 18:16:19 +01:00
Willy Tarreau
9d89ac8f42 MEDIUM: h2: skip the response trailers if any
For now we don't build a HEADERS frame with them, but at least we remove
them from the response so that the L7 chunk parser inside isn't blocked
on these (often two) remaining bytes that don't want to leave the buffer.
It also ensures that trailers delivered progressively will correctly be
skipped.
2017-10-31 18:16:19 +01:00
Willy Tarreau
c652dbde9d MEDIUM: h2: send the H1 response body as DATA frames
The H1 response data are processed (either following content-length or
chunks) and emitted as H2 DATA frames. In the case of content-length,
the maximum size permitted by the mux buffer, the max frame size, the
connection's window and the stream's window it used to determine the
frame size. For chunked encoding, the same limitation applies, but in
addition, each chunk leads to a distinct frame. This could be improved
in the future to aggregate chunks into larger frames.

Streams blocked on the connection's flow control subscribe to the
connection's fctl_list to be woken up when the window opens again.

Streams blocked on their own flow control don't subscribe to anything,
they just sit waiting for window update frames to reopen the window.

The connection-close mode (without content-length) partially works thanks
to the fact that the SHUTW event leads to a close of the stream. In
practice an empty DATA frame should be sent in this case though.
2017-10-31 18:16:19 +01:00
Willy Tarreau
9e5ae1d721 MEDIUM: h2: implement the response HEADERS frame to encode the H1 response
This calls the h1 response parser and feeds the output through the hpack
encoder to produce stateless HPACK bytecode into an output chunk. For now
it's a bit naive but reasonably efficient.

The HPACK encoder relies on hpack_encode_header() so that the most common
response header fields are encoded based on the static header table. The
forbidden header field names (connection, proxy-connection, upgrade,
transfer-encoding, keep-alive) are dropped before calling the hpack
encoder.

A new flag (H2_CF_HEADERS_SENT) is set once such a frame is emitted. It
will be used to know if we can send an empty DATA+ES frame to use as a
shutdown() signal or if we have to use RST_STREAM.
2017-10-31 18:16:19 +01:00
Willy Tarreau
68dd9856ce MEDIUM: h2: don't use trash to decode headers!
The trash is already used by the hpack layer and for Huffman decoding,
it's unsafe to use here as a buffer and results in corrupted data. Use
a safely allocated trash instead.
2017-10-31 18:16:18 +01:00
Willy Tarreau
13278b44b1 MEDIUM: h2: basic processing of HEADERS frame
This takes care of creating a new h2s and a new conn_stream when a
HEADERS frame arrives. The recv() callback from the data layer is then
called to extract the frame into the stream's buffer. It is verified
that the stream ID is strictly greater than the known max stream ID.
And the last_id is updated if the current request is properly converted.
The streams are created in open or half-closed(remote) states.

For now there are some limitations :
  - frames without END_HEADERS are rejected (CONTINUATION not supported
    yet, will require some more changes so that the stream processor
    checks the H2 frame header by itself and steals the frames from the
    connection)
  - padding/stream_dep/priority are currently ignored
  - limited error handling, could be improved

But at least the request is properly decoded, transcoded and processed.
2017-10-31 18:16:18 +01:00
Willy Tarreau
45f752e037 MEDIUM: h2: unblock a connection when its current stream detaches
If a stream is killed for whatever reason and it happens to be the one
currently blocking the connection, we must unblock the connection and
enable polling again so that it can attempt to make progress. This may
happen for example on upload timeout, where the demux is blocked due to
a full stream buffer, and the stream dies on server timeout and quits.
2017-10-31 18:16:18 +01:00
Willy Tarreau
6093514933 MEDIUM: h2: partial implementation of h2_detach()
This does the very minimum required to release a stream and/or a connection
upon the stream's request. The only thing is that it doesn't kill the
connection unless it's already closed or in error or the stream ID reached
the one specified in GOAWAY frame. We're supposed to arm a timer to close
after some idle timeout but it's not done.
2017-10-31 18:16:18 +01:00
Willy Tarreau
61290ec774 MINOR: h2: handle CONTINUATION frames
For now we have nowhere to store partial header frames so we can't
handle CONTINUATION frames and we must reject them. In this case we
respond with a stream error of type INTERNAL_ERROR.
2017-10-31 18:16:18 +01:00
Willy Tarreau
27a84c90ce MINOR: h2: implement h2_send_rst_stream() to send RST_STREAM frames
This one sends an RST_STREAM for a given stream, using the current
demux stream ID. It's also used to send RST_STREAM for streams which
have lost their CS part (ie were aborted).
2017-10-31 18:16:18 +01:00
Willy Tarreau
26f95954fe MEDIUM: h2: honor WINDOW_UPDATE frames
Now they really increase the window size of connections and streams.
If a stream was not queued but requested to send, it means it was
flow-controlled so it's added again into the connection's send list.
2017-10-31 18:16:18 +01:00
Willy Tarreau
f3ee0697f3 MINOR: h2: lookup the stream during demuxing
Several stream-oriented functions will need to perform this lookup, so
better centralize it.
2017-10-31 18:16:18 +01:00
Willy Tarreau
3421aba3de MEDIUM: h2: decode SETTINGS frames and extract relevant settings
The INITIAL_WINDOW_SIZE and MAX_FRAME_SIZE settings are now extracted
from the settings frame, assigned to the connection, and attempted to
be propagated to all existing streams as per the specification. In
practice clients rarely update the settings after sending the first
stream, so the propagation will rarely be used. The ACK is properly
sent after the frame is completely parsed.
2017-10-31 18:16:18 +01:00
Willy Tarreau
cf68c787ae MINOR: h2: implement PING frames
Now we can detect and properly parse PING frames as well as emit a
response containing the same payload.
2017-10-31 18:16:18 +01:00