Commit Graph

3172 Commits

Author SHA1 Message Date
Willy Tarreau
820b391260 MINOR: hpack: provide a function to encode an HTTP scheme
The new function hpack_encode_scheme() supports encoding a scheme
into the ":scheme" header. It knows about "https" and "http" which use
a single byte, and falls back to literal encoding for other ones.
2018-12-11 09:07:02 +01:00
Willy Tarreau
39c80ebff0 MINOR: hpack: provide a function to encode an HTTP method
The new function hpack_encode_method() supports encoding a method.
It knows about GET and POST which use a single byte, and falls back
to literal encoding for other ones.
2018-12-11 09:07:02 +01:00
Willy Tarreau
8895367fb1 MINOR: hpack: provide new functions to encode the ":status" header
This header exists with 7 different values, it's worth taking them
into account for the encoding, hence these functions. One of them
makes use of an integer only and computes the 3 output bytes in case
of literal. The other one benefits from the knowledge of an existing
string, which for example exists in the case of H1 to H2 encoding.
2018-12-11 09:07:02 +01:00
Willy Tarreau
bd5659bbe1 MINOR: hpack: provide a function to encode a long indexed header
For long header values whose index is known, hpack_encodde_long_idx()
may now be used. This function emits the short index and follows with
the header's value.
2018-12-11 09:07:01 +01:00
Willy Tarreau
30eb809fdb MINOR: hpack: provide a function to encode a short indexed header
Most direct calls to HPACK functions are made to encode short header
fields like methods, schemes or statuses, whose lengths and indexes
are known. Let's have a small function to do this.
2018-12-11 09:06:46 +01:00
Willy Tarreau
bad0a381d3 MINOR: hpack: move the length computation and encoding functions to .h
We'll need these functions from other inline functions, let's make them
accessible. len_to_bytes() was renamed to hpack_len_to_bytes() since it's
now exposed.
2018-12-11 09:06:46 +01:00
Willy Tarreau
2df026fbce CLEANUP: hpack: no need to include chunk.h, only include buf.h
Chunk.h used to be needed to declare the struct chunk which we don't
use anymore, let's fall back to the lighter buf.h
2018-12-11 09:06:06 +01:00
Willy Tarreau
071d4b31ff MINOR: compiler: add a new macro ALREADY_CHECKED()
This macro may be used to block constant propagation that lets the compiler
detect a possible NULL dereference on a variable resulting from an explicit
assignment in an impossible check. Sometimes a function is called which does
safety checks and returns NULL if safe conditions are not met. The place
where it's called cannot hit this condition and dereferencing the pointer
without first checking it will make the compiler emit a warning about a
"potential null pointer dereference" which is hard to work around. This
macro "washes" the pointer and prevents the compiler from emitting tests
branching to undefined instructions. It may only be used when the developer
is absolutely certain that the conditions are guaranteed and that the
pointer passed in argument cannot be NULL by design.

A typical use case is a top-level function doing this :

     if (frame->type == HEADERS)
        parse_frame(frame);

Then parse_frame() does this :

    void parse_frame(struct frame *frame)
    {
        const char *frame_hdr;

        frame_hdr = frame_hdr_start(frame);
        if (*frame_hdr == FRAME_HDR_BEGIN)
            process_frame(frame);
    }

and :

    const char *frame_hdr_start(const struct frame *frame)
    {
        if (frame->type == HEADERS)
            return frame->data;
        else
            return NULL;
    }

Above parse_frame() is only called for frame->type == HEADERS so it will
never get a NULL in return from frame_hdr_start(). Thus it's always safe
to dereference *frame_hdr since the check was already performed above.
It's then safe to address it this way instead of inventing dummy error
code paths that may create real bugs :

    void parse_frame(struct frame *frame)
    {
        const char *frame_hdr;

        frame_hdr = frame_hdr_start(frame);
        ALREADY_CHECKED(frame_hdr);
        if (*frame_hdr == FRAME_HDR_BEGIN)
            process_frame(frame);
    }
2018-12-08 15:27:03 +01:00
Willy Tarreau
d6735d611e MEDIUM: ist: use local conversion arrays to case conversion
Calling tolower/toupper for each character is slow, a lookup into a
256-byte table is cheaper, especially for common characters used in
header field names which all fit into a cache line. Let's create these
two variables marked weak so that they're included only once.
2018-12-07 13:25:59 +01:00
Willy Tarreau
3f2d696d72 MINOR: ist: add functions to copy/uppercase/lowercase into a buffer or string
The ist functions were missing functions to copy an IST into a target
buffer, making some code have to resort to memcpy(), which tends to be
overkill for small strings, that the compiler cannot guess. In addition
sometimes there is a need to turn a string to lower or upper case so it
had to be overwritten after the operation.

This patch adds 6 functions to copy an ist to a buffer, as binary or as a
string (i.e. a zero is or is not appended), and optionally to apply a
lower case or upper case transformation on the fly.

A number of tests were performed to optimize the processing for small
strings. The loops are marked unlikely to dissuade the compilers from
over-optimizing them and switching to SIMD instructions. The lower case
or upper case transformations used to rely on external functions for
each character and to crappify the code due to clobbered registers,
which is not acceptable when we know that only a certain class of chars
has to be transformed, so the test was open-coded.
2018-12-07 13:25:59 +01:00
Olivier Houchard
d247be0620 BUG/MEDIUM: connections: Split CS_FL_RCV_MORE into 2 flags.
CS_FL_RCV_MORE is used in two cases, to let the conn_stream
know there may be more data available, and to let it know that
it needs more room. We can't easily differentiate between the
two, and that may leads to hangs, so split it into two flags,
CS_FL_RCV_MORE, that means there may be more data, and
CS_FL_WANT_ROOM, that means we need more room.

This should not be backported.
2018-12-06 16:36:05 +01:00
Willy Tarreau
adc7f3edd2 BUG/MEDIUM: stream-int: don't attempt to receive if the connection is not established
If we try to receive before the connection is established, we lose the
send event and are not woken up anymore once the connection is established.
This was diagnosed by Olivier.

No backport is needed.
2018-12-06 15:25:58 +01:00
Willy Tarreau
a3b62d374a MINOR: stream-int: add a new blocking condition on the remote connection
There are some situations where we need to wait for the other side to
be connected. None of the current blocking flags support this. It used
to work more or less by accident using the old flags. Let's add a new
flag to mention we're blocking on this, it's removed by si_chk_rcv()
when a connection is established. It should be enough for now.
2018-12-06 15:24:01 +01:00
William Lallemand
27f3fa56f5 BUG/MEDIUM: mworker: stop every tasks in the master
The master is not supposed to run (at the moment) any task before the
polling loop, the created tasks should be run only in the workers but in
the master they should be disabled or removed.

No backport needed.
2018-12-06 14:12:58 +01:00
Christopher Faulet
aa75b3d2d5 CLEANUP: htx: Fix indentation here and there in HTX files 2018-12-05 17:33:14 +01:00
Christopher Faulet
b2aedea142 MEDIUM: channel/htx: Add functions for forward HTX data
To ease the fast forwarding and the infinte forwarding on HTX proxies, 2
functions have been added to let the channel be almost aware of the way data are
stored in its buffer. By calling these functions instead of legacy ones, we are
sure to forward the right amount of data.
2018-12-05 17:29:30 +01:00
Christopher Faulet
27ba2dc6d6 MEDIUM: htx: Rework conversion from a buffer to an htx structure
Now, the function htx_from_buf() will set the buffer's length to its size
automatically. In return, the caller should call htx_to_buf() at the end to be
sure to leave the buffer hosting the HTX message in the right state. When the
caller can use the function htxbuf() to get the HTX message without any update
on the underlying buffer.
2018-12-05 17:10:16 +01:00
Willy Tarreau
3906e22f6f MINOR: htx: add buf_room_for_htx_data() to help optimize buffer transfers
The small HTX overhead is enough to make the system perform multiple
reads and unaligned memory copies. Here we provide a function whose
purpose is to reduce the apparent room in a buffer by the size of the
overhead for DATA blocks, which is the struct htx plus 2 blocks (one
for DATA, one for the end of message so that small blocks can fit at
once). The muxes using HTX will be encouraged to use this one instead
of b_room() to compute the available buffer room and avoid filling
their demux buf with more data than can fit at once into the HTX
buffer.
2018-12-05 10:57:42 +01:00
Willy Tarreau
8ae4235f94 MINOR: htx: make htx_from_buf() adjust the size only on new buffers
This one is used a lot during transfers, let's avoid resetting its
size when there are already data in the buffer since it implies the
size is correct.
2018-12-05 10:57:42 +01:00
Christopher Faulet
c59ff23804 MINOR: htx: Rename functions htx_*_to_str() to be H1 specific
"_to_h1" suffix is now used because these function produce H1 strings. It avoids
any ambiguity on the output format.
2018-12-04 05:51:37 +01:00
Joseph Herlant
75a323f04e CLEANUP: Fix a typo in the listener subsystem
Fixes a typo in the code comment of the listener subsystem.
2018-12-02 18:43:28 +01:00
Joseph Herlant
f69b807fa4 CLEANUP: Fix typos in the file descriptor subsystem
Fixes 2 typos in the code comment of the file descriptor subsystem.
2018-12-02 18:43:25 +01:00
Joseph Herlant
0b75e63dc5 CLEANUP: Fix a typo in the checks header file
Fixes a typo in the code comments of the checks header file.
2018-12-02 18:43:21 +01:00
Joseph Herlant
eeac3c722f CLEANUP: Fix a typo in the protocol header file
Fixes a typo in the code comments of the header file holding the general
protocol primitives.
2018-12-02 18:42:49 +01:00
Joseph Herlant
8a95a6e5ed CLEANUP: Fix a typo in the connection subsystem
Fixes a typo in the code comments of the connection subsystem.
2018-12-02 18:42:12 +01:00
Joseph Herlant
41abef77cb CLEANUP: Fix a typo in the mini-clist header
Fixes a typo in the code comments of the mini-clist header.
2018-12-02 18:38:15 +01:00
Joseph Herlant
30bc509c40 CLEANUP: Fix typos in the h1 subsystem
Fixes typos in the code comments of the h1 subsystem.
2018-12-02 18:38:02 +01:00
Joseph Herlant
be7619aaca CLEANUP: Fix typo in the chunk headers file
Fix a typo detected in the chunk.h header file's code comments.
2018-12-02 18:37:56 +01:00
Joseph Herlant
c42c0e9969 CLEANUP: fix typos in the htx subsystem
Fix typos detected in the code comments of the htx subsystem.
2018-12-02 18:37:50 +01:00
Olivier Houchard
0c18a6fe34 MEDIUM: servers: Add a way to keep idle connections alive.
Add a new keyword for servers, "idle-timeout". If set, unused connections are
kept alive until the timeout happens, and will be picked for reuse if no
other connection is available.
2018-12-02 18:16:53 +01:00
Olivier Houchard
8defe4b51a MINOR: mux: add a "max_streams" method.
Add a new method to muxes, "max_streams", that returns the max number of
streams the mux can handle. This will be used to know if a mux is in use
or not.
2018-12-02 17:48:32 +01:00
Olivier Houchard
f3e65b086d MINOR: connection: Fix a comment.
Connections can now have an owner for outgoing connections, so update
the comment tu reflect that.
2018-12-02 17:48:28 +01:00
Willy Tarreau
1329b5be71 MINOR: h2: add new functions to produce an HTX message from an H2 response
The new function h2_prepare_htx_stsline() produces an HTX response message
from an H2 response presented as a list of header fields.
2018-12-02 13:30:17 +01:00
Willy Tarreau
3fbea1d8d0 MINOR: server: the mux_proto entry in the server is const
Same as previous commit. We'll have to update this one soon, let's
avoid any cast and mark it const as it really is.
2018-12-02 13:12:16 +01:00
Willy Tarreau
5fc311c001 MINOR: connection: create conn_get_best_mux_entry()
We currently have conn_get_best_mux() to return the best mux for a
given protocol name, side and proxy mode. But we need the mux entry
as well in order to fix the bind_conf and servers at the end of the
config parsing. Let's split the function in two parts. It's worth
noting that the <conn> argument is never used anymore so this part
is eligible to some cleanup.
2018-12-02 13:12:16 +01:00
Willy Tarreau
a004ae3e66 MINOR: listener: the mux_proto entry in the bind_conf is const
We'll have to update this one soon, let's avoid any cast and mark it
const as it really is.
2018-12-02 13:12:15 +01:00
Willy Tarreau
6deb4129de MINOR: h2: implement H2->HTX request header frame transcoding
Till now we could only produce an HTTP/1 request from a list of H2
request headers. Now the new function h2_make_htx_request() does the
same but using the HTX encoding instead, while respecting the H2
semantics. The code is not much different from the first version,
only the encoding differs.

For now it's not used.
2018-12-01 17:38:32 +01:00
Christopher Faulet
75bc913d23 MAJOR: filters: Adapt filters API to be compatible with the HTX represenation
First, to be called on HTX streams, a filter must explicitly be declared as
compatible by setting the flag STRM_FLT_FL_HAS_FILTERS on the filter's config at
HAProxy startup. This flag is checked when a filter implementation is attached
to a stream.

Then, some changes have been made on HTTP callbacks. The callback http_payload
has been added to filter HTX data. It will be called on HTX streams only. It
replaces the callbacks http_data, http_chunk_trailers and http_forward_data,
called on legacy HTTP streams only and marked as deprecated. The documention
(once updated)) will give all information to implement this new callback. Other
HTTP callbacks will be called for HTX and HTTP legacy streams. So it is the
filter's responsibility to known which kind of data it handles. The macro
IS_HTX_STRM should be used in such cases.

There is at least a noticeable changes in the way data are forwarded. In HTX,
after the call to the callback http_headers, all the headers are considered as
forwarded. So, in http_payload, only the body and eventually the trailers will
be filtered.
2018-12-01 17:37:27 +01:00
Christopher Faulet
e44769b4fa MINOR: mux-h1: Capture bad H1 messages
First of all, an dedicated error snapshot, h1_snapshot, has been added. It
contains more or less the some info than http_snapshot but adapted for H1
messages. Then, the function h1_capture_bad_message() has been added to capture
bad H1 messages. And finally, the function h1_show_error_snapshot() is used to
dump these errors. Only Headers or data parsing are captured.
2018-12-01 17:37:27 +01:00
Christopher Faulet
a7b677cd0d MEDIUM: proto_htx: Convert all HTTP error messages into HTX
During startup, after the configuration parsing, all HTTP error messages
(errorloc, errorfile or default messages) are converted into HTX messages and
stored in dedicated buffers. We use it to return errors in the HTX analyzers
instead of using ugly OOB blocks.
2018-12-01 17:37:27 +01:00
Christopher Faulet
b2db4fa016 MINOR: htx: Add BODYLESS flags on the HTX start-line and the HTTP message
the flags HTX_SL_F_BODYLESS and HTTP_MSGF_BODYLESS have been added. These flags
are set when the corresponding HTTP message has no body at all.
2018-12-01 17:37:27 +01:00
Christopher Faulet
f1ba18d7b3 MEDIUM: htx: Don't rely on h1_sl anymore except during H1 header parsing
Instead, we now use the htx_sl coming from the HTX message. It avoids to have
too H1 specific code in version-agnostic parts. Of course, the concept of the
start-line is higly influenced by the H1, but the structure htx_sl can be
adapted, if necessary. And many things depend on a start-line during HTTP
analyzis. Using the structure htx_sl also avoid boring conversions between HTX
version and H1 version.
2018-12-01 17:37:27 +01:00
Christopher Faulet
54483df5ba MINOR: htx: Add the start-line offset for the HTX message in the HTX structure
If there is no start-line, this offset is set to -1. Otherwise, it is the
relative address where the start-line is stored in the data block. When the
start-line is added, replaced or removed, this offset is updated accordingly. On
remove, if the start-line is no set and if the next block is a start-line, the
offset is updated. Finally, when an HTX structure is defragmented, the offset is
also updated accordingly.
2018-12-01 17:37:27 +01:00
Christopher Faulet
570d1614fa MEDIUM: htx: Change htx_sl to be a struct instead of an union
The HTX start-line is now a struct. It will be easier to extend, if needed. Same
info can be found, of course. In addition it is now possible to set flags on
it. It will be used to set some infos about the message.

Some macros and functions have been added in proto/htx.h to help accessing
different parts of the start-line.
2018-12-01 17:37:27 +01:00
Christopher Faulet
14e88252f2 MINOR: htx: Add a function to find the HTX block corresponding to a data offset
The function htx_find_blk() returns the HTX block containing data with a given
offset, relatively to the beginning of the HTX message. It is a good way to skip
outgoing data and find the first HTX block not already processed.
2018-12-01 17:37:27 +01:00
Christopher Faulet
d16b0a7b2d MINOR: htx: Add function to iterate on an HTX message using HTX blocks
the functions htx_get_next() and htx_get_prev() are used to iterate on an HTX
message using blocks position. With htx_get_next_blk() and htx_get_prev_blk(),
it is possible to do the same, but with HTX blocks. Of course, internally, we
rely on position's versions to do so. But it is handy for callers to not take
care of the blocks position.
2018-12-01 17:37:27 +01:00
Christopher Faulet
24ed835129 MINOR: htx: Add function to add an HTX block just before another one
The function htx_add_data_before() can be used to add an HTX block before
another one. For instance, it could be used to add some data before the
end-of-message marker.
2018-12-01 17:37:27 +01:00
Christopher Faulet
3bc1b11dae MEDIUM: conn_stream: Add a way to get mux's info on a CS from the upper layer
Time to time, the need arises to get some info owned by the multiplexer about a
connection stream from the upper layer. Today we really need to get some dates
and durations specific to the conn_stream. It is only true for the mux H1 and
H2. Otherwise it will be impossible to have correct times reported in the logs.

To do so, the structure cs_info has been defined to provide all info we ever
need on a conn_stream from the upper layer. Of course, it is the first step. So
this structure will certainly envloved. But for now, only the bare minimum is
referenced. On the mux side, the callback get_cs_info() has been added in the
structure mux_ops. Multiplexers can now implement it, if necessary, to return a
pointer on a structure cs_info. And finally, the function si_get_cs_info()
should be used from the upper layer. If the stream interface is not attached to
a connection stream, this function returns NULL, likewise if the callback
get_cs_info() is not defined for the corresponding mux.
2018-12-01 17:37:27 +01:00
Willy Tarreau
c01ed9ff20 MINOR: htx: add a function to cut the beginning of a DATA block
htx_cut_data_blk() is used to cut the beginning of a DATA block after a
part of it was tranferred. It simply advances the address, reduces the
advertised length and updates the htx's total data count.
2018-12-01 17:36:59 +01:00
Willy Tarreau
d3c49d17dc BUG/MINOR: connection: report mux modes when HTX is supported
It looks like we forgot to report HTX when listing the muxes and their
respective protocols, leading to "NONE" being displayed. Let's report
"HTX" and "HTTP|HTX" since both will exist. Also fix a minor typo in
the output message.
2018-12-01 17:33:35 +01:00