Commit Graph

7936 Commits

Author SHA1 Message Date
Willy Tarreau
fd9419d560 MINOR: http: remove the pointer to the error snapshot in http_capture_bad_message()
It's not needed anymore as we know the side thanks to the channel. This
will allow the proxy generic code to better manage the error snapshots.
2018-09-07 18:36:04 +02:00
Willy Tarreau
ef3ca73fc3 MINOR: http: make the HTTP error capture rely on the generic proxy code
Now that we have a generic error capture function, let's simplify
http_capture_bad_message() to make use of it. At this point the API
is not changed at all, but it could be further simplified.
2018-09-07 18:36:04 +02:00
Willy Tarreau
75fb65a51f MINOR: proxy: add a new generic proxy_capture_error()
This function now captures an error regardless of its side and protocol.
The caller must pass a number of elements and may pass a protocol-specific
structure and a callback to display it. Later this function may deal with
more advanced allocation techniques to avoid allocating as many buffers
as proxies.
2018-09-07 18:36:04 +02:00
Willy Tarreau
7ccdd8dad9 MEDIUM: snapshot: implement a show() callback and use it for HTTP
The HTTP dumps are now configurable in the code : "show errors" now
calls a protocol-specific function to emit the decoded output. For
now only HTTP is implemented.
2018-09-07 18:36:01 +02:00
Willy Tarreau
0b5b480594 MEDIUM: snapshot: start to reorder the HTTP snapshot output a little bit
The output of "show errors" was slightly reordered to split the HTTP part
in a single chunk_appendf() call. The useless buffer total input was
replaced to report the buffer's start offset, which is the offset in the
stream of the first input byte (thus not counting output). Also it was
the opportunity to stop calling the stream "session".
2018-09-07 17:48:14 +02:00
Willy Tarreau
7480f323ff MINOR: snapshot: split the error snapshots into common and proto-specific parts
The idea will be to make the error snapshot feature accessible to other
protocols than just HTTP. This patch only introduces an "http_snapshot"
structure and renames a few fields to make things more explicit. The
HTTP part was installed inside a union so that we can easily add more
protocols in the future.
2018-09-07 16:13:45 +02:00
Willy Tarreau
5865a8fe69 MINOR: snapshot: restart on the event ID and not the stream ID
The snapshots have the ability to restart a partial dump and they use
the stream ID as the restart point. Since it's purely HTTP, let's use
the event ID instead.
2018-09-07 15:00:43 +02:00
Willy Tarreau
e9e878a056 BUG/MINOR: http/threads: atomically increment the error snapshot ID
Let's use an atomic increment for the error snapshot, as we'd rather
not assign the same ID to two errors happening in parallel. It's very
unlikely that it will ever happen though.

This patch must be backported to 1.8 with the other one it relies on
("MINOR: thread: implement HA_ATOMIC_XADD()").
2018-09-07 11:31:58 +02:00
Baptiste Assmann
044fd5bc2c BUG/MINOR: dns: check and link servers' resolvers right after config parsing
On the Mailing list, Marcos Moreno reported that haproxy configuration
validation (through "haproxy -c cfgfile") does not detect when a
resolvers section does not exist for a server.
That said, this checking is done after HAProxy has started up.

The problem is that this can create production issue, since init
script can't detect the problem before starting / reloading HAProxy.

To fix this issue, this patch registers the function which validates DNS
configuration validity and run it right after configuration parsing is
finished (through cfg_register_postparser()).
Thanks to it, now "haproxy -c cfgfile" will fail when a server
points to a non-existing resolvers section (or any other validation made
by the function above).

Backport status: 1.8
2018-09-06 19:41:30 +02:00
Olivier Houchard
54620523e2 MINOR: log: One const should be enough.
"const const" doesn't bring much more constness, so only use one.
2018-09-06 18:52:15 +02:00
Willy Tarreau
57f8185625 MINOR: connection: add new function conn_is_back()
This function returns true if the connection is a backend connection
and false if it's a frontend connection.
2018-09-06 14:52:21 +02:00
Willy Tarreau
6ac98ac1be MINOR: connection: add new function conn_get_proxy()
This function returns the proxy associated to a connection. For front
connections it returns the frontend, and for back connections it
returns the backend. This will be used to retrieve some configuration
parameters from within a mux.
2018-09-06 11:48:44 +02:00
Willy Tarreau
be373150c7 MINOR: connection: make the initialization more consistent
Sometimes a connection is prepared before the target is set, sometimes
after. There's no real rule since the few functions involved operate on
different and independent fields. Soon we'll benefit from knowing the
target at the connection layer, in order to figure the associated proxy
and retrieve the various parameters (timeouts etc). This patch slightly
reorders a few calls to conn_prepare() so that we can make sure that the
target is always known to the mux.
2018-09-06 11:45:30 +02:00
Willy Tarreau
950a8a6fde BUG/MINOR: h1: fix buffer shift after realignment
Commit 5e74b0b ("MEDIUM: h1: port to new buffer API.") introduced a
minor bug by which a buffer's head could stay shifted by the amount
of removed CRLF if it started with empty lines. This would cause the
second request (or response) not to work until it would receive a few
extra characters. This most only impacts requests sent by hand though.

This is purely 1.9, no backport is needed.
2018-09-06 10:48:15 +02:00
Willy Tarreau
22de8d3e01 MEDIUM: h2: produce some logs on early errors that prevent streams from being created
The h2 mux currently lacks some basic transparency. Some errors cause the
connection to be aborted but they couldn't be reported. With this patch,
almost all situations where an error will cause a stream or connection to
be aborted without the ability for an existing stream to report it will be
reported in the logs. This at least provides a solution to monitor the
activity and abnormal traffic.
2018-09-06 09:43:41 +02:00
Willy Tarreau
5383935856 MINOR: log: provide a function to emit a log for a session
The new function sess_log() only needs a session to emit a log. It will
ignore the parts that depend on the stream. It is usable to emit a log
to report early errors in muxes. These ones will typically mention
"<BADREQ>" for the request and 0 for the HTTP status code.
2018-09-06 09:43:41 +02:00
Willy Tarreau
09bb27cdea MEDIUM: log: make sess_build_logline() support being called with no stream
Till now it was impossible to emit logs from the lower layers only because
a stream was mandatory. From now on it will at least be possible to emit a
log to report a bad request or some timings for example. When the stream
is null, sess_build_logline() will use default values and will extract the
timing information from the session just like stream_new() does, so the
resulting log line is perfectly valid.

The termination state will indicate a proxy error during the request phase
since it is the only realistic use for such a call with no stream.
2018-09-06 09:43:06 +02:00
Willy Tarreau
5cacab63e1 MINOR: log: use zero as the request counter if there is no stream
When s==NULL we don't have any assigned request counter. Ideally we
should proceed exactly like when a stream is initialized and assign
a unique value. For now we only place it into a local variable.
2018-09-05 20:01:23 +02:00
Willy Tarreau
b8bc52522c MINOR: log: keep a copy of s->flags early to avoid a dereference
By placing s->flags into a local variable we'll be able to force it new
values when s is NULL.
2018-09-05 20:01:23 +02:00
Willy Tarreau
02fdf4f77b MINOR: log: use NULL for the unique_id if there is no stream
Now s->unique_id is used as NULL (not set) if s==NULL.
2018-09-05 20:01:23 +02:00
Willy Tarreau
abd71a5c2e MINOR: log: don't check the stream-int's conn_retries if the stream is NULL
Let's simply forget the conn_retries when there is no stream since we
haven't tried to connect yet.
2018-09-05 20:01:23 +02:00
Willy Tarreau
e1809dfdaf MINOR: log: be sure not to dereference a null stream for a target
The supported targets are either a server or an applet, so both are
NULL if the stream is NULL.
2018-09-05 20:01:23 +02:00
Willy Tarreau
d4f9166f4e MINOR: log: do not dereference a null stream to access captures
If the stream is null, let's simply not check captures. That's already
done if there is no capture.
2018-09-05 20:01:23 +02:00
Willy Tarreau
2393c5b6a9 MINOR: log: keep a copy of the backend connection early in sess_build_logline()
This way we can avoid dereferencing a possibly inexisting stream.
2018-09-05 20:01:23 +02:00
Willy Tarreau
26ffa8544d CLEANUP: log: make the low_level lf_{ip,port,text,text_len} functions take consts
These ones were abusively relying on variables making it hard to integrate
with const arguments.
2018-09-05 20:01:23 +02:00
Willy Tarreau
372ac5abff MINOR: log: don't unconditionally pick log info from s->logs
We'll soon support s==NULL so let's use an intermediary variable for the
logs structure. For now it only points to s->logs but will support a local
variable as an alternative later.
2018-09-05 20:01:23 +02:00
Willy Tarreau
56a91dddc6 MINOR: log: make sess_build_logline() not dereference a NULL stream for txn
If the stream is NULL, the txn is NULL as well. This condition is already
handled everywhere else.
2018-09-05 20:01:23 +02:00
Willy Tarreau
a21c0e60d2 MINOR: log: make the backend fall back to the frontend when there's no stream
This is already what happens before the backend is assigned, except that
now we don't need to dereference a NULL stream to figure this.
2018-09-05 20:01:23 +02:00
Willy Tarreau
43c538eab6 MINOR: log: move the log code to sess_build_logline() to add extra arguments
The current build_logline() can only be used with valid streams, which
means it is not suitable for use from muxes. We start by moving it into
another more generic function which takes the session as an argument,
to avoid complexifying all the internal API for jsut a few use cases.
This new function is not supposed to be called directly from outside so
we'll be able to instrument it to support several calling conventions.

For now the behaviour and conditions remain unchanged.
2018-09-05 20:01:23 +02:00
Willy Tarreau
ec3750c590 BUG/MAJOR: buffer: fix incorrect check in __b_putblk()
This function was split in two at commit f7d0447 ("MINOR: buffers:
split b_putblk() into __b_putblk()") but it's wrong, the first half's
length is not adjusted to the requested size so it copies more than
desired.

This is purely 1.9-specific, no backport is needed.
2018-09-05 20:01:14 +02:00
Willy Tarreau
a0d11b6fd5 BUG/MEDIUM: h2: fix risk of memory leak on malformated wrapped frames
While parsing a headers frame, if the frame is wrapped in the buffer
and needs to be unwrapped, it will be duplicated before being processed.
But if it contains certain combinations of invalid flags, the parser
returns without releasing the temporary buffer leading to a memory
leak.

This fix needs to be backported to 1.8.
2018-09-05 20:01:14 +02:00
Willy Tarreau
590a0514f2 BUG/MEDIUM: session: fix reporting of handshake processing time in the logs
The handshake processing time used to be stored per stream, which was
valid when there was exactly one stream per session. With H2 and
multiplexing it's not the case anymore and the reported handshake times
are wrong in the logs as it's computed between the TCP accept() and the
stream creation. Let's first move the handshake where it belongs, which
is the session.

However, this is not enough because we don't want to report an excessive
idle time either for H2 (since many requests use the connection).

So the solution used here is to have the stream retrieve sess->tv_accept
and the handshake duration when the stream is created, and let the mux
immediately reset them. This way, the handshake time becomes zero for the
second and subsequent requests in H2 (which was already the case in H1),
and the idle time exactly counts how long the connection remained unused
while it could be used, so in H1 it runs from the end of the previous
response and in H2 it runs from the end of the previous request since the
channel is already available.

This patch will need to be backported to 1.8.
2018-09-05 16:30:23 +02:00
Willy Tarreau
90a7c03ec0 BUG/MINOR: stream: use atomic increments for the request counter
The request counter is incremented when creating a new stream and when
resetting a stream, preparing for a new request. Unfortunately during
the thread migration this was missed, leading to non-atomic increments
in case threads are in use. The most visible side effect is that two
requests may have the same ID from time to time in the logs. However
the SPOE also uses this ID to route responses back to the stream so it
may also lead to occasional spurious SPOE timeouts.

Note that it still doesn't guarantee temporal unicity in the stream
identifiers since a long and a short connection could technically use
the same ID. The likeliness that this happens at the same time is almost
null (roughly threads*runqueue_depth/2^32 that it happens in the same
poll loop), but it will have to be addressed later anyway.

This patch must be backported to 1.8 with the other one it relies on
("MINOR: thread: implement HA_ATOMIC_XADD()").
2018-09-05 16:30:19 +02:00
Willy Tarreau
9378df89f6 MINOR: thread: implement HA_ATOMIC_XADD()
We've been missing it several times and now we'll need it to increment
a request counter. Let's do it once for all.

This patch will need to be backported to 1.8 with the associated fix.
2018-09-05 16:30:17 +02:00
Willy Tarreau
f16cb41d19 MINOR: tools: make date2str_log() take some consts
The "tm" and "date" field are not modified, they can be const instead
of forcing their callers to use vars.
2018-09-05 16:30:11 +02:00
Emmanuel Hocdet
9f9b0c6a7f BUG/MEDIUM: ECC cert should work with TLS < v1.2 and openssl >= 1.1.1
With openssl >= 1.1.1 and boringssl multi-cert is natively supported.
ECDSA/RSA selection is done and work correctly with TLS >= v1.2.
TLS < v1.2 have no TLSEXT_TYPE_signature_algorithms extension: ECC
certificate can't be selected, and handshake fail if no RSA cert
is present. Safe ECC certificate selection without client announcement
can be very tricky (browser compatibilty). The safer approach is to
select ECDSA certificate if no other certificate matches, like it is
with openssl < 1.1.1: certificate selection is only done via the SNI.

Thanks to Lukas Tribus for reporting this and analysing the problem.

This patch should be backported to 1.8
2018-09-04 17:47:10 +02:00
Baptiste Assmann
6d0f38f00d BUG/MEDIUM: dns/server: fix incomatibility between SRV resolution and server state file
Server state file has no indication that a server is currently managed
by a DNS SRV resolution.
And thus, both feature (DNS SRV resolution and server state), when used
together, does not provide the expected behavior: a smooth experience...

This patch introduce the "SRV record name" in the server state file and
loads and applies it if found and wherever required.

This patch applies to haproxy-dev branch only. For backport, a specific patch
is provided for 1.8.
2018-09-04 17:40:22 +02:00
Frédéric Lécaille
233afc70ba REGTEST/MINOR: lua: Add reg testing files for 70d318c. 2018-09-04 17:36:56 +02:00
Olivier Houchard
9e643ea172 BUG/MEDIUM: hlua: Don't call RESET_SAFE_LJMP if SET_SAFE_LJMP returns 0.
If SET_SAFE_LJMP returns 0, the spinlock is already unlocked, and lua_atpanic
is already set back to hlua_panic_safe, so there's no need to call
RESET_SAFE_LJMP.

This should be MFC'd into 1.8.
2018-08-31 16:14:58 +02:00
Frédéric Lécaille
dd51e53367 REGTEST/MINOR: Add a reg testing file for 3e60b11.
Thank you to Daniel for having worked on this one.
2018-08-31 13:42:56 +02:00
Frédéric Lécaille
54f2bcf22b BUG/MAJOR: thread: lua: Wrong SSL context initialization.
When calling ->prepare_srv() callback for SSL server which
depends on global "nbthread" value, this latter was not already parsed,
so equal to 1 default value. This lead to bad memory accesses.

Thank you to Pieter (PiBa-NL) for having reported this issue and
for having provided a very helpful reg testing file to reproduce
this issue (reg-test/lua/b00002.*).

Must be backported to 1.8.
2018-08-30 10:06:45 +02:00
Olivier Houchard
c7ffa91763 BUG/MEDIUM: stream_interface: try to call si_cs_send() earlier.
Call si_cs_send() at the beginning of si_cs_wake_cb(), instead of from
stream_int_notify-), so that if we get a connection error while trying to
send, the stream_interface will get SI_FL_ERR, the associated task will
be woken up, and the connection will be properly destroyed.

No backport needed.
2018-08-28 19:46:45 +02:00
Olivier Houchard
4501c3e099 MINOR: checks: Call wake_srv_chk() when we can finally send data.
Instead of calling __event_srv_chk_w, call wake_srv_chk(), which will then
either call tcpcheck_main() or __event_srv_chk_w().
Also make tcpcheck_main() subscribe if it can't send.
2018-08-28 19:43:57 +02:00
Olivier Houchard
594c8c5015 BUG/MEDIUM: hlua: Make sure we drain the output buffer when done.
In hlua_applet_tcp_fct(), drain the output buffer when the applet is done
running, every time we're called.
Overwise, there's a race condition, and the output buffer could be filled
after the applet ran, and as it is never cleared, the stream interface
will never be destroyed.

This should be backported to 1.8 and 1.7.
2018-08-28 16:18:34 +02:00
Patrick Hemmer
155e93e570 MINOR: Add srv_conn_free sample fetch
This adds the 'srv_conn_free([<backend>/]<server>)' sample fetch. This fetch
provides the number of available connections on the designated server.
2018-08-27 16:38:56 +02:00
Patrick Hemmer
4cdf3abaa0 MINOR: add be_conn_free sample fetch
This adds the sample fetch 'be_conn_free([<backend>])'. This sample fetch
provides the total number of unused connections across available servers in the
specified backend.
2018-08-27 14:10:16 +02:00
Patrick Hemmer
e3faf02581 BUG/MEDIUM: lua: reset lua transaction between http requests
Previously LUA code would maintain the transaction state between http
requests, resulting in things like txn:get_priv() retrieving data from
a previous request. This addresses the issue by ensuring the LUA state
is reset between requests.

Co-authored-by: Tim Düsterhus <tim@bastelstu.be>
2018-08-25 07:51:02 +02:00
Frédéric Lécaille
afbfa2a23f REGTEST/MINOR: Add a reg testing file for b406b87 commit. 2018-08-24 16:53:27 +02:00
Willy Tarreau
ad7f0ad1c3 BUG/MEDIUM: mux_pt: dereference the connection with care in mux_pt_wake()
mux_pt_wake() calls data->wake() which can return -1 indicating that the
connection was just destroyed. We need to check for this condition and
immediately exit in this case otherwise we dereference a just freed
connection. Note that this mainly happens on idle connections between
two HTTP requests. It can have random implications between requests as
it may lead a wrong connection's polling to be re-enabled or disabled
for example, especially with threads.

This patch must be backported to 1.8.
2018-08-24 15:48:59 +02:00
Frédéric Lécaille
b083c8316b REGEST/MINOR: Add reg testing files.
Reg testing files for a LUA bug fixed by commit 83ed5d5 ("BUG/MINOR:
lua: Bad HTTP client request duration.")
2018-08-24 14:50:30 +02:00