1
0
mirror of http://git.haproxy.org/git/haproxy.git/ synced 2024-12-22 12:30:07 +00:00
Commit Graph

8877 Commits

Author SHA1 Message Date
Christopher Faulet
7e245dfaca REGTEST: script: Add support of alternatives in requited options list
It is now possible to specify a list of "alternatives" for a required
option. This must be done by separating options by a pipe ('|'). A test will be
executed if at least one of them is available. For instance:

  #REQUIRED_OPTIONS=ZLIB|SLZ,LUA,OPENSSL

The function _findtest() has also been sligthly simplified.
2018-12-21 15:33:26 +01:00
Olivier Houchard
a8f6b43b07 MEDIUM: mux_h1: Implement h1_show_fd.
In the mux_h1, implement h1_show_fd, to give a bit more information about
each H1 stream, when using "show fd".
2018-12-21 15:33:26 +01:00
Christopher Faulet
d238ae3a9b BUG/MINOR: compression/htx: Don't add the last block of data if it is empty
In HTX, when the compression filter analyze the EOM, it flushes the compression
context and add the last block of compressed data. But, this block can be
empty. In this case, we must ignore it.
2018-12-21 15:33:26 +01:00
Christopher Faulet
c963eb2a1d BUG/MINOR: compression/htx: Don't compress responses with unknown body length
In HTX, if the body length of a response cannot be determined, we must not try
to compress it.
2018-12-21 15:33:16 +01:00
Nikhil Agrawal
2fa66c3b93 BUG/MEDIUM: dns: overflowed dns name start position causing invalid dns error
In dns_read_name() when dns name is used with compression and start position of
name is greater than 255 name read is incorrect and causes invalid dns error.
eg: 0xc11b c specifies name compression being used. 11b represent the start
position of name but currently we are using only 1b for start position.

This should be backported as far as 1.7.
2018-12-21 11:36:44 +01:00
J�r�me Magnin
8d4e7dc880 BUG/MEDIUM: dns: Don't prevent reading the last byte of the payload in dns_validate_response()
A regression was introduced with efbbdf72 BUG: dns: Prevent out-of-bounds
read in dns_validate_dns_response() as it prevented from taking into account
the last byte of the payload.  this patch aims at fixing it.

this must be backported in 1.8.
2018-12-20 17:13:02 +01:00
Willy Tarreau
645b33d233 BUG/MEDIUM: mux-h2: Don't forget to quit the send list on error reports
Similar to last fix, we need to quit the send list when reporting an
error via the send side.

This should be backported to 1.9.
2018-12-20 15:35:57 +01:00
Olivier Houchard
f29cd5c8a8 BUG/MEDIUM: h2: Don't forget to quit the sending_list if SUB_CALL_UNSUBSCRIBE.
In mux_h2_unsubscribe, don't forget to leave the sending_list if
SUB_CALL_UNSUBSCRIBE was set. SUB_CALL_UNSUBSCRIBE means we were about
to be woken up for writing, unless the mux was too full to get more data.
If there's an unsubscribe call in the meanwhile, we should leave the list,
or we may be put back in the send_list.

This should be backported to 1.9.
2018-12-20 12:24:43 +01:00
Frédéric Lécaille
15685c7912 REGTEST: Exclude freebsd target for some reg tests.
This patch excludes freebsd target for health-checks reg tests
which rely on TCP_DEFER_ACCEPT usage. This is specific to Linux.
2018-12-20 11:00:17 +01:00
Frédéric Lécaille
a702947bef REGTEST: Missing enclosing quotes for ${tmpdir} macro. 2018-12-20 11:00:17 +01:00
Frédéric Lécaille
9ffb88d3cf REGTEST: Be less Linux specific with a syslog regex.
This patch makes at least this test pass on FreeBSD systems.
The regex to be matched with syslog message was too much Linux specific.
2018-12-20 11:00:17 +01:00
Frédéric Lécaille
3b3a883171 REGTEST: Fix a typo about barrier type.
Fix a typo, furthermore there is no reason to use a barrier with "sock" as type
which are more varnish cache specific: used to synchronize the VCL.
2018-12-20 11:00:17 +01:00
Frédéric Lécaille
a3b4cbff7b REGTEST: Make reg-tests target support argument.
With this patch we can provide a list of argument to reg-tests target.
Useful to run reg tests for a list of VTC files like that:

    $ VARNISHTEST_PROGRAM=<...> make reg-tests reg-tests/checks/*.vtc
2018-12-20 10:37:32 +01:00
Christopher Faulet
8f16148df7 REGTEST: Add option to use HTX prefixed by the macro 'no-htx'
So some tests have been removed.
2018-12-20 10:37:32 +01:00
Christopher Faulet
6bd82cdbe6 REGTEST: script: Print only errors in the results report 2018-12-20 10:37:32 +01:00
Christopher Faulet
fa6798f97c REGTEST: script: Add the option --use-htx to enable the HTX in regtests
A check is done on HAProxy version to be sure it is supported. Internally, by
default, the macro 'no-htx' is set to "#". So in regtests, we can prefix
http-use-htx option line with this macro to inhibit the HTX. Concretly, this
line can be added in the HAProxy configuration:

   ${no-htx} option http-use-htx

When the option --use-htx is used, the macro 'no-htx' is unset.
2018-12-20 10:37:32 +01:00
Christopher Faulet
9c6df5ecb4 REGTEST: script: Add the option --keep-logs to keep all log directories
By default a log directory is kept only if the test fails. With this option it
is possible to always keep it. If this option is used, the result of all tests
are displayed (and not only the failing ones).
2018-12-20 10:37:24 +01:00
Christopher Faulet
2a7cf922c1 REGTEST: script: Add the option --debug to show logs on standard ouput
It implies the option --v. It is handy when you run the same test during a debug
session.
2018-12-20 10:33:18 +01:00
Christopher Faulet
8d0fdf5ef2 REGTEST: script: Add the option --clean to remove previous log direcotries
Running regtests several times leaves many direcotries. It is painful to remove
them by hand. This command do it for you. It ask a confirmation to proceed to be
sure. The template used to create log direcotries has been changed. The prefix
'haregtests-' has been added to help the cleanup function to find existing log
direcotries.
2018-12-20 10:33:18 +01:00
Christopher Faulet
1ecf0ea70a REGTEST: script: Evaluate the varnishtest command to allow quoted parameters
Instead of directly executing varnishtest command, we use a variable to build
the command line and we execute it with the 'eval' builtin. This way it is
possible to have quoted parameters. For instance:

  > ./scripts/run-regtests.sh --varnishtestparams "-Dmacro='some value' -n 10"

And the variable 'varnishtestparams' is also move at the end of the command
line, just before the list of test files. So it is possible to override all
default varnish options set by the script.
2018-12-20 10:33:18 +01:00
Christopher Faulet
8d67cf8187 REGTEST: script: Process script arguments before everything else
The help message is now in the function _help(). It exits the script with the
status 0 to not run regtests when the help is displayed. So it is also handy to
process script arguments before anything else. This avoids to start printing
messages related to regtests execution when it is not appropriate. Another
change, when it detects an invalid argument, the script exits with an error.
2018-12-20 10:33:18 +01:00
Christopher Faulet
fdafd9a3f8 REGTEST: Require the option LUA to run lua tests 2018-12-20 10:33:18 +01:00
Olivier Houchard
8ab8a6eee5 BUG/MAJOR: connections: Close the connection before freeing it.
In si_release_endpoint(), if the end point is a connection, because we don't
know which mux to use it, make sure we close the connection before freeing it,
or else, we'd have a fd left for polling, which would point to a now free'd
connection.

This should be backported to 1.9.
2018-12-20 06:03:14 +01:00
Willy Tarreau
822305067b [RELEASE] Released version 1.9.0
Released version 1.9.0 with the following main changes :
    - BUG/MEDIUM: compression: Use the right buffer pointers to compress input data
    - BUG/MINOR: mux_pt: Set CS_FL_WANT_ROOM when count is zero in rcv_buf() callback
    - BUG/MEDIUM: connection: Add a new CS_FL_ERR_PENDING flag to conn_streams.
    - CONTRIB: debug: teach the "flags" utility about new conn_stream flags
    - BUG/MEDIUM: stream-int: always clear CS_FL_WANT_ROOM before receiving
    - BUG/MEDIUM: mux-h2: also restart demuxing when data are pending in demux
    - BUG/MEDIUM: mux-h2: restart demuxing as soon as demux data are available
    - BUG/MEDIUM: h2: fix aggregated cookie length computation in HTX mode
    - MINOR: mux-h2: report more h2c, last h2s and cs information on "show fd"
    - CONTRIB: debug: report stream-int's flag SI_FL_CLEAN_ABRT
    - MINOR: cli/stream: add the conn_stream in "show sess" output
    - BUG/MINOR: mux-h2: don't report a fantom h2s in "show fd"
    - BUG/MINOR: cli/fd: don't isolate the thread for each individual fd
    - MINOR: objtype: report a few missing types in names and base pointers
    - BUG/MEDIUM: mux-h2: make sure to report synchronous errors after EOS
    - BUG/MEDIUM: mux-h2: report asynchronous errors in h2_wake_some_streams()
    - BUG/MEDIUM: mux-h2: make sure the demux also wakes streams up on errors
    - BUG/MINOR: mux-h1: report the correct frontend in error captures
    - BUG/MEDIUM: stream-int: also wake the stream up on end of transfer
    - MEDIUM: h2: properly check and deduplicate the content-length header in HTX
    - BUG/MEDIUM: stream: Forward the right amount of data before infinite forwarding
    - BUG/MINOR: proto_htx: Call the HTX version of the function managing client cookies
    - BUG/MEDIUM: lua/htx: Handle EOM in receive/get_line calls in HTTP applets
    - BUG/MINOR: lua: Return an error if a legacy HTTP applet doesn't send anything
    - MINOR: compression: Remove the thread_local variable buf_output
    - CLEANUP: connection: rename subscription events values and event field
    - CLEANUP: connection: rename conn->mux_ctx to conn->ctx
    - MINOR: connection: remove an unwelcome dependency on struct stream
    - CLEANUP: stream-int: consistently call the si/stream_int functions
    - BUG/MEDIUM: h1: Don't shutw/shutr the connection if we have keepalive.
    - BUG/MEDIUM: H2: Make sure htx is set even on empty frames.
    - BUG/MEDIUM: mux-h2: pass CS_FL_ERR_PENDING to h2_wake_some_streams()
    - MEDIUM: stream-int: always consider all CS errors on the send side
    - BUG/MEDIUM: h2: Make sure we don't set CS_FL_ERROR if there's still data.
    - CLEANUP: mux-h2: implement h2s_notify_{send,recv} to report events to subscribers
    - MINOR: mux-h2: add a new function h2s_alert() to call the data layer
    - BUG/MEDIUM: mux-h2: make use of h2s_alert() to report aborts
    - MINOR: connection: add cs_set_error() to set the error bits
    - CLEANUP: mux-h2: make use of cs_set_error()
    - BUG/MINOR: mux-h2: make sure we check the conn_stream in early data
    - BUG/MEDIUM: h2: Don't wait for flow control if the connection had a shutr.
    - MINOR: cli/show_fd: report that a connection is back or not
    - SCRIPTS: add the slack channel URL to the announce script
    - CLEANUP: remove my name and address from the copyright banner
    - DOC: mention in the readme that 1.9 is a stable version now
2018-12-19 19:13:17 +01:00
Willy Tarreau
f61afba7d2 DOC: mention in the readme that 1.9 is a stable version now
It's going to happen soon :-)
2018-12-19 19:11:18 +01:00
Willy Tarreau
d22d69bd58 CLEANUP: remove my name and address from the copyright banner
First, it's a pain to always have to think about updating this date,
second for a long time I've not been the only developer there, and third,
some users contact me hoping to get help that I can't deliver. It's about
time to redirect them to the main site where all the useful links should
be.
2018-12-19 19:07:04 +01:00
Willy Tarreau
d6cad12d1a SCRIPTS: add the slack channel URL to the announce script
It's just to provide the URL in the usual URLs when releasing.
2018-12-19 18:59:51 +01:00
Willy Tarreau
e9ca80711d MINOR: cli/show_fd: report that a connection is back or not
It simply adds a "back={0,1}" depending on the connection's side, it's
useful to debug.
2018-12-19 18:40:58 +01:00
Olivier Houchard
6dea2ee939 BUG/MEDIUM: h2: Don't wait for flow control if the connection had a shutr.
In h2_snd_buf(), if we couldn't send the data because of flow control, and
the connection got a shutr, then add CS_FL_ERROR (or CS_FL_ERR_PENDING). We
will never get any window update, so we will never be unlocked, anyway.

No backport is needed.
2018-12-19 18:35:40 +01:00
Willy Tarreau
fde287cc76 BUG/MINOR: mux-h2: make sure we check the conn_stream in early data
When dealing with early data we scan the list of stream to notify them.
We're not supposed to have h2s->cs == NULL here but it doesn't cost much
to make the scan more robust and verify it before notifying.

No backport is needed.
2018-12-19 18:33:16 +01:00
Willy Tarreau
ec988c7a0f CLEANUP: mux-h2: make use of cs_set_error()
It's cleaner than open-coding the conditions and error bits.
2018-12-19 18:13:52 +01:00
Willy Tarreau
e9f4301f0f MINOR: connection: add cs_set_error() to set the error bits
Depending on the CS_FL_EOS status, we either set CS_FL_ERR_PENDING
or CS_FL_ERROR at various places. Let's have a generic function to
do this.
2018-12-19 18:13:52 +01:00
Willy Tarreau
f830f018cf BUG/MEDIUM: mux-h2: make use of h2s_alert() to report aborts
If we had no pending read, it could be complicated to report an
RST_STREAM to a sender since we used to only report it via the
rx side if subscribed. Similarly in h2_wake_some_streams() we
now try all methods, hoping to catch all possible events.

No backport is needed.
2018-12-19 18:13:52 +01:00
Willy Tarreau
8b2757c339 MINOR: mux-h2: add a new function h2s_alert() to call the data layer
In order to report an error to the data layer, we have different ways
depending on the situation. At a lot of places it's open-coded and not
always correct. Let's create a new function h2s_alert() to handle this
task. It tries to wake on recv() first, then on send(), then using
wake().
2018-12-19 18:13:48 +01:00
Willy Tarreau
7e094451d0 CLEANUP: mux-h2: implement h2s_notify_{send,recv} to report events to subscribers
Till now we had to open-code all the manipulation of the wait_event,
let's use standarized functions for this and reduce the risk of bugs.
2018-12-19 18:11:35 +01:00
Olivier Houchard
251064b02d BUG/MEDIUM: h2: Make sure we don't set CS_FL_ERROR if there's still data.
In the mux h2, make sure we set CS_FL_ERR_PENDING and wake the recv task,
instead of setting CS_FL_ERROR, if CS_FL_EOS is not set, so if there's
potentially still some data to be sent.
2018-12-19 17:28:54 +01:00
Willy Tarreau
bddf7fc417 MEDIUM: stream-int: always consider all CS errors on the send side
We still have an issue with asynchronous errors, which is that while
they don't truncate reads anymore, they might be missed during a
send() attempt. This can happen for example when processing a request
followed by undesired data for which the stream doesn't try to receive,
while the send side experiences an error (transfer aborted by the client).
In this case we definitely want all send() attempts to fail as soon as
the error was reported, even if it's only pending. This way we leave an
opportunity to the stream interface to try to receive the last data
pending in the buffer but it cannot send anymore and knows that there
is an error when trying to do so.
2018-12-19 17:23:26 +01:00
Olivier Houchard
9117780bfd BUG/MEDIUM: mux-h2: pass CS_FL_ERR_PENDING to h2_wake_some_streams()
Commiy 8519357c ("BUG/MEDIUM: mux-h2: report asynchronous errors in
h2_wake_some_streams()") addressed an issue with synchronous errors
but forgot to fix the call places to also pass CS_FL_ERR_PENDING
instead of CS_FL_ERROR.

No backport is needed.
2018-12-19 17:06:49 +01:00
Olivier Houchard
2f30883793 BUG/MEDIUM: H2: Make sure htx is set even on empty frames.
When transfering data, make sure htx is set even on empty frames, or we
will never add a HTX_BLK_EOM block.
2018-12-19 17:00:14 +01:00
Olivier Houchard
d2e88c715d BUG/MEDIUM: h1: Don't shutw/shutr the connection if we have keepalive.
In h1_shutw() and h1_shutr(), don't attempt to shutdown() the connection
if we're using keepalive and the connection has no error, or we will close
the connection too soon.
2018-12-19 16:59:13 +01:00
Willy Tarreau
14bfe9af12 CLEANUP: stream-int: consistently call the si/stream_int functions
As long-time changes have accumulated over time, the exported functions
of the stream-interface were almost all prefixed "si_<something>" while
most private ones (mostly callbacks) were called "stream_int_<something>".
There were still a few confusing exceptions, which were addressed to
follow this shcme :
  - stream_sock_read0(), only used internally, was renamed stream_int_read0()
    and made static
  - stream_int_notify() is only private and was made static
  - stream_int_{check_timeouts,report_error,retnclose,register_handler,update}
    were renamed si_<something>.

Now it is clearer when checking one of these if it risks to be used outside
or not.
2018-12-19 15:25:43 +01:00
Willy Tarreau
94031d30d7 MINOR: connection: remove an unwelcome dependency on struct stream
There was a reference to struct stream in conn_free() for the case
where we're freeing a connection that doesn't have a mux attached.
For now we know it's always a stream, and we only need to do it to
put a NULL in s->si[1].end.

Let's do it better by storing the pointer to si[1].end in the context
and specifying that this pointer is always nulled if the mux is null.
This way it allows a connection to detach itself from wherever it's
being used. Maybe we could even get rid of the condition on the mux.
2018-12-19 14:36:29 +01:00
Willy Tarreau
3d2ee55ebd CLEANUP: connection: rename conn->mux_ctx to conn->ctx
We most often store the mux context there but it can also be something
else while setting up the connection. Better call it "ctx" and know
that it's the owner's context than misleadingly call it mux_ctx and
get caught doing suspicious tricks.
2018-12-19 14:13:07 +01:00
Willy Tarreau
4f6516d677 CLEANUP: connection: rename subscription events values and event field
The SUB_CAN_SEND/SUB_CAN_RECV enum values have been confusing a few
times, especially when checking them on reading. After some discussion,
it appears that calling them SUB_RETRY_SEND/SUB_RETRY_RECV more
accurately reflects their purpose since these events may only appear
after a first attempt to perform the I/O operation has failed or was
not completed.

In addition the wait_reason field in struct wait_event which carries
them makes one think that a single reason may happen at once while
it is in fact a set of events. Since the struct is called wait_event
it makes sense that this field is called "events" to indicate it's the
list of events we're subscribed to.

Last, the values for SUB_RETRY_RECV/SEND were swapped so that value
1 corresponds to recv and 2 to send, as is done almost everywhere else
in the code an in the shutdown() call.
2018-12-19 14:09:21 +01:00
Christopher Faulet
b61481c710 MINOR: compression: Remove the thread_local variable buf_output
By doing a c_rew() at the right place, we can avoid to use this variable. This
slightly simplifly the compression for the legacy HTTP.
2018-12-19 13:45:53 +01:00
Christopher Faulet
cc26b13ea5 BUG/MINOR: lua: Return an error if a legacy HTTP applet doesn't send anything
In legacy mode, if an HTTP applet does not send any response, an error 500 is
returned.
2018-12-19 13:45:53 +01:00
Christopher Faulet
e461e34d64 BUG/MEDIUM: lua/htx: Handle EOM in receive/get_line calls in HTTP applets
In HTTP applets, the request's EOM was removed like other blocks when receive or
get_line was called from lua scripts. So it was impossible to stop receiving
data on successive calls when all the request body was already consumed,
blocking infinitly the applet.

Now, we never consume the EOM. So it is easy to interrupt receive/get_line
calls. In all cases, this block is consumed when the applet ends.
2018-12-19 13:45:53 +01:00
Christopher Faulet
b6aadbd19e BUG/MINOR: proto_htx: Call the HTX version of the function managing client cookies
Because of a typo, the legacy version was called instead of the HTX one.
2018-12-19 13:45:53 +01:00
Christopher Faulet
2dba1a50c3 BUG/MEDIUM: stream: Forward the right amount of data before infinite forwarding
Before setting the infinite forward, we first forward all remaining input data
from the channel. Of course for HTX streams, this must be done using the amount
of data in the HTX message not in the channel (which appears as full because of
the HTX).
2018-12-19 13:45:53 +01:00
Willy Tarreau
beefaee4f5 MEDIUM: h2: properly check and deduplicate the content-length header in HTX
When producing an HTX message, we can't rely on the next-level H1 parser
to check and deduplicate the content-length header, so we have to do it
while parsing a message. The algorithm is the exact same as used for H1
messages.
2018-12-19 13:08:08 +01:00