Commit Graph

1813 Commits

Author SHA1 Message Date
Willy Tarreau
c6dac6c7f5 MEDIUM: listeners: remove the now unused ZOMBIE state
The zombie state is not used anymore by the listeners, because in the
last two cases where it was tested it couldn't match as it was covered
by the test on the process mask. Instead now the FD is either in the
LISTEN state or the INIT state. This also avoids forcing the listener
to be single-dimensional because actually belonging to another process
isn't totally exclusive with the other states, which explains some of
the difficulties requiring to check the proc_mask and the fd sometimes.

So let's get rid of it now not to be tempted to reuse it.

The doc on the listeners state was updated.
2020-10-09 11:27:29 +02:00
Sébastien Gross
ab8771285c DOC: Fix typos in configuration.txt
This patch fixes small typos and grammar in configuration.txt for the
http-request return documentation.
2020-10-09 10:02:09 +02:00
Pierre Cheynier
08eb718494 DOC: Add missing stats fields in the management doc
Added latest fields: idle_conn_cur, safe_conn_cur, used_conn_cur, need_conn_est
2020-10-09 09:56:37 +02:00
Willy Tarreau
465dc7d63f DOC: fix a confusing typo on a regsub example
Sbastien reported a confusing example in the doc about regsub when used
with quotes. Nested quotes are already not trivial to grasp, but when
typos are there and result in something valid, it's even worse. The closing
quote ought to have been inside the brackets. However haproxy will not make
any difference because the single quotes delimit a word and the delimited
word remains the same. Let's just not add yet another level of confusion.
2020-10-08 18:05:56 +02:00
Emeric Brun
da46c1ca60 DOC: re-work log forward bind statement documentation.
This patch re-work the documentation about the bind statement
of log forward section.
2020-10-08 08:54:04 +02:00
Emeric Brun
cbb7bf7dd1 MEDIUM: log: syslog TCP support on log forward section.
This patch re-introduce the "bind" statement on log forward
sections to handle syslog TCP listeners as defined in
rfc-6587.

As complement it introduce "maxconn", "backlog" and "timeout
client" statements to parameter those listeners.
2020-10-07 17:17:27 +02:00
William Dauchy
57dd6f1c5a DOC: ssl: fix typo about ocsp files
one of the last I saw in this section while working on github issue #872

might be backported in all still supported versions

Signed-off-by: William Dauchy <w.dauchy@criteo.com>
2020-10-06 15:47:50 +02:00
Amaury Denoyelle
fbd0bc98fe MINOR: dns/stats: integrate dns counters in stats
Use the new stats module API to integrate the dns counters in the
standard stats. This is done in order to avoid code duplication, keep
the code related to cli out of dns and use the full possibility of the
stats function, allowing to print dns stats in csv or json format.
2020-10-05 12:02:14 +02:00
Amaury Denoyelle
0b70a8a314 MINOR: stats: add config "stats show modules"
By default, hide the extra statistics on the html page. Define a new
flag STAT_SHMODULES which is activated if the config "stats show
modules" is set.
2020-10-05 12:02:14 +02:00
Amaury Denoyelle
50660a894d MEDIUM: stats: add delimiter for static proxy stats on csv
Use the character '-' to mark the end of static statistics on proxy
domain. After this marker, the order of the fields is not guaranteed and
should be parsed with care.
2020-10-05 12:02:14 +02:00
Amaury Denoyelle
072f97eddf MINOR: stats: define the concept of domain for statistics
The domain option will be used to have statistics attached to other
objects than proxies/listeners/servers. At the moment, only the PROXY
domain is available.

Add an argument 'domain' on the 'show stats' cli command to specify the
domain. Only 'domain proxy' is available now. If not specified, proxy
will be considered the default domain.

For HTML output, only proxy statistics will be displayed.
2020-10-05 12:02:14 +02:00
William Lallemand
f9ff3ece83 DOC: ssl: new "cert bundle" behavior
Update the documentation with the new bundle behavior which does not use
the same OpenSSL certificate store anymore but loads the PEM separately
as multiple "crt" were specified.

It should fix issue #872.
2020-10-02 18:02:16 +02:00
Christopher Faulet
2079a4ad36 MEDIUM: tcp-rules: Warn if a track-sc* content rule doesn't depend on content
The warning is only emitted for HTTP frontend. Idea is to encourage the usage of
"tcp-request session" rules to track counters that does not depend on the
request content. The documentation has been updated accordingly.

The warning is important because since the multiplexers were added in the
processing chain, the HTTP parsing is performed at a lower level. Thus parsing
errors are detected in the multiplexers, before the stream creation. In HTTP/2,
the error is reported by the multiplexer itself and the stream is never
created. This difference has a certain number of consequences, one of which is
that HTTP request counting in stick tables only works for valid H2 request, and
HTTP error tracking in stick tables never considers invalid H2 requests but only
invalid H1 ones. And the aim is to do the same with the mux-h1. This change will
not be done for the 2.3, but the 2.4. At the end, H1 and H2 parsing errors will
be caught by the multiplexers, at the session level. Thus, tracking counters at
the content level should be reserved for rules using a key based on the request
content or those using ACLs based on the request content.

To be clear, a warning will be emitted for the following rules :

  tcp-request content track-sc0 src
  tcp-request content track-sc0 src if ! { src 10.0.0.0/24 }
  tcp-request content track-sc0 src if { ssl_fc }

But not for the following ones :

  tcp-request content track-sc0 req.hdr(host)
  tcp-request content track-sc0 src if { req.hdr(host) -m found }
2020-10-02 15:50:26 +02:00
Christopher Faulet
7ea509e15f DOC: tcp-rules: Refresh details about L7 matching for tcp-request content rules
Because the parsing of HTTP message is now performed in the HTTP multiplexers,
the content is immediatly available when "tcp-request content" rules are
evaluated for an HTTP frontend. So, it is a good idea to make the documentation
explicit on this point. In addition, because in all cases, the parsing is
already performed, there is no reason to still use "tcp-request content" rules
based on L7 matching, although it is still valid. The recommended way is to use
"http-request" rules instead. Again, it is a good idea to update the
documentation on this point.
2020-10-02 15:50:26 +02:00
Amaury Denoyelle
fa41cb6792 MINOR: tools: support for word expansion of environment in parse_line
Allow the syntax "${...[*]}" to expand an environment variable
containing several values separated by spaces as individual arguments. A
new flag PARSE_OPT_WORD_EXPAND has been added to toggle this feature on
parse_line invocation. In case of an invalid syntax, a new error
PARSE_ERR_WRONG_EXPAND will be triggered.

This feature has been asked on the github issue #165.
2020-10-01 17:24:14 +02:00
William Dauchy
25407965fd DOC: crt: advise to move away from cert bundle
especially when starting to use `new ssl cert` runtime API, it might
become a bit confusing for users to mix bundle and single cert,
especially when it comes to use the commit command:
e.g.:
- start the process with `crt` loading a bundle
- use `set ssl cert my_cert.pem.ecdsa`: API detects it as a replacement
  of a bundle.
- `commit` has to be done on the bundle: `commit ssl cert my_cert.pem`

however:
- add a new cert: `new ssl cert my_cert.pem.rsa`: added as a single
  certificate
- `commit` has to be done on the certificate: `commit ssl cert
  my_cert.pem.rsa`

this should resolve github issue #872

this should probably be backported in >= v2.2 in order to encourage
people to move away from bundle certificates loading.

Signed-off-by: William Dauchy <w.dauchy@criteo.com>
2020-09-27 21:54:10 +02:00
William Dauchy
f8e795ca04 DOC: agent-check: fix typo in "fail" word expected reply
`tcpcheck_agent_expect_reply` expects "fail" not "failed"

This should fix github issue #876

This can be backported to all maintained versions (i.e >= 1.6) as of
today.

Signed-off-by: William Dauchy <w.dauchy@criteo.com>
2020-09-27 21:54:10 +02:00
Christopher Faulet
05f0188943 [RELEASE] Released version 2.3-dev5
Released version 2.3-dev5 with the following main changes :
    - DOC: Fix typo in iif() example
    - CLEANUP: Update .gitignore
    - BUILD: introduce possibility to define ABORT_NOW() conditionally
    - CI: travis-ci: help Coverity to recognize abort()
    - BUG/MINOR: Fix type passed of sizeof() for calloc()
    - CLEANUP: Do not use a fixed type for 'sizeof' in 'calloc'
    - CLEANUP: tree-wide: use VAR_ARRAY instead of [0] in various definitions
    - BUILD: connection: fix build on clang after the VAR_ARRAY cleanup
    - BUG/MINOR: ssl: verifyhost is case sensitive
    - BUILD: makefile: change default value of CC from gcc to cc
    - CI: travis-ci: split asan step out of running tests
    - BUG/MINOR: server: report correct error message for invalid port on "socks4"
    - BUG/MEDIUM: ssl: Don't call ssl_sock_io_cb() directly.
    - BUG/MINOR: ssl/crt-list: crt-list could end without a \n
    - BUG/MINOR: log-forward: fail on unknown keywords
    - MEDIUM: log-forward: use "dgram-bind" instead of "bind" for the listener
    - BUG/MEDIUM: log-forward: always quit on parsing errors
    - MEDIUM: ssl: remove bundle support in crt-list and directories
    - MEDIUM: ssl/cli: remove support for multi certificates bundle
    - MINOR: ssl: crtlist_dup_ssl_conf() duplicates a ssl_bind_conf
    - MINOR: ssl: crtlist_entry_dup() duplicates a crtlist_entry
    - MEDIUM: ssl: emulates the multi-cert bundles in the crtlist
    - MEDIUM: ssl: emulate multi-cert bundles loading in standard loading
    - CLEANUP: ssl: remove test on "multi" variable in ckch functions
    - CLEANUP: ssl/cli: remove test on 'multi' variable in CLI functions
    - CLEANUP: ssl: remove utility functions for bundle
    - DOC: explain bundle emulation in configuration.txt
    - BUILD: fix build with openssl < 1.0.2 since bundle removal
    - BUG/MINOR: log: gracefully handle the "udp@" address format for log servers
    - BUG/MINOR: dns: gracefully handle the "udp@" address format for nameservers
    - MINOR: listener: create a new struct "settings" in bind_conf
    - MINOR: listener: move bind_proc and bind_thread to struct settings
    - MINOR: listener: move the interface to the struct settings
    - MINOR: listener: move the network namespace to the struct settings
    - REORG: listener: create a new struct receiver
    - REORG: listener: move the listening address to a struct receiver
    - REORG: listener: move the receiving FD to struct receiver
    - REORG: listener: move the listener's proto to the receiver
    - MINOR: listener: make sock_find_compatible_fd() check the socket type
    - REORG: listener: move the receiver part to a new file
    - MINOR: receiver: link the receiver to its settings
    - MINOR: receiver: link the receiver to its owner
    - MINOR: listener: prefer to retrieve the socket's settings via the receiver
    - MINOR: receiver: add a receiver-specific flag to indicate the socket is bound
    - MINOR: listener: move the INHERITED flag down to the receiver
    - MINOR: receiver: move the FOREIGN and V6ONLY options from listener to settings
    - MINOR: sock: make sock_find_compatible_fd() only take a receiver
    - MINOR: protocol: rename the ->bind field to ->listen
    - MINOR: protocol: add a new ->bind() entry to bind the receiver
    - MEDIUM: sock_inet: implement sock_inet_bind_receiver()
    - MEDIUM: tcp: make use of sock_inet_bind_receiver()
    - MEDIUM: udp: make use of sock_inet_bind_receiver()
    - MEDIUM: sock_unix: implement sock_unix_bind_receiver()
    - MEDIUM: uxst: make use of sock_unix_bind_receiver()
    - MEDIUM: sockpair: implement sockpair_bind_receiver()
    - MEDIUM: proto_sockpair: make use of sockpair_bind_receiver()
    - MEDIUM: protocol: explicitly start the receiver before the listener
    - MEDIUM: protocol: do not call proto->bind() anymore from bind_listener()
    - MINOR: protocol: add a new proto_fam structure for protocol families
    - MINOR: protocol: retrieve the family-specific fields from the family
    - CLEANUP: protocol: remove family-specific fields from struct protocol
    - MINOR: protocol: add a real family for existing FDs
    - CLEANUP: tools: make str2sa_range() less awful for fd@ and sockpair@
    - MINOR: tools: make str2sa_range() take more options than just resolve
    - MINOR: tools: add several PA_O_PORT_* flags in str2sa_range() callers
    - MEDIUM: tools: make str2sa_range() validate callers' port specifications
    - MEDIUM: config: remove all checks for missing/invalid ports/ranges
    - MINOR: tools: add several PA_O_* flags in str2sa_range() callers
    - MINOR: listener: remove the inherited arg to create_listener()
    - MINOR: tools: make str2sa_range() optionally return the fd
    - MINOR: log: detect LOG_TARGET_FD from the fd and not from the syntax
    - MEDIUM: tools: make str2sa_range() resolve pre-bound listeners
    - MINOR: config: do not test an inherited socket again
    - MEDIUM: tools: make str2sa_range() check for the sockpair's FD usability
    - MINOR: tools: start to distinguish stream and dgram in str2sa_range()
    - MEDIUM: tools: make str2sa_range() only report AF_CUST_UDP on listeners
    - MINOR: tools: remove the central test for "udp" in str2sa_range()
    - MINOR: cfgparse: add str2receiver() to parse dgram receivers
    - MINOR: log-forward: use str2receiver() to parse the dgram-bind address
    - MEDIUM: config: make str2listener() not accept datagram sockets anymore
    - MINOR: listener: pass the chosen protocol to create_listeners()
    - MINOR: tools: make str2sa_range() directly return the protocol
    - MEDIUM: tools: make str2sa_range() check that the protocol has ->connect()
    - MINOR: protocol: add the control layer type in the protocol struct
    - MEDIUM: protocol: store the socket and control type in the protocol array
    - MEDIUM: tools: make str2sa_range() use protocol_lookup()
    - MEDIUM: proto_udp: replace last AF_CUST_UDP* with AF_INET*
    - MINOR: tools: drop listener detection hack from str2sa_range()
    - BUILD: sock_unix: add missing errno.h
    - MINOR: sock_inet: report the errno string in binding errors
    - MINOR: sock_unix: report the errno string in binding errors
    - BUILD: sock_inet: include errno.h
    - MINOR: h2/trace: also display the remaining frame length in traces
    - BUG/MINOR: h2/trace: do not display "stream error" after a frame ACK
    - BUG/MEDIUM: h2: report frame bits only for handled types
    - BUG/MINOR: http-fetch: Don't set the sample type during the htx prefetch
    - BUG/MINOR: Fix memory leaks cfg_parse_peers
    - BUG/MINOR: config: Fix memory leak on config parse listen
    - MINOR: backend: make the "whole" option of balance uri take only one bit
    - MINOR: backend: add a new "path-only" option to "balance uri"
    - REGTESTS: add a few load balancing tests
    - BUG/MEDIUM: listeners: do not pause foreign listeners
    - BUG/MINOR: listeners: properly close listener FDs
    - BUILD: trace: include tools.h
2020-09-25 18:40:47 +02:00
Willy Tarreau
57a374131c MINOR: backend: add a new "path-only" option to "balance uri"
Since we've fixed the way URIs are handled in 2.1, some users have started
to experience inconsistencies in "balance uri" between requests received
over H1 and the same ones received over H2. This is caused by the fact
that H1 rarely uses absolute URIs while H2 always uses them. Similar
issues were reported already around replace-uri etc, leading to "pathq"
recently being introduced, so this isn't new.

Here what this patch does is add a new option to "balance uri" to indicate
that the hashing should only start at the path and not cover the authority.
This makes H1 relative URIs and H2 absolute URI hashes equally again.

Some extra options could be added to normalize URIs by always hashing the
authority (or host) in front of them, which would make sure that both
absolute and relative requests provide the same hash. This is left for
later if needed.
2020-09-23 08:56:29 +02:00
Willy Tarreau
26ff5dabc0 MINOR: log-forward: use str2receiver() to parse the dgram-bind address
Thanks to this we don't need to specify "udp@" as it's implicitly a
datagram type listener that is expected, so any AF_INET/AF_INET4 address
will work.
2020-09-16 22:08:08 +02:00
William Lallemand
98d215e063 DOC: explain bundle emulation in configuration.txt
HAProxy 2.3 does not support OpenSSL 1.0.2 bundle anymore, and requires
OpenSSL 1.1.1 to do serve multiple certificate types with the same SNI.
2020-09-16 17:47:15 +02:00
Willy Tarreau
76aaa7f5b7 MEDIUM: log-forward: use "dgram-bind" instead of "bind" for the listener
The use of "bind" wasn't that wise but was temporary. The problem is that
it will not allow to coexist with tcp. Let's explicitly call it "dgram-bind"
so that datagram listeners are expected here, leaving some room for stream
listeners later. This is the only change.
2020-09-16 15:07:22 +02:00
Tim Duesterhus
870713bbc0 DOC: Fix typo in iif() example
It should read 'iif', not 'iff'.

Reported-by: Miroslav Zagorac <zaga@fly-etf.net>
2020-09-11 18:02:34 +02:00
Willy Tarreau
253c4dc200 [RELEASE] Released version 2.3-dev4
Released version 2.3-dev4 with the following main changes :
    - MINOR: hlua: Add error message relative to the Channel manipulation and HTTP mode
    - BUG/MEDIUM: ssl: crt-list negative filters don't work
    - DOC: overhauling github issue templates
    - MEDIUM: cfgparse: Emit hard error on truncated lines
    - DOC: cache: Use '<name>' instead of '<id>' in error message
    - MINOR: cache: Reject duplicate cache names
    - REGTEST: remove stray leading spaces in converteers_ref_cnt_never_dec.vtc
    - MINOR: stats: prevent favicon.ico requests for stats page
    - BUILD: tools: include auxv a bit later
    - BUILD: task: work around a bogus warning in gcc 4.7/4.8 at -O1
    - MEDIUM: ssl: Support certificate chaining for certificate generation
    - MINOR: ssl: Support SAN extension for certificate generation
    - MINOR: tcp: don't try to set/clear v6only on inherited sockets
    - BUG/MINOR: reload: detect the OS's v6only status before choosing an old socket
    - MINOR: reload: determine the foreing binding status from the socket
    - MEDIUM: reload: stop passing listener options along with FDs
    - BUG/MEDIUM: ssl: fix ssl_bind_conf double free w/ wildcards
    - MEDIUM: fd: replace usages of fd_remove() with fd_stop_both()
    - CLEANUP: fd: remove fd_remove() and rename fd_dodelete() to fd_delete()
    - MINOR: fd: add a new "exported" flag and use it for all regular listeners
    - MEDIUM: reload: pass all exportable FDs, not just listeners
    - DOC: add description of pidfile in master-worker mode
    - BUG/MINOR: reload: do not fail when no socket is sent
    - REORG: tcp: move TCP actions from proto_tcp.c to tcp_act.c
    - CLEANUP: tcp: stop exporting smp_fetch_src()
    - REORG: tcp: move TCP sample fetches from proto_tcp.c to tcp_sample.c
    - REORG: tcp: move TCP bind/server keywords from proto_tcp.c to cfgparse-tcp.c
    - REORG: unix: move UNIX bind/server keywords from proto_uxst.c to cfgparse-unix.c
    - REORG: sock: start to move some generic socket code to sock.c
    - MINOR: sock: introduce sock_inet and sock_unix
    - MINOR: tcp/udp/unix: make use of proto->addrcmp() to compare addresses
    - MINOR: sock_inet: implement sock_inet_get_dst()
    - REORG: inet: replace tcp_is_foreign() with sock_inet_is_foreign()
    - REORG: sock_inet: move v6only_default from proto_tcp.c to sock_inet.c
    - REORG: sock_inet: move default_tcp_maxseg from proto_tcp.c
    - REORG: listener: move xfer_sock_list to sock.{c,h}.
    - MINOR: sock: add interface and namespace length to xfer_sock_list
    - MINOR: sock: implement sock_find_compatible_fd()
    - MINOR: sock_inet: move the IPv4/v6 transparent mode code to sock_inet
    - REORG: sock: move get_old_sockets() from haproxy.c
    - MINOR: sock: do not use LI_O_* in xfer_sock_list anymore
    - MINOR: sock: distinguish dgram from stream types when retrieving old sockets
    - BUILD: sock_unix: fix build issue with isdigit()
    - BUG/MEDIUM: http-ana: Don't wait to send 1xx responses received from servers
    - MINOR: http-htx: Add an option to eval query-string when the path is replaced
    - BUG/MINOR: http-rules: Replace path and query-string in "replace-path" action
    - MINOR: http-htx: Handle an optional reason when replacing the response status
    - MINOR: contrib/spoa-server: allow MAX_FRAME_SIZE override
    - BUG/MAJOR: contrib/spoa-server: Fix unhandled python call leading to memory leak
    - BUG/MINOR: contrib/spoa-server: Ensure ip address references are freed
    - BUG/MINOR: contrib/spoa-server: Do not free reference to NULL
    - BUG/MINOR: contrib/spoa-server: Updating references to free in case of failure
    - BUG/MEDIUM: contrib/spoa-server: Fix ipv4_address used instead of ipv6_address
    - CLEANUP: http: silence a cppcheck warning in get_http_auth()
    - REGTEST: increase some short timeouts to make tests more reliable
    - BUG/MINOR: threads: work around a libgcc_s issue with chrooting
    - BUILD: thread: limit the libgcc_s workaround to glibc only
    - MINOR: protocol: do not call proto->bind_all() anymore
    - MINOR: protocol: do not call proto->unbind_all() anymore
    - CLEANUP: protocol: remove all ->bind_all() and ->unbind_all() functions
    - MAJOR: init: start all listeners via protocols and not via proxies anymore
    - BUG/MINOR: startup: haproxy -s cause 100% cpu
    - Revert "BUG/MINOR: http-rules: Replace path and query-string in "replace-path" action"
    - BUG/MEDIUM: doc: Fix replace-path action description
    - MINOR: http-rules: Add set-pathq and replace-pathq actions
    - MINOR: http-fetch: Add pathq sample fetch
    - REGTEST: Add a test for request path manipulations, with and without the QS
    - MINOR: Commit .gitattributes
    - CLEANUP: Update .gitignore
    - BUG/MEDIUM: dns: Don't store additional records in a linked-list
    - BUG/MEDIUM: dns: Be sure to renew IP address for already known servers
    - MINOR: server: Improve log message sent when server address is updated
    - DOC: ssl-load-extra-files only applies to certificates on bind lines
    - BUG/MINOR: auth: report valid crypto(3) support depending on build options
    - BUG/MEDIUM: mux-h1: always apply the timeout on half-closed connections
    - BUILD: threads: better workaround for late loading of libgcc_s
    - BUILD: compiler: reserve the gcc version checks to the gcc compiler
    - BUILD: compiler: workaround a glibc madness around __attribute__()
    - BUILD: intops: on x86_64, the bswap instruction is called bswapq
    - BUILD: trace: always have an argument before variadic args in macros
    - BUILD: traces: don't pass an empty argument for missing ones
    - BUG/MINOR: haproxy: Free uri_auth->scope during deinit
    - CLEANUP: Free old_argv on deinit
    - CLEANUP: haproxy: Free post_proxy_check_list in deinit()
    - CLEANUP: haproxy: Free per_thread_*_list in deinit()
    - CLEANUP: haproxy: Free post_check_list in deinit()
    - BUG/MEDIUM: pattern: Renew the pattern expression revision when it is pruned
    - REORG: tools: move PARSE_OPT_* from tools.h to tools-t.h
    - MINOR: sample: Add iif(<true>,<false>) converter
2020-09-11 17:05:59 +02:00
Tim Duesterhus
3943e4fc3e MINOR: sample: Add iif(<true>,<false>) converter
iif() takes a boolean as input and returns one of the two argument
strings depending on whether the boolean is true.

This converter most likely is most useful to return the proper scheme
depending on the value returned by the `ssl_fc` fetch, e.g. for use within
the `x-forwarded-proto` request header.

However it can also be useful for use within a template that is sent to
the client using `http-request return` with a `lf-file`. It allows the
administrator to implement a simple condition, without needing to prefill
variables within the regular configuration using `http-request
set-var(req.foo)`.
2020-09-11 16:59:27 +02:00
Jerome Magnin
587be9ce30 DOC: ssl-load-extra-files only applies to certificates on bind lines
Be explicit about ssl-load-extra-files not applying to certificates
referenced with the crt keyword on server lines.

It must be backported in 2.2.
2020-09-08 13:47:55 +02:00
Christopher Faulet
e720c32b78 MINOR: http-fetch: Add pathq sample fetch
The pathq sample fetch extract the relative URI of a request, i.e the path with
the query-string, excluding the scheme and the authority, if any. It is pretty
handy to always get a relative URI independently on the HTTP version. Indeed,
while relative URIs are common in HTTP/1.1, in HTTP/2, most of time clients use
absolute URIs.

This patch may be backported to 2.2.
2020-09-04 11:41:47 +02:00
Christopher Faulet
312294f53d MINOR: http-rules: Add set-pathq and replace-pathq actions
These actions do the same as corresponding "-path" versions except the
query-string is included to the manipulated request path. This means set-pathq
action replaces the path and the query-string and replace-pathq action matches
and replace the path including the query-string.

This patch may be backported to 2.2.
2020-09-04 11:41:46 +02:00
Christopher Faulet
82c8332be0 BUG/MEDIUM: doc: Fix replace-path action description
The description of the replace-path action does not reflect what the code
do. When the request path is replaced, the query-string is preserved. But the
documentation stated the query-string is part of the replacement, if any is
present. Most of time, when the doc and the code differ, the code is fixed. But
here, the replace-path action is pretty confusing because the set-path action is
only applied on the path. The query-string is left intact. And the path sample
fetch also ignores the query-string. In addition, the replace-path action is
quite recent. It was added in the 2.2. Thus, exceptionally, the documentation is
fixed instead.

Note that set-pathq and replace-pathq actions and pathq sample fetch will be
added to manipulate the path with the query-string.

This patch must be backported as far as 2.0.
2020-09-02 17:29:00 +02:00
MIZUTA Takeshi
c32f39431c DOC: add description of pidfile in master-worker mode
Previously, pidfile was only described for daemon mode. In the case of
master-worker mode, the handling of pidfile is different from daemon mode,
so the description has been added.
2020-08-26 18:40:53 +02:00
Willy Tarreau
f104b530e2 [RELEASE] Released version 2.3-dev3
Released version 2.3-dev3 with the following main changes :
    - SCRIPTS: git-show-backports: make -m most only show the left branch
    - SCRIPTS: git-show-backports: emit the shell command to backport a commit
    - BUILD: Makefile: require SSL_LIB, SSL_INC to be explicitly set
    - CI: travis-ci: specify SLZ_LIB, SLZ_INC for travis builds
    - BUG/MEDIUM: mux-h1: Refresh H1 connection timeout after a synchronous send
    - CLEANUP: dns: typo in reported error message
    - BUG/MAJOR: dns: disabled servers through SRV records never recover
    - BUG/MINOR: spoa-server: fix size_t format printing
    - DOC: spoa-server: fix false friends `actually`
    - BUG/MINOR: ssl: fix memory leak at OCSP loading
    - BUG/MEDIUM: ssl: memory leak of ocsp data at SSL_CTX_free()
    - BUG/MEDIUM: map/lua: Return an error if a map is loaded during runtime
    - MINOR: arg: Add an argument type to keep a reference on opaque data
    - BUG/MINOR: converters: Store the sink in an arg pointer for debug() converter
    - BUG/MINOR: lua: Duplicate map name to load it when a new Map object is created
    - BUG/MINOR: arg: Fix leaks during arguments validation for fetches/converters
    - BUG/MINOR: lua: Check argument type to convert it to IPv4/IPv6 arg validation
    - BUG/MINOR: lua: Check argument type to convert it to IP mask in arg validation
    - MINOR: hlua: Don't needlessly copy lua strings in trash during args validation
    - BUG/MINOR: lua: Duplicate lua strings in sample fetches/converters arg array
    - MEDIUM: lua: Don't filter exported fetches and converters
    - MINOR: lua: Add support for userlist as fetches and converters arguments
    - MINOR: lua: Add support for regex as fetches and converters arguments
    - MINOR: arg: Use chunk_destroy() to release string arguments
    - BUG/MINOR: snapshots: leak of snapshots on deinit()
    - CLEANUP: ssl: ssl_sock_crt2der semicolon and spaces
    - MINOR: ssl: add ssl_{c,s}_chain_der fetch methods
    - CLEANUP: fix all duplicated semicolons
    - BUG/MEDIUM: ssl: fix the ssl-skip-self-issued-ca option
    - BUG/MINOR: ssl: ssl-skip-self-issued-ca requires >= 1.0.2
    - BUG/MINOR: stats: use strncmp() instead of memcmp() on health states
    - BUILD: makefile: don't disable -Wstringop-overflow anymore
    - BUG/MINOR: ssl: double free w/ smp_fetch_ssl_x_chain_der()
    - BUG/MEDIUM: htx: smp_prefetch_htx() must always validate the direction
    - BUG/MEDIUM: ssl: never generates the chain from the verify store
    - OPTIM: regex: PCRE2 use JIT match when JIT optimisation occured.
    - BUG/MEDIUM: ssl: does not look for all SNIs before chosing a certificate
    - CLEANUP: ssl: remove poorly readable nested ternary
2020-08-14 18:54:05 +02:00
William Lallemand
9a1d839f61 BUG/MINOR: ssl: ssl-skip-self-issued-ca requires >= 1.0.2
The previous fix for ssl-skip-self-issued-ca requires the use of
SSL_CTX_build_cert_chain() which is only available starting from OpenSSL
1.0.2
2020-08-10 17:31:10 +02:00
William Dauchy
a598b500b4 MINOR: ssl: add ssl_{c,s}_chain_der fetch methods
Following work from Arjen and Mathilde, it adds ssl_{c,s}_chain_der
methods; it returns DER encoded certs from SSL_get_peer_cert_chain

Also update existing vtc tests to add random intermediate certificates

When getting the result through this header:
  http-response add-header x-ssl-chain-der %[ssl_c_chain_der,hex]
One can parse it with any lib accepting ASN.1 DER data, such as in go:
  bin, err := encoding/hex.DecodeString(cert)
  certs_parsed, err := x509.ParseCertificates(bin)

Cc: Arjen Nienhuis <arjen@zorgdoc.nl>
Signed-off-by: Mathilde Gilles <m.gilles@criteo.com>
Signed-off-by: William Dauchy <w.dauchy@criteo.com>
2020-08-07 15:38:40 +02:00
Willy Tarreau
3f3cc8c8c7 [RELEASE] Released version 2.3-dev2
Released version 2.3-dev2 with the following main changes :
    - DOC: ssl: req_ssl_sni needs implicit TLS
    - BUG/MEDIUM: arg: empty args list must be dropped
    - BUG/MEDIUM: resolve: fix init resolving for ring and peers section.
    - BUG/MAJOR: tasks: don't requeue global tasks into the local queue
    - MINOR: tasks/debug: make the thread affinity BUG_ON check a bit stricter
    - MINOR: tasks/debug: add a few BUG_ON() to detect use of wrong timer queue
    - MINOR: tasks/debug: add a BUG_ON() check to detect requeued task on free
    - BUG/MAJOR: dns: Make the do-resolve action thread-safe
    - BUG/MEDIUM: dns: Release answer items when a DNS resolution is freed
    - MEDIUM: htx: Add a flag on a HTX message when no more data are expected
    - BUG/MEDIUM: stream-int: Don't set MSG_MORE flag if no more data are expected
    - BUG/MEDIUM: http-ana: Only set CF_EXPECT_MORE flag on data filtering
    - CLEANUP: dns: remove 45 "return" statements from dns_validate_dns_response()
    - BUG/MINOR: htx: add two missing HTX_FL_EOI and remove an unexpected one
    - BUG/MINOR: mux-fcgi: Don't url-decode the QUERY_STRING parameter anymore
    - BUILD: tools: fix build with static only toolchains
    - DOC: Use gender neutral language
    - BUG/MINOR: debug: Don't dump the lua stack if it is not initialized
    - BUG/MAJOR: dns: fix null pointer dereference in snr_update_srv_status
    - BUG/MAJOR: dns: don't treat Authority records as an error
    - CI : travis-ci : prepare for using stock OpenSSL
    - CI: travis-ci : switch to stock openssl when openssl-1.1.1 is used
    - MEDIUM: lua: Add support for the Lua 5.4
    - BUG/MEDIUM: dns: Don't yield in do-resolve action on a final evaluation
    - BUG/MINOR: lua: Abort execution of actions that yield on a final evaluation
    - MINOR: tcp-rules: Return an internal error if an action yields on a final eval
    - BUG/MINOR: tcp-rules: Preserve the right filter analyser on content eval abort
    - BUG/MINOR: tcp-rules: Set the inspect-delay when a tcp-response action yields
    - MEDIUM: tcp-rules: Use a dedicated expiration date for tcp ruleset
    - MEDIUM: lua: Set the analyse expiration date with smaller wake_time only
    - BUG/MEDIUM: connection: Be sure to always install a mux for sync connect
    - MINOR: connection: Preinstall the mux for non-ssl connect
    - MINOR: stream-int: Be sure to have a mux to do sends and receives
    - BUG/MINOR: lua: Fix a possible null pointer deref on lua ctx
    - SCRIPTS: announce-release: add the link to the wiki in the announce messages
    - CI: travis-ci: use better name for Coverity scan job
    - CI: travis-ci: use proper linking flags for SLZ build
    - BUG/MEDIUM: backend: always attach the transport before installing the mux
    - BUG/MEDIUM: tcp-checks: always attach the transport before installing the mux
    - MINOR: connection: avoid a useless recvfrom() on outgoing connections
    - MINOR: mux-h1: do not even try to receive if the connection is not fully set up
    - MINOR: mux-h1: do not try to receive on backend before sending a request
    - CLEANUP: assorted typo fixes in the code and comments
    - BUG/MEDIUM: ssl: check OCSP calloc in ssl_sock_load_ocsp()
2020-07-31 14:48:32 +02:00
Jackie Tapia
749f74c622 DOC: Use gender neutral language
This patch updates the documentation files and code comments to avoid
the use of gender specific phrasing in favor of "they" or "it".
2020-07-26 22:35:43 +02:00
Lukas Tribus
a267b5df4a DOC: ssl: req_ssl_sni needs implicit TLS
req_ssl_sni is not compatible with protocols negotiating TLS
explicitly, like SMTP on port 25 or 587 and IMAP on port 143.

Fix an example referring to 587 (SMTPS port with implicit TLS
is 465) and amend the req_ssl_sni documentation.

This doc fix should be backported to supported versions.
2020-07-19 06:36:12 +02:00
Willy Tarreau
e732cbd8a9 [RELEASE] Released version 2.3-dev1
Released version 2.3-dev1 with the following main changes :
    - MINOR: config: make strict limits enabled by default
    - BUG/MINOR: acl: Fix freeing of expr->smp in prune_acl_expr
    - BUG/MINOR: sample: Fix freeing of conv_exprs in release_sample_expr
    - BUG/MINOR: haproxy: Free proxy->format_unique_id during deinit
    - BUG/MINOR: haproxy: Add missing free of server->(hostname|resolvers_id)
    - BUG/MINOR: haproxy: Free proxy->unique_id_header during deinit
    - BUG/MINOR: haproxy: Free srule->file during deinit
    - BUG/MINOR: haproxy: Free srule->expr during deinit
    - BUG/MINOR: sample: Free str.area in smp_check_const_bool
    - BUG/MINOR: sample: Free str.area in smp_check_const_meth
    - CLEANUP: haproxy: Free proxy_deinit_list in deinit()
    - CLEANUP: haproxy: Free post_deinit_list in deinit()
    - CLEANUP: haproxy: Free server_deinit_list in deinit()
    - CLEANUP: haproxy: Free post_server_check_list in deinit()
    - CLEANUP: Add static void vars_deinit()
    - CLEANUP: Add static void hlua_deinit()
    - CLEANUP: contrib/prometheus-exporter: typo fixes for ssl reuse metric
    - BUG/MEDIUM: lists: add missing store barrier on MT_LIST_BEHEAD()
    - BUG/MEDIUM: lists: add missing store barrier in MT_LIST_ADD/MT_LIST_ADDQ
    - MINOR: tcp: Support TCP keepalive parameters customization
    - BUILD: tcp: condition TCP keepalive settings to platforms providing them
    - MINOR: lists: rename some MT_LIST operations to clarify them
    - MINOR: buffer: use MT_LIST_ADDQ() for buffer_wait lists additions
    - MINOR: connection: use MT_LIST_ADDQ() to add connections to idle lists
    - MINOR: tasks: use MT_LIST_ADDQ() when killing tasks.
    - CONTRIB: da: fix memory leak in dummy function da_atlas_open()
    - CI: travis-ci: speed up osx build by running brew scripted, switch to latest osx image
    - BUG/MEDIUM: mux-h2: Don't add private connections in available connection list
    - BUG/MEDIUM: mux-fcgi: Don't add private connections in available connection list
    - MINOR: connection: Set the SNI on server connections before installing the mux
    - MINOR: connection: Set new connection as private on reuse never
    - MINOR: connection: Add a wrapper to mark a connection as private
    - MEDIUM: connection: Add private connections synchronously in session server list
    - MINOR: connection: Use a dedicated function to look for a session's connection
    - MINOR: connection: Set the conncetion target during its initialisation
    - MINOR: session: Take care to decrement idle_conns counter in session_unown_conn
    - MINOR: server: Factorize code to deal with reuse of server idle connections
    - MINOR: server: Factorize code to deal with connections removed from an idle list
    - CLEANUP: connection: remove unused field idle_time from the connection struct
    - BUG/MEDIUM: mux-h1: Continue to process request when switching in tunnel mode
    - MINOR: raw_sock: Report the number of bytes emitted using the splicing
    - MINOR: contrib/prometheus-exporter: Add missing global and per-server metrics
    - MINOR: backend: Add sample fetches to get the server's weight
    - BUG/MINOR: mux-fcgi: Handle empty STDERR record
    - BUG/MINOR: mux-fcgi: Set conn state to RECORD_P when skipping the record padding
    - BUG/MINOR: mux-fcgi: Set flags on the right stream field for empty FCGI_STDOUT
    - BUG/MINOR: backend: fix potential null deref on srv_conn
    - BUG/MEDIUM: log: issue mixing sampled to not sampled log servers.
    - MEDIUM: udp: adds minimal proto udp support for message listeners.
    - MEDIUM: log/sink: re-work and merge of build message API.
    - MINOR: log: adds syslog udp message handler and parsing.
    - MEDIUM: log: adds log forwarding section.
    - MINOR: log: adds counters on received syslog messages.
    - BUG/MEDIUM: fcgi-app: fix memory leak in fcgi_flt_http_headers
    - BUG/MEDIUM: server: resolve state file handle leak on reload
    - BUG/MEDIUM: server: fix possibly uninitialized state file on close
    - BUG/MEDIUM: channel: Be aware of SHUTW_NOW flag when output data are peeked
    - BUILD: config: address build warning on raspbian+rpi4
    - BUG/MAJOR: tasks: make sure to always lock the shared wait queue if needed
    - BUILD: config: fix again bugs gcc warnings on calloc
2020-07-17 15:13:19 +02:00
Emeric Brun
12941c82d0 MEDIUM: log: adds log forwarding section.
Log forwarding:

It is possible to declare one or multiple log forwarding section,
haproxy will forward all received log messages to a log servers list.

log-forward <name>
  Creates a new log forwarder proxy identified as <name>.

bind <addr> [param*]
  Used to configure a log udp listener to receive messages to forward.
  Only udp listeners are allowed, address must be prefixed using
  'udp@', 'udp4@' or 'udp6@'. This supports for all "bind" parameters
  found in 5.1 paragraph but most of them are irrelevant for udp/syslog case.

log global
log <address> [len <length>] [format <format>] [sample <ranges>:<smp_size>]
    <facility> [<level> [<minlevel>]]
  Used to configure target log servers. See more details on proxies
  documentation.
  If no format specified, haproxy tries to keep the incoming log format.
  Configured facility is ignored, except if incoming message does not
  present a facility but one is mandatory on the outgoing format.
  If there is no timestamp available in the input format, but the field
  exists in output format, haproxy will use the local date.

  Example:
    global
       log stderr format iso local7

    ring myring
        description "My local buffer"
        format rfc5424
        maxlen 1200
        size 32764
        timeout connect 5s
        timeout server 10s
        # syslog tcp server
        server mysyslogsrv 127.0.0.1:514 log-proto octet-count

    log-forward sylog-loadb
        bind udp4@127.0.0.1:1514
        # all messages on stderr
        log global
        # all messages on local tcp syslog server
        log ring@myring local0
        # load balance messages on 4 udp syslog servers
        log 127.0.0.1:10001 sample 1:4 local0
        log 127.0.0.1:10002 sample 2:4 local0
        log 127.0.0.1:10003 sample 3:4 local0
        log 127.0.0.1:10004 sample 4:4 local0
2020-07-15 17:50:12 +02:00
Emeric Brun
546488559a MEDIUM: log/sink: re-work and merge of build message API.
This patch merges build message code between sink and log
and introduce a new API based on struct ist array to
prepare message header with zero copy, targeting the
log forwarding feature.

Log format 'iso' and 'timed' are now avalaible on logs line.
A new log format 'priority' is also added.
2020-07-15 17:50:12 +02:00
Emeric Brun
3835c0dcb5 MEDIUM: udp: adds minimal proto udp support for message listeners.
This patch introduce proto_udp.c targeting a further support of
log forwarding feature.

This code was originally produced by Frederic Lecaille working on
QUIC support and only minimal requirements for syslog support
have been merged.
2020-07-15 17:50:12 +02:00
Christopher Faulet
1bea865811 MINOR: backend: Add sample fetches to get the server's weight
The following sample fetches have been added :

 * srv_iweight : returns the initial server's weight
 * srv_uweight : returns the user-visible server's weight
 * srv_weight  : returns the current (or effetctive) server's weight

The requested server must be passed as argument, evnetually preceded by the
backend name. For instance :

  srv_weight(back-http/www1)
2020-07-15 14:08:14 +02:00
Willy Tarreau
5254321d14 BUILD: tcp: condition TCP keepalive settings to platforms providing them
Previous commit b24bc0d ("MINOR: tcp: Support TCP keepalive parameters
customization") broke non-Linux builds as TCP_KEEP{CNT,IDLE,INTVL} are
not necessarily defined elsewhere.

This patch adds the required #ifdefs to condition the visibility of the
keywords, and adds a mention in the doc about their dependency on Linux.
2020-07-09 05:58:51 +02:00
MIZUTA Takeshi
b24bc0dfb6 MINOR: tcp: Support TCP keepalive parameters customization
It is now possible to customize TCP keepalive parameters.
These correspond to the socket options TCP_KEEPCNT, TCP_KEEPIDLE, TCP_KEEPINTVL
and are valid for the defaults, listen, frontend and backend sections.

This patch fixes GitHub issue #670.
2020-07-09 05:22:16 +02:00
William Dauchy
a5194607ab MINOR: config: make strict limits enabled by default
as agreed a few months ago, enable strict-limits for v2.3
update configuration manual accordingly

Signed-off-by: William Dauchy <w.dauchy@criteo.com>
2020-07-07 16:52:35 +02:00
Willy Tarreau
33205c23a7 [RELEASE] Released version 2.3-dev0
Released version 2.3-dev0 with the following main changes :
    - exact copy of 2.2.0
2020-07-07 16:35:28 +02:00
Willy Tarreau
3a00c915fd [RELEASE] Released version 2.2.0
Released version 2.2.0 with the following main changes :
    - BUILD: mux-h2: fix typo breaking build when using DEBUG_LOCK
    - CLEANUP: makefile: update the outdated list of DEBUG_xxx options
    - BUILD: tools: make resolve_sym_name() return a const
    - CLEANUP: auth: fix useless self-include of auth-t.h
    - BUILD: tree-wide: cast arguments to tolower/toupper to unsigned char
    - CLEANUP: assorted typo fixes in the code and comments
    - WIP/MINOR: ssl: add sample fetches for keylog in frontend
    - DOC: fix tune.ssl.keylog sample fetches array
    - BUG/MINOR: ssl: check conn in keylog sample fetch
    - DOC: configuration: various typo fixes
    - MINOR: log: Remove unused case statement during the log-format string parsing
    - BUG/MINOR: mux-h1: Fix the splicing in TUNNEL mode
    - BUG/MINOR: mux-h1: Don't read data from a pipe if the mux is unable to receive
    - BUG/MINOR: mux-h1: Disable splicing only if input data was processed
    - BUG/MEDIUM: mux-h1: Disable splicing for the conn-stream if read0 is received
    - MINOR: mux-h1: Improve traces about the splicing
    - BUG/MINOR: backend: Remove CO_FL_SESS_IDLE if a client remains on the last server
    - BUG/MEDIUM: connection: Don't consider new private connections as available
    - BUG/MINOR: connection: See new connection as available only on reuse always
    - DOC: configuration: remove obsolete mentions of H2 being converted to HTTP/1.x
    - CLEANUP: ssl: remove unrelevant comment in smp_fetch_ssl_x_keylog()
    - DOC: update INSTALL with new compiler versions
    - DOC: minor update to coding style file
    - MINOR: version: mention that it's an LTS release now
2020-07-07 16:33:14 +02:00
Willy Tarreau
bdc62096fc DOC: minor update to coding style file
It used to still refrence includes/{types,proto,common}/, now turned
to import/ and haproxy/.
2020-07-07 16:23:11 +02:00
Willy Tarreau
253c2519c2 DOC: configuration: remove obsolete mentions of H2 being converted to HTTP/1.x
The first H2 implementation in version 1.8 used to turn HTTP/2 requests
to HTTP/1.1, causing many limitations. This is not true anymore and we
don't suffer from the lack of server-side H2 nor are we forced to close
mode anymore, so let's remove such obsolete mentions.

This could be backported to 2.0.
2020-07-07 15:55:23 +02:00
Daniel Corbett
67a8271cc7 DOC: configuration: various typo fixes
Quick round of typo corrections for configuration.txt
2020-07-07 11:55:16 +02:00
William Lallemand
d742b6caf7 DOC: fix tune.ssl.keylog sample fetches array
The labels EXPORTER_SECRET and EARLY_EXPORTER_SECRET were swapped in the
array.
2020-07-07 10:25:15 +02:00