Commit Graph

1677 Commits

Author SHA1 Message Date
Christopher Faulet
b7d30098f3 MEDIUM: checks: Support expression to set the port
Since we have a session attached to tcp-check healthchecks, It is possible use
sample expression and variables. In addition, it is possible to add tcp-check
set-var rules to define custom variables. So, now, a sample expression can be
used to define the port to use to establish a connection for a tcp-check connect
rule. For instance:

    tcp-check set-var(check.port) int(8888)
    tcp-check connect port var(check.port)
2020-04-27 09:39:37 +02:00
Christopher Faulet
5c28874a69 MINOR: checks: Add the addr option for tcp-check connect rule
With this option, it is now possible to use a specific address to open the
connection for a tcp-check connect rule. If the port option is also specified,
it is used in priority.
2020-04-27 09:39:37 +02:00
Christopher Faulet
4dce5923c8 MINOR: checks: Add the default option for tcp-check connect rules
With this option, it is possible to open a connection from a tcp-check connect
rule using all parameter of the server line, like any other healthcheck. For
now, this parameter is exclusive with all other option for a tcp-check connect
rule.
2020-04-27 09:39:37 +02:00
Christopher Faulet
9857232c9f MINOR: checks: Add the alpn option for tcp-check connect rules
This option defines which protocols to advertise with ALPN on the SSL conection
opened by a tcp-check connect rule.
2020-04-27 09:39:37 +02:00
Christopher Faulet
085426aea9 MINOR: checks: Add the via-socks4 option for tcp-check connect rules
With this option, it is possible to establish the connection opened by a
tcp-check connect rule using upstream socks4 proxy. Info from the socks4
parameter on the server are used.
2020-04-27 09:39:37 +02:00
Christopher Faulet
79b31d4ee5 MINOR: checks: Add the sni option for tcp-check connect rules
With this option, it is possible to specify the SNI to be used for SSL
conncection opened by a tcp-check connect rule.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
0c39eccdd4 MINOR: checks: Add support to set-var and unset-var rules in tcp-checks
Evaluate the registered action_ptr associated with each CHK_ACTION_KW rules from
a ruleset. Currently only the 'set-var' and 'unset-var' are parsed by the
tcp-check parser. Thus it is now possible to set or unset variables. It is
possible to use such rules before the first connect of the ruleset.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
efab6c61d9 MINOR: checks: add rbinary expect match type
The rbinary match works similarly to the rstring match type, however the
received data is rewritten as hex-string before the match operation is
done.

This allows using regexes on binary content even with the POSIX regex
engine.

[Cf: I slightly updated the patch. mem2hex function was removed and dump_binary
is used instead.]
2020-04-27 09:39:37 +02:00
Gaetan Rivet
f8ba6773e5 MINOR: checks: add linger option to tcp connect
Allow declaring tcpcheck connect commands with a new parameter,
"linger". This option will configure the connection to avoid using an
RST segment to close, instead following the four-way termination
handshake. Some servers would otherwise log each healthcheck as
an error.
2020-04-27 09:39:37 +02:00
Gaetan Rivet
1afd826ae4 MINOR: checks: add min-recv tcp-check expect option
Some expect rules cannot be satisfied due to inherent ambiguity towards
the received data: in the absence of match, the current behavior is to
be forced to wait either the end of the connection or a buffer full,
whichever comes first. Only then does the matching diagnostic is
considered  conclusive. For instance :

    tcp-check connect
    tcp-check expect !rstring "^error"
    tcp-check expect string "valid"

This check will only succeed if the connection is closed by the server before
the check timeout. Otherwise the first expect rule will wait for more data until
"^error" regex matches or the check expires.

Allow the user to explicitly define an amount of data that will be
considered enough to determine the value of the check.

This allows succeeding on negative rstring rules, as previously
in valid condition no match happened, and the matching was repeated
until the end of the connection. This could timeout the check
while no error was happening.

[Cf: I slighly updated the patch. The parameter was renamed and the value is a
signed integer to support -1 as default value to ignore the parameter.]
2020-04-27 09:39:37 +02:00
Christopher Faulet
8acb1284bc MINOR: checks: Add a way to send custom headers and payload during http chekcs
The 'http-check send' directive have been added to add headers and optionnaly a
payload to the request sent during HTTP healthchecks. The request line may be
customized by the "option httpchk" directive but there was not official way to
add extra headers. An old trick consisted to hide these headers at the end of
the version string, on the "option httpchk" line. And it was impossible to add
an extra payload with an "http-check expect" directive because of the
"Connection: close" header appended to the request (See issue #16 for details).

So to make things official and fully support payload additions, the "http-check
send" directive have been added :

    option httpchk POST /status HTTP/1.1

    http-check send hdr Content-Type "application/json;charset=UTF-8" \
        hdr X-test-1 value1 hdr X-test-2 value2 \
        body "{id: 1, field: \"value\"}"

When a payload is defined, the Content-Length header is automatically added. So
chunk-encoded requests are not supported yet. For now, there is no special
validity checks on the extra headers.

This patch is inspired by Kiran Gavali's work. It should fix the issue #16 and
as far as possible, it may be backported, at least as far as 1.8.
2020-04-27 09:39:37 +02:00
Jerome Magnin
95fb57b923 DOC: option logasap does not depend on mode
The documentation for option logasap misleads into thinking it is
only valid for mode http. It is actually valid for mode tcp too,
so this patch tries to disambiguate the current wording.
2020-04-23 20:50:03 +02:00
Willy Tarreau
62ba9ba6ca BUG/MINOR: http: make url_decode() optionally convert '+' to SP
The url_decode() function used by the url_dec converter and a few other
call points is ambiguous on its processing of the '+' character which
itself isn't stable in the spec. This one belongs to the reserved
characters for the query string but not for the path nor the scheme,
in which it must be left as-is. It's only in argument strings that
follow the application/x-www-form-urlencoded encoding that it must be
turned into a space, that is, in query strings and POST arguments.

The problem is that the function is used to process full URLs and
paths in various configs, and to process query strings from the stats
page for example.

This patch updates the function to differentiate the situation where
it's parsing a path and a query string. A new argument indicates if a
query string should be assumed, otherwise it's only assumed after seeing
a question mark.

The various locations in the code making use of this function were
updated to take care of this (most call places were using it to decode
POST arguments).

The url_dec converter is usually called on path or url samples, so it
needs to remain compatible with this and will default to parsing a path
and turning the '+' to a space only after a question mark. However in
situations where it would explicitly be extracted from a POST or a
query string, it now becomes possible to enforce the decoding by passing
a non-null value in argument.

It seems to be what was reported in issue #585. This fix may be
backported to older stable releases.
2020-04-23 20:03:27 +02:00
William Lallemand
d7f8bd9b40 DOC: internals: update the SSL architecture schema
This commit updates the SSL files architecture schema and adds the
crtlist structures in it.
2020-04-23 16:30:12 +02:00
Jerome Magnin
b203ff6e20 MINOR: config: add a global directive to set default SSL curves
This commit adds a new keyword to the global section to set default
curves for ssl binds:
  - ssl-default-bind-curves
2020-04-22 17:26:08 +02:00
Emmanuel Hocdet
c3b7e74455 MINOR: ssl: add ssl-skip-self-issued-ca global option
This option activate the feature introduce in commit 16739778:
"MINOR: ssl: skip self issued CA in cert chain for ssl_ctx".
The patch disable the feature per default.
2020-04-22 15:35:56 +02:00
Olivier Doucet
56e3120f9e DOC: Improve documentation on http-request set-src
This patch adds more explanation on how to use "http-request set-src"
and a link to "option forwardfor".

This patch can be applied to all previous version starting at 1.6

Reviewed-by: Tim Duesterhus <tim@bastelstu.be>
2020-04-21 17:02:35 +02:00
Willy Tarreau
d008930265 [RELEASE] Released version 2.2-dev6
Released version 2.2-dev6 with the following main changes :
    - BUG/MINOR: ssl: memory leak when find_chain is NULL
    - CLEANUP: ssl: rename ssl_get_issuer_chain to ssl_get0_issuer_chain
    - MINOR: ssl: rework add cert chain to CTX to be libssl independent
    - BUG/MINOR: peers: init bind_proc to 1 if it wasn't initialized
    - BUG/MINOR: peers: avoid an infinite loop with peers_fe is NULL
    - BUG/MINOR: peers: Use after free of "peers" section.
    - CI: github actions: add weekly h2spec test
    - BUG/MEDIUM: mux_h1: Process a new request if we already received it.
    - MINOR: build: Fix build in mux_h1
    - CLEANUP: remove obsolete comments
    - BUG/MEDIUM: dns: improper parsing of aditional records
    - MINOR: ssl: skip self issued CA in cert chain for ssl_ctx
    - MINOR: listener: add so_name sample fetch
    - MEDIUM: stream: support use-server rules with dynamic names
    - MINOR: servers: Add a counter for the number of currently used connections.
    - MEDIUM: connections: Revamp the way idle connections are killed
    - MINOR: cli: add a general purpose pointer in the CLI struct
    - MINOR: ssl: add a list of bind_conf in struct crtlist
    - REORG: ssl: move SETCERT enum to ssl_sock.h
    - BUG/MINOR: ssl: ckch_inst wrongly inserted in crtlist_entry
    - REORG: ssl: move some functions above crtlist_load_cert_dir()
    - MINOR: ssl: use crtlist_free() upon error in directory loading
    - MINOR: ssl: add a list of crtlist_entry in ckch_store
    - MINOR: ssl: store a ptr to crtlist in crtlist_entry
    - MINOR: ssl/cli: update pointer to store in 'commit ssl cert'
    - MEDIUM: ssl/cli: 'add ssl crt-list' command
    - REGTEST: ssl/cli: test the 'add ssl crt-list' command
    - BUG/MINOR: ssl: entry->ckch_inst not initialized
    - REGTEST: ssl/cli: change test type to devel
    - REGTEST: make the PROXY TLV validation depend on version 2.2
    - CLEANUP: assorted typo fixes in the code and comments
    - BUG/MINOR: stats: Fix color of draining servers on stats page
    - DOC: internals: Fix spelling errors in filters.txt
    - MINOR: connections: Don't mark conn flags 0x00000001 and 0x00000002 as unused.
    - REGTEST: make the unique-id test depend on version 2.0
    - BUG/MEDIUM: dns: Consider the fact that dns answers are case-insensitive
    - MINOR: ssl: split the line parsing of the crt-list
    - MINOR: ssl/cli: support filters and options in add ssl crt-list
    - MINOR: ssl: add a comment above the ssl_bind_conf keywords
    - REGTEST: ssl/cli: tests options and filters w/ add ssl crt-list
    - REGTEST: ssl: pollute the crt-list file
    - BUG/CRITICAL: hpack: never index a header into the headroom after wrapping
    - BUG/MINOR: protocol_buffer: Wrong maximum shifting.
    - CLEANUP: src/fd.c: mask setsockopt with DISGUISE
    - BUG/MINOR: ssl/cli: initialize fcount int crtlist_entry
    - REGTEST: ssl/cli: add other cases of 'add ssl crt-list'
    - CLEANUP: assorted typo fixes in the code and comments
    - DOC: management: add the new crt-list CLI commands
    - BUG/MINOR: ssl/cli: fix spaces in 'show ssl crt-list'
    - MINOR: ssl/cli: 'del ssl crt-list' delete an entry
    - MINOR: ssl/cli: replace dump/show ssl crt-list by '-n' option
    - CI: use better SSL library definition
    - CI: travis-ci: enable DEBUG_STRICT=1 for CI builds
    - CI: travis-ci: upgrade openssl to 1.1.1f
    - MINOR: ssl: improve the errors when a crt can't be open
    - CI: cirrus-ci: rename openssl package after it is renamed in FreeBSD
    - CI: adopt openssl download script to download all versions
    - BUG/MINOR: ssl/cli: lock the ckch structures during crt-list delete
    - MINOR: ssl/cli: improve error for bundle in add/del ssl crt-list
    - MINOR: ssl/cli: 'del ssl cert' deletes a certificate
    - BUG/MINOR: ssl: trailing slashes in directory names wrongly cached
    - BUG/MINOR: ssl/cli: memory leak in 'set ssl cert'
    - CLEANUP: ssl: use the refcount for the SSL_CTX'
    - CLEANUP: ssl/cli: use the list of filters in the crtlist_entry
    - BUG/MINOR: ssl: memleak of the struct cert_key_and_chain
    - CLEANUP: ssl: remove a commentary in struct ckch_inst
    - MINOR: ssl: initialize all list in ckch_inst_new()
    - MINOR: ssl: free instances and SNIs with ckch_inst_free()
    - MINOR: ssl: replace ckchs_free() by ckch_store_free()
    - BUG/MEDIUM: ssl/cli: trying to access to free'd memory
    - MINOR: ssl: ckch_store_new() alloc and init a ckch_store
    - MINOR: ssl: crtlist_new() alloc and initialize a struct crtlist
    - REORG: ssl: move some free/new functions
    - MINOR: ssl: crtlist_entry_{new, free}
    - BUG/MINOR: ssl: ssl_conf always set to NULL on crt-list parsing
    - MINOR: ssl: don't alloc ssl_conf if no option found
    - BUG/MINOR: connection: always send address-less LOCAL PROXY connections
    - BUG/MINOR: peers: Incomplete peers sections should be validated.
    - MINOR: init: report in "haproxy -c" whether there were warnings or not
    - MINOR: init: add -dW and "zero-warning" to reject configs with warnings
    - MINOR: init: report the compiler version in haproxy -vv
    - CLEANUP: assorted typo fixes in the code and comments
    - MINOR: init: report the haproxy version and executable path once on errors
    - DOC: Make how "option redispatch" works more explicit
    - BUILD: Makefile: add linux-musl to TARGET
    - CLEANUP: assorted typo fixes in the code and comments
    - CLEANUP: http: Fixed small typo in parse_http_return
    - DOC: hashing: update link to hashing functions
2020-04-17 14:19:38 +02:00
Adam Mills
bbf697e752 DOC: hashing: update link to hashing functions
Bret Mulvey, the author of the article cited in this pulication
has migrated his work to papa.bretmulvey.com. I was able to
view an archival version of Bret M.'s original post
(http://home.comcast.net/~bretm/hash/3.html) and have validated
that this is the same paper that is originally cited.
2020-04-17 13:59:46 +02:00
Olivier Carrère
6e6f59b7e0 DOC: Make how "option redispatch" works more explicit
People are often misled and think that this option can redirect
connections to backup servers.

This patch makes the documentation more specific about how the option
handles backup servers.
2020-04-16 11:03:57 +02:00
Willy Tarreau
3eb10b8e98 MINOR: init: add -dW and "zero-warning" to reject configs with warnings
Since some systems switched to service managers which hide all warnings
by default, some users are not aware of some possibly important warnings
and get caught too late with errors that could have been detected earlier.

This patch adds a new global keyword, "zero-warning" and an equivalent
command-line option "-dW" to refuse to start in case any warning is
detected. It is recommended to use these with configurations that are
managed by humans in order to catch mistakes very early.
2020-04-15 16:42:39 +02:00
Willy Tarreau
bebd212064 MINOR: init: report in "haproxy -c" whether there were warnings or not
This helps quickly checking if the config produces any warning. For
this we reuse the "warned" bit field to add a new WARN_ANY bit that is
set by ha_warning(). The rest of the bit field was also cleaned from
unused bits.
2020-04-15 16:42:00 +02:00
William Lallemand
419e6349f6 MINOR: ssl/cli: 'del ssl cert' deletes a certificate
Delete a certificate store from HAProxy and free its memory. The
certificate must be unused and removed from any crt-list or directory.
The deletion doesn't work with a certificate referenced directly with
the "crt" directive in the configuration.
2020-04-08 12:08:03 +02:00
William Lallemand
c69f02d0f0 MINOR: ssl/cli: replace dump/show ssl crt-list by '-n' option
The dump and show ssl crt-list commands does the same thing, they dump
the content of a crt-list, but the 'show' displays an ID in the first
column. Delete the 'dump' command so it is replaced by the 'show' one.
The old 'show' command is replaced by an '-n' option to dump the ID.
And the ID which was a pointer is replaced by a line number and placed
after colons in the filename.

Example:
  $ echo "show ssl crt-list -n kikyo.crt-list" | socat /tmp/sock1 -
  # kikyo.crt-list
  kikyo.pem.rsa:1 secure.domain.tld
  kikyo.pem.ecdsa:2 secure.domain.tld
2020-04-06 19:33:33 +02:00
William Lallemand
0a9b9414f0 MINOR: ssl/cli: 'del ssl crt-list' delete an entry
Delete an entry in a crt-list, this is done by iterating over the
ckch_inst in the crtlist_entry. For each ckch_inst the bind_conf lock is
held during the deletion of the sni_ctx in the SNI trees. Everything
is free'd.

If there is several entries with the same certificate, a line number
must be provided to chose with entry delete.
2020-04-06 19:33:28 +02:00
William Lallemand
accac23aff DOC: management: add the new crt-list CLI commands
Add the new crt-lists command in the management manual:
"new ssl cert", "add ssl crt-list", "dump ssl crt-list", "show ssl
crt-list".
2020-04-02 17:50:02 +02:00
Miroslav Zagorac
d80f5c0d0c DOC: internals: Fix spelling errors in filters.txt 2020-03-31 17:24:07 +02:00
Jerome Magnin
824186bb08 MEDIUM: stream: support use-server rules with dynamic names
With server-template was introduced the possibility to scale the
number of servers in a backend without needing a configuration change
and associated reload. On the other hand it became impractical to
write use-server rules for these servers as they would only accept
existing server labels as argument. This patch allows the use of
log-format notation to describe targets of a use-server rules, such
as in the example below:

  listen test
    bind *:1234
    use-server %[hdr(srv)] if { hdr(srv) -m found }
    use-server s1 if { path / }
    server s1 127.0.0.1:18080
    server s2 127.0.0.1:18081

If a use-server rule is applied because it was conditionned by an
ACL returning true, but the target of the use-server rule cannot be
resolved, no other use-server rule is evaluated and we fall back to
load balancing.

This feature was requested on the ML, and bumped with issue #563.
2020-03-29 09:55:10 +02:00
Jerome Magnin
eb421b2fe0 MINOR: listener: add so_name sample fetch
Add a sample fetch for the name of a bind. This can be useful to
take decisions when PROXY protocol is used and we can't rely on dst,
such as the sample config below.

  defaults
    mode http
  listen bar
    bind 127.0.0.1:1111
    server s1 127.0.1.1:1234 send-proxy

  listen foo
    bind 127.0.1.1:1234 name foo accept-proxy
    http-request return status 200 hdr dst %[dst] if { dst 127.0.1.1 }
2020-03-29 05:47:29 +02:00
Willy Tarreau
3328f18596 [RELEASE] Released version 2.2-dev5
Released version 2.2-dev5 with the following main changes :
    - CLEANUP: ssl: is_default is a bit in ckch_inst
    - BUG/MINOR: ssl/cli: sni_ctx' mustn't always be used as filters
    - DOC: ssl: clarify security implications of TLS tickets
    - CLEANUP: remove support for Linux i686 vsyscalls
    - CLEANUP: drop support for USE_MY_ACCEPT4
    - CLEANUP: remove support for USE_MY_EPOLL
    - CLEANUP: remove support for USE_MY_SPLICE
    - CLEANUP: remove the now unused common/syscall.h
    - BUILD: make dladdr1 depend on glibc version and not __USE_GNU
    - BUILD: wdt: only test for SI_TKILL when compiled with thread support
    - BUILD: Makefile: the compiler-specific flags should all be in SPEC_CFLAGS
    - CLEANUP: ssl: separate the directory loading in a new function
    - BUG/MINOR: buffers: MT_LIST_DEL_SAFE() expects the temporary pointer.
    - BUG/MEDIUM: mt_lists: Make sure we set the deleted element to NULL;
    - MINOR: init: move the maxsock calculation code to compute_ideal_maxsock()
    - MEDIUM: init: always try to push the FD limit when maxconn is set from -m
    - BUG/MAJOR: list: fix invalid element address calculation
    - BUILD: stream-int: fix a few includes dependencies
    - MINOR: mt_lists: Appease gcc.
    - MINOR: lists: Implement function to convert list => mt_list and mt_list => list
    - MINOR: servers: Kill priv_conns.
    - MINOR: lists: fix indentation.
    - BUG/MEDIUM: random: align the state on 2*64 bits for ARM64
    - BUG/MEDIUM: connections: Don't assume the connection has a valid session.
    - BUG/MEDIUM: pools: Always update free_list in pool_gc().
    - BUG/MINOR: haproxy: always initialize sleeping_thread_mask
    - BUG/MINOR: listener/mq: do not dispatch connections to remote threads when stopping
    - BUG/MINOR: haproxy/threads: try to make all threads leave together
    - Revert "BUILD: travis-ci: enable s390x builds"
    - BUILD: travis-ci: enable regular s390x builds
    - DOC: proxy_protocol: Reserve TLV type 0x05 as PP2_TYPE_UNIQUE_ID
    - MINOR: proxy_protocol: Ingest PP2_TYPE_UNIQUE_ID on incoming connections
    - MEDIUM: proxy_protocol: Support sending unique IDs using PPv2
    - CLEANUP: connection: Add blank line after declarations in PP handling
    - CLEANUP: assorted typo fixes in the code and comments
    - CI: add spellcheck github action
    - DOC: correct typo in alert message about rspirep
    - CI: travis: switch linux builds to clang-9
    - MINOR: debug: add a new DISGUISE() macro to pass a value as identity
    - MINOR: debug: consume the write() result in BUG_ON() to silence a warning
    - MINOR: use DISGUISE() everywhere we deliberately want to ignore a result
    - BUILD: pools: silence build warnings with DEBUG_MEMORY_POOLS and DEBUG_UAF
    - CLEANUP: connection: Stop directly setting an ist's .ptr
    - CI: travis: revert to clang-7 for BoringSSL tests
    - BUILD: on ARM, must be linked to libatomic.
    - BUILD: makefile: fix regex syntax in ARM platform detection
    - BUG/MEDIUM: peers: resync ended with RESYNC_PARTIAL in wrong cases.
    - REORG: ssl: move ssl_sock_load_cert()
    - MINOR: ssl: pass ckch_inst to ssl_sock_load_ckchs()
    - MEDIUM: ssl: allow crt-list caching
    - MINOR: ssl: directories are loaded like crt-list
    - BUG/MINOR: ssl: can't open directories anymore
    - BUG/MEDIUM: spoe: dup agent's engine_id string from trash.area
    - MINOR: fd: Use a separate lock for logs instead of abusing the fd lock.
    - MINOR: mux_pt: Don't try to remove the connection from the idle list.
    - MINOR: ssl/cli: show/dump ssl crt-list
    - BUG/MINOR: ssl/cli: free the trash chunk in dump_crtlist
    - MEDIUM: fd: Introduce a running mask, and use it instead of the spinlock.
    - BUG/MINOR: ssl: memory leak in crtlist_parse_file()
    - MINOR: tasks: Provide the tasklet to the callback.
    - BUG/MINOR: ssl: memleak of struct crtlist_entry
    - BUG/MINOR: pattern: Do not pass len = 0 to calloc()
    - BUILD: makefile: fix expression again to detect ARM platform
    - CI: travis: re-enable ASAN on clang
    - CI: travis: proper group output redirection together with travis_wait
    - DOC: assorted typo fixes in the documentation
    - MINOR: wdt: Move the definitions of WDTSIG and DEBUGSIG into types/signal.h.
    - BUG/MEDIUM: wdt: Don't ignore WDTSIG and DEBUGSIG in __signal_process_queue().
    - MINOR: memory: Change the flush_lock to a spinlock, and don't get it in alloc.
    - MINOR: ssl/cli: 'new ssl cert' command
    - MINOR: ssl/cli: show certificate status in 'show ssl cert'
    - MEDIUM: sessions: Don't be responsible for connections anymore.
    - MEDIUM: servers: Split the connections into idle, safe, and available.
    - MINOR: fd: Implement fd_takeover().
    - MINOR: connections: Add a new mux method, "takeover".
    - MINOR: connections: Make the "list" element a struct mt_list instead of list.
    - MINOR: connections: Add a flag to know if we're in the safe or idle list.
    - MEDIUM: connections: Attempt to get idle connections from other threads.
    - MEDIUM: mux_h1: Implement the takeover() method.
    - MEDIUM: mux_h2: Implement the takeover() method.
    - MEDIUM: mux_fcgi: Implement the takeover() method.
    - MEDIUM: connections: Kill connections even if we are reusing one.
    - BUG/MEDIUM: connections: Don't forget to decrement idle connection counters.
    - BUG/MINOR: ssl: Do not free garbage pointers on memory allocation failure
    - BUG/MINOR: ssl: Correctly add the 1 for the sentinel to the number of elements
    - BUG/MINOR: ssl: crtlist_dup_filters() must return NULL with fcount == 0
    - BUG/MEDIUM: build: Fix compilation by spelling decl correctly.
    - BUILD/MEDIUM: fd: Declare fd_mig_lock as extern.
    - CI: run travis-ci builds on push only, skip pull requests
    - CI: temporarily disable unstable travis arm64 builds
    - BUG/MINOR: ssl/cli: free BIO upon error in 'show ssl cert'
    - BUG/MINOR: connections: Make sure we free the connection on failure.
    - BUG/MINOR: ssl/cli: fix a potential NULL dereference
    - BUG/MEDIUM: h1: Make sure we subscribe before going into idle list.
    - BUG/MINOR: connections: Set idle_time before adding to idle list.
    - MINOR: muxes: Note that we can't usee a connection when added to the srv idle.
    - REGTEST: increase timeouts on the seamless-reload test
    - BUG/MINOR: haproxy/threads: close a possible race in soft-stop detection
    - CLEANUP: haproxy/threads: don't check global_tasks_mask twice
2020-03-23 09:43:45 +01:00
Ilya Shipitsin
1fae8db7b7 DOC: assorted typo fixes in the documentation
This is the fourth round of cleanups in various docs
2020-03-18 11:34:33 +01:00
Tim Duesterhus
cf6e0c8a83 MEDIUM: proxy_protocol: Support sending unique IDs using PPv2
This patch adds the `unique-id` option to `proxy-v2-options`. If this
option is set a unique ID will be generated based on the `unique-id-format`
while sending the proxy protocol v2 header and stored as the unique id for
the first stream of the connection.

This feature is meant to be used in `tcp` mode. It works on HTTP mode, but
might result in inconsistent unique IDs for the first request on a keep-alive
connection, because the unique ID for the first stream is generated earlier
than the others.

Now that we can send unique IDs in `tcp` mode the `%ID` log variable is made
available in TCP mode.
2020-03-13 17:26:43 +01:00
Tim Duesterhus
d1b15b6e9b MINOR: proxy_protocol: Ingest PP2_TYPE_UNIQUE_ID on incoming connections
This patch reads a proxy protocol v2 provided unique ID and makes it
available using the `fc_pp_unique_id` fetch.
2020-03-13 17:25:23 +01:00
Tim Duesterhus
b435f77620 DOC: proxy_protocol: Reserve TLV type 0x05 as PP2_TYPE_UNIQUE_ID
This reserves and defines TLV type 0x05.
2020-03-13 17:25:23 +01:00
Lukas Tribus
bdb386d3d9 DOC: ssl: clarify security implications of TLS tickets
Clarifies security implications of TLS ticket usage when not
rotating TLS ticket keys, after commit 7b5e136458 ("DOC:
improve description of no-tls-tickets").
2020-03-10 03:52:22 +01:00
Willy Tarreau
5a753bd7b7 [RELEASE] Released version 2.2-dev4
Released version 2.2-dev4 with the following main changes :
    - MEDIUM: buffer: remove the buffer_wq lock
    - MINOR: ssl: move find certificate chain code to its own function
    - MINOR: ssl: resolve issuers chain later
    - MINOR: ssl: resolve ocsp_issuer later
    - MINOR: ssl/cli: "show ssl cert" command should print the "Chain Filename:"
    - BUG/MINOR: h2: reject again empty :path pseudo-headers
    - MINOR: wdt: always clear sigev_value to make valgrind happy
    - MINOR: epoll: always initialize all of epoll_event to please valgrind
    - BUG/MINOR: sample: Make sure to return stable IDs in the unique-id fetch
    - BUG/MEDIUM: ssl: chain must be initialized with sk_X509_new_null()
    - BUILD: cirrus-ci: suppress OS version check when installing packages
    - BUG/MINOR: http_ana: make sure redirect flags don't have overlapping bits
    - CLEANUP: fd: remove the FD_EV_STATUS aggregate
    - CLEANUP: fd: remove some unneeded definitions of FD_EV_* flags
    - MINOR: fd: merge the read and write error bits into RW error
    - BUG/MINOR: dns: ignore trailing dot
    - MINOR: contrib/prometheus-exporter: Add the last heathcheck duration metric
    - BUG/MINOR: http-htx: Do case-insensive comparisons on Host header name
    - MINOR: mux-h1: Remove useless case-insensitive comparisons
    - MINOR: rawsock: always mark the FD not ready when we're certain it happens
    - MEDIUM: connection: make the subscribe() call able to wakeup if ready
    - MEDIUM: connection: don't stop receiving events in the FD handler
    - MEDIUM: mux-h1: do not blindly wake up the tasklet at end of request anymore
    - BUG/MINOR: arg: don't reject missing optional args
    - MINOR: tools: make sure to correctly check the returned 'ms' in date2std_log
    - MINOR: debug: report the task handler's pointer relative to main
    - BUG/MEDIUM: debug: make the debug_handler check for the thread in threads_to_dump
    - MINOR: haproxy: export main to ease access from debugger
    - MINOR: haproxy: export run_poll_loop
    - MINOR: task: export run_tasks_from_list
    - BUILD: tools: remove obsolete and conflicting trace() from standard.c
    - MINOR: tools: add new function dump_addr_and_bytes()
    - MINOR: tools: add resolve_sym_name() to resolve function pointers
    - MINOR: debug: use resolve_sym_name() to dump task handlers
    - MINOR: cli: make "show fd" rely on resolve_sym_name()
    - MEDIUM: debug: add support for dumping backtraces of stuck threads
    - MINOR: debug: call backtrace() once upon startup
    - MINOR: ssl: add "ca-verify-file" directive
    - BUG/MINOR: wdt: do not return an error when the watchdog couldn't be enabled
    - BUILD: Makefile: include librt before libpthread
    - MEDIUM: wdt: fall back to CLOCK_REALTIME if CLOCK_THREAD_CPUTIME is not available
    - MINOR: wdt: do not depend on USE_THREAD
    - MINOR: debug: report the number of entries in the backtrace
    - MINOR: debug: improve backtrace() on aarch64 and possibly other systems
    - MINOR: debug: use our own backtrace function on clang+x86_64
    - MINOR: debug: dump the whole trace if we can't spot the starting point
    - BUILD: tools: unbreak resolve_sym_name() on non-GNU platforms
    - BUILD: tools: rely on __ELF__ not USE_DL to enable use of dladdr()
    - CLEANUP: contrib/spoa_example: Fix several typos
    - BUILD: makefile: do not modify the build options during make reg-tests
    - BUG/MEDIUM: connection: stop polling for sending when the event is ready
    - MEDIUM: stream-int: make sure to try to immediately validate the connection
    - MINOR: tcp/uxst/sockpair: only ask for I/O when really waiting for a connect()
    - MEDIUM: connection: only call ->wake() for connect() without I/O
    - OPTIM: connection: disable receiving on disabled events when the run queue is too high
    - OPTIM: mux-h1: subscribe rather than waking up at a few other places
    - REGTEST: Add unique-id reg-test
    - MINOR: stream: Add stream_generate_unique_id function
    - MINOR: stream: Use stream_generate_unique_id
    - BUG/MINOR: connection/debug: do not enforce !event_type on subscribe() anymore
    - MINOR: ssl/cli: support crt-list filters
    - MINOR: ssl: reach a ckch_store from a sni_ctx
    - DOC: fix incorrect indentation of http_auth_*
    - BUG/MINOR: ssl-sock: do not return an uninitialized pointer in ckch_inst_sni_ctx_to_sni_filters
    - MINOR: debug: add CLI command "debug dev write" to write an arbitrary size
    - MINOR: ist: Add `IST_NULL` macro
    - MINOR: ist: Add `int isttest(const struct ist)`
    - MINOR: ist: Add `struct ist istalloc(size_t)` and `void istfree(struct ist*)`
    - CLEANUP: Use `isttest()` and `istfree()`
    - MINOR: ist: Add `struct ist istdup(const struct ist)`
    - MINOR: proxy: Make `header_unique_id` a `struct ist`
    - MEDIUM: stream: Make the `unique_id` member of `struct stream` a `struct ist`
    - OPTIM: startup: fast unique_id allocation for acl.
    - DOC: configuration.txt: fix various typos
    - DOC: assorted typo fixes in the documentation and Makefile
    - BUG/MINOR: init: make the automatic maxconn consider the max of soft/hard limits
    - BUG/MAJOR: proxy_protocol: Properly validate TLV lengths
    - CLEANUP: proxy_protocol: Use `size_t` when parsing TLVs
    - MINOR: buf: Add function to insert a string at an absolute offset in a buffer
    - MINOR: htx: Add a function to return a block at a specific offset
    - MINOR: htx: Use htx_find_offset() to truncate an HTX message
    - MINOR: flt_trace: Use htx_find_offset() to get the available payload length
    - BUG/MINOR: filters: Use filter offset to decude the amount of forwarded data
    - BUG/MINOR: filters: Forward everything if no data filters are called
    - BUG/MEDIUM: cache/filters: Fix loop on HTX blocks caching the response payload
    - BUG/MEDIUM: compression/filters: Fix loop on HTX blocks compressing the payload
    - BUG/MINOR: http-ana: Reset request analysers on a response side error
    - BUG/MINOR: lua: Abort when txn:done() is called from a Lua action
    - BUG/MINOR: lua: Ignore the reserve to know if a channel is full or not
    - MINOR: lua: Add function to know if a channel is a response one
    - MINOR: lua: Stop using the lua txn in hlua_http_get_headers()
    - MINOR: lua: Stop using the lua txn in hlua_http_rep_hdr()
    - MINOR: lua: Stop using lua txn in hlua_http_del_hdr() and hlua_http_add_hdr()
    - MINOR: lua: Remove the flag HLUA_TXN_HTTP_RDY
    - MINOR: lua: Rename hlua_action_wake_time() to hlua_set_wake_time()
    - BUG/MINOR: lua: Init the lua wake_time value before calling a lua function
    - BUG/MINOR: http-rules: Return ACT_RET_ABRT to abort a transaction
    - BUG/MINOR: http-rules: Preserve FLT_END analyzers on reject action
    - BUG/MINOR: http-rules: Fix a typo in the reject action function
    - MINOR: cache/filters: Initialize the cache filter when stream is created
    - MINOR: compression/filters: Initialize the comp filter when stream is created
    - BUG/MINOR: rules: Preserve FLT_END analyzers on silent-drop action
    - BUG/MINOR: rules: Return ACT_RET_ABRT when a silent-drop action is executed
    - BUG/MINOR: rules: Increment be_counters if backend is assigned for a silent-drop
    - BUG/MINOR: http-rules: Abort transaction when a redirect is applied on response
    - BUILD: buffer: types/{ring.h,checks.h} should include buf.h, not buffer.h
    - BUILD: ssl: include mini-clist.h
    - BUILD: global: must not include common/standard.h but only types/freq_ctr.h
    - BUILD: freq_ctr: proto/freq_ctr needs to include common/standard.h
    - BUILD: listener: types/listener.h must not include standard.h
    - BUG/MEDIUM: random: initialize the random pool a bit better
    - BUG/MEDIUM: random: implement per-thread and per-process random sequences
    - Revert "BUG/MEDIUM: random: implement per-thread and per-process random sequences"
    - BUILD: cirrus-ci: get rid of unstable freebsd images
    - MINOR: tools: add 64-bit rotate operators
    - BUG/MEDIUM: random: implement a thread-safe and process-safe PRNG
    - MINOR: backend: use a single call to ha_random32() for the random LB algo
    - BUG/MINOR: checks/threads: use ha_random() and not rand()
    - MINOR: sample: make all bits random on the rand() sample fetch
    - MINOR: tools: add a generic function to generate UUIDs
    - DOC: fix typo about no-tls-tickets
    - DOC: improve description of no-tls-tickets
    - DOC: assorted typo fixes in the documentation
    - CLEANUP: remove unused code in 'my_ffsl/my_flsl' functions
2020-03-09 14:57:20 +01:00
Ilya Shipitsin
2075ca8a93 DOC: assorted typo fixes in the documentation
This is the third round of cleanups in various docs
2020-03-09 14:45:58 +01:00
Björn Jacke
7b5e136458 DOC: improve description of no-tls-tickets
It was not obvious, that this setting only affects TLS versions <= 1.2 and it
we should also mention the security implication of session tickets here.

Signed-off-by: Bjoern Jacke <bjacke@samba.org>
2020-03-09 14:45:05 +01:00
Bjoern Jacke
5ab7eb6860 DOC: fix typo about no-tls-tickets
It's "no-tls-tickets", not "no-tlsv-tickets"

Signed-off-by: Bjoern Jacke <bjacke@samba.org>
2020-03-09 14:45:02 +01:00
Ilya Shipitsin
2a950d02a9 DOC: assorted typo fixes in the documentation and Makefile
This is another round of cleanups in various docs and comments in the
Makefile.
2020-03-06 10:49:55 +01:00
Ilya Shipitsin
8525fd95b2 DOC: configuration.txt: fix various typos
This was done using automatic spellcheck.
2020-03-06 08:25:14 +01:00
Willy Tarreau
c9c6cdbf9c DOC: fix incorrect indentation of http_auth_*
These ones were incorrectly indented and thus not displayed optimally
in the HTML version. This addresses issue #533.
2020-03-05 16:05:05 +01:00
Emmanuel Hocdet
842e94ee06 MINOR: ssl: add "ca-verify-file" directive
It's only available for bind line. "ca-verify-file" allows to separate
CA certificates from "ca-file". CA names sent in server hello message is
only compute from "ca-file". Typically, "ca-file" must be defined with
intermediate certificates and "ca-verify-file" with certificates to
ending the chain, like root CA.

Fix issue #404.
2020-03-04 11:53:11 +01:00
Willy Tarreau
32bf97fb60 [RELEASE] Released version 2.2-dev3
Released version 2.2-dev3 with the following main changes :
    - SCRIPTS: announce-release: place the send command in the mail's header
    - SCRIPTS: announce-release: allow the user to force to overwrite old files
    - SCRIPTS: backport: fix the master branch detection
    - BUG/MINOR: http-act: Set stream error flag before returning an error
    - BUG/MINOR: http-act: Fix bugs on error path during parsing of return actions
    - BUG/MEDIUM: ssl/cli: 'commit ssl cert' wrong SSL_CTX init
    - BUG/MEDIUM: tcp-rules: Fix track-sc* actions for L4/L5 TCP rules
    - DOC: schematic of the SSL certificates architecture
    - BUG/MAJOR: mux-h2: don't wake streams after connection was destroyed
    - BUG/MINOR: unix: better catch situations where the unix socket path length is close to the limit
    - BUILD: cirrus-ci: switch to "snap" images to unify openssl naming
    - BUILD: cirrus-ci: workaround "pkg install" bug
    - BUILD: cirrus-ci: add ERR=1 to freebsd builds
    - BUG/MINOR: connection: correctly retry I/O on signals
    - CLEANUP: mini-clist: simplify nested do { while(1) {} } while (0)
    - BUILD: http_act: cast file sizes when reporting file size error
    - BUG/MEDIUM: listener: only consider running threads when resuming listeners
    - BUG/MINOR: listener: enforce all_threads_mask on bind_thread on init
    - BUG/MINOR: tcp: avoid closing fd when socket failed in tcp_bind_listener
    - MINOR: build: add aix72-gcc build TARGET and power{8,9} CPUs
    - BUILD: travis-ci: no more allowed failures for openssl-1.0.2
    - BUILD: travis-ci: harden builds, add ERR=1 (warning ought to be errors)
    - BUILD: scripts/build-ssl.sh: use "uname" instead of ${TRAVIS_OS_NAME}
    - BUG/MINOR: tcp: don't try to set defaultmss when value is negative
    - SCRIPTS: make announce-release executable again
    - BUG/MINOR: namespace: avoid closing fd when socket failed in my_socketat
    - BUG/MEDIUM: muxes: Use the right argument when calling the destroy method.
    - BUG/MINOR: mux-fcgi: Forbid special characters when matching PATH_INFO param
    - CLEANUP: ssl: remove unused functions in openssl-compat.h
    - MINOR: mux-fcgi: Make the capture of the path-info optional in pathinfo regex
    - MINOR: tools: add is_idchar() to tell if a char may belong to an identifier
    - MINOR: chunk: implement chunk_strncpy() to copy partial strings
    - MINOR: sample/acl: use is_idchar() to locate the fetch/conv name
    - MEDIUM: arg: make make_arg_list() stop after its own arguments
    - MEDIUM: arg: copy parsed arguments into the trash instead of allocating them
    - MEDIUM: arg: make make_arg_list() support quotes in arguments
    - MINOR: sample: make sample_parse_expr() able to return an end pointer
    - MEDIUM: log-format: make the LF parser aware of sample expressions' end
    - BUG/MINOR: arg: report an error if an argument is larger than bufsize
    - SCRIPTS: announce-release: use mutt -H instead of -i to include the draft
    - BUILD: enable ERR=1 in github cygwin builds
    - BUG/MINOR: arg: fix again incorrect argument length check
    - MINOR: sample: regsub now supports backreferences
    - BUG/MINOR: tools: also accept '+' as a valid character in an identifier
    - MINOR: http-htx: Add a function to retrieve the headers size of an HTX message
    - MINOR: filters: Forward data only if the last filter forwards something
    - BUG/MINOR: filters: Count HTTP headers as filtered data but don't forward them
    - BUG/MINOR: http-htx: Don't return error if authority is updated without changes
    - BUG/MINOR: stream: Don't incr frontend cum_req counter when stream is closed
    - BUG/MINOR: sample: exit regsub() in case of trash allocation error
    - MINOR: ssl: add "issuers-chain-path" directive.
    - REGTESTS: use "command -v" instead of "which"
    - BUG/MINOR: http-ana: Matching on monitor-uri should be case-sensitive
    - MINOR: http-ana: Match on the path if the monitor-uri starts by a /
    - BUG/MINOR: ssl: Stop passing dynamic strings as format arguments
    - BUG/MAJOR: http-ana: Always abort the request when a tarpit is triggered
    - BUG/MINOR: mux: do not call conn_xprt_stop_recv() on buffer shortage
    - MINOR: checks: do not call conn_xprt_stop_send() anymore
    - CLEANUP: epoll: place the struct epoll_event in the stack
    - MEDIUM: connection: remove the intermediary polling state from the connection
    - MINOR: raw_sock: directly call fd_stop_send() and not conn_xprt_stop_send()
    - MINOR: tcp/uxst/sockpair: use fd_want_send() instead of conn_xprt_want_send()
    - MINOR: connection: remove the last calls to conn_xprt_{want,stop}_*
    - CLEANUP: connection: remove the definitions of conn_xprt_{stop,want}_{send,recv}
    - MINOR: connection: introduce a new receive flag: CO_RFL_READ_ONCE
    - MINOR: mux-h1: pass CO_RFL_READ_ONCE to the lower layers when relevant
    - MINOR: ist: add an iststop() function
    - BUG/MINOR: http: http-request replace-path duplicates the query string
    - CLEANUP: sample: use iststop instead of a for loop
    - BUG/MEDIUM: shctx: make sure to keep all blocks aligned
    - MINOR: compiler: move CPU capabilities definition from config.h and complete them
    - BUG/MEDIUM: ebtree: don't set attribute packed without unaligned access support
    - CLEANUP: http/h1: rely on HA_UNALIGNED_LE instead of checking for CPU families
    - BUILD: fix recent build failure on unaligned archs
    - MINOR: ssl: load the key from a dedicated file
    - BUG/MINOR: ssl: load .key in a directory only after PEM
    - MINOR: compiler: drop special cases of likely/unlikely for older compilers
    - CLEANUP: conn: Do not pass a pointer to likely
    - CLEANUP: net_helper: Do not negate the result of unlikely
    - BUILD: remove obsolete support for -mregparm / USE_REGPARM
    - CLEANUP: cfgparse: Fix type of second calloc() parameter
    - BUILD: ssl: only pass unsigned chars to isspace()
    - BUILD: general: always pass unsigned chars to is* functions
    - BUG/MINOR: sample: fix the json converter's endian-sensitivity
    - BUG/MEDIUM: ssl: fix several bad pointer aliases in a few sample fetch functions
    - CLEANUP: fd: use a union in fd_rm_from_fd_list() to shut aliasing warnings
    - CLEANUP: cache: use read_u32/write_u32 to access the cache entry's hash
    - CLEANUP: stick-tables: use read_u32() to display a node's key
    - CLEANUP: sample: use read_u64() in ipmask() to apply an IPv6 mask
    - MINOR: pattern: fix all remaining strict aliasing issues
    - CLEANUP: lua: fix aliasing issues in the address matching code
    - CLEANUP: connection: use read_u32() instead of a cast in the netscaler parser
    - BUILD: makefile: re-enable strict aliasing
    - BUG/MINOR: connection: make sure to correctly tag local PROXY connections
    - MINOR: compiler: add new alignment macros
    - BUILD: ebtree: improve architecture-specific alignment
    - MINOR: config: mark global.debug as deprecated
    - BUILD: travis-ci: enable s390x builds
    - MINOR: ssl/cli: 'show ssl cert' displays the chain
    - MINOR: ssl/cli: 'show ssl cert'displays the issuer in the chain
    - MINOR: ssl/cli: reorder 'show ssl cert' output
    - CLEANUP: ssl: move issuer_chain tree and definition
    - DOC: proxy-protocol: clarify IPv6 address representation in the spec
2020-02-25 18:14:02 +01:00
Willy Tarreau
269a9b61b8 DOC: proxy-protocol: clarify IPv6 address representation in the spec
Daniel Barclay reported that the wording around "IPv6 addresses must be
indicated as series of 4 hex digits" is confusing and can be interpreted
two ways (only 4 digits or series of sets of 4 digits), so let's adjust
the wording to resolve this ambiguity.
2020-02-25 18:06:32 +01:00
Willy Tarreau
1b85785bc2 MINOR: config: mark global.debug as deprecated
This directive has never made any sense and has already caused trouble
by forcing the process to stay in foreground during the boot process.
Let's emit a warning mentioning it's deprecated and will be removed in
2.3.
2020-02-25 11:28:58 +01:00
William Lallemand
3f25ae31bd BUG/MINOR: ssl: load .key in a directory only after PEM
Don't try to load a .key in a directory without loading its associated
certificate file.

This patch ignores the .key files when iterating over the files in a
directory.

Introduced by 4c5adbf ("MINOR: ssl: load the key from a dedicated
file").
2020-02-24 16:34:16 +01:00
William Lallemand
4c5adbf595 MINOR: ssl: load the key from a dedicated file
For a certificate on a bind line, if the private key was not found in
the PEM file, look for a .key and load it.

This default behavior can be changed by using the ssl-load-extra-files
directive in the global section

This feature was mentionned in the issue #221.
2020-02-24 15:39:53 +01:00
Christopher Faulet
6072beb214 MINOR: http-ana: Match on the path if the monitor-uri starts by a /
if the monitor-uri starts by a slash ('/'), the matching is performed against
the request's path instead of the request's uri. It is a workaround to let the
HTTP/2 requests match the monitor-uri. Indeed, in HTTP/2, clients are encouraged
to send absolute URIs only.

This patch is not tagged as a bug, because the previous behavior matched exactly
what the doc describes. But it may surprise that HTTP/2 requests don't match the
monitor-uri.

This patch may be backported to 2.1 because URIs of HTTP/2 are stored using the
absolute-form starting this version. For previous versions, this patch will only
helps explicitely absolute HTTP/1 requests (and only the HTX part because on the
legacy HTTP, all the URI is matched).

It should fix the issue #509.
2020-02-18 16:29:29 +01:00
Emmanuel Hocdet
70df7bf19c MINOR: ssl: add "issuers-chain-path" directive.
Certificates loaded with "crt" and "crt-list" commonly share the same
intermediate certificate in PEM file. "issuers-chain-path" is a global
directive to share intermediate chain certificates in a directory. If
certificates chain is not included in certificate PEM file, haproxy
will complete chain if issuer match the first certificate of the chain
stored via "issuers-chain-path" directive. Such chains will be shared
in memory.
2020-02-18 14:33:05 +01:00