Commit Graph

1754 Commits

Author SHA1 Message Date
Willy Tarreau
c6e7a1b8e9 MINOR: cli: make "show sess" stop at the last known session
"show sess" and particularly "show sess all" can be very slow when dumping
lots of information, and while dumping, new sessions might appear, making
the output really endless. When threads are used, this causes a double
problem:
  - all threads are paused during the dump, so an overly long dump degrades
    the quality of service ;

  - since all threads are paused, more events get postponed, possibly
    resulting in more streams to be dumped on next invocation of the dump
    function.

This patch addresses this long-lasting issue by doing something simple:
the CLI's stream is moved at the end of the steams list, serving as an
identifiable marker to end the dump, because all entries past it were
added after the command was entered. As a result, the CLI's stream always
appears as the last one.

It may make sense to backport this to stable branches where dumping live
streams is difficult as well.
2020-06-28 01:41:56 +02:00
Willy Tarreau
4462af8da2 [RELEASE] Released version 2.2-dev11
Released version 2.2-dev11 with the following main changes :
    - REGTEST: Add a simple script to tests errorfile directives in proxy sections
    - BUG/MEDIUM: fcgi-app: Resolve the sink if a fcgi-app logs in a ring buffer
    - BUG/MINOR: spoe: correction of setting bits for analyzer
    - BUG/MINOR: cfgparse: Support configurations without newline at EOF
    - MINOR: cfgparse: Warn on truncated lines / files
    - BUG/MINOR: http_ana: clarify connection pointer check on L7 retry
    - MINOR: debug: add a new DEBUG_FD build option
    - BUG/MINOR: tasks: make sure never to exceed max_processed
    - MINOR: task: add a new pointer to current tasklet queue
    - BUG/MEDIUM: task: be careful not to run too many tasks at TL_URGENT
    - BUG/MINOR: cfgparse: Fix argument reference in PARSE_ERR_TOOMANY message
    - BUG/MINOR: cfgparse: Fix calculation of position for PARSE_ERR_TOOMANY message
    - BUG/MEDIUM: ssl: fix ssl_bind_conf double free
    - MINOR: ssl: free bind_conf_node in crtlist_free()
    - MINOR: ssl: free the crtlist and the ckch during the deinit()
    - BUG/MINOR: ssl: fix build with ckch_deinit() and crtlist_deinit()
    - BUG/MINOR: ssl/cli: certs added from the CLI can't be deleted
    - MINOR: ssl: move the ckch/crtlist deinit to ssl_sock.c
    - MEDIUM: tasks: apply a fair CPU distribution between tasklet classes
    - MINOR: tasks: make current_queue an index instead of a pointer
    - MINOR: tasks: add a mask of the queues with active tasklets
    - MINOR: tasks: pass the queue index to run_task_from_list()
    - MINOR: tasks: make run_tasks_from_lists() scan the queues itself
    - MEDIUM: tasks: add a tune.sched.low-latency option
    - BUG/MEDIUM: ssl/cli: 'commit ssl cert' crashes when no private key
    - BUG/MINOR: cfgparse: don't increment linenum on incomplete lines
    - MINOR: tools: make parse_line() always terminate the args list
    - BUG/MINOR: cfgparse: report extraneous args *after* the string is allocated
    - MINOR: cfgparse: sanitize the output a little bit
    - MINOR: cli/ssl: handle trailing slashes in crt-list commands
    - MINOR: ssl: add the ssl_s_* sample fetches for server side certificate
    - BUG/MEDIUM: http-ana: Don't loop trying to generate a malformed 500 response
    - BUG/MINOR: stream-int: Don't wait to send truncated HTTP messages
    - BUG/MINOR: http-ana: Set CF_EOI on response channel for generated responses
    - BUG/MINOR: http-ana: Don't wait to send 1xx responses generated by HAProxy
    - MINOR: spoe: Don't systematically create new applets if processing rate is low
    - DOC: fix some typos in the ssl_s_{s|i}_dn documentation
    - BUILD: fix ssl_sample.c when building against BoringSSL
    - CI: travis-ci: switch BoringSSL builds to ninja
    - CI: extend spellchecker whitelist
    - DOC: assorted typo fixes in the documentation
    - CLEANUP: assorted typo fixes in the code and comments
    - MINOR: http: Add support for http 413 status
    - REGTEST: ssl: tests the ssl_f_* sample fetches
    - REGTEST: ssl: add some ssl_c_* sample fetches test
    - DOC: ssl: update the documentation of "commit ssl cert"
    - BUG/MINOR: cfgparse: correctly deal with empty lines
    - BUG/MEDIUM: fetch: Fix hdr_ip misparsing IPv4 addresses due to missing NUL
2020-06-26 22:01:04 +02:00
William Lallemand
c184d87558 DOC: ssl: update the documentation of "commit ssl cert"
Update the documentation of "commit ssl cert" in management.txt to
explain the behavior with new certificates.
2020-06-26 15:39:57 +02:00
Anthonin Bonnefoy
85048f80c9 MINOR: http: Add support for http 413 status
Add 413 http "payload too large" status code. This will allow 413 to be
used in deny_status and errorfile.
2020-06-26 11:30:02 +02:00
Ilya Shipitsin
11057a3590 DOC: assorted typo fixes in the documentation
this is 10th iteration of typo fixes
2020-06-26 11:27:10 +02:00
William Lallemand
8f600c833a DOC: fix some typos in the ssl_s_{s|i}_dn documentation
Fix some typos in the ssl_s_{s|i}_dn documentation.
2020-06-26 09:55:06 +02:00
William Lallemand
bfa3e81a7b MINOR: ssl: add the ssl_s_* sample fetches for server side certificate
This commit adds some sample fetches that were lacking on the server
side:

ssl_s_key_alg, ssl_s_notafter, ssl_s_notbefore, ssl_s_sig_alg,
ssl_s_i_dn, ssl_s_s_dn, ssl_s_serial, ssl_s_sha1, ssl_s_der,
ssl_s_version
2020-06-25 21:22:24 +02:00
Willy Tarreau
e7723bddd7 MEDIUM: tasks: add a tune.sched.low-latency option
Now that all tasklet queues are scanned at once by run_tasks_from_lists(),
it becomes possible to always check for lower priority classes and jump
back to them when they exist.

This patch adds tune.sched.low-latency global setting to enable this
behavior. What it does is stick to the lowest ranked priority list in
which tasks are still present with an available budget, and leave the
loop to refill the tasklet lists if the trees got new tasks or if new
work arrived into the shared urgent queue.

Doing so allows to cut the latency in half when running with extremely
deep run queues (10k-100k), thus allowing forwarding of small and large
objects to coexist better. It remains off by default since it does have
a small impact on large traffic by default (shorter batches).
2020-06-24 12:21:26 +02:00
Willy Tarreau
dc0936c255 [RELEASE] Released version 2.2-dev10
Released version 2.2-dev10 with the following main changes :
    - BUILD: include: add sys/types before netinet/tcp.h
    - BUG/MEDIUM: log: don't hold the log lock during writev() on a file descriptor
    - BUILD: Remove nowarn for warnings that do not trigger
    - BUG/MEDIUM: pattern: fix thread safety of pattern matching
    - BUILD: Re-enable -Wimplicit-fallthrough
    - BUG/MINOR: ssl: fix ssl-{min,max}-ver with openssl < 1.1.0
    - BUILD: thread: add parenthesis around values of locking macros
    - BUILD: proto_uxst: shut up yet another gcc's absurd warning
    - BUG/MEDIUM: checks: Fix off-by-one in allocation of SMTP greeting cmd
    - CI: travis-ci: use "-O1" for clang builds
    - MINOR: haproxy: Add void deinit_and_exit(int)
    - MINOR: haproxy: Make use of deinit_and_exit() for clean exits
    - BUG/MINOR: haproxy: Free rule->arg.vars.expr during deinit_act_rules
    - BUILD: compression: make gcc 10 happy with free_zlib()
    - BUILD: atomic: add string.h for memcpy() on ARM64
    - BUG/MINOR: http: make smp_fetch_body() report that the contents may change
    - BUG/MINOR: tcp-rules: tcp-response must check the buffer's fullness
    - BUILD: haproxy: mark deinit_and_exit() as noreturn
    - BUG/MAJOR: vars: Fix bogus free() during deinit() for http-request rules
    - BUG/MEDIUM: ebtree: use a byte-per-byte memcmp() to compare memory blocks
    - MINOR: tools: add a new configurable line parse, parse_line()
    - BUG/MEDIUM: cfgparse: use parse_line() to expand/unquote/unescape config lines
    - BUG/MEDIUM: cfgparse: stop after a reasonable amount of fatal error
    - MINOR: http: do not close connections anymore after internal responses
    - BUG/MINOR: cfgparse: Add missing fatal++ in PARSE_ERR_HEX case
    - BUG/MINOR: spoe: add missing key length check before checking key names
    - MINOR: version: put the compiler version output into version.c not haproxy.c
    - MINOR: compiler: always define __has_feature()
    - MINOR: version: report the presence of the compiler's address sanitizer
    - BUILD: Fix build by including haproxy/global.h
    - BUG/MAJOR: connection: always disable ready events once reported
    - CLEANUP: activity: remove unused counter fd_lock
    - DOC: fd: make it clear that some fields ordering must absolutely be respected
    - MINOR: activity: report the number of times poll() reports I/O
    - MINOR: activity: rename confusing poll_* fields in the output
    - MINOR: fd: Fix a typo in a coment.
    - BUG/MEDIUM: fd: Don't fd_stop_recv() a fd we don't own.
    - BUG/MEDIUM: fd: Call fd_stop_recv() when we just got a fd.
    - MINOR: activity: group the per-loop counters at the top
    - MINOR: activity: rename the "stream" field to "stream_calls"
    - MEDIUM: fd: refine the fd_takeover() migration lock
    - MINOR: fd: slightly optimize the fd_takeover double-CAS loop
    - MINOR: fd: factorize the fd_takeover() exit path to make it safer
    - MINOR: peers: do not use localpeer as an array anymore
    - MEDIUM: peers: add the "localpeer" global option
    - MEDIUM: fd: add experimental support for edge-triggered polling
    - CONTRIB: debug: add the missing flags CO_FL_SAFE_LIST and CO_FL_IDLE_LIST
    - MINOR: haproxy: process signals before runnable tasks
    - MEDIUM: tasks: clean up the front side of the wait queue in wake_expired_tasks()
    - MEDIUM: tasks: also process late wakeups in process_runnable_tasks()
    - BUG/MINOR: cli: allow space escaping on the CLI
    - BUG/MINOR: mworker/cli: fix the escaping in the master CLI
    - BUG/MINOR: mworker/cli: fix semicolon escaping in master CLI
    - REGTEST: http-rules: test spaces in ACLs
    - REGTEST: http-rules: test spaces in ACLs with master CLI
    - BUG/MAJOR: init: properly compute the default global.maxpipes value
    - MEDIUM: map: make the "clear map" operation yield
    - BUG/MEDIUM: stream-int: fix loss of CO_SFL_MSG_MORE flag in forwarding
    - MINOR: mux_h1: Set H1_F_CO_MSG_MORE if we know we have more to send.
    - BUG/MINOR: systemd: Wait for network to be online
    - DOC: configuration: Unindent non-code sentences in the protobuf example
    - DOC: configuration: http-check send was missing from matrix
2020-06-19 21:43:26 +02:00
Peter Gervai
8912ae6987 DOC: configuration: http-check send was missing from matrix
The new directive and its doc were added by commit 8acb1284b ("MINOR:
checks: Add a way to send custom headers and payload during http chekcs")
but the index was not updated.
2020-06-19 21:38:05 +02:00
Peter Gervai
df4c9d2a28 DOC: configuration: Unindent non-code sentences in the protobuf example
Unindent to make the explanation go back to text from code formatted
example in tyhe HTMLized version. Still it's not perfect since these
are not haproxy examples but protobuf config, but... way better.
2020-06-19 21:33:37 +02:00
Willy Tarreau
bc52bec163 MEDIUM: fd: add experimental support for edge-triggered polling
Some of the recent optimizations around the polling to save a few
epoll_ctl() calls have shown that they could also cause some trouble.
However, over time our code base has become totally asynchronous with
I/Os always attempted from the upper layers and only retried at the
bottom, making it look like we're getting closer to EPOLLET support.

There are showstoppers there such as the listeners which cannot support
this. But given that most of the epoll_ctl() dance comes from the
connections, we can try to enable edge-triggered polling on connections.

What this patch does is to add a new global tunable "tune.fd.edge-triggered",
that makes fd_insert() automatically set an et_possible bit on the fd if
the I/O callback is conn_fd_handler. When the epoll code sees an update
for such an FD, it immediately registers it in both directions the first
time and doesn't update it anymore.

On a few tests it proved quite useful with a 14% request rate increase in
a H2->H1 scenario, reducing the epoll_ctl() calls from 2 per request to
2 per connection.

The option is obviously disabled by default as bugs are still expected,
particularly around the subscribe() code where it is possible that some
layers do not always re-attempt reading data after being woken up.
2020-06-19 14:21:46 +02:00
Dragan Dosen
13cd54c08b MEDIUM: peers: add the "localpeer" global option
localpeer <name>
  Sets the local instance's peer name. It will be ignored if the "-L"
  command line argument is specified or if used after "peers" section
  definitions. In such cases, a warning message will be emitted during
  the configuration parsing.

  This option will also set the HAPROXY_LOCALPEER environment variable.
  See also "-L" in the management guide and "peers" section in the
  configuration manual.
2020-06-19 11:37:30 +02:00
Willy Tarreau
1385c88560 [RELEASE] Released version 2.2-dev9
Released version 2.2-dev9 with the following main changes :
    - BUG/MINOR: http-htx: Don't forget to release the http reply in release function
    - BUG/MINOR: http-htx: Fix a leak on error path during http reply parsing
    - MINOR: checks: Remove dead code from process_chk_conn()
    - REGTESTS: checks: Fix tls_health_checks when IPv6 addresses are used
    - REGTESTS: Add missing OPENSSL to REQUIRE_OPTIONS for lua/txn_get_priv
    - MINOR: lua: Use vars_unset_by_name_ifexist()
    - CLEANUP: vars: Remove void vars_unset_by_name(const char*, size_t, struct sample*)
    - MINOR: vars: Make vars_(un|)set_by_name(_ifexist|) return a success value
    - MINOR: lua: Make `set_var()` and `unset_var()` return success
    - MEDIUM: lua: Add `ifexist` parameter to `set_var`
    - MEDIUM: ring: new section ring to declare custom ring buffers.
    - REGTESTS: Add missing OPENSSL to REQUIRE_OPTIONS for compression/lua_validation
    - REGTESTS: Require the version 2.2 to execute lua/set_var
    - BUG/MEDIUM: checks: Refresh the conn-stream and the connection after a connect
    - MINOR: checks: Remove useless tests on the connection and conn-stream
    - BUG/MEDIUM: contrib/spoa: do not register python3.8 if --embed fail
    - BUG/MEDIUM: connection: Ignore PP2 unique ID for stream-less connections
    - BUG/MINOR: connection: Always get the stream when available to send PP2 line
    - BUG/MEDIUM: backend: set the connection owner to the session when using alpn.
    - MINOR: pools: compute an estimate of each pool's average needed objects
    - MEDIUM: pools: directly free objects when pools are too much crowded
    - REGTEST: Add connection/proxy_protocol_send_unique_id_alpn
    - MINOR: http-ana: Make the function http_reply_to_htx() public
    - MINOR: http-ana: Use proxy's error replies to emit 401/407 responses
    - MINOR: http-rules: Use an action function to eval http-request auth rules
    - CLEANUP: http: Remove unused HTTP message templates
    - BUG/MEDIUM: checks: Don't blindly subscribe for receive if waiting for connect
    - MINOR: checks: I/O callback function only rely on the data layer wake callback
    - BUG/MINOR: lua: Add missing string length for lua sticktable lookup
    - BUG/MEDIUM: logs: fix trailing zeros on log message.
    - CI: cirrus-ci: skip reg-tests/connection/proxy_protocol_send_unique_id_alpn.vtc on CentOS 6
    - BUG/MINOR: nameservers: fix error handling in parsing of resolv.conf
    - BUG/MEDIUM: checks: Don't add a tcpcheck ruleset twice in the shared tree
    - MEDIUM: ssl: use TLSv1.2 as the minimum default on bind lines
    - CLEANUP: pools: use the regular lock for the flush operation on lockless pools
    - SCRIPTS: publish-release: pass -n to gzip to remove timestamp
    - MINOR: ring: re-work ring attach generic API.
    - BUG/MINOR: error on unknown statement in ring section.
    - MEDIUM: ring: add server statement to forward messages from a ring
    - MEDIUM: ring: add new srv statement to support octet counting forward
    - MINOR: ssl: set ssl-min-ver in ambiguous configurations
    - CLEANUP: ssl: remove comment from dump_crtlist_sslconf()
    - BUILD: sink: address build warning on 32-bit architectures
    - BUG/MINOR: peers: fix internal/network key type mapping.
    - CLEANUP: regex: remove outdated support for regex actions
    - Revert "MINOR: ssl: rework add cert chain to CTX to be libssl independent"
    - MINOR: mux-h1/proxy: Add a proxy option to disable clear h2 upgrade
    - BUG/MEDIUM: lua: Reset analyse expiration timeout before executing a lua action
    - DOC: add a line about comments in crt-list
    - BUG/MEDIUM: hlua: Lock pattern references to perform set/add/del operations
    - BUG/MINOR: checks: Fix test on http-check rulesets during config validity check
    - BUG/MEDIUM: contrib/prometheus-exporter: Properly set flags to dump metrics
    - BUG/MEDIUM: mworker: fix the copy of options in copy_argv()
    - BUG/MINOR: init: -x can have a parameter starting with a dash
    - BUG/MINOR: init: -S can have a parameter starting with a dash
    - BUG/MEDIUM: mworker: fix the reload with an -- option
    - BUG/MINOR: ssl: fix a trash buffer leak in some error cases
    - BUG/MINOR: mworker: fix a memleak when execvp() failed
    - MINOR: sample: Add secure_memcmp converter
    - REORG: ebtree: move the C files from ebtree/ to src/
    - REORG: ebtree: move the include files from ebtree to include/import/
    - REORG: ebtree: clean up remains of the ebtree/ directory
    - REORG: include: create new file haproxy/api-t.h
    - REORG: include: create new file haproxy/api.h
    - REORG: include: update all files to use haproxy/api.h or api-t.h if needed
    - CLEANUP: include: remove common/config.h
    - CLEANUP: include: remove unused template.h
    - REORG: include: move MIN/MAX from tools.h to compat.h
    - REORG: include: move SWAP/MID_RANGE/MAX_RANGE from tools.h to standard.h
    - CLEANUP: include: remove unused common/tools.h
    - REORG: include: move the base files from common/ to haproxy/
    - REORG: include: move version.h to haproxy/
    - REORG: include: move base64.h, errors.h and hash.h from common to to haproxy/
    - REORG: include: move openssl-compat.h from common/ to haproxy/
    - REORG: include: move ist.h from common/ to import/
    - REORG: include: move the BUG_ON() code to haproxy/bug.h
    - REORG: include: move debug.h from common/ to haproxy/
    - CLEANUP: debug: drop unused function p_malloc()
    - REORG: include: split buf.h into haproxy/buf-t.h and haproxy/buf.h
    - REORG: include: move istbuf.h to haproxy/
    - REORG: include: split mini-clist into haproxy/list and list-t.h
    - REORG: threads: extract atomic ops from hathreads.h
    - CLEANUP: threads: remove a few needless includes of hathreads.h
    - REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h
    - CLEANUP: thread: rename __decl_hathreads() to __decl_thread()
    - REORG: include: move time.h from common/ to haproxy/
    - REORG: include: move integer manipulation functions from standard.h to intops.h
    - CLEANUP: include: remove excessive includes of common/standard.h
    - REORG: include: move freq_ctr to haproxy/
    - CLEANUP: pool: include freq_ctr.h and remove locally duplicated functions
    - REORG: memory: move the pool type definitions to haproxy/pool-t.h
    - REORG: memory: move the OS-level allocator to haproxy/pool-os.h
    - MINOR: memory: don't let __pool_get_first() pick from the cache
    - MEDIUM: memory: don't let pool_put_to_cache() free the objects itself
    - MINOR: memory: move pool-specific path of the locked pool_free() to __pool_free()
    - MEDIUM: memory: make local pools independent on lockless pools
    - REORG: include: move common/memory.h to haproxy/pool.h
    - REORG: include: move common/chunk.h to haproxy/chunk.h
    - REORG: include: move activity to haproxy/
    - REORG: include: move common/buffer.h to haproxy/dynbuf{,-t}.h
    - REORG: include: move common/net_helper.h to haproxy/net_helper.h
    - REORG: include: move common/namespace.h to haproxy/namespace{,-t}.h
    - REORG: include: split common/regex.h into haproxy/regex{,-t}.h
    - REORG: include: split common/xref.h into haproxy/xref{,-t}.h
    - REORG: include: move common/ticks.h to haproxy/ticks.h
    - REORG: include: split common/http.h into haproxy/http{,-t}.h
    - REORG: include: split common/http-hdr.h into haproxy/http-hdr{,-t}.h
    - REORG: include: move common/h1.h to haproxy/h1.h
    - REORG: include: split common/htx.h into haproxy/htx{,-t}.h
    - REORG: include: move hpack*.h to haproxy/ and split hpack-tbl
    - REORG: include: move common/h2.h to haproxy/h2.h
    - REORG: include: move common/fcgi.h to haproxy/
    - REORG: include: move protocol.h to haproxy/protocol{,-t}.h
    - REORG: tools: split common/standard.h into haproxy/tools{,-t}.h
    - REORG: include: move dict.h to hparoxy/dict{,-t}.h
    - REORG: include: move shctx to haproxy/shctx{,-t}.h
    - REORG: include: move port_range.h to haproxy/port_range{,-t}.h
    - REORG: include: move fd.h to haproxy/fd{,-t}.h
    - REORG: include: move ring to haproxy/ring{,-t}.h
    - REORG: include: move sink.h to haproxy/sink{,-t}.h
    - REORG: include: move pipe.h to haproxy/pipe{,-t}.h
    - CLEANUP: include: remove empty raw_sock.h
    - REORG: include: move proto_udp.h to haproxy/proto_udp{,-t}.h
    - REORG: include: move proto/proto_sockpair.h to haproxy/proto_sockpair.h
    - REORG: include: move compression.h to haproxy/compression{,-t}.h
    - REORG: include: move h1_htx.h to haproxy/h1_htx.h
    - REORG: include: move http_htx.h to haproxy/http_htx{,-t}.h
    - REORG: include: move hlua.h to haproxy/hlua{,-t}.h
    - REORG: include: move hlua_fcn.h to haproxy/hlua_fcn.h
    - REORG: include: move action.h to haproxy/action{,-t}.h
    - REORG: include: move arg.h to haproxy/arg{,-t}.h
    - REORG: include: move auth.h to haproxy/auth{,-t}.h
    - REORG: include: move dns.h to haproxy/dns{,-t}.h
    - REORG: include: move flt_http_comp.h to haproxy/
    - REORG: include: move counters.h to haproxy/counters-t.h
    - REORG: include: split mailers.h into haproxy/mailers{,-t}.h
    - REORG: include: move capture.h to haproxy/capture{,-t}.h
    - REORG: include: move frontend.h to haproxy/frontend.h
    - REORG: include: move obj_type.h to haproxy/obj_type{,-t}.h
    - REORG: include: move http_rules.h to haproxy/http_rules.h
    - CLEANUP: include: remove unused mux_pt.h
    - REORG: include: move mworker.h to haproxy/mworker{,-t}.h
    - REORG: include: move ssl_utils.h to haproxy/ssl_utils.h
    - REORG: include: move ssl_ckch.h to haproxy/ssl_ckch{,-t}.h
    - REORG: move ssl_crtlist.h to haproxy/ssl_crtlist{,-t}.h
    - REORG: include: move lb_chash.h to haproxy/lb_chash{,-t}.h
    - REORG: include: move lb_fas.h to haproxy/lb_fas{,-t}.h
    - REORG: include: move lb_fwlc.h to haproxy/lb_fwlc{,-t}.h
    - REORG: include: move lb_fwrr.h to haproxy/lb_fwrr{,-t}.h
    - REORG: include: move listener.h to haproxy/listener{,-t}.h
    - REORG: include: move pattern.h to haproxy/pattern{,-t}.h
    - REORG: include: move map to haproxy/map{,-t}.h
    - REORG: include: move payload.h to haproxy/payload.h
    - REORG: include: move sample.h to haproxy/sample{,-t}.h
    - REORG: include: move protocol_buffers.h to haproxy/protobuf{,-t}.h
    - REORG: include: move vars.h to haproxy/vars{,-t}.h
    - REORG: include: split global.h into haproxy/global{,-t}.h
    - REORG: include: move task.h to haproxy/task{,-t}.h
    - REORG: include: move proto_tcp.h to haproxy/proto_tcp.h
    - REORG: include: move signal.h to haproxy/signal{,-t}.h
    - REORG: include: move tcp_rules.h to haproxy/tcp_rules.h
    - REORG: include: move connection.h to haproxy/connection{,-t}.h
    - REORG: include: move checks.h to haproxy/check{,-t}.h
    - REORG: include: move http_fetch.h to haproxy/http_fetch.h
    - REORG: include: move peers.h to haproxy/peers{,-t}.h
    - REORG: include: move stick_table.h to haproxy/stick_table{,-t}.h
    - REORG: include: move session.h to haproxy/session{,-t}.h
    - REORG: include: move trace.h to haproxy/trace{,-t}.h
    - REORG: include: move acl.h to haproxy/acl.h{,-t}.h
    - REORG: include: split common/uri_auth.h into haproxy/uri_auth{,-t}.h
    - REORG: move applet.h to haproxy/applet{,-t}.h
    - REORG: include: move stats.h to haproxy/stats{,-t}.h
    - REORG: include: move cli.h to haproxy/cli{,-t}.h
    - REORG: include: move lb_map.h to haproxy/lb_map{,-t}.h
    - REORG: include: move ssl_sock.h to haproxy/ssl_sock{,-t}.h
    - REORG: include: move stream_interface.h to haproxy/stream_interface{,-t}.h
    - REORG: include: move channel.h to haproxy/channel{,-t}.h
    - REORG: include: move http_ana.h to haproxy/http_ana{,-t}.h
    - REORG: include: move filters.h to haproxy/filters{,-t}.h
    - REORG: include: move fcgi-app.h to haproxy/fcgi-app{,-t}.h
    - REORG: include: move log.h to haproxy/log{,-t}.h
    - REORG: include: move proxy.h to haproxy/proxy{,-t}.h
    - REORG: include: move spoe.h to haproxy/spoe{,-t}.h
    - REORG: include: move backend.h to haproxy/backend{,-t}.h
    - REORG: include: move queue.h to haproxy/queue{,-t}.h
    - REORG: include: move server.h to haproxy/server{,-t}.h
    - REORG: include: move stream.h to haproxy/stream{,-t}.h
    - REORG: include: move cfgparse.h to haproxy/cfgparse.h
    - CLEANUP: hpack: export debug functions and move inlines to .h
    - REORG: check: move the e-mail alerting code to mailers.c
    - REORG: check: move tcpchecks away from check.c
    - REORG: check: move email_alert* from proxy-t.h to mailers-t.h
    - REORG: check: extract the external checks from check.{c,h}
    - CLEANUP: include: don't include stddef.h directly
    - CLEANUP: include: don't include proxy-t.h in global-t.h
    - CLEANUP: include: move sample_data out of sample-t.h
    - REORG: include: move the error reporting functions to from log.h to errors.h
    - BUILD: reorder objects in the Makefile for faster builds
    - CLEANUP: compiler: add a THREAD_ALIGNED macro and use it where appropriate
    - CLEANUP: include: make atomic.h part of the base API
    - REORG: include: move MAX_THREADS to defaults.h
    - REORG: include: move THREAD_LOCAL and __decl_thread() to compiler.h
    - CLEANUP: include: tree-wide alphabetical sort of include files
    - REORG: include: make list-t.h part of the base API
    - REORG: dgram: rename proto_udp to dgram
2020-06-11 10:22:10 +02:00
Tim Duesterhus
f38175cf6e MINOR: sample: Add secure_memcmp converter
secure_memcmp compares two binary strings in constant time. It's only
available when haproxy is compiled with USE_OPENSSL.
2020-06-09 22:04:13 +02:00
William Lallemand
7c26ed73f7 DOC: add a line about comments in crt-list
Add a line about comments in crt-list.

Fix issue #514.
2020-06-03 17:36:39 +02:00
Christopher Faulet
89aed32bff MINOR: mux-h1/proxy: Add a proxy option to disable clear h2 upgrade
By default, HAProxy is able to implicitly upgrade an H1 client connection to an
H2 connection if the first request it receives from a given HTTP connection
matches the HTTP/2 connection preface. This way, it is possible to support H1
and H2 clients on a non-SSL connections. It could be a problem if for any
reason, the H2 upgrade is not acceptable. "option disable-h2-upgrade" may now be
used to disable it, per proxy. The main puprose of this option is to let an
admin to totally disable the H2 support for security reasons. Recently, a
critical issue in the HPACK decoder was fixed, forcing everyone to upgrade their
HAProxy version to fix the bug. It is possible to disable H2 for SSL
connections, but not on clear ones. This option would have been a viable
workaround.
2020-06-03 10:23:39 +02:00
Emeric Brun
530ba38a70 BUG/MINOR: peers: fix internal/network key type mapping.
Network types were directly and mistakenly mapped on sample types:

This patch fix the doc with values effectively used to keep backward
compatiblitiy on existing implementations.

In addition it adds an internal/network mapping for key types to avoid
further mistakes adding or modifying internals types.

This patch should be backported on all maintained branches,
particularly until v1.8 included for documentation part.
2020-06-02 16:25:19 +02:00
William Lallemand
50df1cb1e5 MINOR: ssl: set ssl-min-ver in ambiguous configurations
Using ssl-max-ver without ssl-min-ver is ambiguous.

When the ssl-min-ver is not configured, and ssl-max-ver is set to a
value lower than the default ssl-min-ver (which is TLSv1.2 currently),
set the ssl-min-ver to the value of ssl-max-ver, and emit a warning.
2020-06-02 11:13:12 +02:00
Emeric Brun
975564784f MEDIUM: ring: add new srv statement to support octet counting forward
log-proto <logproto>
  The "log-proto" specifies the protocol used to forward event messages to
  a server configured in a ring section. Possible values are "legacy"
  and "octet-count" corresponding respectively to "Non-transparent-framing"
  and "Octet counting" in rfc6587. "legacy" is the default.

Notes: a separated io_handler was created to avoid per messages test
and to prepare code to set different log protocols such as
request- response based ones.
2020-05-31 10:49:43 +02:00
Emeric Brun
494c505703 MEDIUM: ring: add server statement to forward messages from a ring
This patch adds new statement "server" into ring section, and the
related "timeout connect" and "timeout server".

server <name> <address> [param*]
  Used to configure a syslog tcp server to forward messages from ring buffer.
  This supports for all "server" parameters found in 5.2 paragraph.
  Some of these parameters are irrelevant for "ring" sections.

timeout connect <timeout>
  Set the maximum time to wait for a connection attempt to a server to succeed.

  Arguments :
    <timeout> is the timeout value specified in milliseconds by default, but
              can be in any other unit if the number is suffixed by the unit,
              as explained at the top of this document.

timeout server <timeout>
  Set the maximum time for pending data staying into output buffer.

  Arguments :
    <timeout> is the timeout value specified in milliseconds by default, but
              can be in any other unit if the number is suffixed by the unit,
              as explained at the top of this document.

  Example:
    global
        log ring@myring local7

    ring myring
        description "My local buffer"
        format rfc3164
        maxlen 1200
        size 32764
        timeout connect 5s
        timeout server 10s
        server mysyslogsrv 127.0.0.1:6514
2020-05-31 10:46:13 +02:00
Christopher Faulet
612f2eafe9 MINOR: http-ana: Use proxy's error replies to emit 401/407 responses
There is no reason to not use proxy's error replies to emit 401/407
responses. The function http_reply_40x_unauthorized(), responsible to emit those
responses, is not really complex. It only adds a
WWW-Authenticate/Proxy-Authenticate header to a generic message.

So now, error replies can be defined for 401 and 407 status codes, using
errorfile or http-error directives. When an http-request auth rule is evaluated,
the corresponding error reply is used. For 401 responses, all occurrences of the
WWW-Authenticate header are removed and replaced by a new one with a basic
authentication challenge for the configured realm. For 407 responses, the same
is done on the Proxy-Authenticate header. If the error reply must not be
altered, "http-request return" rule must be used instead.
2020-05-28 15:07:20 +02:00
Emeric Brun
99c453df9d MEDIUM: ring: new section ring to declare custom ring buffers.
It is possible to globally declare ring-buffers, to be used as target for log
servers or traces.

ring <ringname>
  Creates a new ring-buffer with name <ringname>.

description <text>
  The descritpition is an optional description string of the ring. It will
  appear on CLI. By default, <name> is reused to fill this field.

format <format>
  Format used to store events into the ring buffer.

  Arguments:
    <format> is the log format used when generating syslog messages. It may be
             one of the following :

      iso     A message containing only the ISO date, followed by the text.
              The PID, process name and system name are omitted. This is
              designed to be used with a local log server.

      raw     A message containing only the text. The level, PID, date, time,
              process name and system name are omitted. This is designed to be
              used in containers or during development, where the severity
              only depends on the file descriptor used (stdout/stderr). This
              is the default.

      rfc3164 The RFC3164 syslog message format. This is the default.
              (https://tools.ietf.org/html/rfc3164)

      rfc5424 The RFC5424 syslog message format.
              (https://tools.ietf.org/html/rfc5424)

      short   A message containing only a level between angle brackets such as
              '<3>', followed by the text. The PID, date, time, process name
              and system name are omitted. This is designed to be used with a
              local log server. This format is compatible with what the systemd
              logger consumes.

      timed   A message containing only a level between angle brackets such as
              '<3>', followed by ISO date and by the text. The PID, process
              name and system name are omitted. This is designed to be
              used with a local log server.

maxlen <length>
  The maximum length of an event message stored into the ring,
  including formatted header. If an event message is longer than
  <length>, it will be truncated to this length.

size <size>
  This is the optional size in bytes for the ring-buffer. Default value is
  set to BUFSIZE.

  Example:
    global
        log ring@myring local7

    ring myring
        description "My local buffer"
        format rfc3164
        maxlen 1200

Note: ring names are resolved during post configuration processing.
2020-05-26 08:03:15 +02:00
Tim Duesterhus
4e172c93f9 MEDIUM: lua: Add ifexist parameter to set_var
As discussed in GitHub issue #624 Lua scripts should not use
variables that are never going to be read, because the memory
for variable names is never going to be freed.

Add an optional `ifexist` parameter to the `set_var` function
that allows a Lua developer to set variables that are going to
be ignored if the variable name was not used elsewhere before.

Usually this mean that there is no `var()` sample fetch for the
variable in question within the configuration.
2020-05-25 08:12:35 +02:00
Willy Tarreau
73b943b964 [RELEASE] Released version 2.2-dev8
Released version 2.2-dev8 with the following main changes :
    - MINOR: checks: Improve report of unexpected errors for expect rules
    - MEDIUM: checks: Add matching on log-format string for expect rules
    - DOC: Fix req.body and co documentation to be accurate
    - MEDIUM: checks: Remove dedicated sample fetches and use response ones instead
    - CLEANUP: checks: sort and rename tcpcheck_expect_type types
    - MINOR: checks: Use dedicated actions to send log-format strings in send rules
    - MINOR: checks: Simplify matching on HTTP headers in HTTP expect rules
    - MINOR: checks/sample: Remove unnecessary tests on the sample session
    - REGTEST: checks: Adapt SSL error message reported when connection is rejected
    - MINOR: mworker: replace ha_alert by ha_warning when exiting successfuly
    - MINOR: checks: Support log-format string to set the URI for HTTP send rules
    - MINOR: checks: Support log-format string to set the body for HTTP send rules
    - DOC: Be more explicit about configurable check ok/error/timeout status
    - MINOR: checks: Make matching on HTTP headers for expect rules less obscure
    - BUG/MEDIUM: lua: Fix dumping of stick table entries for STD_T_DICT
    - BUG/MINOR: config: Make use_backend and use-server post-parsing less obscur
    - REGTESTS: make the http-check-send test require version 2.2
    - BUG/MINOR: http-ana: fix NTLM response parsing again
    - BUG/MEDIUM: http_ana: make the detection of NTLM variants safer
    - BUG/MINOR: cfgparse: Abort parsing the current line if an invalid \x sequence is encountered
    - MINOR: cfgparse: Improve error message for invalid \x sequences
    - CI: travis-ci: enable arm64 builds again
    - MEDIUM: ssl: increase default-dh-param to 2048
    - CI: travis-ci: skip pcre2 on arm64 build
    - CI: travis-ci: extend the build time for SSL to 60 minutes
    - CLEANUP: config: drop unused setting CONFIG_HAP_MEM_OPTIM
    - CLEANUP: config: drop unused setting CONFIG_HAP_INLINE_FD_SET
    - CLENAUP: config: move CONFIG_HAP_LOCKLESS_POOLS out of config.h
    - CLEANUP: remove THREAD_LOCAL from config.h
    - CI: travis-ci: upgrade LibreSSL versions
    - DOC: assorted typo fixes in the documentation
    - CI: extend spellchecker whitelist
    - CLEANUP: assorted typo fixes in the code and comments
    - MAJOR: contrib: porting spoa_server to support python3
    - BUG/MEDIUM: checks: Subscribe to I/O events on an unfinished connect
    - BUG/MINOR: checks: Don't subscribe to I/O events if it is already done
    - BUG/MINOR: checks: Rely on next I/O oriented rule when waiting for a connection
    - MINOR: checks: Don't try to send outgoing data if waiting to be able to send
    - MINOR: sample: Move aes_gcm_dec implementation into sample.c
    - MINOR: sample: Add digest and hmac converters
    - BUG/MEDIUM: checks: Subscribe to I/O events only if a mux was installed
    - BUG/MINOR: sample/ssl: Fix digest converter for openssl < 1.1.0
    - BUG/MINOR: pools: use %u not %d to report pool stats in "show pools"
    - BUG/MINOR: pollers: remove uneeded free in global init
    - CLEANUP: select: enhance readability in init
    - BUG/MINOR: soft-stop: always wake up waiting threads on stopping
    - MINOR: soft-stop: let the first stopper only signal other threads
    - BUILD: select: only declare existing local labels to appease clang
    - BUG/MEDIUM: streams: Remove SF_ADDR_SET if we're retrying due to L7 retry.
    - BUG/MEDIUM: stream: Only allow L7 retries when using HTTP.
    - DOC: retry-on can only be used with mode http
    - MEDIUM: ssl: allow to register callbacks for SSL/TLS protocol messages
    - MEDIUM: ssl: split ssl_sock_msgcbk() and use a new callback mechanism
    - MINOR: ssl: add a new function ssl_sock_get_ssl_object()
    - MEDIUM: ssl: use ssl_sock_get_ssl_object() in fetchers where appropriate
    - REORG: ssl: move macros and structure definitions to ssl_sock.h
    - CLEANUP: ssl: remove the shsess_* macros
    - REORG: move the crt-list structures in their own .h
    - REORG: ssl: move the ckch structures to types/ssl_ckch.h
    - CLEANUP: ssl: add ckch prototypes in proto/ssl_ckch.h
    - REORG: ssl: move crtlist functions to src/ssl_crtlist.c
    - CLEANUP: ssl: avoid circular dependencies in ssl_crtlist.h
    - REORG: ssl: move the ckch_store related functions to src/ssl_ckch.c
    - REORG: ssl: move ckch_inst functions to src/ssl_ckch.c
    - REORG: ssl: move the crt-list CLI functions in src/ssl_crtlist.c
    - REORG: ssl: move the CLI 'cert' functions to src/ssl_ckch.c
    - REORG: ssl: move ssl configuration to cfgparse-ssl.c
    - MINOR: ssl: remove static keyword in some SSL utility functions
    - REORG: ssl: move ssl_sock_ctx and fix cross-dependencies issues
    - REORG: ssl: move sample fetches to src/ssl_sample.c
    - REORG: ssl: move utility functions to src/ssl_utils.c
    - DOC: ssl: update MAINTAINERS file
    - CI: travis-ci: switch arm64 builds to use openssl from distro
    - MINOR: stats: Prepare for more accurate moving averages
    - MINOR: stats: Expose native cum_req metric for a server
    - MEDIUM: stats: Enable more accurate moving average calculation for stats
    - BUILD: ssl: include buffer common headers for ssl_sock_ctx
    - BUILD: ssl: include errno.h in ssl_crtlist.c
    - CLEANUP: acl: remove unused assignment
    - DOC/MINOR: halog: Add long help info for ic flag
    - BUILD: ssl: fix build without OPENSSL_NO_ENGINE
    - DOC: SPOE is no longer experimental
    - BUG/MINOR: cache: Don't needlessly test "cache" keyword in parse_cache_flt()
    - MINOR: config: Don't dump keywords if argument is NULL
    - MEDIUM: checks: Make post-41 the default mode for mysql checks
    - BUG/MINOR: logs: prevent double line returns in some events.
    - MEDIUM: sink: build header in sink_write for log formats
    - MEDIUM: logs: buffer targets now rely on new sink_write
    - MEDIUM: sink: add global statement to create a new ring (sink buffer)
    - MEDIUM: hpack: use a pool for the hpack table
    - BUG/MAJOR: mux-fcgi: Stop sending loop if FCGI stream is blocked for any reason
    - BUG/MEDIUM: ring: write-lock the ring while attaching/detaching
    - MINOR: applet: adopt the wait list entry from the CLI
    - MINOR: ring: make the applet code not depend on the CLI
    - Revert "MEDIUM: sink: add global statement to create a new ring (sink buffer)"
    - CI: travis-ci: fix libslz download URL
    - MINOR: ssl: split config and runtime variable for ssl-{min,max}-ver
    - CLEANUP: http_ana: Remove unused TXN flags
    - BUG/MINOR: http-rules: Mark http return rules as final
    - MINOR: http-htx: Add http_reply type based on what is used for http return rules
    - CLEANUP: http-htx: Rename http_error structure into http_error_msg
    - MINOR: http-rules: Use http_reply structure for http return rules
    - MINOR: http-htx: Use a dedicated function to release http_reply objects
    - MINOR: http-htx: Use a dedicated function to parse http reply arguments
    - MINOR: http-htx: Use a dedicated function to check http reply validity
    - MINOR: http-ana: Use a dedicated function to send a response from an http reply
    - MEDIUM: http-rules: Rely on http reply for http deny/tarpit rules
    - MINOR: http-htx: Store default error messages in a global http reply array
    - MINOR: http-htx: Store messages of an http-errors section in a http reply array
    - MINOR: http-htx: Store errorloc/errorfile messages in http replies
    - MINOR: proxy: Add references on http replies for proxy error messages
    - MINOR: http-htx: Use http reply from the http-errors section
    - MINOR: http-ana: Use a TXN flag to prevent after-response ruleset evaluation
    - MEDIUM: http-ana: Use http replies for HTTP error messages
    - CLEANUP: http-htx: Remove unused storage of error messages in buffers
    - MINOR: htx: Add a function to copy a buffer in an HTX message
    - CLEANUP: channel: Remove channel_htx_copy_msg() function
    - MINOR: http-ana: Add a function to write an http reply in an HTX message
    - MINOR: http-htx/proxy: Add http-error directive using http return syntax
    - DOC: Fix "errorfile" description in the configuration manual
    - BUG/MINOR: checks: Respect check-ssl param when a port or an addr is specified
    - BUILD: hpack: make sure the hpack table can still be built standalone
    - CONTRIB: hpack: make use of the simplified standalone HPACK API
    - MINOR: connection: add pp2-never-send-local to support old PP2 behavior
2020-05-22 16:19:04 +02:00
Willy Tarreau
119e50e0cc MINOR: connection: add pp2-never-send-local to support old PP2 behavior
A bug in the PROXY protocol v2 implementation was present in HAProxy up to
version 2.1, causing it to emit a PROXY command instead of a LOCAL command
for health checks. This is particularly minor but confuses some servers'
logs. Sadly, the bug was discovered very late and revealed that some servers
which possibly only tested their PROXY protocol implementation against
HAProxy fail to properly handle the LOCAL command, and permanently remain in
the "down" state when HAProxy checks them. When this happens, it is possible
to enable this global option to revert to the older (bogus) behavior for the
time it takes to contact the affected components' vendors and get them fixed.
This option is disabled by default and acts on all servers having the
"send-proxy-v2" statement.

Older versions were reverted to the old behavior and should not attempt to
be fixed by default again. However a variant of this patch could possibly
be implemented to ask to explicitly send LOCAL if needed by some servers.

More context here:
   https://www.mail-archive.com/haproxy@formilux.org/msg36890.html
   https://www.mail-archive.com/haproxy@formilux.org/msg37218.html
2020-05-22 13:55:32 +02:00
Christopher Faulet
7017067d68 DOC: Fix "errorfile" description in the configuration manual
"errorfile" description is outdated. Now, error messages are parsed at startup
and converted to HTX messages. So they must be valid according to the HTTP
standards. In addition, there is now a comment about the reserved buffer space
that should remain available to allow header rewrites.
2020-05-20 18:27:14 +02:00
Christopher Faulet
3b967c1210 MINOR: http-htx/proxy: Add http-error directive using http return syntax
The http-error directive can now be used instead of errorfile to define an error
message in a proxy section (including default sections). This directive uses the
same syntax that http return rules. The only real difference is the limitation
on status code that may be specified. Only status codes supported by errorfile
directives are supported for this new directive. Parsing of errorfile directive
remains independent from http-error parsing. But functionally, it may be
expressed in terms of http-errors :

  errorfile <status> <file> ==> http-errror status <status> errorfile <file>
2020-05-20 18:27:14 +02:00
Christopher Faulet
5cb513abeb MEDIUM: http-rules: Rely on http reply for http deny/tarpit rules
"http-request deny", "http-request tarpit" and "http-response deny" rules now
use the same syntax than http return rules and internally rely on the http
replies. The behaviour is not the same when no argument is specified (or only
the status code). For http replies, a dummy response is produced, with no
payload. For old deny/tarpit rules, the proxy's error messages are used. Thus,
to be compatible with existing configuration, the "default-errorfiles" parameter
is implied. For instance :

  http-request deny deny_status 404

is now an alias of

  http-request deny status 404 default-errorfiles
2020-05-20 18:27:13 +02:00
Willy Tarreau
d68a6927f7 Revert "MEDIUM: sink: add global statement to create a new ring (sink buffer)"
This reverts commit 957ec59571.

As discussed with Emeric, the current syntax is not extensible enough,
this will be turned to a section instead in a forthcoming patch.
2020-05-20 12:06:16 +02:00
Emeric Brun
957ec59571 MEDIUM: sink: add global statement to create a new ring (sink buffer)
This patch adds the new global statement:
ring <name> [desc <desc>] [format <format>] [size <size>] [maxlen <length>]
  Creates a named ring buffer which could be used on log line for instance.

  <desc> is an optionnal description string of the ring. It will appear on
         CLI. By default, <name> is reused to fill this field.

  <format> is the log format used when generating syslog messages. It may be
           one of the following :

    iso       A message containing only the ISO date, followed by the text.
              The PID, process name and system name are omitted. This is
              designed to be used with a local log server.

    raw       A message containing only the text. The level, PID, date, time,
              process name and system name are omitted. This is designed to be
              used in containers or during development, where the severity only
              depends on the file descriptor used (stdout/stderr). This is
              the default.

    rfc3164   The RFC3164 syslog message format. This is the default.
              (https://tools.ietf.org/html/rfc3164)

    rfc5424   The RFC5424 syslog message format.
              (https://tools.ietf.org/html/rfc5424)

    short     A message containing only a level between angle brackets such as
              '<3>', followed by the text. The PID, date, time, process name
              and system name are omitted. This is designed to be used with a
              local log server. This format is compatible with what the systemd
              logger consumes.

    timed     A message containing only a level between angle brackets such as
              '<3>', followed by ISO date and by the text. The PID, process
              name and system name are omitted. This is designed to be
              used with a local log server.

  <length> is the maximum length of event message stored into the ring,
           including formatted header. If the event message is longer
           than <length>, it would be truncated to this length.

  <name> is the ring identifier, which follows the same naming convention as
         proxies and servers.

  <size> is the optionnal size in bytes. Default value is set to BUFSIZE.

Note: Historically sink's name and desc were refs on const strings. But with new
configurable rings a dynamic allocation is needed.
2020-05-19 11:04:11 +02:00
Christopher Faulet
62f79fe68a MEDIUM: checks: Make post-41 the default mode for mysql checks
MySQL 4.1 is old enough to be the default mode for mysql checks. So now, once a
username is defined, post-41 mode is automatically used. To do mysql checks on
previous MySQL version, the argument "pre-41" must be used.

Note, it is a compatibility breakage for everyone using an antique and
unsupported MySQL version.
2020-05-18 18:32:09 +02:00
Christopher Faulet
3b78809de0 DOC: SPOE is no longer experimental
The SPOE was marked as experiemental since the begining. But, it is no longer
true. This can be an obstacle to its use.
2020-05-18 17:47:18 +02:00
Jerome Magnin
5ce3c14aa9 DOC: retry-on can only be used with mode http
The documentation for retry-on hints at it being meant to be used
in conjuction with mode http, but since we've a had bug report
involving mode tcp and retry-on, lets make it explicit in the
documentation that it only works with mode http and will be
ignored otherwise.
2020-05-14 09:43:13 +02:00
Patrick Gansterer
8e36651ed3 MINOR: sample: Add digest and hmac converters
Make the digest and HMAC function of OpenSSL accessible to the user via
converters. They can be used to sign and validate content.

Reviewed-by: Tim Duesterhus <tim@bastelstu.be>
2020-05-12 10:08:11 +02:00
Ilya Shipitsin
4329a9af5a DOC: assorted typo fixes in the documentation
This is the 8th round of cleanups in various docs
2020-05-11 10:10:15 +02:00
Willy Tarreau
3ba77d29ac MEDIUM: ssl: increase default-dh-param to 2048
For 6 years now we've been seeing a warning suggesting to set dh-param
beyond 1024 if possible when it was not set. It's about time to do it
and get rid of this warning since most users seem to already use 2048.
It will remain possible to set a lower value of course, so only those
who were experiencing the warning and were relying on the default value
may notice a change (higher CPU usage). For more context, please refer
to this thread :

  https://www.mail-archive.com/haproxy@formilux.org/msg37226.html

This commit removes a big chunk of code which happened to be needed
exclusively to figure if it was required to emit a warning or not :-)
2020-05-08 09:36:37 +02:00
Christopher Faulet
d888f0fc6e DOC: Be more explicit about configurable check ok/error/timeout status
It is possible to configure the check status on success, on error and on
timeout, for http-check and tcp-check expect rules. But the documentation relies
on internal names. These names are reported on the stats and are describe in the
management guide. But it is probably a good idea to be more explicit too in the
doc describing these options.
2020-05-07 07:40:18 +02:00
Christopher Faulet
574e7bd7f3 MINOR: checks: Support log-format string to set the body for HTTP send rules
For http-check send rules, it is now possible to use a log-format string to set
the request's body. the keyword "body-lf" should be used instead of "body". If the
string eval fails, no body is added.
2020-05-06 18:04:05 +02:00
Christopher Faulet
7c95f5f22b MINOR: checks: Support log-format string to set the URI for HTTP send rules
For http-check send rules, it is now possible to use a log-format string to set
the request URI. the keyword "uri-lf" should be used instead of "uri". If the
string eval fails, we fall back on the default uri "/".
2020-05-06 18:04:05 +02:00
Christopher Faulet
b5594265d2 MINOR: checks: Simplify matching on HTTP headers in HTTP expect rules
Extra parameters on http-check expect rules, for the header matching method, to
use log-format string or to match full header line have been removed. There is
now separate matching methods to match a full header line or to match each
comma-separated values. "http-check expect fhdr" must be used in the first case,
and "http-check expect hdr" in the second one. In addition, to match log-format
header name or value, "-lf" suffix must be added to "name" or "value"
keyword. For intance:

   http-check expect hdr name "set-cookie" value-lf -m beg "sessid=%[var(check.cookie)]"

Thanks to this changes, each parameter may only be interpreted in one way.
2020-05-06 12:42:36 +02:00
Christopher Faulet
b50b3e6d0a MINOR: checks: Use dedicated actions to send log-format strings in send rules
Following actions have been added to send log-format strings from a tcp-check
ruleset instead the log-format parameter:

  * tcp-check send-lf <fmt>
  * tcp-check send-binary-lf <fmt>

It is easier for tools generating configurations. Each action may only be
interpreted in one way.
2020-05-06 12:41:57 +02:00
Christopher Faulet
e596d184be MEDIUM: checks: Remove dedicated sample fetches and use response ones instead
All sample fetches in the scope "check." have been removed. Response sample
fetches must be used instead. It avoids keyword duplication. So, for instance,
res.hdr() must be now used instead of check.hdr().

To do so, following sample fetches have been added on the response :

  * res.body, res.body_len and res.body_size
  * res.hdrs and res.hdrs_bin

Sample feches dealing with the response's body are only useful in the health
checks context. When called from a stream context, there is no warranty on the
body presence. There is no option to wait the response's body.
2020-05-06 12:37:43 +02:00
Christopher Faulet
af4dc4ccaa DOC: Fix req.body and co documentation to be accurate
Because the HTX is the only mode to represent HTTP data, "option
http-request-buffer" is no longer mandatory to have body data. Without this
option, there is no warranty on the body presence. So it is recommanded to use
it. But it is not a requirement. In addition, the note about chunked body is
removed because outdated.
2020-05-06 09:06:01 +02:00
Christopher Faulet
aaab0836d9 MEDIUM: checks: Add matching on log-format string for expect rules
It is now possible to use log-format string (or hexadecimal string for the
binary version) to match a content in tcp-check based expect rules. For
hexadecimal log-format string, the conversion in binary is performed after the
string evaluation, during health check execution. The pattern keywords to use
are "string-lf" for the log-format string and "binary-lf" for the hexadecimal
log-format string.
2020-05-06 08:31:29 +02:00
Willy Tarreau
fc0b8f39a6 [RELEASE] Released version 2.2-dev7
Released version 2.2-dev7 with the following main changes :
    - MINOR: version: Show uname output in display_version()
    - CI: run weekly OpenSSL "no-deprecated" builds
    - CLEANUP: log: fix comment of parse_logformat_string()
    - DOC: Improve documentation on http-request set-src
    - MINOR: ssl/cli: disallow SSL options for directory in 'add ssl crt-list'
    - MINOR: ssl/cli: restrain certificate path when inserting into a directory
    - MINOR: ssl: add ssl-skip-self-issued-ca global option
    - BUG/MINOR: ssl: default settings for ssl server options are not used
    - MINOR: config: add a global directive to set default SSL curves
    - BUG/MEDIUM: http-ana: Handle NTLM messages correctly.
    - DOC: internals: update the SSL architecture schema
    - BUG/MINOR: tools: fix the i386 version of the div64_32 function
    - BUG/MINOR: mux-fcgi/trace: fix wrong set of trace flags in fcgi_strm_add_eom()
    - BUG/MINOR: http: make url_decode() optionally convert '+' to SP
    - DOC: option logasap does not depend on mode
    - MEDIUM: memory: make pool_gc() run under thread isolation
    - MINOR: contrib: make the peers wireshark dissector a plugin
    - BUG/MINOR: http-ana: Throw a 500 error if after-response ruleset fails on errors
    - BUG/MINOR: check: Update server address and port to execute an external check
    - MINOR: mini-clist: Add functions to iterate backward on a list
    - MINOR: checks: Add a way to send custom headers and payload during http chekcs
    - MINOR: server: respect warning and alert semantic
    - BUG/MINOR: checks: Respect the no-check-ssl option
    - BUG/MEDIUM: server/checks: Init server check during config validity check
    - CLEANUP: checks: Don't export anymore init_check and srv_check_healthcheck_port
    - BUG/MINOR: checks: chained expect will not properly wait for enough data
    - BUG/MINOR: checks: Forbid tcp-check lines in default section as documented
    - MINOR: checks: Use an enum to describe the tcp-check rule type
    - MINOR: checks: Simplify connection flag parsing in tcp-check connect
    - MEDIUM: checks: rewind to the first inverse expect rule of a chain on new data
    - MINOR: checks: simplify tcp expect config parser
    - MINOR: checks: add min-recv tcp-check expect option
    - MINOR: checks: add linger option to tcp connect
    - MINOR: checks: define a tcp expect type
    - MEDIUM: checks: rewrite tcp-check expect block
    - MINOR: checks: Stop xform buffers to null-terminated string for tcp-check rules
    - MINOR: checks: add rbinary expect match type
    - MINOR: checks: Simplify functions to get step id and comment
    - MEDIUM: checks: capture groups in expect regexes
    - MINOR: checks: Don't use a static tcp rule list head
    - MEDIUM: checks: Use a non-comment rule iterator to get next rule
    - MEDIUM: proxy/checks: Register a keyword to parse tcp-check rules
    - MINOR: checks: Set the tcp-check rule index during parsing
    - MINOR: checks: define tcp-check send type
    - MINOR: checks: define a tcp-check connect type
    - MEDIUM: checks: Add implicit tcp-check connect rule
    - MAJOR: checks: Refactor and simplify the tcp-check loop
    - MEDIUM: checks: Associate a session to each tcp-check healthcheck
    - MINOR: checks/vars: Add a check scope for variables
    - MEDIUM: checks: Parse custom action rules in tcp-checks
    - MINOR: checks: Add support to set-var and unset-var rules in tcp-checks
    - MINOR: checks: Add the sni option for tcp-check connect rules
    - MINOR: checks: Add the via-socks4 option for tcp-check connect rules
    - MINOR: checks: Add the alpn option for tcp-check connect rules
    - MINOR: ssl: Export a generic function to parse an alpn string
    - MINOR: checks: Add the default option for tcp-check connect rules
    - MINOR: checks: Add the addr option for tcp-check connect rule
    - MEDIUM: checks: Support expression to set the port
    - MEDIUM: checks: Support log-format strings for tcp-check send rules
    - MINOR: log: Don't depends on a stream to process samples in log-format string
    - MINOR: log: Don't systematically set LW_REQ when a sample expr is added
    - MEDIUM: checks: Add a shared list of tcp-check rules
    - MINOR: sample: add htonl converter
    - MINOR: sample: add cut_crlf converter
    - MINOR: sample: add ltrim converter
    - MINOR: sample: add rtrim converter
    - MINOR: checks: Use a name for the healthcheck status enum
    - MINOR: checks: Add option to tcp-check expect rules to customize error status
    - MINOR: checks: Merge tcp-check comment rules with the others at config parsing
    - MINOR: checks: Add a sample fetch to extract a block from the input check buffer
    - MEDIUM: checks: Add on-error/on-success option on tcp-check expect rules
    - MEDIUM: checks: Add status-code sample expression on tcp-check expect rules
    - MINOR: checks: Relax the default option for tcp-check connect rules
    - MEDIUM: checks: Add a list of vars to set before executing a tpc-check ruleset
    - MINOR: checks: Export the tcpcheck_eval_ret enum
    - MINOR: checks: Use dedicated function to handle onsuccess/onerror messages
    - MINOR: checks: Support custom functions to eval a tcp-check expect rules
    - MEDIUM: checks: Implement redis check using tcp-check rules
    - MEDIUM: checks: Implement ssl-hello check using tcp-check rules
    - MEDIUM: checks: Implement smtp check using tcp-check rules
    - MEDIUM: checks: Implement postgres check using tcp-check rules
    - MEDIUM: checks: Implement MySQL check using tcp-check rules
    - MEDIUM: checks: Implement LDAP check using tcp-check rules
    - MEDIUM: checks: Implement SPOP check using tcp-check rules
    - MINOR: server/checks: Move parsing of agent keywords in checks.c
    - MINOR: server/checks: Move parsing of server check keywords in checks.c
    - MEDIUM: checks: Implement agent check using tcp-check rules
    - REGTEST: Adapt regtests about checks to recent changes
    - MINOR: Produce tcp-check info message for pure tcp-check rules only
    - MINOR: checks: Add an option to set success status of tcp-check expect rules
    - MINOR: checks: Improve log message of tcp-checks on success
    - MINOR: proxy/checks: Move parsing of httpchk option in checks.c
    - MINOR: proxy/checks: Move parsing of tcp-check option in checks.c
    - MINOR: proxy/checks: Register a keyword to parse http-check rules
    - MINOR: proxy/checks: Move parsing of external-check option in checks.c
    - MINOR: proxy/checks: Register a keyword to parse external-check rules
    - MEDIUM: checks: Use a shared ruleset to store tcp-check rules
    - MINOR: checks: Use an indirect string to represent the expect matching string
    - MINOR: checks: Introduce flags to configure in tcp-check expect rules
    - MINOR: standard: Add my_memspn and my_memcspn
    - MINOR: checks: Add a reverse non-comment rule iterator to get last rule
    - MAJOR: checks: Implement HTTP check using tcp-check rules
    - MINOR: checks: Make resume conditions more explicit in tcpcheck_main()
    - MINOR: connection: Add macros to know if a conn or a cs uses an HTX mux
    - MEDIUM: checks: Refactor how data are received in tcpcheck_main()
    - MINOR: checks/obj_type: Add a new object type for checks
    - BUG/MINOR: obj_type: Handle stream object in obj_base_ptr() function
    - MINOR: checks: Use the check as origin when a session is created
    - MINOR: checks: Add a mux proto to health-check and tcp-check connect rule
    - MINOR: connection: Add a function to install a mux for a health-check
    - MAJOR: checks: Use the best mux depending on the protocol for health checks
    - MEDIUM: checks: Implement default TCP check using tcp-check rules
    - MINOR: checks: Remove unused code about pure TCP checks
    - CLEANUP: checks: Reorg checks.c file to be more readable
    - REGTEST: Fix reg-tests about health-checks to adapt them to recent changes
    - MINOR: ist: Add a function to retrieve the ist pointer
    - MINOR: checks: Use ist API as far as possible
    - BUG/MEDIUM: checks: Be sure to subscribe for sends if outgoing data remains
    - MINOR: checks: Use a tree instead of a list to store tcp-check rulesets
    - BUG/MINOR: checks: Send the right amount of outgoing data for HTTP checks
    - REGTEST: Add scripts to test based tcp-check health-checks
    - Revert "MEDIUM: checks: capture groups in expect regexes"
    - DOC: Add documentation about comments for tcp-check and http-check directives
    - DOC: Fix the tcp-check and http-check directives layout
    - BUG/MEDIUM: checks: Use the mux protocol specified on the server line
    - MINOR: checks: Support mux protocol definition for tcp and http health checks
    - BUG/MINOR: mux-fcgi: Be sure to have a connection as session's origin to use it
    - MINOR: checks: Support list of status codes on http-check expect rules
    - BUG/MEDIUM: checks: Unsubscribe to mux events when a conn-stream is destroyed
    - REGTEST: Add a script to validate agent checks
    - BUG/MINOR: server: Fix server_finalize_init() to avoid unused variable
    - BUG/MEDIUM: checks: unsubscribe for events on the old conn-stream on connect
    - BUG/MINOR: checks: Only use ssl_sock_is_ssl() if compiled with SSL support
    - BUG/MINOR: checks/server: use_ssl member must be signed
    - BUG/MEDIUM: sessions: Always pass the mux context as argument to destroy a mux
    - BUG/MEDIUM: checks: Destroy the conn-stream before the session
    - BUG/MINOR: checks: Fix PostgreSQL regex on the authentication packet
    - CI: cirrus-ci: remove reg-tests/checks/tcp-check-ssl.vtc on CentOS 6
    - MINOR: checks: Support HTTP/2 version (without '.0') for http-check send rules
    - MINOR: checks: Use ver keyword to specify the HTTP version for http checks
    - BUG/MINOR: checks: Remove wrong variable redeclaration
    - BUG/MINOR: checks: Properly handle truncated mysql server messages
    - CLEANUP: checks: Remove unused code when ldap server message is parsed
    - MINOR: checks: Make the use of the check's server more explicit on connect
    - BUG/MINOR: checks: Avoid incompatible cast when a binary string is parsed
    - BUG/MINOR: checks: Remove bad call to free() when an expect rule is parsed
    - BUG/MINOR: checks: Don't lose warning on proxy capability
    - MINOR: log: Add "Tu" timer
    - BUG/MINOR: checks: Set the output buffer length before calling parse_binary()
    - BUG/MEDIUM: mux-h1: make sure we always have a timeout on front connections
    - REGTEST: ssl: test the client certificate authentication
    - DOC: give a more accurate description of what check does
    - BUG/MEDIUM: capture: capture-req/capture-res converters crash without a stream
    - BUG/MEDIUM: capture: capture.{req,res}.* crash without a stream
    - BUG/MEDIUM: http: the "http_first_req" sample fetch could crash without a steeam
    - BUG/MEDIUM: http: the "unique-id" sample fetch could crash without a steeam
    - CLEANUP: http: add a few comments on certain functions' assumptions about streams
    - BUG/MEDIUM: sample: make the CPU and latency sample fetches check for a stream
    - MINOR: http-htx: Export functions to update message authority and host
    - MINOR: checks: Don't support multiple host header for http-check send rule
    - MINOR: checks: Skip some headers for http-check send rules
    - MINOR: checks: Keep the Host header and the request uri synchronized
    - CLEANUP: checks: Fix checks includes
    - DOC: Fix send rules in the http-check connect example
    - DOC: Add more info about request formatting in http-check send description
    - REGTEST: http-rules: Require PCRE or PCRE2 option to run map_redirect script
    - REGTEST: ssl: remove curl from the "add ssl crt-list" test
    - REGTEST: ssl: improve the "set ssl cert" test
    - CLEANUP: ssl: silence a build warning when threads are disabled
    - BUG/MEDIUM: listener: mark the thread as not stuck inside the loop
    - MINOR: threads: export the POSIX thread ID in panic dumps
    - BUG/MINOR: debug: properly use long long instead of long for the thread ID
    - BUG/MEDIUM: shctx: really check the lock's value while waiting
    - BUG/MEDIUM: shctx: bound the number of loops that can happen around the lock
    - MINOR: stream: report the list of active filters on stream crashes
    - BUG/MEDIUM: mux-fcgi: Return from detach if server don't keep the connection
    - BUG/MEDIUM: mux_fcgi: Free the FCGI connection at the end of fcgi_release()
    - BUG/MEDIUM: mux-fcgi: Fix wrong test on FCGI_CF_KEEP_CONN in fcgi_detach()
    - BUG/MEDIUM: connections: force connections cleanup on server changes
    - BUG/MEDIUM: h1: Don't compare host and authority if only h1 headers are parsed
    - BUG/MEDIUM: ssl: fix the id length check within smp_fetch_ssl_fc_session_id()
    - CLEANUP: connections: align function declaration
    - BUG/MINOR: sample: Set the correct type when a binary is converted to a string
    - MEDIUM: checks/http-fetch: Support htx prefetch from a check for HTTP samples
    - DOC: Document the log-format parameter for tcp-check send/send-binary rules
    - MINOR: checks: Add support of payload-based sample fetches
    - MINOR: checks: Add support of be_id, be_name, srv_id and srv_name sample fetches
    - MINOR: checks: Add support of server side ssl sample fetches
    - MINOR: checks: Add support of HTTP response sample fetches
    - MINOR: http-htx: Support different methods to look for header names
    - MINOR: checks: Set by default expect rule status to UNKNOWN during parsing
    - BUG/MINOR: checks: Support multiple HTTP expect rules
    - REGTEST: checks: Fix sync condition for agent-check
    - MEDIUM: checks: Support matching on headers for http-check expect rules
    - MINOR: lua: allow changing port with set_addr
    - BUG/MINOR: da: Fix HTX message prefetch
    - BUG/MINOR: wurfl: Fix HTX message prefetch
    - BUG/MINOR: 51d: Fix HTX message prefetch
    - MINOR: ist: add istadv() function
    - MINOR: ist: add istissame() function
    - MINOR: istbuf: add ist2buf() function
    - BUG/MINOR: threads: fix multiple use of argument inside HA_ATOMIC_CAS()
    - BUG/MINOR: threads: fix multiple use of argument inside HA_ATOMIC_UPDATE_{MIN,MAX}()
    - DOC: update intro.txt for 2.2
    - DOC: intro: add a contacts section
2020-05-05 21:49:10 +02:00
Willy Tarreau
6562623f41 DOC: intro: add a contacts section
This indicates where to find help and where to report bugs.
2020-05-05 18:08:07 +02:00
Willy Tarreau
ec8962cb5a DOC: update intro.txt for 2.2
A number of things have changed since last update, for example caching
and fastcgi were not mentioned.
2020-05-05 17:39:16 +02:00
Joseph C. Sible
49bbf528e4 MINOR: lua: allow changing port with set_addr
Add an optional port parameter, which can be either a number or a
string (to support '+' and '-' for port mapping).

This fixes issue #586.
2020-05-05 11:24:39 +02:00
Christopher Faulet
3970819a55 MEDIUM: checks: Support matching on headers for http-check expect rules
It is now possible to add http-check expect rules matching HTTP header names and
values. Here is the format of these rules:

  http-check expect header name [ -m <meth> ] <name> [log-format] \
                           [ value [ -m <meth> ] <value> [log-format] [full] ]

the name pattern (name ...) is mandatory but the value pattern (value ...) is
optionnal. If not specified, only the header presence is verified. <meth> is the
matching method, applied on the header name or the header value. Supported
matching methods are:

  * "str" (exact match)
  * "beg" (prefix match)
  * "end" (suffix match)
  * "sub" (substring match)
  * "reg" (regex match)

If not specified, exact matching method is used. If the "log-format" option is
used, the pattern (<name> or <value>) is evaluated as a log-format string. This
option cannot be used with the regex matching method. Finally, by default, the
header value is considered as comma-separated list. Each part may be tested. The
"full" option may be used to test the full header line. Note that matchings are
case insensitive on the header names.
2020-05-05 11:19:27 +02:00