Commit Graph

427 Commits

Author SHA1 Message Date
Christopher Faulet
a36b311b9f BUG/MINOR: buffers: Fix bi/bo_contig_space to handle full buffers
These functions was added in commit 637f8f2c ("BUG/MEDIUM: buffers: Fix how
input/output data are injected into buffers").

This patch fixes hidden bugs. When a buffer is full (buf->i + buf->o ==
buf->size), instead of returning 0, these functions can return buf->size. Today,
this never happens because callers already check if the buffer is full before
calling bi/bo_contig_space. But to avoid possible bugs if calling conditions
changed, we slightly refactored these functions.
2017-06-14 16:20:20 +02:00
Willy Tarreau
ed936c5d37 MINOR: tools: make debug_hexdump() take a string prefix
When dumping data at various places in the code, it's hard to figure
what is present where. To make this easier, this patch slightly modifies
debug_hexdump() to take a prefix string which is prepended in front of
each output line.
2017-06-02 15:49:31 +02:00
Willy Tarreau
9faef1e391 MINOR: tools: make debug_hexdump() use a const char for the string
There's no reason the string to be dumped should be a char *, it's
a const.
2017-06-02 15:49:31 +02:00
Jarno Huuskonen
577d5ac8ae CLEANUP: str2mask return code comment: non-zero -> zero. 2017-06-02 15:43:46 +02:00
Stéphane Cottin
23e9e93128 MINOR: log: Add logurilen tunable.
The default len of request uri in log messages is 1024. In some use
cases, you need to keep the long trail of GET parameters. The only
way to increase this len is to recompile with DEFINE=-DREQURI_LEN=2048.

This commit introduces a tune.http.logurilen configuration directive,
allowing to tune this at runtime.
2017-06-02 11:06:36 +02:00
Lukas Tribus
23953686da DOC: update RFC references
A few doc and code comment updates bumping RFC references to the new
ones.
2017-04-28 18:58:11 +02:00
Thierry FOURNIER
6ab2bae084 REORG: spoe: move spoe_encode_varint / spoe_decode_varint from spoe to common
These encoding functions does general stuff and can be used in
other context than spoe. This patch moves the function spoe_encode_varint
and spoe_decode_varint from spoe to common. It also remove the prefix spoe.

These functions will be used for encoding values in new binary sample fetch.
2017-04-27 11:50:41 +02:00
Frédéric Lécaille
b82f742b78 MINOR: server: Add 'server-template' new keyword supported in backend sections.
This patch makes backend sections support 'server-template' new keyword.
Such 'server-template' objects are parsed similarly to a 'server' object
by parse_server() function, but its first arguments are as follows:
    server-template <ID prefix> <nb | range> <ip | fqdn>:<port> ...

The remaining arguments are the same as for 'server' lines.

With such server template declarations, servers may be allocated with IDs
built from <ID prefix> and <nb | range> arguments.

For instance declaring:
    server-template foo 1-5 google.com:80 ...
or
    server-template foo 5 google.com:80 ...

would be equivalent to declare:
    server foo1 google.com:80 ...
    server foo2 google.com:80 ...
    server foo3 google.com:80 ...
    server foo4 google.com:80 ...
    server foo5 google.com:80 ...
2017-04-21 15:42:10 +02:00
Willy Tarreau
7b677265fd [RELEASE] Released version 1.8-dev1
Released version 1.8-dev1 with the following main changes :
    - BUG/MEDIUM: proxy: return "none" and "unknown" for unknown LB algos
    - BUG/MINOR: stats: make field_str() return an empty string on NULL
    - DOC: Spelling fixes
    - BUG/MEDIUM: http: Fix tunnel mode when the CONNECT method is used
    - BUG/MINOR: http: Keep the same behavior between 1.6 and 1.7 for tunneled txn
    - BUG/MINOR: filters: Protect args in macros HAS_DATA_FILTERS and IS_DATA_FILTER
    - BUG/MINOR: filters: Invert evaluation order of HTTP_XFER_BODY and XFER_DATA analyzers
    - BUG/MINOR: http: Call XFER_DATA analyzer when HTTP txn is switched in tunnel mode
    - BUG/MAJOR: stream: fix session abort on resource shortage
    - OPTIM: stream-int: don't disable polling anymore on DONT_READ
    - BUG/MINOR: cli: allow the backslash to be escaped on the CLI
    - BUG/MEDIUM: cli: fix "show stat resolvers" and "show tls-keys"
    - DOC: Fix map table's format
    - DOC: Added 51Degrees conv and fetch functions to documentation.
    - BUG/MINOR: http: don't send an extra CRLF after a Set-Cookie in a redirect
    - DOC: mention that req_tot is for both frontends and backends
    - BUG/MEDIUM: variables: some variable name can hide another ones
    - MINOR: lua: Allow argument for actions
    - BUILD: rearrange target files by build time
    - CLEANUP: hlua: just indent functions
    - MINOR: lua: give HAProxy variable access to the applets
    - BUG/MINOR: stats: fix be/sessions/max output in html stats
    - MINOR: proxy: Add fe_name/be_name fetchers next to existing fe_id/be_id
    - DOC: lua: Documentation about some entry missing
    - DOC: lua: Add documentation about variable manipulation from applet
    - MINOR: Do not forward the header "Expect: 100-continue" when the option http-buffer-request is set
    - DOC: Add undocumented argument of the trace filter
    - DOC: Fix some typo in SPOE documentation
    - MINOR: cli: Remove useless call to bi_putchk
    - BUG/MINOR: cli: be sure to always warn the cli applet when input buffer is full
    - MINOR: applet: Count number of (active) applets
    - MINOR: task: Rename run_queue and run_queue_cur counters
    - BUG/MEDIUM: stream: Save unprocessed events for a stream
    - BUG/MAJOR: Fix how the list of entities waiting for a buffer is handled
    - BUILD/MEDIUM: Fixing the build using LibreSSL
    - BUG/MEDIUM: lua: In some case, the return of sample-fetches is ignored (2)
    - SCRIPTS: git-show-backports: fix a harmless typo
    - SCRIPTS: git-show-backports: add -H to use the hash of the commit message
    - BUG/MINOR: stream-int: automatically release SI_FL_WAIT_DATA on SHUTW_NOW
    - CLEANUP: applet/lua: create a dedicated ->fcn entry in hlua_cli context
    - CLEANUP: applet/table: add an "action" entry in ->table context
    - CLEANUP: applet: remove the now unused appctx->private field
    - DOC: lua: documentation about time parser functions
    - DOC: lua: improve links
    - DOC: lua: section declared twice
    - MEDIUM: cli: 'show cli sockets' list the CLI sockets
    - BUG/MINOR: cli: "show cli sockets" wouldn't list all processes
    - BUG/MINOR: cli: "show cli sockets" would always report process 64
    - CLEANUP: lua: rename one of the lua appctx union
    - BUG/MINOR: lua/cli: bad error message
    - MEDIUM: lua: use memory pool for hlua struct in applets
    - MINOR: lua/signals: Remove Lua part from signals.
    - DOC: cli: show cli sockets
    - MINOR: cli: automatically enable a CLI I/O handler when there's no parser
    - CLEANUP: memory: remove the now unused cli_parse_show_pools() function
    - CLEANUP: applet: group all CLI contexts together
    - CLEANUP: stats: move a misplaced stats context initialization
    - MINOR: cli: add two general purpose pointers and integers in the CLI struct
    - MINOR: appctx/cli: remove the cli_socket entry from the appctx union
    - MINOR: appctx/cli: remove the env entry from the appctx union
    - MINOR: appctx/cli: remove the "be" entry from the appctx union
    - MINOR: appctx/cli: remove the "dns" entry from the appctx union
    - MINOR: appctx/cli: remove the "server_state" entry from the appctx union
    - MINOR: appctx/cli: remove the "tlskeys" entry from the appctx union
    - CONTRIB: tcploop: add limits.h to fix build issue with some compilers
    - MINOR/DOC: lua: just precise one thing
    - DOC: fix small typo in fe_id (backend instead of frontend)
    - BUG/MINOR: Fix the sending function in Lua's cosocket
    - BUG/MINOR: lua: memory leak executing tasks
    - BUG/MINOR: lua: bad return code
    - BUG/MINOR: lua: memleak when Lua/cli fails
    - MEDIUM: lua: remove Lua struct from session, and allocate it with memory pools
    - CLEANUP: haproxy: statify unexported functions
    - MINOR: haproxy: add a registration for build options
    - CLEANUP: wurfl: use the build options list to report it
    - CLEANUP: 51d: use the build options list to report it
    - CLEANUP: da: use the build options list to report it
    - CLEANUP: namespaces: use the build options list to report it
    - CLEANUP: tcp: use the build options list to report transparent modes
    - CLEANUP: lua: use the build options list to report it
    - CLEANUP: regex: use the build options list to report the regex type
    - CLEANUP: ssl: use the build options list to report the SSL details
    - CLEANUP: compression: use the build options list to report the algos
    - CLEANUP: auth: use the build options list to report its support
    - MINOR: haproxy: add a registration for post-check functions
    - CLEANUP: checks: make use of the post-init registration to start checks
    - CLEANUP: filters: use the function registration to initialize all proxies
    - CLEANUP: wurfl: make use of the late init registration
    - CLEANUP: 51d: make use of the late init registration
    - CLEANUP: da: make use of the late init registration code
    - MINOR: haproxy: add a registration for post-deinit functions
    - CLEANUP: wurfl: register the deinit function via the dedicated list
    - CLEANUP: 51d: register the deinitialization function
    - CLEANUP: da: register the deinitialization function
    - CLEANUP: wurfl: move global settings out of the global section
    - CLEANUP: 51d: move global settings out of the global section
    - CLEANUP: da: move global settings out of the global section
    - MINOR: cfgparse: add two new functions to check arguments count
    - MINOR: cfgparse: move parsing of "ca-base" and "crt-base" to ssl_sock
    - MEDIUM: cfgparse: move all tune.ssl.* keywords to ssl_sock
    - MEDIUM: cfgparse: move maxsslconn parsing to ssl_sock
    - MINOR: cfgparse: move parsing of ssl-default-{bind,server}-ciphers to ssl_sock
    - MEDIUM: cfgparse: move ssl-dh-param-file parsing to ssl_sock
    - MEDIUM: compression: move the zlib-specific stuff from global.h to compression.c
    - BUG/MEDIUM: ssl: properly reset the reused_sess during a forced handshake
    - BUG/MEDIUM: ssl: avoid double free when releasing bind_confs
    - BUG/MINOR: stats: fix be/sessions/current out in typed stats
    - MINOR: tcp-rules: check that the listener exists before updating its counters
    - MEDIUM: spoe: don't create a dummy listener for outgoing connections
    - MINOR: listener: move the transport layer pointer to the bind_conf
    - MEDIUM: move listener->frontend to bind_conf->frontend
    - MEDIUM: ssl: remote the proxy argument from most functions
    - MINOR: connection: add a new prepare_bind_conf() entry to xprt_ops
    - MEDIUM: ssl_sock: implement ssl_sock_prepare_bind_conf()
    - MINOR: connection: add a new destroy_bind_conf() entry to xprt_ops
    - MINOR: ssl_sock: implement ssl_sock_destroy_bind_conf()
    - MINOR: server: move the use_ssl field out of the ifdef USE_OPENSSL
    - MINOR: connection: add a minimal transport layer registration system
    - CLEANUP: connection: remove all direct references to raw_sock and ssl_sock
    - CLEANUP: connection: unexport raw_sock and ssl_sock
    - MINOR: connection: add new prepare_srv()/destroy_srv() entries to xprt_ops
    - MINOR: ssl_sock: implement and use prepare_srv()/destroy_srv()
    - CLEANUP: ssl: move tlskeys_finalize_config() to a post_check callback
    - CLEANUP: ssl: move most ssl-specific global settings to ssl_sock.c
    - BUG/MINOR: backend: nbsrv() should return 0 if backend is disabled
    - BUG/MEDIUM: ssl: for a handshake when server-side SNI changes
    - BUG/MINOR: systemd: potential zombie processes
    - DOC: Add timings events schemas
    - BUILD: lua: build failed on FreeBSD.
    - MINOR: samples: add xx-hash functions
    - MEDIUM: regex: pcre2 support
    - BUG/MINOR: option prefer-last-server must be ignored in some case
    - MINOR: stats: Support "select all" for backend actions
    - BUG/MINOR: sample-fetches/stick-tables: bad type for the sample fetches sc*_get_gpt0
    - BUG/MAJOR: channel: Fix the definition order of channel analyzers
    - BUG/MINOR: http: report real parser state in error captures
    - BUILD: scripts: automatically update the branch in version.h when releasing
    - MINOR: tools: add a generic hexdump function for debugging
    - BUG/MAJOR: http: fix risk of getting invalid reports of bad requests
    - MINOR: http: custom status reason.
    - MINOR: connection: add sample fetch "fc_rcvd_proxy"
    - BUG/MINOR: config: emit a warning if http-reuse is enabled with incompatible options
    - BUG/MINOR: tools: fix off-by-one in port size check
    - BUG/MEDIUM: server: consider AF_UNSPEC as a valid address family
    - MEDIUM: server: split the address and the port into two different fields
    - MINOR: tools: make str2sa_range() return the port in a separate argument
    - MINOR: server: take the destination port from the port field, not the addr
    - MEDIUM: server: disable protocol validations when the server doesn't resolve
    - BUG/MEDIUM: tools: do not force an unresolved address to AF_INET:0.0.0.0
    - BUG/MINOR: ssl: EVP_PKEY must be freed after X509_get_pubkey usage
    - BUG/MINOR: ssl: assert on SSL_set_shutdown with BoringSSL
    - MINOR: Use "500 Internal Server Error" for 500 error/status code message.
    - MINOR: proto_http.c 502 error txt typo.
    - DOC: add deprecation notice to "block"
    - MINOR: compression: fix -vv output without zlib/slz
    - BUG/MINOR: Reset errno variable before calling strtol(3)
    - MINOR: ssl: don't show prefer-server-ciphers output
    - OPTIM/MINOR: config: Optimize fullconn automatic computation loading configuration
    - BUG/MINOR: stream: Fix how backend-specific analyzers are set on a stream
    - MAJOR: ssl: bind configuration per certificat
    - MINOR: ssl: add curve suite for ECDHE negotiation
    - MINOR: checks: Add agent-addr config directive
    - MINOR: cli: Add possiblity to change agent config via CLI/socket
    - MINOR: doc: Add docs for agent-addr configuration variable
    - MINOR: doc: Add docs for agent-addr and agent-send CLI commands
    - BUILD: ssl: fix to build (again) with boringssl
    - BUILD: ssl: fix build on OpenSSL 1.0.0
    - BUILD: ssl: silence a warning reported for ERR_remove_state()
    - BUILD: ssl: eliminate warning with OpenSSL 1.1.0 regarding RAND_pseudo_bytes()
    - BUILD: ssl: kill a build warning introduced by BoringSSL compatibility
    - BUG/MEDIUM: tcp: don't poll for write when connect() succeeds
    - BUG/MINOR: unix: fix connect's polling in case no data are scheduled
    - MINOR: server: extend the flags to 32 bits
    - BUG/MINOR: lua: Map.end are not reliable because "end" is a reserved keyword
    - MINOR: dns: give ability to dns_init_resolvers() to close a socket when requested
    - BUG/MAJOR: dns: restart sockets after fork()
    - MINOR: chunks: implement a simple dynamic allocator for trash buffers
    - BUG/MEDIUM: http: prevent redirect from overwriting a buffer
    - BUG/MEDIUM: filters: Do not truncate HTTP response when body length is undefined
    - BUG/MEDIUM: http: Prevent replace-header from overwriting a buffer
    - BUG/MINOR: http: Return an error when a replace-header rule failed on the response
    - BUG/MINOR: sendmail: The return of vsnprintf is not cleanly tested
    - BUG/MAJOR: ssl: fix a regression in ssl_sock_shutw()
    - BUG/MAJOR: lua segmentation fault when the request is like 'GET ?arg=val HTTP/1.1'
    - BUG/MEDIUM: config: reject anything but "if" or "unless" after a use-backend rule
    - MINOR: http: don't close when redirect location doesn't start with "/"
    - MEDIUM: boringssl: support native multi-cert selection without bundling
    - BUG/MEDIUM: ssl: fix verify/ca-file per certificate
    - BUG/MEDIUM: ssl: switchctx should not return SSL_TLSEXT_ERR_ALERT_WARNING
    - MINOR: ssl: removes SSL_CTX_set_ssl_version call and cleanup CTX creation.
    - BUILD: ssl: fix build with -DOPENSSL_NO_DH
    - MEDIUM: ssl: add new sample-fetch which captures the cipherlist
    - MEDIUM: ssl: remove ssl-options from crt-list
    - BUG/MEDIUM: ssl: in bind line, ssl-options after 'crt' are ignored.
    - BUG/MINOR: ssl: fix cipherlist captures with sustainable SSL calls
    - MINOR: ssl: improved cipherlist captures
    - BUG/MINOR: spoe: Fix soft stop handler using a specific id for spoe filters
    - BUG/MINOR: spoe: Fix parsing of arguments in spoe-message section
    - MAJOR: spoe: Add support of pipelined and asynchronous exchanges with agents
    - MINOR: spoe: Add support for pipelining/async capabilities in the SPOA example
    - MINOR: spoe: Remove SPOE details from the appctx structure
    - MINOR: spoe: Add status code in error variable instead of hardcoded value
    - MINOR: spoe: Send a log message when an error occurred during event processing
    - MINOR: spoe: Check the scope of sample fetches used in SPOE messages
    - MEDIUM: spoe: Be sure to wakeup the good entity waiting for a buffer
    - MINOR: spoe: Use the min of all known max_frame_size to encode messages
    - MAJOR: spoe: Add support of payload fragmentation in NOTIFY frames
    - MINOR: spoe: Add support for fragmentation capability in the SPOA example
    - MAJOR: spoe: refactor the filter to clean up the code
    - MINOR: spoe: Handle NOTIFY frames cancellation using ABORT bit in ACK frames
    - REORG: spoe: Move struct and enum definitions in dedicated header file
    - REORG: spoe: Move low-level encoding/decoding functions in dedicated header file
    - MINOR: spoe: Improve implementation of the payload fragmentation
    - MINOR: spoe: Add support of negation for options in SPOE configuration file
    - MINOR: spoe: Add "pipelining" and "async" options in spoe-agent section
    - MINOR: spoe: Rely on alertif_too_many_arg during configuration parsing
    - MINOR: spoe: Add "send-frag-payload" option in spoe-agent section
    - MINOR: spoe: Add "max-frame-size" statement in spoe-agent section
    - DOC: spoe: Update SPOE documentation to reflect recent changes
    - MINOR: config: warn when some HTTP rules are used in a TCP proxy
    - BUG/MEDIUM: ssl: Clear OpenSSL error stack after trying to parse OCSP file
    - BUG/MEDIUM: cli: Prevent double free in CLI ACL lookup
    - BUG/MINOR: Fix "get map <map> <value>" CLI command
    - MINOR: Add nbsrv sample converter
    - CLEANUP: Replace repeated code to count usable servers with be_usable_srv()
    - MINOR: Add hostname sample fetch
    - CLEANUP: Remove comment that's no longer valid
    - MEDIUM: http_error_message: txn->status / http_get_status_idx.
    - MINOR: http-request tarpit deny_status.
    - CLEANUP: http: make http_server_error() not set the status anymore
    - MEDIUM: stats: Add JSON output option to show (info|stat)
    - MEDIUM: stats: Add show json schema
    - BUG/MAJOR: connection: update CO_FL_CONNECTED before calling the data layer
    - MINOR: server: Add dynamic session cookies.
    - MINOR: cli: Let configure the dynamic cookies from the cli.
    - BUG/MINOR: checks: attempt clean shutw for SSL check
    - CONTRIB: tcploop: make it build on FreeBSD
    - CONTRIB: tcploop: fix time format to silence build warnings
    - CONTRIB: tcploop: report action 'K' (kill) in usage message
    - CONTRIB: tcploop: fix connect's address length
    - CONTRIB: tcploop: use the trash instead of NULL for recv()
    - BUG/MEDIUM: listener: do not try to rebind another process' socket
    - BUG/MEDIUM server: Fix crash when dynamic is defined, but not key is provided.
    - CLEANUP: config: Typo in comment.
    - BUG/MEDIUM: filters: Fix channels synchronization in flt_end_analyze
    - TESTS: add a test configuration to stress handshake combinations
    - BUG/MAJOR: stream-int: do not depend on connection flags to detect connection
    - BUG/MEDIUM: connection: ensure to always report the end of handshakes
    - MEDIUM: connection: don't test for CO_FL_WAKE_DATA
    - CLEANUP: connection: completely remove CO_FL_WAKE_DATA
    - BUG: payload: fix payload not retrieving arbitrary lengths
    - BUILD: ssl: simplify SSL_CTX_set_ecdh_auto compatibility
    - BUILD: ssl: fix OPENSSL_NO_SSL_TRACE for boringssl and libressl
    - BUG/MAJOR: http: fix typo in http_apply_redirect_rule
    - MINOR: doc: 2.4. Examples should be 2.5. Examples
    - BUG/MEDIUM: stream: fix client-fin/server-fin handling
    - MINOR: fd: add a new flag HAP_POLL_F_RDHUP to struct poller
    - BUG/MINOR: raw_sock: always perfom the last recv if RDHUP is not available
    - OPTIM: poll: enable support for POLLRDHUP
    - MINOR: kqueue: exclusively rely on the kqueue returned status
    - MEDIUM: kqueue: take care of EV_EOF to improve polling status accuracy
    - MEDIUM: kqueue: only set FD_POLL_IN when there are pending data
    - DOC/MINOR: Fix typos in proxy protocol doc
    - DOC: Protocol doc: add checksum, TLV type ranges
    - DOC: Protocol doc: add SSL TLVs, rename CHECKSUM
    - DOC: Protocol doc: add noop TLV
    - MEDIUM: global: add a 'hard-stop-after' option to cap the soft-stop time
    - MINOR: dns: improve DNS response parsing to use as many available records as possible
    - BUG/MINOR: cfgparse: loop in tracked servers lists not detected by check_config_validity().
    - MINOR: server: irrelevant error message with 'default-server' config file keyword.
    - MINOR: server: Make 'default-server' support 'backup' keyword.
    - MINOR: server: Make 'default-server' support 'check-send-proxy' keyword.
    - CLEANUP: server: code alignement.
    - MINOR: server: Make 'default-server' support 'non-stick' keyword.
    - MINOR: server: Make 'default-server' support 'send-proxy' and 'send-proxy-v2 keywords.
    - MINOR: server: Make 'default-server' support 'check-ssl' keyword.
    - MINOR: server: Make 'default-server' support 'force-sslv3' and 'force-tlsv1[0-2]' keywords.
    - CLEANUP: server: code alignement.
    - MINOR: server: Make 'default-server' support 'no-ssl*' and 'no-tlsv*' keywords.
    - MINOR: server: Make 'default-server' support 'ssl' keyword.
    - MINOR: server: Make 'default-server' support 'send-proxy-v2-ssl*' keywords.
    - CLEANUP: server: code alignement.
    - MINOR: server: Make 'default-server' support 'verify' keyword.
    - MINOR: server: Make 'default-server' support 'verifyhost' setting.
    - MINOR: server: Make 'default-server' support 'check' keyword.
    - MINOR: server: Make 'default-server' support 'track' setting.
    - MINOR: server: Make 'default-server' support 'ca-file', 'crl-file' and 'crt' settings.
    - MINOR: server: Make 'default-server' support 'redir' keyword.
    - MINOR: server: Make 'default-server' support 'observe' keyword.
    - MINOR: server: Make 'default-server' support 'cookie' keyword.
    - MINOR: server: Make 'default-server' support 'ciphers' keyword.
    - MINOR: server: Make 'default-server' support 'tcp-ut' keyword.
    - MINOR: server: Make 'default-server' support 'namespace' keyword.
    - MINOR: server: Make 'default-server' support 'source' keyword.
    - MINOR: server: Make 'default-server' support 'sni' keyword.
    - MINOR: server: Make 'default-server' support 'addr' keyword.
    - MINOR: server: Make 'default-server' support 'disabled' keyword.
    - MINOR: server: Add 'no-agent-check' server keyword.
    - DOC: server: Add docs for "server" and "default-server" new "no-*" and other settings.
    - MINOR: doc: fix use-server example (imap vs mail)
    - BUG/MEDIUM: tcp: don't require privileges to bind to device
    - BUILD: make the release script use shortlog for the final changelog
    - BUILD: scripts: fix typo in announce-release error message
    - CLEANUP: time: curr_sec_ms doesn't need to be exported
    - BUG/MEDIUM: server: Wrong server default CRT filenames initialization.
    - BUG/MEDIUM: peers: fix buffer overflow control in intdecode.
    - BUG/MEDIUM: buffers: Fix how input/output data are injected into buffers
    - BUG/MINOR: http: Fix conditions to clean up a txn and to handle the next request
    - CLEANUP: http: Remove channel_congested function
    - CLEANUP: buffers: Remove buffer_bounce_realign function
    - CLEANUP: buffers: Remove buffer_contig_area and buffer_work_area functions
    - MINOR: http: remove useless check on HTTP_MSGF_XFER_LEN for the request
    - MINOR: http: Add debug messages when HTTP body analyzers are called
    - BUG/MEDIUM: http: Fix blocked HTTP/1.0 responses when compression is enabled
    - BUG/MINOR: filters: Don't force the stream's wakeup when we wait in flt_end_analyze
    - DOC: fix parenthesis and add missing "Example" tags
    - DOC: update the contributing file
    - DOC: log-format/tcplog/httplog update
    - MINOR: config parsing: add warning when log-format/tcplog/httplog is overriden in "defaults" sections
2017-04-03 09:27:49 +02:00
Christopher Faulet
a545569f1e CLEANUP: buffers: Remove buffer_contig_area and buffer_work_area functions
Not used anymore since last commit.
2017-03-31 14:38:30 +02:00
Christopher Faulet
aaf4a325ca CLEANUP: buffers: Remove buffer_bounce_realign function
Not used anymore since last commit.
2017-03-31 14:38:22 +02:00
Christopher Faulet
637f8f2ca7 BUG/MEDIUM: buffers: Fix how input/output data are injected into buffers
The function buffer_contig_space is buggy and could lead to pernicious bugs
(never hitted until now, AFAIK). This function should return the number of bytes
that can be written into the buffer at once (without wrapping).

First, this function is used to inject input data (bi_putblk) and to inject
output data (bo_putblk and bo_inject). But there is no context. So it cannot
decide where contiguous space should placed. For input data, it should be after
bi_end(buf) (ie, buf->p + buf->i modulo wrapping calculation). For output data,
it should be after bo_end(buf) (ie, buf->p) and input data are assumed to not
exist (else there is no space at all).

Then, considering we need to inject input data, this function does not always
returns the right value. And when we need to inject output data, we must be sure
to have no input data at all (buf->i == 0), else the result can also be wrong
(but this is the caller responsibility, so everything should be fine here).

The buffer can be in 3 different states:

 1) no wrapping

              <---- o ----><----- i ----->
 +------------+------------+-------------+------------+
 |            |oooooooooooo|iiiiiiiiiiiii|xxxxxxxxxxxx|
 +------------+------------+-------------+------------+
                           ^             <contig_space>
                           p             ^            ^
			                 l            r

 2) input wrapping

 ...--->            <---- o ----><-------- i -------...
 +-----+------------+------------+--------------------+
 |iiiii|xxxxxxxxxxxx|oooooooooooo|iiiiiiiiiiiiiiiiiiii|
 +-----+------------+------------+--------------------+
       <contig_space>            ^
       ^            ^            p
       l            r

 3) output wrapping

 ...------ o ------><----- i ----->            <----...
 +------------------+-------------+------------+------+
 |oooooooooooooooooo|iiiiiiiiiiiii|xxxxxxxxxxxx|oooooo|
 +------------------+-------------+------------+------+
                    ^             <contig_space>
                    p             ^            ^
		                  l            r

buffer_contig_space returns (l - r). The cases 1 and 3 are correctly
handled. But for the second case, r is wrong. It points on the buffer's end
(buf->data + buf->size). It should be bo_end(buf) (ie, buf->p - buf->o).

To fix the bug, the function has been splitted. Now, bi_contig_space and
bo_contig_space should be used to know the contiguous space available to insert,
respectively, input data and output data. For bo_contig_space, input data are
assumed to not exist. And the right version is used, depending what we want to
do.

In addition, to clarify the buffer's API, buffer_realign does not return value
anymore. So it has the same API than buffer_slow_realign.

This patch can be backported in 1.7, 1.6 and 1.5.
2017-03-31 14:36:04 +02:00
Willy Tarreau
b686afd568 MINOR: chunks: implement a simple dynamic allocator for trash buffers
The trash buffers are becoming increasingly complex to deal with due to
the code's modularity allowing some functions to be chained and causing
the same chunk buffers to be used multiple times along the chain, possibly
corrupting each other. In fact the trash were designed from scratch for
explicitly not surviving a function call but string manipulation makes
this impossible most of the time while not fullfilling the need for
reliable temporary chunks.

Here we introduce the ability to allocate a temporary trash chunk which
is reserved, so that it will not conflict with the trash chunks other
functions use, and will even support reentrant calls (eg: build_logline).

For this, we create a new pool which is exactly the size of a usual chunk
buffer plus the size of the chunk struct so that these chunks when allocated
are exactly the same size as the ones returned by get_trash_buffer(). These
chunks may fail so the caller must check them, and the caller is also
responsible for freeing them.

The code focuses on minimal changes and ease of reliable backporting
because it will be needed in stable versions in order to support next
patch.
2017-02-08 11:16:29 +01:00
Emmanuel Hocdet
98263291cc MAJOR: ssl: bind configuration per certificat
crt-list is extend to support ssl configuration. You can now have
such line in crt-list <file>:
mycert.pem [npn h2,http/1.1]

Support include "npn", "alpn", "verify", "ca_file", "crl_file",
"ecdhe", "ciphers" configuration and ssl options.

"crt-base" is also supported to fetch certificates.
2017-01-13 11:40:34 +01:00
Willy Tarreau
48ef4c95b6 MINOR: tools: make str2sa_range() return the port in a separate argument
This will be needed so that we're don't have to extract it from the
returned address where it will not always be anymore (eg: for unresolved
servers).
2017-01-06 19:29:34 +01:00
Willy Tarreau
0ebb511b3e MINOR: tools: add a generic hexdump function for debugging
debug_hexdump() prints to the requested output stream (typically stdout
or stderr) an hex dump of the blob passed in argument. This is useful
to help debug binary protocols.
2017-01-05 20:12:20 +01:00
David Carlier
f2592b29f1 MEDIUM: regex: pcre2 support
this adds a support of the newest pcre2 library,
more secure than its older sibling in a cost of a
more complex API.
It works pretty similarly to pcre's part to keep
the overall change smooth,  except :

- we define the string class supported at compile time.
- after matching the ovec data is properly sized, althought
we do not take advantage of it here.
- the lack of jit support is treated less 'dramatically'
as pcre2_jit_compile in this case is 'no-op'.
2016-12-28 12:51:51 +01:00
Willy Tarreau
ece9b07c71 MINOR: cfgparse: add two new functions to check arguments count
We already had alertif_too_many_args{,_idx}(), but these ones are
specifically designed for use in cfgparse. Outside of it we're
trying to avoid calling Alert() all the time so we need an
equivalent using a pointer to an error message.

These new functions called too_many_args{,_idx)() do exactly this.
They don't take the file name nor the line number which they have
no use for but instead they take an optional pointer to an error
message and the pointer to the error code is optional as well.
With (NULL, NULL) they'll simply check the validity and return a
verdict. They are quite convenient for use in isolated keyword
parsers.

These two new functions as well as the previous ones have all been
exported.
2016-12-21 23:39:26 +01:00
Christopher Faulet
a73e59b690 BUG/MAJOR: Fix how the list of entities waiting for a buffer is handled
When an entity tries to get a buffer, if it cannot be allocted, for example
because the number of buffers which may be allocated per process is limited,
this entity is added in a list (called <buffer_wq>) and wait for an available
buffer.

Historically, the <buffer_wq> list was logically attached to streams because it
were the only entities likely to be added in it. Now, applets can also be
waiting for a free buffer. And with filters, we could imagine to have more other
entities waiting for a buffer. So it make sense to have a generic list.

Anyway, with the current design there is a bug. When an applet failed to get a
buffer, it will wait. But we add the stream attached to the applet in
<buffer_wq>, instead of the applet itself. So when a buffer is available, we
wake up the stream and not the waiting applet. So, it is possible to have
waiting applets and never awakened.

So, now, <buffer_wq> is independant from streams. And we really add the waiting
entity in <buffer_wq>. To be generic, the entity is responsible to define the
callback used to awaken it.

In addition, applets will still request an input buffer when they become
active. But they will not be sleeped anymore if no buffer are available. So this
is the responsibility to the applet I/O handler to check if this buffer is
allocated or not. This way, an applet can decide if this buffer is required or
not and can do additional processing if not.

[wt: backport to 1.7 and 1.6]
2016-12-12 19:11:04 +01:00
Willy Tarreau
97c2ae13bc REORG: cli: move dump_text(), dump_text_line(), and dump_binary() to standard.c
These are general purpose functions, move them away.
2016-11-24 16:59:27 +01:00
Christopher Faulet
79bdef3cad MINOR: cfgparse: Parse scope lines and save the last one parsed
A scope is a section name between square bracket, alone on its line, ie:

  [scope-name]
  ...

The spaces at the beginning and at the end of the line are skipped. Comments at
the end of the line are also skipped.

When a scope is parsed, its name is saved in the global variable
cfg_scope. Initially, cfg_scope is NULL and it remains NULL until a valid scope
line is parsed.

This feature remains unused in the HAProxy configuration file and
undocumented. However, it will be used during SPOE configuration parsing.
2016-11-09 22:56:59 +01:00
Christopher Faulet
7110b40d06 MINOR: cfgparse: Add functions to backup and restore registered sections
This feature will be used by the stream processing offload engine (SPOE) to
parse dedicated configuration files without mixing HAProxy sections with SPOE
sections.

So, here we can back up all sections known by HAProxy, unregister all of them
and add new ones, dedicted to the SPOE. Once the SPOE configuration file parsed,
we can roll back all changes by restoring HAProxy sections.
2016-11-09 22:56:59 +01:00
Christopher Faulet
898566e7e6 CLEANUP: remove last references to 'ruleset' section 2016-11-09 22:50:54 +01:00
Willy Tarreau
620408f406 MEDIUM: tcp: add registration and processing of TCP L5 rules
This commit introduces "tcp-request session" rules. These are very
much like "tcp-request connection" rules except that they're processed
after the handshake, so it is possible to consider SSL information and
addresses rewritten by the proxy protocol header in actions. This is
particularly useful to track proxied sources as this was not possible
before, given that tcp-request content rules are processed after each
HTTP request. Similarly it is possible to assign the proxied source
address or the client's cert to a variable.
2016-10-21 18:19:24 +02:00
Lukas Tribus
dcbc5c5ecf MINOR: show Built with PCRE version
Inspired by PCRE's pcre_version.c and improved with Willy's
suggestions. Reusable parts have been added to
include/common/standard.h.
2016-09-13 07:55:51 +02:00
Baptiste Assmann
7819c125c2 MINOR: chunk: new strncat function
Purpose of this function is to append data to the end of a chunk when
we know only the pointer to the beginning of the string and the string
length.
2016-09-12 19:51:59 +02:00
Baptiste Assmann
08396c87d0 MINOR: standard.c: ipcpy() function to copy an IP address from a struct sockaddr_storage into an other one
The function ipcpy() simply duplicates the IP address found in one
struct sockaddr_storage into an other struct sockaddr_storage.
It also update the family on the destination structure.

Memory of destination structure must be allocated and cleared by the
caller.
2016-08-14 12:16:43 +02:00
Baptiste Assmann
08b24cfdb2 MINOR: standard.c: ipcmp() function to compare 2 IP addresses stored in 2 struct sockaddr_storage
new ipcmp() function to compare 2 IP addresses stored in struct
sockaddr_storage.
Returns 0 if both addresses doesn't match and 1 if they do.
2016-08-14 12:16:27 +02:00
Willy Tarreau
9d87ca0685 BUILD: tcp: define SOL_TCP when only IPPROTO_TCP exists
FreeBSD prefers to use IPPROTO_TCP over SOL_TCP, just like it does
with their *_IP counterparts. It's worth noting that there are a few
inconsistencies between SOL_TCP and IPPROTO_TCP in the code, eg on
TCP_QUICKACK. The two values are the same but it's worth applying
what implementations recommend.

No backport is needed, this was uncovered by the recent tcp_info stuff.
2016-08-10 21:11:38 +02:00
Willy Tarreau
16e015635c MINOR: tcp: add dst_is_local and src_is_local
It is sometimes needed in application server environments to easily tell
if a source is local to the machine or a remote one, without necessarily
knowing all the local addresses (dhcp, vrrp, etc). Similarly in transparent
proxy configurations it is sometimes desired to tell the difference between
local and remote destination addresses.

This patch adds two new sample fetch functions for this :

dst_is_local : boolean
  Returns true if the destination address of the incoming connection is local
  to the system, or false if the address doesn't exist on the system, meaning
  that it was intercepted in transparent mode. It can be useful to apply
  certain rules by default to forwarded traffic and other rules to the traffic
  targetting the real address of the machine. For example the stats page could
  be delivered only on this address, or SSH access could be locally redirected.
  Please note that the check involves a few system calls, so it's better to do
  it only once per connection.

src_is_local : boolean
  Returns true if the source address of the incoming connection is local to the
  system, or false if the address doesn't exist on the system, meaning that it
  comes from a remote machine. Note that UNIX addresses are considered local.
  It can be useful to apply certain access restrictions based on where the
  client comes from (eg: require auth or https for remote machines). Please
  note that the check involves a few system calls, so it's better to do it only
  once per connection.
2016-08-09 16:50:08 +02:00
Dragan Dosen
1a5d06032b MINOR: standard: add function "escape_string"
Similar to "escape_chunk", this function tries to prefix all characters
tagged in the <map> with the <escape> character. The specified <string>
contains the input to be escaped.
2016-07-26 15:25:32 +02:00
Willy Tarreau
eec1d3869d BUG/MEDIUM: dns: fix alignment issues in the DNS response parser
Alexander Lebedev reported that the DNS parser crashes in 1.6 with a bus
error on Sparc when it receives a response. This is obviously caused by
some alignment issues. The issue can also be reproduced on ARMv5 when
setting /proc/cpu/alignment to 4 (which helps debugging).

Two places cause this crash in turn, the first one is when the IP address
from the packet is compared to the current one, and the second place is
when the address is assigned because an unaligned address is passed to
update_server_addr().

This patch modifies these places to properly use memcpy() and memcmp()
to manipulate the unaligned data.

Nenad Merdanovic found another set of places specific to 1.7 in functions
in_net_ipv4() and in_net_ipv6(), which are used to compare networks. 1.6
has the functions but does not use them. There we perform a temporary copy
to a local variable to fix the problem. The type of the function's argument
is wrong since it's not necessarily aligned, so we change it for a const
void * instead.

This fix must be backported to 1.6. Note that in 1.6 the code is slightly
different, there's no rec[] array, the pointer is used directly from the
buffer.
2016-07-13 12:13:24 +02:00
Hubert Verstraete
2eae3a0497 MINOR: new function my_realloc2 = realloc + free upon failure
When realloc fails to allocate memory, the original pointer is not
freed. Sometime people override the original pointer with the pointer
returned by realloc which is NULL in case of failure. This results
in a memory leak because the memory pointed by the original pointer
cannot be freed.
2016-06-29 10:45:15 +02:00
Emmanuel Hocdet
5e0e6e409b MINOR: ssl: crt-list parsing factor
LINESIZE and MAX_LINE_ARGS are too low for parsing crt-list.
2016-06-20 17:29:56 +02:00
Willy Tarreau
5f6e9054b9 BUILD: fix build on Solaris 11
htonll()/ntohll() already exist on Solaris 11 with a different declaration,
causing a build error as reported by Jonathan Fisher. They used to exist on
OSX with a #define which allowed us to detect them. It was a bad idea to give
these functions a name subject to conflicts like this. Simply rename them
my_htonll()/my_ntohll() to definitely get rid of the conflict.

This patch must be backported to 1.6.
2016-05-26 07:15:57 +02:00
Maxime de Roucy
dc88785f9c MINOR: add list_append_word function
int list_append_word(struct list *li, const char *str, char **err)

Append a copy of string <str> (inside a wordlist) at the end of
the list <li>.
The caller is responsible for freeing the <err> and <str> copy memory
area using free().

On failure : return 0 and <err> filled with an error message.
2016-05-14 00:00:54 +02:00
David Carlier
8ab1043c6b CLEANUP: chunk: adding NULL check to chunk_dup allocation.
Avoiding harmful memcpy call if the allocation failed.
Resetting the size which avoids further harmful freeing
invalid pointer. Closer to the comment behavior description.
2016-03-24 10:18:44 +01:00
Benoit GARNIER
e2e5bde3f2 BUG/MINOR: log: Don't use strftime() which can clobber timezone if chrooted
The strftime() function can call tzset() internally on some platforms.
When haproxy is chrooted, the /etc/localtime file is not found, and some
implementations will clobber the content of the current timezone.

The GMT offset is computed by diffing the times returned by gmtime_r() and
localtime_r(). These variants are guaranteed to not call tzset() and were
already used in haproxy while chrooted, so they should be safe.

This patch must be backported to 1.6 and 1.5.
2016-03-17 05:30:03 +01:00
Benoit GARNIER
b413c2a759 BUG/MINOR: log: GMT offset not updated when entering/leaving DST
GMT offset used in local time formats was computed at startup, but was not updated when DST status changed while running.

For example these two RFC5424 syslog traces where emitted 5 seconds apart, just before and after DST changed:
  <14>1 2016-03-27T01:59:58+01:00 bunch-VirtualBox haproxy 2098 - - Connect ...
  <14>1 2016-03-27T03:00:03+01:00 bunch-VirtualBox haproxy 2098 - - Connect ...

It looked like they were emitted more than 1 hour apart, unlike with the fix:
  <14>1 2016-03-27T01:59:58+01:00 bunch-VirtualBox haproxy 3381 - - Connect ...
  <14>1 2016-03-27T03:00:03+02:00 bunch-VirtualBox haproxy 3381 - - Connect ...

This patch should be backported to 1.6 and partially to 1.5 (no fix needed in log.c).
2016-03-13 23:48:05 +01:00
Willy Tarreau
508a63fb96 MINOR: stats: add ST_SHOWADMIN to pass the admin info in the regular flags
It's easier to have a new flag in <flags> to indicate whether or not we
want to display the admin column in HTML dumps. We already have similar
flags to show the version or the legends.
2016-03-11 17:08:05 +01:00
Willy Tarreau
320ec2a745 BUG/MEDIUM: chunks: always reject negative-length chunks
The recent addition of "show env" on the CLI has revealed an interesting
design bug. Chunks are supposed to support a negative length to indicate
that they carry no data. chunk_printf() sets this size to -1 if the string
is too large for the buffer. At a few places in the http engine we may end
up with trash.len = -1. But bi_putchk(), chunk_appendf() and a few other
chunks consumers don't consider this case as possible and will use such a
chunk, possibly restoring an invalid string or trying to copy -1 bytes.

This fix takes care of clarifying the situation in a backportable way
where such sizes are used, so that a negative length indicating an error
remains present until the chunk is reinitialized or overwritten. But a
cleaner design adjustment needs to be done so that there's a clear contract
on how to use these chunks. At first glance it doesn't seem *that* useful
to support negative sizes, so probably this is what should change.

This fix must be backported to 1.6 and 1.5.
2016-02-25 16:24:14 +01:00
Thierry Fournier
70473a5f8c MINOR: common: mask conversion
Add function which converts network mask from bit length form
to struct in*_addr form.
2016-02-19 14:37:41 +01:00
Pieter Baauw
46af170e41 MINOR: mailers: increase default timeout to 10 seconds
This allows the tcp connection to send multiple SYN packets, so 1 lost
packet does not cause the mail to be lost. It changes the socket timeout
from 2 to 10 seconds, this allows for 3 syn packets to be send and
waiting a little for their reply.

This patch should be backported to 1.6.

Acked-by: Simon Horman <horms@verge.net.au>
2016-02-17 10:19:08 +01:00
Dragan Dosen
0edd10925d MINOR: standard: add function "escape_chunk"
This function tries to prefix all characters tagged in the <map> with the
<escape> character. The specified <chunk> contains the input to be
escaped.
2016-02-12 13:36:47 +01:00
Thierry Fournier
9312794ed7 MINOR: standard: add RFC HTTP date parser
This parser takes a string containing an HTTP date. It returns
a broken-down time struct. We must considers considers this
time as GMT. Maybe later the timezone will be taken in account.
2016-02-12 11:08:53 +01:00
Willy Tarreau
581bf81d34 MEDIUM: pools: add a new flag to avoid rounding pool size up
Usually it's desirable to merge similarly sized pools, which is the
reason why their size is rounded up to the next multiple of 16. But
for the buffers this is problematic because we add the size of
struct buffer to the user-requested size, and the rounding results
in 8 extra bytes that are usable in the end. So the user gets more
bytes than asked for, and in case of SSL it results in short writes
for the extra bytes that are sent above multiples of 16 kB.

So we add a new flag MEM_F_EXACT to request that the size is not
rounded up when creating the entry. Thus it doesn't disable merging.
2016-01-25 02:31:18 +01:00
Willy Tarreau
898529b4a8 MEDIUM: tools: add csv_enc_append() to preserve the original chunk
We have csv_enc() but there's no way to append some CSV-encoded data
to an existing chunk, so here we modify the existing function for this
and create an inlined version of csv_enc() which first resets the output
chunk. It will be handy to append data to an existing chunk without
having to use an extra temporary chunk, or to encode multiple strings
into a single chunk with chunk_newstr().

The patch is quite small, in fact most changes are typo fixes in the
comments.
2016-01-06 20:58:55 +01:00
Willy Tarreau
70af633ebe MINOR: chunk: make chunk_initstr() take a const string
chunk_initstr() prepares a read-only chunk from a string of
fixed length. Thus it must be prepared to accept a read-only
string on the input, otherwise the caller has to force-cast
some const char* and that's not a good idea.
2016-01-06 20:58:55 +01:00
Willy Tarreau
601360b41d MINOR: chunks: add chunk_strcat() and chunk_newstr()
These two new functions will make it easier to manipulate small strings
from within functions, because at many places, multiple short strings
are needed which do not deserve a malloc() nor a free(), and alloca()
is often discouraged. Since we already have trash chunks, it's convenient
to be able to allocate substrings from a chunk and use them later since
our functions already perform all the length checks. chunk_newstr() adds
a trailing zero at the end of a chunk and returns the pointer to the next
character, which can be used as an independant string. chunk_strcat()
does what it says.
2016-01-06 13:53:37 +01:00
Willy Tarreau
0b6044fa24 MINOR: chunks: ensure that chunk_strcpy() adds a trailing zero
Since thus function bears the name of a well-known string function, it
must at least promise compatible semantics. Here it means always adding
the trailing zero so that anyone willing to use chunk->str as a regular
string can do it. Of course the zero is not counted in the chunk's length.
2016-01-06 13:53:37 +01:00