Disabling idle-pool sharing can result in awful performance in presence
of a not so high number of threads, because the number of available idle
connections will be shared among threads, resulting in most of them
abandonning their connections after a request is done if there are already
enough total available. This is a case where pool-low-conn ought to be
used to preserve a number of connections for each thread, but this relation
isn't obvious as is. Let's add mentions about this with both keywords.
The DeviceAtlas Detection API now supports also the pcre2 library,
and some users wish to have exclusively this version in their
environment.
Also, there is no longer new development happening in the legacy
pcre(1) counterpart.
Simple check in the build process as the mutual exclusivity check between the
two are already taking care of early on. Moving the check to the part
only when we build haproxy + the API from source as the other case the API is
already built with the chosen regex library separately.
Released version 2.4-dev8 with the following main changes :
- BUILD: ssl: fix typo in HAVE_SSL_CTX_ADD_SERVER_CUSTOM_EXT macro
- BUILD: ssl: guard SSL_CTX_add_server_custom_ext with special macro
- BUG/MINOR: mux-h1: Don't emit extra CRLF for empty chunked messages
- MINOR: contrib/prometheus-exporter: use stats desc when possible followup
- MEDIUM: contrib/prometheus-exporter: export base stick table stats
- CLEANUP: assorted typo fixes in the code and comments
- CLEANUP: check: fix some typo in comments
- CLEANUP: tools: typo in `strl2irc` mention
- BUILD: ssl: guard SSL_CTX_set_msg_callback with SSL_CTRL_SET_MSG_CALLBACK macro
- MEDIUM: ssl: add a rwlock for SSL server session cache
- BUG/MINOR: intops: fix mul32hi()'s off-by-one
- BUG/MINOR: freq_ctr: fix a wrong delay calculation in next_event_delay()
- MINOR: stick-tables/counters: add http_fail_cnt and http_fail_rate data types
- MINOR: ssl: add SSL_SERVER_LOCK label in threads.h
- BUG/MINOR: mux-h1: Don't increment HTTP error counter for 408/500/501 errors
- BUG/MINOR: http-ana: Don't increment HTTP error counter on internal errors
- BUG/MEDIUM: mux-h1: Always set CS_FL_EOI for response in MSG_DONE state
- BUG/MINOR: mux-h1: Fix data skipping for bodyless responses
- BUG/MINOR: mux-h1: Don't blindly skip EOT block for non-chunked messages
- BUG/MEDIUM: mux-h2: Add EOT block when EOM flag is set on an empty HTX message
- MINOR: mux-h1: Be sure EOM flag is set when processing end of outgoing message
- REGTESTS: Add a script to test payload skipping for bodyless HTTP responses
- BUG/MINOR: server: re-align state file fields number
- CLEANUP: muxes: Remove useless calls to b_realign_if_empty()
- BUG/MINOR: tools: Fix a memory leak on error path in parse_dotted_uints()
- CLEANUP: remove unused variable assigned found by Coverity
- CLEANUP: queue: Remove useless tests on p or pp in pendconn_process_next_strm()
- BUG/MINOR: backend: hold correctly lock when killing idle conn
- MEDIUM: connection: protect idle conn lists with locks
- MEDIUM: connection: replace idle conn lists by eb trees
- MINOR: backend: search conn in idle/safe trees after available
- MINOR: backend: search conn in idle tree after safe on always reuse
- MINOR: connection: prepare hash calcul for server conns
- MINOR: connection: use the srv pointer for the srv conn hash
- MINOR: backend: compare conn hash for session conn reuse
- MINOR: connection: use sni as parameter for srv conn hash
- MINOR: reg-tests: test http-reuse with sni
- MINOR: backend: rewrite alloc of stream target address
- MINOR: connection: use dst addr as parameter for srv conn hash
- MINOR: reg-test: test http-reuse with specific dst addr
- MINOR: backend: rewrite alloc of connection src address
- MINOR: connection: use src addr as parameter for srv conn hash
- MINOR: connection: use proxy protocol as parameter for srv conn hash
- MINOR: reg-tests: test http-reuse with proxy protocol
- MINOR: doc: update http reuse for new eligilible connections
- BUG/MINOR: backend: fix compilation without ssl
- REGTESTS: adjust http_reuse_conn_hash requirements
- REGTESTS: deactivate a failed test on CI in http_reuse_conn_hash
- REGTESTS: fix sni used in http_reuse_conn_hash for libressl 3.3.0
- CI: cirrus: update FreeBSD image to 12.2
- MEDIUM: cli: add check-addr command
- MEDIUM: cli: add agent-port command
- MEDIUM: server: add server-states version 2
- MEDIUM: server: support {check,agent}_addr, agent_port in server state
- MINOR: server: enhance error precision when applying server state
- BUG/MINOR: server: Fix server-state-file-name directive
- CLEANUP: deinit: release global and per-proxy server-state variables on deinit
- BUG/MEDIUM: config: don't pick unset values from last defaults section
- BUG/MINOR: stats: revert the change on ST_CONVDONE
- BUG/MINOR: cfgparse: do not mention "addr:port" as supported on proxy lines
- BUG/MINOR: http-htx: defpx must be a const in proxy_dup_default_conf_errors()
- BUG/MINOR: tcpheck: the source list must be a const in dup_tcpcheck_var()
- BUILD: proxy: add missing compression-t.h to proxy-t.h
- REORG: move init_default_instance() to proxy.c and pass it the defproxy pointer
- REORG: proxy: centralize the proxy allocation code into alloc_new_proxy()
- MEDIUM: proxy: only take defaults when a default proxy is passed.
- MINOR: proxy: move the defproxy freeing code to proxy.c
- MINOR: proxy: always properly reset the just freed default instance pointers
- BUG/MINOR: extcheck: proxy_parse_extcheck() must take a const for the defproxy
- BUG/MINOR: tcpcheck: proxy_parse_*check*() must take a const for the defproxy
- BUG/MINOR: server: parse_server() must take a const for the defproxy
- MINOR: cfgparse: move defproxy to cfgparse-listen as a static
- MINOR: proxy: add a new capability PR_CAP_DEF
- MINOR: cfgparse: check PR_CAP_DEF instead of comparing poiner against defproxy
- MINOR: cfgparse: use a pointer to the current default proxy
- MINOR: proxy: also store the name for a defaults section
- MINOR: proxy: support storing defaults sections into their own tree
- MEDIUM: proxy: store the default proxies in a tree by name
- MEDIUM: cfgparse: allow a proxy to designate the defaults section to use
- MINOR: http: add baseq sample fetch
- CLEANUP: tcpcheck: Remove a useless test on port variable
- BUG/MINOR: server: Don't call fopen() with server-state filepath set to NULL
- CLEANUP: server: Remove useless "filepath" variable in apply_server_state()
- MINOR: peers/cli: do not dump the peers dictionaries by default on "show peers"
- MINOR: cfgparse: implement a simple if/elif/else/endif macro block handler
- DOC: tune: explain the origin of block size for ssl.cachesize
- MINOR: tcp: add support for defer-accept on FreeBSD.
- MINOR: ring: adds new ring_init function.
- CLEANUP: channel: fix comment in ci_putblk.
- BUG/MINOR: dns: add missing sent counter and parent id to dns counters.
- BUG/MINOR: resolvers: fix attribute packed struct for dns
- MINOR: resolvers: renames some resolvers internal types and removes dns prefix
- MINOR: resolvers: renames type dns_resolvers to resolvers.
- MINOR: resolvers: renames some resolvers specific types to not use dns prefix
- MINOR: resolvers: renames some dns prefixed types using resolv prefix.
- MINOR: resolvers: renames resolvers DNS_RESP_* errcodes RSLV_RESP_*
- MINOR: resolvers: renames resolvers DNS_UPD_* returncodes to RSLV_UPD_*
- MINOR: resolvers: rework prototype suffixes to split resolving and dns.
- MEDIUM: resolvers: move resolvers section parsing from cfgparse.c to dns.c
- MINOR: resolvers: replace nameserver's resolver ref by generic parent pointer
- MINOR: resolvers: rework dns stats prototype because specific to resolvers
- MEDIUM: resolvers: split resolving and dns message exchange layers.
- MEDIUM: resolvers/dns: split dns.c into dns.c and resolvers.c
- MEDIUM: dns: adds code to support pipelined DNS requests over TCP.
- MEDIUM: resolvers: add supports of TCP nameservers in resolvers.
This patch introduce the new line "server" to set a TCP
nameserver in a "resolvers" section:
server <name> <address> [param*]
Used to configure a DNS TCP or stream server. This supports for all
"server" parameters found in 5.2 paragraph. Some of these parameters
are irrelevant for DNS resolving. Note: currently 4 queries are pipelined
on the same connections. A batch of idle connections are removed every
5 seconds. "maxconn" can be configured to limit the amount of those
concurrent connections and TLS should also usable if the server supports
. The current implementation limits to 4 pipelined
The name of the line in configuration is open to discussion
and could be changed before the next release.
A user could eventually ask himself where those 200 bytes block size are
coming from. This patch tries to better explain the origin in case
people are curious or want to double check the reality.
Signed-off-by: William Dauchy <wdauchy@gmail.com>
Very often, especially since reg-tests, it would be desirable to be able
to conditionally comment out a config block, such as removing an SSL
binding when SSL is disabled, or enabling HTX only for certain versions,
etc.
This patch introduces a very simple nested block management which takes
".if", ".elif", ".else" and ".endif" directives to take or ignore a block.
For now the conditions are limited to empty string or "0" for false versus
a non-nul integer for true, which already suffices to test environment
variables. Still, it needs to be a bit more advanced with defines, versions
etc.
A set of ".notice", ".warning" and ".alert" statements are provided to
emit messages, often in order to provide advice about how to fix certain
conditions.
The "show peers" output has become huge due to the dictionaries making it
less readable. Now this feature has reached a certain level of maturity
which doesn't warrant to dump it all the time, given that it was essentially
needed by developers. Let's make it optional, and disabled by default, only
when "show peers dict" is requested. The default output reminds about the
command. The output has been divided by 5 :
$ socat - /tmp/sock1 <<< "show peers dict" | wc -l
125
$ socat - /tmp/sock1 <<< "show peers" | wc -l
26
It could be useful to backport this to recent stable versions.
Now it becomes possible to specify "from foo" on a frontend/listen/backend
or even on a "defaults" line, to mention that defaults section "foo" needs
to be used to preset the proxy's settings.
When not set, the last section remains used. In case the designated name
is found at multiple places, it is rejected and an error indicates two
occurrences of the same name. Similarly, if the section name is found,
its name must only use valid characters. This allows multiple named
defaults section to continue to coexist without the risk that they will
cause trouble by accident.
When it comes to "defaults" relying on another defaults, what happens is
just that a new defaults section is created from the designated one. This
will make it possible for example to reuse some settings such as log-format
like below:
defaults tcp-clear
log stdout local0 info
log-format "%ci:%cp/%b/%si:%sp %ST %ts %U/%B %{+Q}r"
defaults tcp-ssl
log stdout local0 info
log-format "%ci:%cp/%b/%si:%sp %ST %ts %U/%B %{+Q}r ssl=%sslv"
defaults http-clear from tcp-clear
mode http
defaults http-ssl from tcp-ssl
mode http
frontend fe1 from http-clear
bind :8001
frontend fe2 from http-ssl
bind :8002
A small corner case remains in the error detection, if a second defaults
section appears with the same name after the point where it was used, and
nobody references it, the duplicate will not be detected. This could be
addressed by performing the syntactic checks in check_config_validity(),
and by postponing the freeing of the defaults, after tagging a defaults
section as explicitly looked up by another section. This doesn't seem
that important at the moment though.
Since the beginning, this directive is documented to accept an optional file
name. But it should also be possible to use it without any argument to use
the backend name as file name. However, when no argument is provided, an
error is reported during the configuration parsing requesting an argument, a
file name or "use-backend-name". And This last special argument is not
documented.
So, to respect the documentation and to avoid configuration breakages, all
modes are now supported. If this directive is called with no argument or
with "use-backend-name", the backend name is use as file name for the
server-state file. Otherwise, the provided string is used.
In addition, we take care to release any previously allocated file name in
case this directive is defines multiple times in the same backend. And an
error is reported if more than one argument are defined. Finally, the
documentation is updated accordingly. Sections supporting this directive are
also mentioned.
This patch should be backported as far as 1.6.
logical followup from cli commands addition, so that the state server
file stays compatible with the changes made at runtime; use previously
added helper to load server attributes.
also alloc a specific chunk to avoid mixing with other called functions
using it
Signed-off-by: William Dauchy <wdauchy@gmail.com>
this patch allows to set agent port at runtime. In order to align with
both `addr` and `check-addr` commands, also add the possibility to
optionnaly set port on `agent-addr` command. This led to a small
refactor in order to use the same function for both `agent-addr` and
`agent-port` commands.
Signed-off-by: William Dauchy <wdauchy@gmail.com>
this patch allows to set server health check address at runtime. In
order to align with `addr` command, also allow to set port optionnaly.
This led to a small refactor in order to use the same function for both
`check-addr` and `check-port` commands.
for `check-port`, we however don't permit the change anymore if checks
are not enabled on the server.
This command becomes more and more useful for people having a consul
like architecture:
- the backend server is located on a container with its own IP
- the health checks are done the consul instance located on the host
with the host IP
Signed-off-by: William Dauchy <wdauchy@gmail.com>
Historically we've been counting lots of client-triggered events in stick
tables to help detect misbehaving ones, but we've been missing the same on
the server side, and there's been repeated requests for being able to count
the server errors per URL in order to precisely monitor the quality of
service or even to avoid routing requests to certain dead services, which
is also called "circuit breaking" nowadays.
This commit introduces http_fail_cnt and http_fail_rate, which work like
http_err_cnt and http_err_rate in that they respectively count events and
their frequency, but they only consider server-side issues such as network
errors, unparsable and truncated responses, and 5xx status codes other
than 501 and 505 (since these ones are usually triggered by the client).
Note that retryable errors are purposely not accounted for, so that only
what the client really sees is considered.
With this it becomes very simple to put some protective measures in place
to perform a redirect or return an excuse page when the error rate goes
beyond a certain threshold for a given URL, and give more chances to the
server to recover from this condition. Typically it could look like this
to bypass a URL causing more than 10 requests per second:
stick-table type string len 80 size 4k expire 1m store http_fail_rate(1m)
http-request track-sc0 base # track host+path, ignore query string
http-request return status 503 content-type text/html \
lf-file excuse.html if { sc0_http_fail_rate gt 10 }
A more advanced mechanism using gpt0 could even implement high/low rates
to disable/enable the service.
Reg-test converteers_ref_cnt_never_dec.vtc was updated to test it.
Released version 2.4-dev7 with the following main changes :
- BUG/MINOR: stats: Continue to fill frontend stats on unimplemented metric
- BUILD: ssl: guard Client Hello callbacks with HAVE_SSL_CLIENT_HELLO_CB macro instead of openssl version
- BUG/MINOR: stats: Init the metric variable when frontend stats are filled
- MINOR: contrib/prometheus-exporter: better output of Not-a-Number
- CLEANUP: stats: improve field selection for frontend http fields
- CLEANUP: assorted typo fixes in the code and comments
- DOC: Improve documentation of the various hdr() fetches
- MEDIUM: stats: allow to select one field in `stats_fill_be_stats`
- MINOR: contrib/prometheus-exporter: use fill_be_stats for backend dump
- MEDIUM: stats: allow to select one field in `stats_fill_sv_stats`
- MINOR: contrib/prometheus-exporter: use fill_sv_stats for server dump
- MINOR: abort() on my_unreachable() when DEBUG_USE_ABORT is set.
- BUG/MEDIUM: filters/htx: Fix data forwarding when payload length is unknown
- BUG/MINOR: config: fix leak on proxy.conn_src.bind_hdr_name
- MINOR: reg-tests: add http-reuse test
- CLEANUP: srv: fix comment for pool-max-conn
- CLEANUP: backend: remove an obsolete comment on conn_backend_get
- REORG: backend: simplify conn_backend_get
- MINOR: ssl: Server ssl context prepare function refactoring
- MINOR: ssl: Certificate chain loading refactorization
- MEDIUM: ssl: Load client certificates in a ckch for backend servers
- MEDIUM: ssl: Enable backend certificate hot update
- MINOR: ssl: Remove client_crt member of the server's ssl context
- CLEANUP: ssl/cli: rework free in cli_io_handler_commit_cert()
- CLEANUP: ssl: remove SSL_CTX function parameter
- CLEANUP: ssl: make load_srv_{ckchs,cert} match their bind counterpart
- BUILD: Include stdlib.h in compiler.h if DEBUG_USE_ABORT is set
- CI: Fix DEBUG_STRICT definition for Coverity
- BUG/MINOR: stats: Remove a break preventing ST_F_QCUR to be set for servers
- BUG/MINOR: stats: Add a break after filling ST_F_MODE field for servers
- CLEANUP: ssl: remove dead code in ckch_inst_new_load_srv_store()
- BUG/MINOR: ssl: init tmp chunk correctly in ssl_sock_load_sctl_from_file()
- BUG/MEDIUM: session: only retrieve ready idle conn from session
- BUG/MEDIUM: backend: never reuse a connection for tcp mode
- REGTESTS: set_ssl_server_cert.vtc: remove the abort command
- REGTESTS: set_ssl_server_cert.vtc: check the Sha1 Fingerprint
- REGTESTS: set_ssl_server_cert.vtc: check the sha1 from the server
- MEDIUM: stream-int: Take care of EOS if the SI wake callback function
- MINOR: mux-h1: Try to wake up data layer first before calling its wake callback
- MINOR: mux-h1: Wake up H1C after its creation if input buffer is not empty
- MEDIUM: mux-h1: Add ST_READY state for the H1 connections
- MINOR: stream: Add a function to validate TCP to H1 upgrades
- MEDIUM: http-ana: Do nothing in wait-for-request analyzer if not htx
- BUG/MEDIUM: stream: Don't immediatly ack the TCP to H1 upgrades
- BUG/MAJOR: mux-h1: Properly handle TCP to H1 upgrades
- MINOR: htx/http-ana: Save info about Upgrade option in the Connection header
- MEDIUM: http-ana: Refuse invalid 101-switching-protocols responses
- BUG/MINOR: h2/mux-h2: Reject 101 responses with a PROTOCOL_ERROR h2s error
- MINOR: mux-h1/mux-fcgi: Don't set TUNNEL mode if payload length is unknown
- MINOR: mux-h1: Split H1C_F_WAIT_OPPOSITE flag to separate input/output sides
- MINOR: mux-h2: Add 2 flags to help to properly handle tunnel mode
- MEDIUM: mux-h2: Block client data on server side waiting tunnel establishment
- MEDIUM: mux-h2: Close streams when processing data for an aborted tunnel
- MEDIUM: mux-h1: Properly handle tunnel establishments and aborts
- BUG/MAJOR: mux-h1/mux-h2/htx: Fix HTTP tunnel management at the mux level
- MINOR: htx: Rename HTX_FL_EOI flag into HTX_FL_EOM
- REGTESTS: Don't run http_msg_full_on_eom script on the 2.4 anymore
- MINOR: htx: Add a function to know if a block is the only one in a message
- MAJOR: htx: Remove the EOM block type and use HTX_FL_EOM instead
- MINOR: mux-h1: Add a flag on H1 streams with a response known to be bodyless
- MEDIUM: mux-h1: Don't emit any payload for bodyless responses
- MINOR: mux-h1: Don't emit C-L and T-E headers for 204 and 1xx responses
- MINOR: mux-h1: Don't add Connection close/keep-alive header for 1xx messages
- MINOR: h2/mux-h2: Add flags to notify the response is known to have no body
- MEDIUM: mux-h2: Don't emit DATA frame for bodyless responses
- MEDIUM: http-ana: Deal with L7 retries in HTTP analysers
- MINOR: h1: reject websocket handshake if missing key
- MEDIUM: h1: generate WebSocket key on response if needed
- MINOR: mux_h2: define H2_SF_EXT_CONNECT_SENT stream flag
- MEDIUM: h2: parse Extended CONNECT reponse to htx
- MEDIUM: mux_h2: generate Extended CONNECT from htx upgrade
- MEDIUM: h1: add a WebSocket key on handshake if needed
- MEDIUM: mux_h2: generate Extended CONNECT response
- MEDIUM: h2: parse Extended CONNECT request to htx
- MEDIUM: h2: send connect protocol h2 settings
- MINOR: vtc: add test for h1/h2 protocol upgrade translation
- MINOR: vtc: add websocket test
- REGTESTS: Fix required versions for several scripts
- REGTEST: Don't use the websocket to validate http-check
- MINOR: mux-h1/trace: add traces at level ERROR for all kind of errors
- MINOR: mux-fcgi/trace: add traces at level ERROR for all kind of errors
- MINOR: h1: Raise the chunk size limit up to (2^52 - 1)
- BUG/MEDIUM: listener: do not accept connections faster than we can process them
- REGTESTS: set_ssl_server_cert.vtc: set as broken
- Revert "BUG/MEDIUM: listener: do not accept connections faster than we can process them"
- BUG/MINOR: backend: check available list allocation for reuse
- CI: Fix the coverity builds
- DOC: management: fix "show resolvers" alphabetical ordering
- MINOR: tools: add print_time_short() to print a condensed duration value
- MINOR: activity: make profiling more manageable
- MINOR: activity: declare a new structure to collect per-function activity
- MEDIUM: tasks/activity: collect per-task statistics when profiling is enabled
- MINOR: activity: also report collected tasks stats in "show profiling"
- MINOR: activity: flush scheduler stats on "set profiling tasks on"
- MINOR: activity: add a new "show tasks" command to list currently active tasks
- MINOR: listener: export accept_queue_process
- MINOR: session: export session_expire_embryonic()
- MINOR: muxes: export the timeout and shutr task handlers
- MINOR: checks: export a few functions that appear often in trace dumps
- MINOR: peers: export process_peer_sync() to improve traces
- MINOR: stick-tables: export process_table_expire()
- MINOR: mux-h1: Remove first useless test on count in h1_process_output()
- BUG/MINOR: stick-table: Always call smp_fetch_src() with a valid arg list
- MINOR: http-fetch: Don't check if argument list is set in sample fetches
- MINOR: http-conv: Don't check if argument list is set in sample converters
- MINOR: sample: Don't check if argument list is set in sample fetches
- MINOR: ssl-sample: Don't check if argument list is set in sample fetches
- MINOR: mux-h2: Don't tests the start-line when sending HEADERS frame
- MINOR: mux-h2: Slightly improve request HEADERS frames sending
- MINOR: contrib/prometheus-exporter: declare states for objects
- MAJOR: contrib/prometheus-exporter: move ftd/bkd/srv states to labels
- MEDIUM: contrib/prometheus-exporter: Use dynamic labels instead of static ones
- MINOR: listener: export manage_global_listener_queue()
- BUG/MINOR: activity: take care of late wakeups in "show tasks"
- REGTESTS: set_ssl_server_cert.vtc: remove SSL caching and set as working
- REGTESTS: set_ssl_server_cert: cleanup the SSL caching option
- MINOR: checks: Add function to get the result code corresponding to a status
- MAJOR: contrib/prometheus-exporter: move health check status to labels
- MINOR: contrib/prometheus-exporter: improve service status description field
- MINOR: stats: improve pending connections description
- MINOR: stats: improve max stats descriptions
- MINOR: contrib/prometheus-exporter: use stats desc when possible
- MINOR: contrib/prometheus-exporter: add uweight field
- MINOR: contrib/prometheus-exporter: add recv logs_logs_total field
- CLEANUP: contrib/prometheus-exporter: remove unused includes
- CLEANUP: contrib/prometheus-exporter: align and reorder fields
- CLEANUP: contrib/prometheus-exporter: remove description in README
- DOC: contrib/prometheus-exporter: Add missing metrics in README
- BUG/MINOR: contrib/prometheus-exporter: Add missing label for ST_F_HRSP_1XX
- BUG/MINOR: contrib/prometheus-exporter: Restart labels dump at the right pos
- BUG/MEDIUM: ssl/cli: abort ssl cert is freeing the old store
- BUG/MEDIUM: ssl: check a connection's status before computing a handshake
- BUG/MINOR: mux_h2: fix incorrect stat titles
- MINOR: ssl/cli: flush the server session cache upon 'commit ssl cert'
- BUG/MINOR: cli: fix set server addr/port coherency with health checks
- MINOR: server: Don't set the check port during the update from a state file
- MINOR: dns: Don't set the check port during a server dns resolution
- MEDIUM: check: remove checkport checkaddr flag
- MEDIUM: server: adding support for check_port in server state
- BUG/MINOR: check: consitent way to set agentaddr
- MEDIUM: check: align agentaddr and agentport behaviour
- DOC: server: Add missing params in comment of the server state line parsing
- BUG/MINOR: xxhash: make sure armv6 uses memcpy()
- REGTESTS: mark http-check-send.vtc as 2.4-only
- REGTESTS: mark sample_fetches/hashes.vtc as 2.4-only
- BUG/MINOR: ssl: do not try to use early data if not configured
- REGTESTS: unbreak http-check-send.vtc
- MINOR: cli/show_fd: report local and report ports when known
- BUILD: Makefile: move REGTESTST_TYPE default setting
- BUG/MEDIUM: mux-h2: handle remaining read0 cases
- CLEANUP: http-htx: Set buffer area to NULL instead of malloc(0)
- BUG/MINOR: sock: Unclosed fd in case of connection allocation failure
- BUG/MEDIUM: mux-h2: do not quit the demux loop before setting END_REACHED
in the same manner of agentaddr, we now:
- permit to set agentport through `port` keyword, like it is the case
for agentaddr through `addr`
- set the priority on `agent-port` keyword when used
- add a flag to be able to test when the value is set like for agentaddr
it makes the behaviour between `addr` and `port` more consistent.
Signed-off-by: William Dauchy <wdauchy@gmail.com>
We can currently change the check-port using the cli command `set server
check-port` but there is a consistency issue when using server state.
This patch aims to fix this problem but will be also a good preparation
work to get rid of checkport flag, so we are able to know when checkport
was set by config.
I am fully aware this is not making github #953 moving forward, I
however think this might be acceptable while waiting for a proper
solution and resolve consistency problem faced with port settings.
Signed-off-by: William Dauchy <wdauchy@gmail.com>
This finally adds the long-awaited solution to inspect the run queues
and figure what is eating the CPU or causing latencies. We can even see
the experienced latencies when profiling is enabled. Example on a
saturated process:
> show tasks
Running tasks: 14983 (4 threads)
function places % lat_tot lat_avg
process_stream 4948 33.0 5.840m 70.82ms
h1_io_cb 2535 16.9 - -
main+0x9e670 2508 16.7 2.930m 70.10ms
ssl_sock_io_cb 2499 16.6 - -
si_cs_io_cb 2493 16.6 - -
If a user enables profiling by hand, it makes sense to reset the stats
counters to provide fresh new measurements. Therefore it's worth using
this as the standard method to reset counters.
"show profiling" will now dump the stats collected by the scheduler if
profiling was previously enabled. This will immediately make it obvious
what functions are responsible for others' high latencies or which ones
are suffering from others, and should help spot issues like undesired
wakeups.
Example:
Per-task CPU profiling : on # set profiling tasks {on|auto|off}
Tasks activity:
function calls cpu_tot cpu_avg lat_tot lat_avg
si_cs_io_cb 5569479 23.37s 4.196us - -
h1_io_cb 5558654 13.60s 2.446us - -
process_stream 250841 1.476s 5.882us 3.499s 13.95us
main+0x9e670 198 - - 5.526ms 27.91us
task_run_applet 17 1.509ms 88.77us 205.8us 12.11us
srv_cleanup_idle_connections 12 44.51us 3.708us 25.71us 2.142us
main+0x158c80 9 48.72us 5.413us - -
srv_cleanup_toremove_connections 5 165.1us 33.02us 123.6us 24.72us
The EOM block may be removed. The HTX_FL_EOM flags is enough. Most of time,
to know if the end of the message is reached, we just need to have an empty
HTX message with HTX_FL_EOM flag set. It may also be detected when the last
block of a message with HTX_FL_EOM flag is manipulated.
Removing EOM blocks simplifies the HTX message filling. Indeed, there is no
more edge problems when the message ends but there is no more space to write
the EOM block. However, some part are more tricky. Especially the
compression filter or the FCGI mux. The compression filter must finish the
compression on the last DATA block. Before it was performed on the EOM
block, an extra DATA block with the checksum was added. Now, we must detect
the last DATA block to be sure to finish the compression. The FCGI mux on
its part must be sure to reserve the space for the empty STDIN record on the
last DATA block while this record was inserted on the EOM block.
The H2 multiplexer is probably the part that benefits the most from this
change. Indeed, it is now fairly easier to known when to set the ES flag.
The HTX documentaion has been updated accordingly.
GitHub issue #796 notes that many administrators miss the fact that the `hdr()`
fetch (without the `f`) splits the header value at commas. This is only
mentioned at the end of a long paragraph.
This patch attempts to improve the documentation by:
- Explaning the "comma issue" as early as possible.
- Adding newlines to split the explanation into distinct sections.
- Reducing duplication by making the `res` siblings refer to their `req`
counterparts.
This patch may be backported as long as it applies cleanly. During the
refactoring I needed to adjust several explanations for consistency and not all
of them might be available in older branches.
Released version 2.4-dev6 with the following main changes :
- MINOR: converter: adding support for url_enc
- BUILD: SSL: guard TLS13 ciphersuites with HAVE_SSL_CTX_SET_CIPHERSUITES
- BUILD: ssl: guard EVP_PKEY_get_default_digest_nid with ASN1_PKEY_CTRL_DEFAULT_MD_NID
- BUILD: ssl: guard openssl specific with SSL_READ_EARLY_DATA_SUCCESS
- BUILD: Makefile: exclude broken tests by default
- CLEANUP: cfgparse: replace "realloc" with "my_realloc2" to fix to memory leak on error
- BUG/MINOR: hlua: Fix memory leak in hlua_alloc
- MINOR: contrib/prometheus-exporter: export build_info
- DOC: fix some spelling issues over multiple files
- CLEANUP: Fix spelling errors in comments
- SCRIPTS: announce-release: fix typo in help message
- CI: github: add a few more words to the codespell ignore list
- DOC: Add maintainers for the Prometheus exporter
- BUG/MINOR: sample: fix concat() converter's corruption with non-string variables
- BUG/MINOR: server: Memory leak of proxy.used_server_addr during deinit
- CLEANUP: sample: remove uneeded check in json validation
- MINOR: reg-tests: add a way to add service dependency
- BUG/MINOR: sample: check alloc_trash_chunk return value in concat()
- BUG/MINOR: reg-tests: fix service dependency script
- MINOR: reg-tests: add base prometheus test
- Revert "BUG/MINOR: dns: SRV records ignores duplicated AR records"
- BUG/MINOR: sample: Memory leak of sample_expr structure in case of error
- BUG/MINOR: check: Don't perform any check on servers defined in a frontend
- BUG/MINOR: init: enforce strict-limits when using master-worker
- MINOR: contrib/prometheus-exporter: avoid connection close header
- MINOR: contrib/prometheus-exporter: use fill_info for process dump
- BUG/MINOR: init: Use a dynamic buffer to set HAPROXY_CFGFILES env variable
- MINOR: config: Add failifnotcap() to emit an alert on proxy capabilities
- MINOR: server: Forbid server definitions in frontend sections
- BUG/MINOR: threads: Fixes the number of possible cpus report for Mac.
- CLEANUP: pattern: rename pat_ref_commit() to pat_ref_commit_elt()
- MINOR: pattern: add the missing generation ID manipulation functions
- MINOR: peers: Add traces for peer control messages.
- BUG/MINOR: dns: SRV records ignores duplicated AR records (v2)
- BUILD: peers: fix build warning about unused variable
- BUG/MEDIUM: stats: add missing INF_BUILD_INFO definition
- MINOR: cache: Do not store responses with an unknown encoding
- BUG/MINOR: peers: Possible appctx pointer dereference.
- MINOR: build: discard echoing in help target
- MINOR: cache: Remove the `hash` part of the accept-encoding secondary key
- CLEANUP: cache: Use proper data types in secondary_key_cmp()
- CLEANUP: Rename accept_encoding_hash_cmp to accept_encoding_bitmap_cmp
- BUG/MINOR: peers: Wrong "new_conn" value for "show peers" CLI command.
- MINOR: contrib: Make the wireshark peers dissector compile for more distribs.
- BUG/MINOR: mux_h2: missing space between "st" and ".flg" in the "show fd" helper
- CLEANUP: tools: make resolve_sym_name() take a const pointer
- CLEANUP: cli: make "show fd" use a const connection to access other fields
- MINOR: cli: make "show fd" also report the xprt and xprt_ctx
- MINOR: xprt: add a new show_fd() helper to complete some "show fd" dumps.
- MINOR: ssl: provide a "show fd" helper to report important SSL information
- MINOR: xprt/mux: export all *_io_cb functions so that "show fd" resolves them
- MINOR: mux-h2: make the "show fd" helper also decode the h2s subscriber when known
- MINOR: mux-h1: make the "show fd" helper also decode the h1s subscriber when known
- MINOR: mux-fcgi: make the "show fd" helper also decode the fstrm subscriber when known
- CI: Pin VTest to a known good commit
- MINOR: cli: give the show_fd helpers the ability to report a suspicious entry
- MINOR: cli/show_fd: report some easily detectable suspicious states
- MINOR: ssl/show_fd: report some FDs as suspicious when possible
- MINOR: mux-h2/show_fd: report as suspicious an entry with too many calls
- MINOR: mux-h1/show_fd: report as suspicious an entry with too many calls
- BUG/MINOR: mworker: define _GNU_SOURCE for strsignal()
- BUG/MEDIUM: tcpcheck: Don't destroy connection in the wake callback context
- BUG/MEDIUM: mux-h2: Xfer rxbuf to the upper layer when creating a front stream
- MINOR: http: Add HTTP 501-not-implemented error message
- MINOR: muxes: Add exit status for errors about not implemented features
- MINOR: mux-h1: Be prepared to return 501-not-implemented error during parsing
- MEDIUM: mux-h1: Return a 501-not-implemented for upgrade requests with a body
- DOC: Remove space after comma in converter signature
- DOC: Rename '<var name>' to '<var>' in converter signature
- MINOR: stats: duplicate 3 fields in bytes in info
- MINOR: stats: add new start time field
- MINOR: contrib/prometheus-exporter: merge info description from stats
- MEDIUM: stats: allow to select one field in `stats_fill_fe_stats`
- MINOR: contrib/prometheus-exporter: use fill_fe_stats for frontend dump
- MINOR: contrib/prometheus-exporter: Don't needlessly set empty label for metrics
- MINOR: contrib/prometheus-exporter: Split the PROMEX_FL_STATS_METRIC flag
- MINOR: contrib/prometheus-exporter: Add promex_metric struct defining a metric
- MEDIUM: contrib/prometheus-exporter: Rework matrices defining Promex metrics
- BUG/MINOR: stream: Don't update counters when TCP to H2 upgrades are performed
- BUG/MEDIUM: mux-h2: fix read0 handling on partial frames
- MINOR: debug: always export the my_backtrace function
- MINOR: debug: extract the backtrace dumping code to its own function
- MINOR: debug: create ha_backtrace_to_stderr() to dump an instant backtrace
- MEDIUM: debug: now always print a backtrace on CRASH_NOW() and friends
- MINOR: debug: let ha_dump_backtrace() dump a bit further for some callers
- BUILD: debug: fix build warning by consuming the write() result
- MINOR: lua: remove unused variable
- BUILD/MINOR: lua: define _GNU_SOURCE for LLONG_MAX
Add the support for the 501-not-implemented status code with the
corresponding default message. The documentation is updated accordingly
because it is now part of status codes HAProxy may emit via an errorfile or
a deny/return HTTP action.
Now the show_fd helpers at the transport and mux levels return an integer
which indicates whether or not the inspected entry looks suspicious. When
an entry is reported as suspicious, "show fd" will suffix it with an
exclamation mark ('!') in the dump, that is supposed to help detecting
them.
For now, helpers were adjusted to adapt to the new API but none of them
reports any suspicious entry yet.
If a server varies on the accept-encoding header and it sends a response
with an encoding we do not know (see parse_encoding_value function), we
will not store it. This will prevent unexpected errors caused by
cache collisions that could happen in accept_encoding_hash_cmp.
add base support for url encode following RFC3986, supporting `query`
type only.
- add test checking url_enc/url_dec/url_enc
- update documentation
- leave the door open for future changes
this should resolve github issue #941
Signed-off-by: William Dauchy <wdauchy@gmail.com>
Released version 2.4-dev5 with the following main changes :
- BUG/MEDIUM: mux_h2: Add missing braces in h2_snd_buf()around trace+wakeup
- BUILD: hpack: hpack-tbl-t.h uses VAR_ARRAY but does not include compiler.h
- MINOR: time: increase the minimum wakeup interval to 60s
- MINOR: check: do not ignore a connection header for http-check send
- REGTESTS: complete http-check test
- CI: travis-ci: drop coverity scan builds
- MINOR: atomic: don't use ; to separate instruction on aarch64.
- IMPORT: xxhash: update to v0.8.0 that introduces stable XXH3 variant
- MEDIUM: xxhash: use the XXH3 functions to generate 64-bit hashes
- MEDIUM: xxhash: use the XXH_INLINE_ALL macro to inline all functions
- CLEANUP: xxhash: remove the unused src/xxhash.c
- MINOR: sample: add the xxh3 converter
- REGTESTS: add tests for the xxh3 converter
- MINOR: protocol: Create proto_quic QUIC protocol layer.
- MINOR: connection: Attach a "quic_conn" struct to "connection" struct.
- MINOR: quic: Redefine control layer callbacks which are QUIC specific.
- MINOR: ssl_sock: Initialize BIO and SSL objects outside of ssl_sock_init()
- MINOR: connection: Add a new xprt to connection.
- MINOR: ssl: Export definitions required by QUIC.
- MINOR: cfgparse: Do not modify the QUIC xprt when parsing "ssl".
- MINOR: tools: Add support for QUIC addresses parsing.
- MINOR: quic: Add definitions for QUIC protocol.
- MINOR: quic: Import C source code files for QUIC protocol.
- MINOR: listener: Add QUIC info to listeners and receivers.
- MINOR: server: Add QUIC definitions to servers.
- MINOR: ssl: SSL CTX initialization modifications for QUIC.
- MINOR: ssl: QUIC transport parameters parsing.
- MINOR: quic: QUIC socket management finalization.
- MINOR: cfgparse: QUIC default server transport parameters init.
- MINOR: quic: Enable the compilation of QUIC modules.
- MAJOR: quic: Make usage of ebtrees to store QUIC ACK ranges.
- MINOR: quic: Attempt to make trace more readable
- MINOR: quic: Make usage of the congestion control window.
- MINOR: quic: Flag RX packet as ack-eliciting from the generic parser.
- MINOR: quic: Code reordering to help in reviewing/modifying.
- MINOR: quic: Add traces to congestion avoidance NewReno callback.
- MINOR: quic: Display the SSL alert in ->ssl_send_alert() callback.
- MINOR: quic: Update the initial salt to that of draft-29.
- MINOR: quic: Add traces for in flght ack-eliciting packet counter.
- MINOR: quic: make a packet build fails when qc_build_frm() fails.
- MINOR: quic: Add traces for quic_packet_encrypt().
- MINOR: cache: Refactoring of secondary_key building functions
- MINOR: cache: Avoid storing responses whose secondary key was not correctly calculated
- BUG/MINOR: cache: Manage multiple headers in accept-encoding normalization
- MINOR: cache: Add specific secondary key comparison mechanism
- MINOR: http: Add helper functions to trim spaces and tabs
- MEDIUM: cache: Manage a subset of encodings in accept-encoding normalizer
- REGTESTS: cache: Simplify vary.vtc file
- REGTESTS: cache: Add a specific test for the accept-encoding normalizer
- MINOR: cache: Remove redundant test in http_action_req_cache_use
- MINOR: cache: Replace the "process-vary" option's expected values
- CI: GitHub Actions: enable daily Coverity scan
- BUG/MEDIUM: cache: Fix hash collision in `accept-encoding` handling for `Vary`
- MEDIUM: stick-tables: Add srvkey option to stick-table
- REGTESTS: add test for stickiness using "srvkey addr"
- BUILD: Makefile: disable -Warray-bounds until it's fixed in gcc 11
- BUG/MINOR: sink: Return an allocation failure in __sink_new if strdup() fails
- BUG/MINOR: lua: Fix memory leak error cases in hlua_config_prepend_path
- MINOR: lua: Use consistent error message 'memory allocation failed'
- CLEANUP: Compare the return value of `XXXcmp()` functions with zero
- CLEANUP: Apply the coccinelle patch for `XXXcmp()` on include/
- CLEANUP: Apply the coccinelle patch for `XXXcmp()` on contrib/
- MINOR: qpack: Add static header table definitions for QPACK.
- CLEANUP: qpack: Wrong comment about the draft for QPACK static header table.
- CLEANUP: quic: Remove useless QUIC event trace definitions.
- BUG/MINOR: quic: Possible CRYPTO frame building errors.
- MINOR: quic: Pass quic_conn struct to frame parsers.
- BUG/MINOR: quic: Wrong STREAM frames parsing.
- MINOR: quic: Drop packets with STREAM frames with wrong direction.
- CLEANUP: ssl: Remove useless loop in tlskeys_list_get_next()
- CLEANUP: ssl: Remove useless local variable in tlskeys_list_get_next()
- MINOR: ssl: make tlskeys_list_get_next() take a list element
- Revert "BUILD: Makefile: disable -Warray-bounds until it's fixed in gcc 11"
- BUG/MINOR: cfgparse: Fail if the strdup() for `rule->be.name` for `use_backend` fails
- CLEANUP: mworker: remove duplicate pointer tests in cfg_parse_program()
- CLEANUP: Reduce scope of `header_name` in http_action_store_cache()
- CLEANUP: Reduce scope of `hdr_age` in http_action_store_cache()
- CLEANUP: spoe: fix typo on `var_check_arg` comment
- BUG/MINOR: tcpcheck: Report a L7OK if the last evaluated rule is a send rule
- CI: github actions: build several popular "contrib" tools
- DOC: Improve the message printed when running `make` w/o `TARGET`
- BUG/MEDIUM: server: srv_set_addr_desc() crashes when a server has no address
- REGTESTS: add unresolvable servers to srvkey-addr
- BUG/MINOR: stats: Make stat_l variable used to dump a stat line thread local
- BUG/MINOR: quic: NULL pointer dereferences when building post handshake frames.
- SCRIPTS: improve announce-release to support different tag and versions
- SCRIPTS: make announce release support preparing announces before tag exists
- CLEANUP: assorted typo fixes in the code and comments
- BUG/MINOR: srv: do not init address if backend is disabled
- BUG/MINOR: srv: do not cleanup idle conns if pool max is null
- CLEANUP: assorted typo fixes in the code and comments
- CLEANUP: few extra typo and fixes over last one ("ot" -> "to")
This allows using the address of the server rather than the name of the
server for keeping track of servers in a backend for stickiness.
The peers code was also extended to support feeding the dictionary using
this key instead of the name.
Fixes#814
Allow the user to specify a custom Connection header for http-check
send. This is useful for example to implement a websocket upgrade check.
If no connection header has been set, a 'Connection: close' header is
automatically appended to allow the server to close the connection
immediately after the request/response.
Update the documentation related to http-check send.
This fixes the github issue #1009.
Released version 2.4-dev4 with the following main changes :
- BUG/MEDIUM: lb-leastconn: Reposition a server using the right eweight
- BUG/MEDIUM: mux-h1: Fix a deadlock when a 408 error is pending for a client
- BUG/MEDIUM: ssl/crt-list: bad behavior with "commit ssl cert"
- BUG/MAJOR: cache: Crash because of disabled entry not removed from the tree
- BUILD: SSL: fine guard for SSL_CTX_add_server_custom_ext call
- MEDIUM: cache: Add a secondary entry counter and insertion limitation
- MEDIUM: cache: Avoid going over duplicates lists too often
- MINOR: cache: Add a max-secondary-entries cache option
- CI: cirrus: drop CentOS 6 builds
- BUILD: Makefile: have "make clean" destroy .o/.a/.s in contrib subdirs as well
- MINOR: vars: replace static functions with global ones
- MINOR: opentracing: add ARGC_OT enum
- CONTRIB: opentracing: add the OpenTracing filter
- DOC: opentracing: add the OpenTracing filter section
- REGTESTS: make use of HAPROXY_ARGS and pass -dM by default
- BUG/MINOR: http: Establish a tunnel for all 2xx responses to a CONNECT
- BUG/MINOR: mux-h1: Don't set CS_FL_EOI too early for protocol upgrade requests
- BUG/MEDIUM: http-ana: Never for sending data in TUNNEL mode
- CLEANUP: mux-h2: Rename h2s_frt_make_resp_data() to be generic
- CLEANUP: mux-h2: Rename h2c_frt_handle_data() to be generic
- BUG/MEDIUM: mux-h1: Handle h1_process() failures on a pipelined request
- CLEANUP: debug: mark the RNG's seed as unsigned
- CONTRIB: halog: fix build issue caused by %L printf format
- CONTRIB: halog: mark the has_zero* functions unused
- CONTRIB: halog: fix signed/unsigned build warnings on counts and timestamps
- CONTRIB: debug: address "poll" utility build on non-linux platforms
- BUILD: plock: remove dead code that causes a warning in gcc 11
- BUILD: ssl: fine guard for SSL_CTX_get0_privatekey call
- BUG/MINOR: dns: SRV records ignores duplicated AR records
- DOC: fix "smp_size" vs "sample_size" in "log" directive arguments
- CLEANUP: assorted typo fixes in the code and comments
- DOC: assorted typo fixes in the documentation
- CI: codespell: whitelist "te" and "nd" words
This new option allows to tune the maximum number of simultaneous
entries with the same primary key in the cache (secondary entries).
When we try to store a response in the cache and there are already
max-secondary-entries living entries in the cache, the storage will
fail (but the response will still be sent to the client).
It defaults to 10 and does not have a maximum number.
Released version 2.4-dev3 with the following main changes :
- MINOR: log: Logging HTTP path only with %HPO
- BUG/MINOR: mux-h2/stats: make stream/connection proto errors more accurate
- MINOR: traces: add a new level "error" below the "user" level
- MINOR: mux-h2/trace: add traces at level ERROR for protocol errors
- BUG/MINOR: mux-h2/stats: not all GOAWAY frames are errors
- BUG/MINOR: lua: missing "\n" in error message
- BUG/MINOR: lua: lua-load doesn't check its parameters
- BUG/MINOR: lua: Post init register function are not executed beyond the first one
- BUG/MINOR: lua: Some lua init operation are processed unsafe
- MINOR: actions: Export actions lookup functions
- MINOR: actions: add a function returning a service pointer from its name
- MINOR: cli: add a function to look up a CLI service description
- BUG/MINOR: lua: warn when registering action, conv, sf, cli or applet multiple times
- MINOR: cache: Improve accept_encoding_normalizer
- MINOR: cache: Add entry to the tree as soon as possible
- BUG/MINOR: trace: Wrong displayed trace level
- BUG/MAJOR: ring: tcp forward on ring can break the reader counter.
- MINOR: lua: simplify hlua_alloc() to only rely on realloc()
- MEDIUM: lua-thread: use atomics for memory accounting
- MINOR: lua-thread: remove struct hlua from function hlua_prepend_path()
- MEDIUM: lua-thread: make hlua_post_init() no longer use the runtime execution function
- MINOR: lua-thread: hlua_ctx_renew() is never called with main gL lua state
- MINOR: lua-thread: Use NULL context for main lua state
- MINOR: lua-thread: Stop usage of struct hlua for the global lua state
- MINOR: lua-thread: Replace embedded struct hlua_function by a pointer
- MINOR: lua-thread: Split hlua_init() function in two parts
- MINOR: lua-thread: make hlua_ctx_init() get L from its caller
- MINOR: lua-thread: Split hlua_load function in two parts
- MINOR: lua-thread: Split hlua_post_init() function in two parts
- MINOR: lua-thread: Add the "thread" core variable
- MEDIUM: lua-thread: No longer use locked context in initialization parts
- MEDIUM: lua-thread: Apply lock only if the parent state is the main thread
- MINOR: lua-thread: Replace global gL var with an array of states
- MINOR: lua-thread: Replace "struct hlua_function" allocation by dedicated function
- MINOR: lua-thread: Replace state_from by state_id
- MINOR: lua-thread: Store each function reference and init reference in array
- MEDIUM: lua-thread: Add the lua-load-per-thread directive
- MINOR: lua-thread: Add verbosity in errors
- REGTESTS: add a test for the threaded Lua code
- BUILD/MINOR: haproxy DragonFlyBSD affinity build update.
- DOC/MINOR: Fix formatting in Management Guide
- MINOR: cache: Do not store stale entry
- MINOR: cache: Add extra "cache-control" value checks
- MEDIUM: cache: Remove cache entry in case of POST on the same resource
- MINOR: cache: Consider invalid Age values as stale
- BUG/MEDIUM: lua-thread: some parts must be initialized once
- BUG/MINOR: lua-thread: close all states on deinit
- BUG/MINOR: listener: use sockaddr_in6 for IPv6
- BUG/MINOR: mux-h1: Handle keep-alive timeout for idle frontend connections
- MINOR: session: Add the idle duration field into the session
- MINOR: mux-h1: Update session idle duration when data are received
- MINOR: mux-h1: Reset session dates and durations info when the CS is detached
- MINOR: logs: Use session idle duration when no stream is provided
- MINOR: stream: Always get idle duration from the session
- MINOR: stream: Don't retrieve anymore timing info from the mux csinfo
- MINOR: mux-h1: Don't provide anymore timing info using cs_info structure
- MINOR: muxes: Remove get_cs_info callback function now useless
- MINOR: stream: Pass an optional input buffer when a stream is created
- MINOR: mux-h1: Add a flag to disable reads to wait opposite side
- MEDIUM: mux-h1: Use a h1c flag to block reads when splicing is in-progress
- MINOR: mux-h1: Introduce H1C_F_IS_BACK flag on the H1 connection
- MINOR: mux-h1: Separate parsing and formatting errors at H1 stream level
- MINOR: mux-h1: Split front/back h1 stream creation in 2 functions
- MINOR: mux-h1: Add a rxbuf into the H1 stream
- MINOR: mux-h1: Don't set CS flags in internal parsing functions
- MINOR: mux-h1: Add embryonic and attached states on the H1 connection
- MINOR: mux-h1: rework the h1_timeout_task() function
- MINOR: mux-h1: Reset more H1C flags when a H1 stream is destroyed
- MINOR: mux-h1: Disable reads if an error was reported on the H1 stream
- MINOR: mux-h1: Rework how shutdowns are handled
- MINOR: mux-h1: Rework h1_refresh_timeout to be easier to read
- MINOR: mux-h1: Process next request for IDLE connection only
- MINOR: mux-h1: Add a idle expiration date on the H1 connection
- MINOR: stick-tables: Add functions to update some values of a tracked counter
- MINOR: session: Add functions to increase http values of tracked counters
- MINOR: mux: Add a ctl parameter to get the exit status of the multiplexers
- MINOR: logs: Get the multiplexer exist status when no stream is provided
- MINOR: mux-h1: Add functions to send HTTP errors from the mux
- MAJOR: mux-h1: Create the client stream as later as possible
- DOC: config: Add notes about errors emitted by H1 mux
- CLEANUP: mux-h1: Rename H1C_F_CS_* flags and reorder H1C flags
- MINOR: http-ana: Remove useless update of t_idle duration of the stream
- CLEANUP: htx: Remove HTX_FL_UPGRADE unsued flag
- MEDIUM: http-ana: Don't process partial or empty request anymore
- CLEANUP: http-ana: Remove TX_WAIT_NEXT_RQ unsued flag
- CLEANUP: connection: Remove CS_FL_READ_PARTIAL flag
- REGTESTS: Fix proxy_protocol_tlv_validation
- MINOR: http-ana: Properly set message flags from the start-line flags
- MINOR: h1-htx/http-ana: Set BODYLESS flag on message in TUNNEL state
- MINOR: protocol: add a ->set_port() helper to address families
- MINOR: listener: automatically set the port when creating listeners
- MINOR: listener: now use a generic add_listener() function
- MEDIUM: ssl: fatal error with bundle + openssl < 1.1.1
- BUG/MEDIUM: stream: Xfer the input buffer to a fully created stream
- BUG/MINOR: stream: Don't use input buffer after the ownership xfer
- MINOR: protocol: remove the redundant ->sock_domain field
- MINOR: protocol: export protocol definitions
- CLEANUP: protocol: group protocol struct members by usage
- MINOR: protocol: add a set of ctrl_init/ctrl_close methods for setup/teardown
- MINOR: connection: use the control layer's init/close
- MINOR: udp: export udp_suspend_receiver() and udp_resume_receiver()
- BUG/MAJOR: spoa/python: Fixing return None
- DOC: spoa/python: Fixing typo in IP related error messages
- DOC: spoa/python: Rephrasing memory related error messages
- DOC: spoa/python: Fixing typos in comments
- BUG/MINOR: spoa/python: Cleanup references for failed Module Addobject operations
- BUG/MINOR: spoa/python: Cleanup ipaddress objects if initialization fails
- BUG/MEDIUM: spoa/python: Fixing PyObject_Call positional arguments
- BUG/MEDIUM: spoa/python: Fixing references to None
- DOC: email change of the DeviceAtlas maintainer
- MINOR: cache: Dump secondary entries in "show cache"
- CLEANUP: connection: use fd_stop_both() instead of conn_stop_polling()
- MINOR: stream-int: don't touch polling anymore on shutdown
- MINOR: connection: implement cs_drain_and_close()
- MINOR: mux-pt: take care of CS_SHR_DRAIN in shutr()
- MINOR: checks: use cs_drain_and_close() instead of draining the connection
- MINOR: checks: don't call conn_cond_update_polling() anymore
- CLEANUP: connection: open-code conn_cond_update_polling() and update the comment
- CLEANUP: connection: remove the unused conn_{stop,cond_update}_polling()
- BUG/MINOR: http-check: Use right condition to consider HTX message as full
- BUG/MINOR: tcpcheck: Don't rearm the check timeout on each read
- MINOR: tcpcheck: Only wait for more payload data on HTTP expect rules
- BUG/MINOR: tools: make parse_time_err() more strict on the timer validity
- BUG/MINOR: tools: Reject size format not starting by a digit
- MINOR: action: define enum for timeout type of the set-timeout rule
- MINOR: stream: prepare the hot refresh of timeouts
- MEDIUM: stream: support a dynamic server timeout
- MEDIUM: stream: support a dynamic tunnel timeout
- MEDIUM: http_act: define set-timeout server/tunnel action
- MINOR: frontend: add client timeout sample fetch
- MINOR: backend: add timeout sample fetches
- MINOR: stream: add sample fetches
- MINOR: stream: add timeout sample fetches
- REGTESTS: add regtest for http-request set-timeout
- CLEANUP: remove the unused fd_stop_send() in conn_xprt_shutw{,_hard}()
- CLEANUP: connection: remove the unneeded fd_stop_{recv,send} on read0/shutw
- MINOR: connection: remove sock-specific code from conn_sock_send()
- REORG: connection: move the socket iocb (conn_fd_handler) to sock.c
- MINOR: protocol: add a ->drain() function at the connection control layer
- MINOR: connection: make conn_sock_drain() use the control layer's ->drain()
- MINOR: protocol: add a pair of check_events/ignore_events functions at the ctrl layer
- MEDIUM: connection: make use of the control layer check_events/ignore_events
Add cur_server_timeout and cur_tunnel_timeout.
These sample fetches return the current timeout value for a stream. This
is useful to retrieve the value of a timeout which was changed via a
set-timeout rule.
Add a new http-request action 'set-timeout [server/tunnel]'. This action
can be used to update the server or tunnel timeout of a stream. It takes
two parameters, the timeout name to update and the new timeout value.
This rule is only valid for a proxy with backend capabilities. The
timeout value cannot be null. A sample expression can also be used
instead of a plain value.
The duplicated entries (in case of vary) were not taken into account by
the "show cache" command. They are now dumped too.
A new "vary" column is added to the output. It contains the complete
seocndary key (in hex format).
Now, some errors are handled by the H1 multiplexer. During the headers
parsing request, there is no stream attached to the H1 mux. Thus, if an
error is reported at this stage, it is handled by the mux itself. If
possible the corresponding frontend errorfile is used, but it should be a
static message. Custom error messages are not supported. Otherwise, default
error messages are used.
In addition, the http analysis has not started yet, so http-after-response
ruleset is not evaluated and cannot alter these early responses.
The goal is to allow execution of one main lua state per thread.
This patch contains the main job. The lua init is done using these
steps:
- "lua-load-per-thread" loads the lua code in the first thread
- it creates the structs
- it stores loaded files
- the 1st step load is completed (execution of hlua_post_init)
and now, we known the number of threads
- we initilize lua states for all remaining threads
- for each one, we load the lua file
- for each one, we execute post-init
Once all is loaded, we control consistency of functions references.
The rules are:
- a function reference cannot be in the shared lua state and in
a per-thread lua state at the same time.
- if a function reference is declared in a per-thread lua state, it
must be declared in all per-thread lua states
The goal is to allow execution of one main lua state per thread.
This commit introduces this variable in the core. Lua state initialized
by thread will have access to this variable, which reports the executing
thread. 0 indicates the shared thread. Programs which must be executed
only once can check for core.thread <= 1.
The goal is to no longer use "struct hlua" with global main lua_state.
This patch returns NULL value when some code tries go get the hlua struct
associated with a task through hlua_gethlua(). This functions is useful
only during runtime because the struct hlua contains only runtime states.
Some Lua functions allowed to yield are called from init environment.
I'm not sure this is a good practice. Maybe it will be clever to
disallow calling this kind of functions.
This patch adds a new logging variable '%HPO' for logging HTTP path only
(without query string) from relative or absolute URI.
For example:
log-format "hpo=%HPO hp=%HP hu=%HU hq=%HQ"
GET /r/1 HTTP/1.1
=>
hpo=/r/1 hp=/r/1 hu=/r/1 hq=
GET /r/2?q=2 HTTP/1.1
=>
hpo=/r/2 hp=/r/2 hu=/r/2?q=2 hq=?q=2
GET http://host/r/3 HTTP/1.1
=>
hpo=/r/3 hp=http://host/r/3 hu=http://host/r/3 hq=
GET http://host/r/4?q=4 HTTP/1.1
=>
hpo=/r/4 hp=http://host/r/4 hu=http://host/r/4?q=4 hq=?q=4
Released version 2.4-dev2 with the following main changes :
- BUILD: Make DEBUG part of .build_opts
- BUILD: Show the value of DEBUG= in haproxy -vv
- CI: Set DEBUG=-DDEBUG_STRICT=1 in GitHub Actions
- MINOR: stream: Add level 7 retries on http error 401, 403
- CLEANUP: remove unused function "ssl_sock_is_ckch_valid"
- BUILD: SSL: add BoringSSL guarding to "RAND_keep_random_devices_open"
- BUILD: SSL: do not "update" BoringSSL version equivalent anymore
- BUG/MEDIUM: http_act: Restore init of log-format list
- DOC: better describes how to configure a fallback crt
- BUG/MAJOR: filters: Always keep all offsets up to date during data filtering
- MINOR: cache: Prepare helper functions for Vary support
- MEDIUM: cache: Add the Vary header support
- MINOR: cache: Add a process-vary option that can enable/disable Vary processing
- BUG/CRITICAL: cache: Fix trivial crash by sending accept-encoding header
- BUG/MAJOR: peers: fix partial message decoding
- DOC: cache: Add new caching limitation information
- DOC: cache: Add information about Vary support
- DOC: better document the config file format and escaping/quoting rules
- DOC: Clarify %HP description in log-format
- CI: github actions: update LibreSSL to 3.3.0
- CI: github actions: enable 51degrees feature
- MINOR: fd/threads: silence a build warning with threads disabled
- BUG/MINOR: tcpcheck: Don't forget to reset tcp-check flags on new kind of check
- MINOR: tcpcheck: Don't handle anymore in-progress send rules in tcpcheck_main
- BUG/MAJOR: tcpcheck: Allocate input and output buffers from the buffer pool
- MINOR: tcpcheck: Don't handle anymore in-progress connect rules in tcpcheck_main
- MINOR: config: Deprecate and ignore tune.chksize global option
- MINOR: config: Add a warning if tune.chksize is used
- REORG: tcpcheck: Move check option parsing functions based on tcp-check
- MINOR: check: Always increment check health counter on CONPASS
- MINOR: tcpcheck: Add support of L7OKC on expect rules error-status argument
- DOC: config: Make disable-on-404 option clearer on transition conditions
- DOC: config: Move req.hdrs and req.hdrs_bin in L7 samples fetches section
- BUG/MINOR: http-fetch: Fix smp_fetch_body() when called from a health-check
- MINOR: plock: use an ARMv8 instruction barrier for the pause instruction
- MINOR: debug: add "debug dev sched" to stress the scheduler.
- MINOR: debug: add a trivial PRNG for scheduler stress-tests
- BUG/MEDIUM: lists: Lock the element while we check if it is in a list.
- MINOR: task: remove tasklet_insert_into_tasklet_list()
- MINOR: task: perform atomic counter increments only once per wakeup
- MINOR: task: remove __tasklet_remove_from_tasklet_list()
- BUG/MEDIUM: task: close a possible data race condition on a tasklet's list link
- BUG/MEDIUM: local log format regression.
Since 2.3 default local log format always adds hostame field.
This behavior change was due to log/sink re-work, because according
to rfc3164 the hostname field is mandatory.
This patch re-introduce a legacy "local" format which is analog
to rfc3164 but with hostname stripped. This is the new
default if logs are generated by haproxy.
To stay compliant with previous configurations, the option
"log-send-hostname" acts as if the default format is switched
to rfc3164.
This patch addresses the github issue #963
This patch should be backported in branches >= 2.3.
L7OKC may now be used as an error status for an HTTP/TCP expect rule. Thus
it is for instance possible to write:
option httpchk GET /isalive
http-check expect status 200,404
http-check expect status 200 error-status L7OKC
It is more or less the same than the disable-on-404 option except that if a
DOWN is up again but still replying a 404 will be set to NOLB state. While
it will stay in DOWN state with the disable-on-404 option.
This option is now ignored because I/O check buffers are now allocated using the
buffer pool. Thus, it is marked as deprecated in the documentation and ignored
during the configuration parsing. The field is also removed from the global
structure.
Because this option is ignored since a recent fix, backported as fare as 2.2,
this patch should be backported too. Especially because it updates the
documentation.
%HP is used to report HTTP request URI in logs, which might be relative
or absolute. Description in documentation should not suggest that it
behaves exactly the same as "path" sample fetch.
This is even more important after 30ee1efe67
because right now, when HTTP2 is a standard, %HP usually returns absolute
URI.
This might be backported as far as 2.1
It's always a pain to figure how to proceed when special characters need
to be embedded inside arguments of an expression. Let's document the
configuration file format and how unquoting/unescaping works at each
level (top level and argument level) so that everyone hopefully finds
suitable reminders or examples for complex cases.
This is related to github issue #200 and addresses issues #712 and #966.
Responses that do not have an explicit expiration time or a validator
will not be cached anymore.
Must be backported if cc9bf2e ("MEDIUM: cache: Change caching
conditions") is backported.
The cache section's process-vary option takes a 0 or 1 value to disable
or enable the vary processing.
When disabled, a response containing such a header will never be cached.
When enabled, we will calculate a preliminary hash for a subset of request
headers on all the incoming requests (which might come with a cpu cost) which
will be used to build a secondary key for a given request (see RFC 7234#4.1).
The default value is 0 (disabled).
A default certificate is always the first one declared in the bind line,
either from `crt` or from `crt-line` option. This commit updates the
description of how to configure a fallback certificate, clarifying that
it needs to be the first one of the bind line.
Should be merged as far as the first SNI filter implementation.
Level-7 retries are only possible with a restricted number of HTTP
return codes. While it is usually not safe to retry on 401 and 403, I
came up with an authentication backend which was not synchronizing
authentication of users. While not perfect, being allowed to also retry
on those return codes is really helpful and acts as a hotfix until we
can fix the backend.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
Released version 2.4-dev1 with the following main changes :
- MINOR: ist: Add istend() function to return a pointer to the end of the string
- MINOR: sample: Add converters to parse FIX messages
- REGTEST: converter: Add a regtest for fix converters
- MINOR: sample: Add converts to parses MQTT messages
- REGTEST: converter: Add a regtest for MQTT converters
- MINOR: compat: automatically include malloc.h on glibc
- MEDIUM: pools: call malloc_trim() from pool_gc()
- MEDIUM: pattern: call malloc_trim() on pat_ref_reload()
- MINOR: pattern: move the update revision to the pat_ref, not the expression
- CLEANUP: pattern: delete the back refs at once during pat_ref_reload()
- MINOR: pattern: new sflag PAT_SF_REGFREE indicates regex_free() is needed
- MINOR: pattern: make the delete and prune functions more generic
- MEDIUM: pattern: link all final elements from the reference
- MEDIUM: pattern: change the pat_del_* functions to delete from the references
- MINOR: pattern: remerge the list and tree deletion functions
- MINOR: pattern: perform a single call to pat_delete_gen() under the expression
- CLEANUP: acl: don't reference the generic pattern deletion function anymore
- CLEANUP: pattern: remove pat_delete_fcts[] and pattern_head->delete()
- MINOR: pattern: introduce pat_ref_delete_by_ptr() to delete a valid reference
- MINOR: pattern: store a generation number in the reference patterns
- MEDIUM: pattern: only match patterns that match the current generation
- MINOR: pattern: add pat_ref_commit() to commit a previously inserted element
- MINOR: pattern: implement pat_ref_load() to load a pattern at a given generation
- MINOR: pattern: add pat_ref_purge_older() to purge old entries
- MEDIUM: pattern: make pat_ref_prune() rely on pat_ref_purge_older()
- MINOR: pattern: during reload, delete elements frem the ref, not the expression
- MINOR: pattern: prepare removal of a pattern from the list head
- MEDIUM: pattern: turn the pattern chaining to single-linked list
- CLEANUP: cfgparse: remove duplicate registration for transparent build options
- BUG/MINOR: ssl: don't report 1024 bits DH param load error when it's higher
- MINOR: http-htx: Add understandable errors for the errorfiles parsing
- MINOR: ssl: instantiate stats module
- MINOR: ssl: count client hello for stats
- MINOR: ssl: add counters for ssl sessions
- DOC: config: Fix a typo on ssl_c_chain_der
- MINOR: server: remove idle lock in srv_cleanup_connections
- BUILD: ssl: silence build warning on uninitialised counters
- BUILD: http-htx: fix build warning regarding long type in printf
- REGTEST: ssl: test wildcard and multi-type + exclusions
- BUG/MEDIUM: ssl/crt-list: correctly insert crt-list line if crt already loaded
- CI: Expand use of GitHub Actions for CI
- REGTEST: ssl: mark reg-tests/ssl/ssl_crt-list_filters.vtc as broken
- BUG/MINOR: pattern: a sample marked as const could be written
- BUG/MINOR: lua: set buffer size during map lookups
- MEDIUM: cache: Change caching conditions
- BUG/MINOR: stats: free dynamically stats fields/lines on shutdown
- BUG/MEDIUM: stats: prevent crash if counters not alloc with dummy one
- MINOR: peers: Add traces to peer_treat_updatemsg().
- BUG/MINOR: peers: Do not ignore a protocol error for dictionary entries.
- BUG/MINOR: peers: Missing TX cache entries reset.
- BUG/MEDIUM: peers: fix decoding of multi-byte length in stick-table messages
- BUG/MINOR: http-fetch: Extract cookie value even when no cookie name
- BUG/MINOR: http-fetch: Fix calls w/o parentheses of the cookie sample fetches
- BUG/MEDIUM: check: reuse srv proto only if using same mode
- MINOR: check: report error on incompatible proto
- MINOR: check: report error on incompatible connect proto
- BUG/MINOR: http-htx: Handle warnings when parsing http-error and http-errors
- BUG/MAJOR: spoe: Be sure to remove all references on a released spoe applet
- MINOR: spoe: Don't close connection in sync mode on processing timeout
- BUG/MINOR: tcpcheck: Don't warn on unused rules if check option is after
- MINOR: init: Fix the prototype for per-thread free callbacks
- MINOR: config/mux-h2: Return ERR_ flags from init_h2() instead of a status
- CLEANUP: config: Return ERR_NONE from config callbacks instead of 0
- MINOR: cfgparse: tighten the scope of newnameserver variable, free it on error.
- REGTEST: make ssl_client_samples and ssl_server_samples require to 2.2
- REGTESTS: Add sample_fetches/cook.vtc
- BUG/MEDIUM: filters: Forward all filtered data at the end of http filtering
- BUG/MINOR: http-ana: Don't wait for the body of CONNECT requests
- CLEANUP: flt-trace: Remove unused random-parsing option
- MINOR: flt-trace: Add an option to inhibits trace messages
- MINOR: flt-trace: Use a bitfield for the trace options
- REGTESTS: Add a script to test the random forwarding with several filters
- REGTESTS: mark the abns test as broken again
- REGTESTS: converter: add url_dec test
- CI: Stop hijacking the hosts file
- CI: Make the h2spec workflow more consistent with the VTest workflow
- CI: travis-ci: remove amd64, osx builds
- CI: travis-ci: arm64 are not allowed to fail anymore
- DOC: add missing 3.10 in the summary
- MINOR: ssl: remove client hello counters
- MEDIUM: stats: add counters for failed handshake
- MINOR: ssl: create common ssl_ctx init
- MEDIUM: cli/ssl: configure ssl on server at runtime
- REGTEST: server/cli_set_ssl.vtc requires OpenSSL
- DOC: coding-style: update a few rules about pointers
- BUG/MINOR: ssl: segv on startup when AKID but no keyid
- BUILD: ssl: use SSL_MODE_ASYNC macro instead of OPENSSL_VERSION
- BUG/MEDIUM: http-ana: Don't eval http-after-response ruleset on empty messages
- BUG/MEDIUM: ssl/crt-list: bundle support broken in crt-list
- BUG/MEDIUM: ssl: error when no certificate are found
- BUG/MINOR: ssl/crt-list: load bundle in crt-list only if activated
- BUG/MEDIUM: ssl/crt-list: fix error when no file found
- CI: Github Actions: enable prometheus exporter
- CI: Github Actions: remove LibreSSL-3.0.2 builds
- CI: Github Actions: enable BoringSSL builds
- CI: travis-ci: remove builds migrated to GH actions
- BUILD: makefile: enable crypt(3) for OpenBSD
- CI: Github Action: run "apt-get update" before packages restore
- BUILD: SSL: guard TLS13 ciphersuites with HAVE_SSL_CTX_SET_CIPHERSUITES
- CI: Pass the github.event_name to matrix.py
- CI: Clean up Windows CI
- DOC: clarify how to create a fallback crt
- CLEANUP: connection: do not use conn->owner when the session is known
- BUG/MAJOR: connection: reset conn->owner when detaching from session list
- REGTESTS: mark proxy_protocol_random_fail as broken
- BUG/MINOR: http_htx: Fix searching headers by substring
- MINOR: http_act: Add -m flag for del-header name matching method
This patch adds -m flag which allows to specify header name
matching method when deleting headers from http request/response.
Currently beg, end, sub, str and reg are supported.
This is related to GitHub issue #909
HAProxy uses CN and SAN of the certificates to match incoming SNI, and
use the matching certificate in the TLS handshake. `crt-list` goes
further and allows to configure SNI filters to explicitly define the
FQDNs that should match a certificate.
The first declared certificate of the `crt-list` option follows the same
rules, and it's also used as a fallback - the certificate that should be
used if SNI isn't provided or the provided one cannot match any
certificate or SNI filter. If a provided SNI matches the CN or SAN of
the first certificate, the first certificate would be used even if a
matching SNI filter is declared later.
This change clarifies this scenario and documents a filter that can be
used to convert the first declared certificate as a proper fallback.
Should be merged as far as the first SNI filter implementation.
It's really annoying to see that in 2020 we're still facing bugs caused
by dangling pointers in the code that result from poorly written rules
about how these pointers are supposed to be handled, set and reset. Let's
add a few supposedly obvious (but apparently not) rules about how pointers
have to be used through out the code in hope to make such bad practices
disappear (or at least have something to point the authors to after
reviewing their code).
in the context of a progressive backend migration, we want to be able to
activate SSL on outgoing connections to the server at runtime without
reloading.
This patch adds a `set server ssl` command; in order to allow that:
- add `srv_use_ssl` to `show servers state` command for compatibility,
also update associated parsing
- when using default-server ssl setting, and `no-ssl` on server line,
init SSL ctx without activating it
- when triggering ssl API, de/activate SSL connections as requested
- clean ongoing connections as it is done for addr/port changes, without
checking prior server state
example config:
backend be_foo
default-server ssl
server srv0 127.0.0.1:6011 weight 1 no-ssl
show servers state:
5 be_foo 1 srv0 127.0.0.1 2 0 1 1 15 1 0 4 0 0 0 0 - 6011 - -1
where srv0 can switch to ssl later during the runtime:
set server be_foo/srv0 ssl on
5 be_foo 1 srv0 127.0.0.1 2 0 1 1 15 1 0 4 0 0 0 0 - 6011 - 1
Also update existing tests and create a new one.
Signed-off-by: William Dauchy <wdauchy@gmail.com>
There is a typo on the ssl_c_chain_der sample fetch
(s/ssl_c_der_chain/ssl_c_chain_der/). This implies a move of the fetch to keep
it at the right place.
This should be backported as far as 2.2 or anywhere the commit a598b500b
("MINOR: ssl: add ssl_{c,s}_chain_der fetch methods") is.
This patch implements a couple of converters to validate and extract data from a
MQTT (Message Queuing Telemetry Transport) message. The validation consists of a
few checks as well as "packet size" validation. The extraction can get any field
from the variable header and the payload.
This is limited to CONNECT and CONNACK packet types only. All other messages are
considered as invalid. It is not a problem for now because only the first packet
on each side can be parsed (CONNECT for the client and CONNACK for the server).
MQTT 3.1.1 and 5.0 are supported.
Reviewed and Fixed by Christopher Faulet <cfaulet@haproxy.com>
This patch implements a couple of converters to validate and extract tag value
from a FIX (Financial Information eXchange) message. The validation consists in
a few checks such as mandatory fields and checksum computation. The extraction
can get any tag value based on a tag string or tag id.
This patch requires the istend() function. Thus it depends on "MINOR: ist: Add
istend() function to return a pointer to the end of the string".
Reviewed and Fixed by Christopher Faulet <cfaulet@haproxy.com>
Released version 2.4-dev0 with the following main changes :
- MINOR: version: it's development again.
- DOC: mention in INSTALL that it's development again
Released version 2.3.0 with the following main changes :
- CLEANUP: pattern: remove unused entry "tree" in pattern.val
- BUILD: ssl: use SSL_CTRL_GET_RAW_CIPHERLIST instead of OpenSSL versions
- BUG/MEDIUM: filters: Don't try to init filters for disabled proxies
- BUG/MINOR: proxy/server: Skip per-proxy/server post-check for disabled proxies
- BUG/MINOR: checks: Report a socket error before any connection attempt
- BUG/MINOR: server: Set server without addr but with dns in RMAINT on startup
- MINOR: server: Copy configuration file and line for server templates
- BUG/MEDIUM: mux-pt: Release the tasklet during an HTTP upgrade
- BUILD: ssl: use HAVE_OPENSSL_KEYLOG instead of OpenSSL versions
- MINOR: debug: don't count free(NULL) in memstats
- BUG/MINOR: filters: Skip disabled proxies during startup only
- MINOR: mux_h2: capitalize frame type in stats
- MINOR: mux_h2: add stat for total count of connections/streams
- MINOR: stats: do not display empty stat module title on html
- BUG/MEDIUM: stick-table: limit the time spent purging old entries
- BUG/MEDIUM: listener: only enable a listening listener if needed
- BUG/MEDIUM: listener: never suspend inherited sockets
- BUG/MEDIUM: listener: make the master also keep workers' inherited FDs
- MINOR: fd: add fd_want_recv_safe()
- MEDIUM: listeners: make use of fd_want_recv_safe() to enable early receivers
- REGTESTS: mark abns_socket as working now
- CLEANUP: mux-h2: Remove the h1 parser state from the h2 stream
- MINOR: sock: add a check against cross worker<->master socket activities
- CI: github actions: limit OpenSSL no-deprecated builds to "default,bug,devel" reg-tests
- BUG/MEDIUM: server: make it possible to kill last idle connections
- MINOR: mworker/cli: the master CLI use its own applet
- MINOR: ssl: define SSL_CTX_set1_curves_list to itself on BoringSSL
- BUILD: ssl: use feature macros for detecting ec curves manipulation support
- DOC: Add dns as an available domain to show stat
- BUILD: makefile: usual reorder of objects for faster builds
- DOC: update INSTALL to mention that TCC is supported
- DOC: mention in INSTALL that haproxy 2.3 is a stable version
- MINOR: version: mention that it's stable now
Within management.txt, proxy was listed as the only available option. "dns"
is now supported so let's add that. This change also updates the command to list
the available options <dns|proxy> for "domain" as previously it only specified
<domain>, which could be confusing as a user may think this field accepts
dynamic options when it actually requires a specific keyword.
Released version 2.3-dev9 with the following main changes :
- CLEANUP: http_ana: remove unused assignation of `att_beg`
- BUG/MEDIUM: ssl: OCSP must work with BoringSSL
- BUG/MINOR: log: fix memory leak on logsrv parse error
- BUG/MINOR: log: fix risk of null deref on error path
- BUILD: ssl: more elegant OpenSSL early data support check
- CI: github actions: update h2spec to 2.6.0
- BUG/MINOR: cache: Check the return value of http_replace_res_status
- MINOR: cache: Store the "Last-Modified" date in the cache_entry
- MINOR: cache: Process the If-Modified-Since header in conditional requests
- MINOR: cache: Create res.cache_hit and res.cache_name sample fetches
- MINOR: mux-h2: register a stats module
- MINOR: mux-h2: add counters instance to h2c
- MINOR: mux-h2: add stats for received frame types
- MINOR: mux-h2: report detected error on stats
- MINOR: mux-h2: count open connections/streams on stats
- BUG/MINOR: server: fix srv downtime calcul on starting
- BUG/MINOR: server: fix down_time report for stats
- BUG/MINOR: lua: initialize sample before using it
- MINOR: cache: Add Expires header value parsing
- MINOR: ist: Add a case insensitive istmatch function
- BUG/MINOR: cache: Manage multiple values in cache-control header value
- BUG/MINOR: cache: Inverted variables in http_calc_maxage function
- MINOR: pattern: make pat_ref_append() return the newly added element
- MINOR: pattern: make pat_ref_add() rely on pat_ref_append()
- MINOR: pattern: export pat_ref_push()
- CLEANUP: pattern: use calloc() rather than malloc for structures
- CLEANUP: pattern: fix spelling/grammatical/copy-paste in comments
Res.cache_hit sample fetch returns a boolean which is true when the HTTP
response was built out of a cache. The cache's name is returned by the
res.cache_name sample_fetch.
This resolves GitHub issue #900.
Released version 2.3-dev8 with the following main changes :
- MINOR: backend: replace the lbprm lock with an rwlock
- MINOR: lb/map: use seek lock and read locks where appropriate
- MINOR: lb/leastconn: only take a read lock in fwlc_get_next_server()
- MINOR: lb/first: use a read lock in fas_get_next_server()
- MINOR: lb/chash: use a read lock in chash_get_server_hash()
- BUG/MINOR: disable dynamic OCSP load with BoringSSL
- BUILD: ssl: make BoringSSL use its own version numbers
- CLEANUP: threads: don't register an initcall when not debugging
- MINOR: threads: change lock_t to an unsigned int
- CLEANUP: tree-wide: reorder a few structures to plug some holes around locks
- CLEANUP: task: remove the unused and mishandled global_rqueue_size
- BUG/MEDIUM: connection: Never cleanup server lists when freeing private conns
- MEDIUM: config: report that "nbproc" is deprecated
- BUG/MINOR: listener: close before free in `listener_accept`
- MINOR: ssl: 'ssl-load-extra-del-ext' removes the certificate extension
- BUG/MINOR: queue: properly report redistributed connections
- CONTRIB: tcploop: remove unused local variables in tcp_pause()
- BUILD: makefile: add entries to build common debugging tools
- BUG/MEDIUM: server: support changing the slowstart value from state-file
- MINOR: http: Add `enum etag_type http_get_etag_type(const struct ist)`
- MINOR: http: Add etag comparison function
- MEDIUM: cache: Store the ETag information in the cache_entry
- MEDIUM: cache: Add support for 'If-None-Match' request header
- REGTEST: cache: Add if-none-match test case
- CLEANUP: compression: Make use of http_get_etag_type()
- BUG/MINOR: http-ana: Don't send payload for internal responses to HEAD requests
- BUG/MAJOR: mux-h2: Don't try to send data if we know it is no longer possible
- MINOR: threads/debug: only report used lock stats
- MINOR: threads/debug: only report lock stats for used operations
- MINOR: proxy; replace the spinlock with an rwlock
- MINOR: server: read-lock the cookie during srv_set_dyncookie()
- MINOR: proxy/cli: only take a read lock in "show errors"
- OPTIM: queue: don't call pendconn_unlink() when the pendconn is not queued
- MINOR: queue: split __pendconn_unlink() in per-srv and per-prx
- MINOR: queue: reduce the locked area in pendconn_add()
- OPTIM: queue: make the nbpend counters atomic
- OPTIM: queue: decrement the nbpend and totpend counters outside of the lock
- MINOR: leastconn: take the queue length into account when queuing servers
- MEDIUM: fwlc: re-enable per-server queuing up to maxqueue
- Revert "OPTIM: queue: don't call pendconn_unlink() when the pendconn is not queued"
- MINOR: stats: support the "up" output modifier for "show stat"
- MINOR: stats: also support a "no-maint" show stat modifier
- MINOR: stats: indicate the number of servers in a backend's status
- MEDIUM: ssl: ssl-load-extra-del-ext work only with .crt
- REGTEST: ssl: test "set ssl cert" with separate key / crt
- DOC: management: apply the "show stat" modifiers to "show stat", not "show info"
- MINOR: stats: report server's user-configured weight next to effective weight
- CI: travis-ci: switch to Ubuntu 20.04
- CONTRIB: release-estimator: Add release estimating tool
- BUG/MEDIUM: queue: fix unsafe proxy pointer when counting nbpend
- BUG/MINOR: extcheck: add missing checks on extchk_setenv()
The "weight" column on the stats page is somewhat confusing when using
slowstart becaue it reports the effective weight, without being really
explicit about it. In some situations the user-configured weight is more
relevant (especially with long slowstarts where it's important to know
if the configured weight is correct).
This adds a new uweight stat which reports a server's user-configured
weight, and in a backend it receives the sum of all servers' uweights.
In addition it adds the mention of "effective" in a few descriptions
for the "weight" column (help and doc).
As a result, the list of servers in a backend is now always scanned
when dumping the stats. But this is not a problem given that these
servers are already scanned anyway and for way heavier processing.
By mistake I added the "up" then "maint" output modifiers to the "show info"
block instead of the "show stat" one in the two previous commits 65141ffc4
("MINOR: stats: support the "up" output modifier for "show stat"") and
3e3203670 ("MINOR: stats: also support a "no-maint" show stat modifier").
No backport is needed.
In order to be compatible with the "set ssl cert" command of the CLI,
this patch restrict the ssl-load-extra-del-ext to files with a ".crt"
extension in the configuration.
Related to issue #785.
Should be backported where 8e8581e ("MINOR: ssl: 'ssl-load-extra-del-ext'
removes the certificate extension") was backported.
"no-maint" is a bit similar to "up" except that it will only hide
servers that are in maintenance (or disabled in the configuration), and
not those that are enabled but failed a check. One benefit here is to
significantly reduce the output of the "show stat" command when using
large server-templates containing entries that are not yet provisioned.
Note that the prometheus exporter also has such an option which does
the exact same.
We already had it on the HTTP interface but it was not accessible on the
CLI. It can be very convenient to hide servers which are down, do not
resolve, or are in maintenance.
Leastconn has the nice propery of being able to sort servers by their
current usage. It's really a shame to force all requests into the backend
queue when the algo would be able to also consider their current queue.
In order not to change existing behavior but extend it, this patch allows
leastconn to elect servers which are already full if they have an explicitly
configured maxqueue setting above zero and their queue hasn't reached that
threshold. This will significantly reduce the pressure in the backend queue
when queuing a lot with lots of servers.
A test on 8 threads with 100 servers configured with maxconn 1 jumped
from 165krps to 330krps with maxqueue 15 with this patch.
This partially undoes commit 82cd5c13a ("OPTIM: backend: skip LB when we
know the backend is full") but allows to scale much better even by setting
a single-digit maxqueue value. Some better heuristics could be used to
maintain the behavior of the bypass in the patch above, consisting in
keeping it if it's known that there is no server with a configured
maxqueue in the farm (or in the backend).
When servers are queued into the leastconn tree, it's important to also
consider their queue length. There could be some servers with lots of
queued requests that we don't want to hammer with extra connections. In
order not to add extra stress to the LB algorithm, we don't update the
value when adding to the queue, only when updating the connection count
(i.e. picking from the queue or releasing a connection). This will be
sufficient to significantly improve the fairness in such situations.
In issue #785, users are reporting that it's not convenient to load a
".crt.key" when the configuration contains a ".crt".
This option allows to remove the extension of the certificate before
trying to load any extra SSL file (.key, .ocsp, .sctl, .issuer etc.)
The patch changes a little bit the way ssl_sock_load_files_into_ckch()
looks for the file.
As previously discussed, nbproc usage is bad, deprecated, and scheduled
for removal in 2.5.
If "nbproc" is found with more than one process while nbthread is not
set, a warning will be emitted encouraging to remove it or to migrate
to nbthread instead. This makes sure the user has an opportunity to
both see the message and silence it.
Released version 2.3-dev7 with the following main changes :
- CI: travis-ci: replace not defined SSL_LIB, SSL_INC for BotringSSL builds
- BUG/MINOR: init: only keep rlim_fd_cur if max is unlimited
- BUG/MINOR: mux-h2: do not stop outgoing connections on stopping
- MINOR: fd: report an error message when failing initial allocations
- MINOR: proto-tcp: make use of connect(AF_UNSPEC) for the pause
- MINOR: sock: add sock_accept_conn() to test a listening socket
- MINOR: protocol: make proto_tcp & proto_uxst report listening sockets
- MINOR: sockpair: implement the .rx_listening function
- CLEANUP: tcp: make use of sock_accept_conn() where relevant
- CLEANUP: unix: make use of sock_accept_conn() where relevant
- BUG/MINOR: listener: detect and handle shared sockets stopped in other processes
- CONTRIB: tcploop: implement a disconnect operation 'D'
- CLEANUP: protocol: intitialize all of the sockaddr when disconnecting
- BUG/MEDIUM: deinit: check fdtab before fdtab[fd].owner
- BUG/MINOR: connection: fix loop iter on connection takeover
- BUG/MEDIUM: connection: fix srv idle count on conn takeover
- MINOR: connection: improve list api usage
- MINOR: mux/connection: add a new mux flag for HOL risk
- MINOR: connection: don't check priv flag on free
- MEDIUM: backend: add new conn to session if mux marked as HOL blocking
- MEDIUM: backend: add reused conn to sess if mux marked as HOL blocking
- MEDIUM: h2: remove conn from session on detach
- MEDIUM: fcgi: remove conn from session on detach
- DOC: Describe reuse safe for HOL handling
- MEDIUM: proxy: remove obsolete "mode health"
- MEDIUM: proxy: remove obsolete "monitor-net"
- CLEANUP: protocol: remove the ->drain() function
- CLEANUP: fd: finally get rid of fd_done_recv()
- MINOR: connection: make sockaddr_alloc() take the address to be copied
- MEDIUM: listener: allocate the connection before queuing a new connection
- MINOR: session: simplify error path in session_accept_fd()
- MINOR: connection: add new error codes for accept_conn()
- MINOR: sock: rename sock_accept_conn() to sock_accepting_conn()
- MINOR: protocol: add a new function accept_conn()
- MINOR: sock: implement sock_accept_conn() to accept a connection
- MINOR: sockpair: implement sockpair_accept_conn() to accept a connection
- MEDIUM: listener: use protocol->accept_conn() to accept a connection
- MEDIUM: listener: remove the second pass of fd manipulation at the end
- MINOR: protocol: add a default I/O callback and put it into the receiver
- MINOR: log: set the UDP receiver's I/O handler in the receiver
- MINOR: protocol: register the receiver's I/O handler and not the protocol's
- CLEANUP: protocol: remove the now unused <handler> field of proto_fam->bind()
- DOC: improve the documentation for "option nolinger"
- BUG/MEDIUM: proxy: properly stop backends
- BUG/MEDIUM: task: bound the number of tasks picked from the wait queue at once
- MINOR: threads: augment rwlock debugging stats to report seek lock stats
- MINOR: threads: add the transitions to/from the seek state
- MEDIUM: task: use an upgradable seek lock when scanning the wait queue
- BUILD: listener: avoir a build warning when threads are disabled
- BUG/MINOR: peers: Possible unexpected peer seesion reset after collisions.
- MINOR: ssl: add volatile flags to ssl samples
- MEDIUM: backend: reuse connection if using a static sni
- BUG/MEDIUM: spoe: Unset variable instead of set it if no data provided
- BUG/MEDIUM: mux-h1: Get the session from the H1S when capturing bad messages
- BUG/MEDIUM: lb: Always lock the server when calling server_{take,drop}_conn
- DOC: fix typo in MAX_SESS_STKCTR
Detect if the sni used a constant value and if so, allow to reuse this
connection for later sessions. Use a combination of SMP_USE_INTRN +
!SMP_F_VOLATILE to consider a sample as a constant value.
This features has been requested on github issue #371.
fd_done_recv() used to be useful with the FD cache because it used to
allow to keep a file descriptor active in the poller without being
marked as ready in the cache, saving it from ringing immediately,
without incurring any system call. It was a way to make it yield
to wait for new events leaving a bit of time for others. The only
user left was the connection accepter (listen_accept()). We used
to suspect that with the FD cache removal it had become totally
useless since changing its readiness or not wouldn't change its
status regarding the poller itself, which would be the only one
deciding to report it again.
Careful tests showed that it indeed has exactly zero effect nowadays,
the syscall numbers are exactly the same with and without, including
when enabling edge-triggered polling.
Given that there's no more API available to manipulate it and that it
was directly called as an optimization from listener_accept(), it's
about time to remove it.
As discussed here during 2.1-dev, "monitor-net" is totally obsolete:
https://www.mail-archive.com/haproxy@formilux.org/msg35204.html
It's fundamentally incompatible with usage of SSL, and imposes the
presence of file descriptors with hard-coded syscalls directly in the
generic accept path.
It's very unlikely that anyone has used it in the last 10 years for
anything beyond testing. In the worst case if anyone would depend
on it, replacing it with "http-request return status 200 if ..." and
"mode http" would certainly do the trick.
The keyword is still detected as special by the config parser to help
users update their configurations appropriately.
