Often we need to understand why some transfers were aborted or what
constitutes server response errors. With those two counters, it is
now possible to detect an unexpected transfer abort during a data
phase (eg: too short HTTP response), and to know what part of the
server response errors may in fact be assigned to aborted transfers.
Released version 1.4.0 with the following main changes :
- [MINOR] stats: report maint state for tracking servers too
- [DOC] fix summary to add pattern extraction
- [DOC] Documentation cleanups
- [BUG] cfgparse memory leak and missing free calls in deinit()
- [BUG] pxid/puid/luid: don't shift IDs when some of them are forced
- [EXAMPLES] add auth.cfg
- [BUG] uri_auth: ST_SHLGNDS should be 0x00000008 not 0x0000008
- [BUG] uri_auth: do not attemp to convert uri_auth -> http-request more than once
- [BUILD] auth: don't use unnamed unions
- [BUG] config: report unresolvable host names as errors
- [BUILD] fix build breakage with DEBUG_FULL
- [DOC] fix a typo about timeout check and clarify the explanation.
- [MEDIUM] http: don't use trash to realign large buffers
- [STATS] report HTTP requests (total and rate) in frontends
- [STATS] separate frontend and backend HTTP stats
- [MEDIUM] http: revert to use a swap buffer for realignment
- [MINOR] stats: report the request rate in frontends as cell titles
- [MINOR] stats: mark areas with an underline when tooltips are available
- [DOC] reorder some entries to maintain the alphabetical order
- [DOC] cleanup of the keyword matrix
Hi Willy,
This patch make some cleanups on the documentation :
- strictly respect the 80 columns constraint
- use space indentation everywhere
- add some missing "(deprecated)"
- fix the "monitor fail" keyword syntax
- fix a keyword in a "See Also" line (interval instead of inter)
- the "no" prefix of "option http-use-proxy-header" is displayed as the others
I've also noticed that 3 keywords are missing from the keywords reference (I don't know what to do for that) :
- dispatch
- reqsetbe
- reqisetbe
--
Cyril Bonté
Released version 1.4-rc1 with the following main changes :
- [MEDIUM] add a maintenance mode to servers
- [MINOR] http-auth: last fix was wrong
- [CONTRIB] add base64rev-gen.c that was used to generate the base64rev table.
- [MINOR] Base64 decode
- [MINOR] generic auth support with groups and encrypted passwords
- [MINOR] add ACL_TEST_F_NULL_MATCH
- [MINOR] http-request: allow/deny/auth support for frontend/backend/listen
- [MINOR] acl: add http_auth and http_auth_group
- [MAJOR] use the new auth framework for http stats
- [DOC] add info about userlists, http-request and http_auth/http_auth_group acls
- [STATS] make it possible to change a CLI connection timeout
- [BUG] patterns: copy-paste typo in type conversion arguments
- [MINOR] pattern: make the converter more flexible by supporting void* and int args
- [MINOR] standard: str2mask: string to netmask converter
- [MINOR] pattern: add support for argument parsers for converters
- [MINOR] pattern: add the "ipmask()" converting function
- [MINOR] config: off-by-one in "stick-table" after list of converters
- [CLEANUP] acl, patterns: make use of my_strndup() instead of malloc+memcpy
- [BUG] restore accidentely removed line in last patch !
- [MINOR] checks: make the HTTP check code add the CRLF itself
- [MINOR] checks: add the server's status in the checks
- [BUILD] halog: make without arch-specific optimizations
- [BUG] halog: fix segfault in case of empty log in PCT mode (cherry picked from commit fe362fe476)
- [MINOR] http: disable keep-alive when process is going down
- [MINOR] acl: add build_acl_cond() to make it easier to add ACLs in config
- [CLEANUP] config: use build_acl_cond() instead of parse_acl_cond()
- [CLEANUP] config: use warnif_cond_requires_resp() to check for bad ACLs
- [MINOR] prepare req_*/rsp_* to receive a condition
- [CLEANUP] config: specify correct const char types to warnif_* functions
- [MEDIUM] config: factor out the parsing of 20 req*/rsp* keywords
- [MEDIUM] http: make the request filter loop check for optional conditions
- [MEDIUM] http: add support for conditional request filter execution
- [DOC] add some build info about the AIX platform (cherry picked from commit e41914c77e)
- [MEDIUM] http: add support for conditional request header addition
- [MEDIUM] http: add support for conditional response header rewriting
- [DOC] add some missing ACLs about response header matching
- [MEDIUM] http: add support for proxy authentication
- [MINOR] http-auth: make the 'unless' keyword work as expected
- [CLEANUP] config: use build_acl_cond() to simplify http-request ACL parsing
- [MEDIUM] add support for anonymous ACLs
- [MEDIUM] http: switch to tunnel mode after status 101 responses
- [MEDIUM] http: stricter processing of the CONNECT method
- [BUG] config: reset check request to avoid double free when switching to ssl/sql
- [MINOR] config: fix too large ssl-hello-check message.
- [BUG] fix error response in case of server error
A 101 response is accompanied with an Upgrade header indicating
a new protocol that is spoken on the connection after the exchange
completes. At least we should switch to tunnel mode after such a
response.
Anonymous ACLs allow the declaration of rules which rely directly on
ACL expressions without passing via the declaration of an ACL. Example :
With named ACLs :
acl site_dead nbsrv(dynamic) lt 2
acl site_dead nbsrv(static) lt 2
monitor fail if site_dead
With anonymous ACLs :
monitor fail if { nbsrv(dynamic) lt 2 } || { nbsrv(static) lt 2 }
This is a first attempt to add a maintenance mode on servers, using
the stat socket (in admin level).
It can be done with the following command :
- disable server <backend>/<server>
- enable server <backend>/<server>
In this mode, no more checks will be performed on the server and it
will be marked as a special DOWN state (MAINT).
If some servers were tracking it, they'll go DOWN until the server
leaves the maintenance mode. The stats page and the CSV export also
display this special state.
This can be used to disable the server in haproxy before doing some
operations on this server itself. This is a good complement to the
"http-check disable-on-404" keyword and works in TCP mode.
We're already able to know if a request is a proxy request or a
normal one, and we have an option "http-use-proxy-header" which states
that proxy headers must be checked. So let's switch to use the proxy
authentication headers and responses when this option is set and we're
facing a proxy request. That allows haproxy to enforce auth in front
of a proxy.
Just as for the req* rules, we can now condition rsp* rules with ACLs.
ACLs match on response, so volatile request information cannot be used.
A warning is emitted if a configuration contains such an anomaly.
All the req* rules except the reqadd rules can now be specified with
an if/unless condition. If a condition is specified and does not match,
the filter is ignored. This is particularly useful with reqidel, reqirep
and reqtarpit.
Now a server can check the contents of the header X-Haproxy-Server-State
to know how haproxy sees it. The same values as those reported in the stats
are provided :
- up/down status + check counts
- throttle
- weight vs backend weight
- active sessions vs backend sessions
- queue length
- haproxy node name
This converter can be applied on top of an IPv4-type pattern. It
applies a netmask which is suited for IP address storage and matching.
This can be used to make all hosts within a certain mask to share the
same table entries and as such use the same server.
The mask can be passed in dotted form (eg: 255.255.255.0) or in CIDR
form (eg: 24).
Hi Willy,
I've made a quick pass on the "defaults" column in the Proxy keywords matrix (chapter 4.1. in the documentation).
This patch resyncs the code and the documentation. I let you decide if some keywords that still work in the "defaults" section should be forbidden.
- default_backend : in the matrix, "defaults" was not supported but the keyword details say it is.
Tests also shows it works, then I've updated the matrix.
- capture cookie : in the keyword details, we can read `It is not possible to specify a capture in a "defaults" section.'.
Ok, even if the tests worked, I've added an alert in the configuration parser (as it is for capture request/response header).
- description : not supported in "defaults", I added an alert in the parser.
I've also noticed that this keyword doesn't appear in the documentation.
There's one "description" entry, but for the "global" section, which is for a different use (the patch doesn't update the documentation).
- grace : even if this is maybe useless, it works in "defaults". Documentation is updated.
- redirect : alert is added in the parser.
- rsprep : alert added in the parser.
--
Cyril Bonté
Despite what is explicitly stated in HTTP specifications,
browsers still use the undocumented Proxy-Connection header
instead of the Connection header when they connect through
a proxy. As such, proxies generally implement support for
this stupid header name, breaking the standards and making
it harder to support keep-alive between clients and proxies.
Thus, we add a new "option http-use-proxy-header" to tell
haproxy that if it sees requests which look like proxy
requests, it should use the Proxy-Connection header instead
of the Connection header.
Released version 1.4-dev7 with the following main changes :
- [BUG] appsession: possible memory leak in case of out of memory condition
- [MINOR] config: don't accept 'appsession' in defaults section
- [MINOR] Add function to parse a size in configuration
- [MEDIUM] Add stick table (persistence) management functions and types
- [MEDIUM] Add pattern fetch management types and functions
- [MEDIUM] Add src dst and dport pattern fetches.
- [MEDIUM] Add stick table configuration and init.
- [MEDIUM] Add stick and store rules analysers.
- [MINOR] add option "mysql-check" to use MySQL health checks
- [BUG] health checks: fix requeued message
- [OPTIM] remove SSP_O_VIA and SSP_O_STATUS
- [BUG] checks: fix newline termination
- [MINOR] acl: add fe_id/so_id to match frontend's and socket's id
- [BUG] appsession's sessid must be reset at end of transaction
- [BUILD] appsession did not build anymore under gcc-2.95
- [BUG] server redirection used an uninitialized string.
- [MEDIUM] http: fix handling of message pointers
- [MINOR] http: fix double slash prefix with server redirect
- [MINOR] http redirect: add the ability to append a '/' to the URL
- [BUG] stream_interface: fix retnclose and remove cond_close
- [MINOR] http redirect: don't explicitly state keep-alive on 1.1
- [MINOR] http: move appsession 'sessid' from session to http_txn
- [OPTIM] reorder http_txn to optimize cache lines placement
- [MINOR] http: differentiate waiting for new request and waiting for a complete requst
- [MINOR] http: add a separate "http-keep-alive" timeout
- [MINOR] config: remove undocumented and buggy 'timeout appsession'
- [DOC] fix various too large lines
- [DOC] remove several trailing spaces
- [DOC] add the doc about stickiness
- [BUILD] remove a warning in standard.h on AIX
- [BUG] checks: chars are unsigned on AIX, check was always true
- [CLEANUP] stream_sock: MSG_NOSIGNAL is only for send(), not recv()
- [BUG] check: we must not check for error before reading a response
- [BUG] buffers: remove remains of wrong obsolete length check
- [OPTIM] stream_sock: don't shutdown(write) when the socket is in error
- [BUG] http: don't count req errors on client resets or t/o during keep-alive
- [MEDIUM] http: don't switch to tunnel mode upon close
- [DOC] add documentation about connection header processing
- [MINOR] http: add http_remove_header2() to remove a header value.
- [MINOR] tools: add a "word_match()" function to match words and ignore spaces
- [MAJOR] http: rework request Connection header handling
- [MAJOR] http: rework response Connection header handling
- [MINOR] add the ability to force kernel socket buffer size.
- [BUG] http_server_error() must not purge a previous pending response
- [OPTIM] http: don't delay response if next request is incomplete
- [MINOR] add the "force-persist" statement to force persistence on down servers
- [MINOR] http: logs must report persistent connections to down servers
- [BUG] buffer_replace2 must never change the ->w entry
This is used to force access to down servers for some requests. This
is useful when validating that a change on a server correctly works
before enabling the server again.
Sometimes we need to be able to change the default kernel socket
buffer size (recv and send). Four new global settings have been
added for this :
- tune.rcvbuf.client
- tune.rcvbuf.server
- tune.sndbuf.client
- tune.sndbuf.server
Those can be used to reduce kernel memory footprint with large numbers
of concurrent connections, and to reduce risks of write timeouts with
very slow clients due to excessive kernel buffering.
This one is the next step of previous patch. It correctly computes
the response mode and the Connection flag transformations depending
on the request mode and version, and the response version and headers.
We're now also able to add "Connection: keep-alive", and to convert
server's close during a keep-alive connection to a server-close
connection.
The connection header is complex to handle, especially in the response
path, depending on request and response HTTP versions, desired mode,
etc... Let's document it. Note that only a subset of this document is
currently implemented.
Sometimes it can be desired to return a location which is the same
as the request with a slash appended when there was not one in the
request. A typical use of this is for sending a 301 so that people
don't reference links without the trailing slash. The name of the
new option is "append-slash" and it can be used on "redirect"
statements in prefix mode.
Released version 1.4-dev6 with the following main changes :
- [BUILD] warning in stream_interface.h
- [BUILD] warning ultoa_r returns char *
- [MINOR] hana: only report stats if it is enabled
- [MINOR] stats: add "a link" & "a href" for sockets
- [MINOR]: stats: add show-legends to report additional informations
- [MEDIUM] default-server support
- [BUG]: add 'observer', 'on-error', 'error-limit' to supported options list
- [MINOR] stats: add href to tracked server
- [BUG] stats: show UP/DOWN status also in tracking servers
- [DOC] Restore ability to search a keyword at the beginning of a line
- [BUG] stats: cookie should be reported under backend not under proxy
- [BUG] cfgparser/stats: fix error message
- [BUG] http: disable auto-closing during chunk analysis
- [BUG] http: fix hopefully last closing issue on data forwarding
- [DEBUG] add an http_silent_debug function to debug HTTP states
- [MAJOR] http: fix again the forward analysers
- [BUG] http_process_res_common() must not skip the forward analyser
- [BUG] http: some possible missed close remain in the forward chain
- [BUG] http: redirect needed to be updated after recent changes
- [BUG] http: don't set no-linger on response in case of forced close
- [MEDIUM] http: restore the original behaviour of option httpclose
- [TESTS] add a file to test various connection modes
- [BUG] http: check options before the connection header
- [MAJOR] session: fix the order by which the analysers are run
- [MEDIUM] session: also consider request analysers added during response
- [MEDIUM] http: make safer use of the DONT_READ and AUTO_CLOSE flags
- [BUG] http: memory leak with captures when using keep-alive
- [BUG] http: fix for capture memory leak was incorrect
- [MINOR] http redirect: use proper call to return last response
- [MEDIUM] http: wait for some flush of the response buffer before a new request
- [MEDIUM] session: limit the number of analyser loops
This patch implements default-server support allowing to change
default server options. It can be used in [defaults] or [backend]/[listen]
sections. Currently the following options are supported:
- error-limit
- fall
- inter
- fastinter
- downinter
- maxconn
- maxqueue
- minconn
- on-error
- port
- rise
- slowstart
- weight
Supported informations, available via "tr/td title":
- cap: capabilities (proxy)
- mode: one of tcp, http or health (proxy)
- id: SNMP ID (proxy, socket, server)
- IP (socket, server)
- cookie (backend, server)
Historically, "option httpclose" has always worked the same way. It
only mangles the "Connection" header in the request and the response
if needed, but does not affect the connection by itself, and ignores
any further data. It is dangerous to change this behaviour without
leaving any other alternative. If an active close is desired, it's
better to make use of "option forceclose" which does exactly what
it intends to do.
So as of now, "option httpclose" will only mangle the headers as
before, and will only affect the connection by itself when combined
with another connection-related option (eg: keepalive or server-close).
Released version 1.4-dev5 with the following main changes :
- [MINOR] server tracking: don't care about the tracked server's mode
- [MEDIUM] appsession: add "len", "prefix" and "mode" options
- [MEDIUM] appsession: add the "request-learn" option
- [BUG] Configuration parser bug when escaping characters
- [MINOR] CSS & HTML fun
- [MINOR] Collect & provide http response codes received from servers
- [BUG] Fix silly typo: hspr_other -> hrsp_other
- [MINOR] Add "a name" to stats page
- [MINOR] add additional "a href"s to stats page
- [MINOR] Collect & provide http response codes for frontends, fix backends
- [DOC] some small spell fixes and unifications
- [MEDIUM] Decrease server health based on http responses / events, version 3
- [BUG] format '%d' expects type 'int', but argument 5 has type 'long int'
- [BUG] config: fix erroneous check on cookie domain names, again
- [BUG] Healthchecks: get a proper error code if connection cannot be completed immediately
- [DOC] trivial fix for man page
- [MINOR] config: report all supported options for the "bind" keyword
- [MINOR] tcp: add support for the defer_accept bind option
- [MINOR] unix socket: report the socket path in case of bind error
- [CONTRIB] halog: support searching by response time
- [DOC] add a reminder about obsolete documents
- [DOC] point to 1.4 doc, not 1.3
- [DOC] option tcp-smart-connect was missing from index
- [MINOR] http: detect connection: close earlier
- [CLEANUP] sepoll: clean up the fd_clr/fd_set functions
- [OPTIM] move some rarely used fields out of fdtab
- [MEDIUM] fd: merge fd_list into fdtab
- [MAJOR] buffer: flag BF_DONT_READ to disable reads when not required
- [MINOR] http: add new transaction flags for keep-alive and content-length
- [MEDIUM] http request: parse connection, content-length and transfer-encoding
- [MINOR] http request: update the TX_SRV_CONN_KA flag on rewrite
- [MINOR] http request: simplify the test of no-data
- [MEDIUM] http request: simplify POST length detection
- [MEDIUM] http request: make use of pre-parsed transfer-encoding header
- [MAJOR] http: create the analyser which waits for a response
- [MINOR] http: pre-set the persistent flags in the transaction
- [MEDIUM] http response: check body length and set transaction flags
- [MINOR] http response: update the TX_CLI_CONN_KA flag on rewrite
- [MINOR] http: remove the last call to stream_int_return
- [IMPORT] import ebtree v5.0 into directory ebtree/
- [MEDIUM] build: switch ebtree users to use new ebtree version
- [CLEANUP] ebtree: remove old unused files
- [BUG] definitely fix regparm issues between haproxy core and ebtree
- [CLEANUP] ebtree: cast to char * to get rid of gcc warning
- [BUILD] missing #ifndef in ebmbtree.h
- [BUILD] missing #ifndef in ebsttree.h
- [MINOR] tools: add hex2i() function to convert hex char to int
- [MINOR] http: create new MSG_BODY sub-states
- [BUG] stream_sock: BUF_INFINITE_FORWARD broke splice on 64-bit platforms
- [DOC] option is "defer-accept", not "defer_accept"
- [MINOR] http: keep pointer to beginning of data
- [BUG] x-original-to: name was not set in default instance
- [MINOR] http: detect tunnel mode and set it in the session
- [BUG] config: fix error message when config file is not found
- [BUG] config: fix wrong handling of too large argument count
- [BUG] config: disable 'option httplog' on TCP proxies
- [BUG] config: fix erroneous check on cookie domain names
- [BUG] config: cookie domain was ignored in defaults sections
- [MINOR] config: support passing multiple "domain" statements to cookies
- [MINOR] ebtree: add functions to lookup non-null terminated strings
- [MINOR] config: don't report error on all subsequent files on failure
- [BUG] second fix for the printf format warning
- [BUG] check_post: limit analysis to the buffer length
- [MEDIUM] http: process request body in a specific analyser
- [MEDIUM] backend: remove HTTP POST parsing from get_server_ph_post()
- [MAJOR] http: completely process the "connection" header
- [MINOR] http: only consider chunk encoding with HTTP/1.1
- [MAJOR] buffers: automatically compute the maximum buffer length
- [MINOR] http: move the http transaction init/cleanup code to proto_http
- [MINOR] http: move 1xx handling earlier to eliminate a lot of ifs
- [MINOR] http: introduce a new synchronisation state : HTTP_MSG_DONE
- [MEDIUM] http: rework chunk-size parser
- [MEDIUM] http: add a new transaction flags indicating if we know the transfer length
- [MINOR] buffers: add buffer_ignore() to skip some bytes
- [BUG] http: offsets are relative to the buffer, not to ->som
- [MEDIUM] http: automatically re-aling request buffer
- [BUG] http: body parsing must consider the start of message
- [MINOR] new function stream_int_cond_close()
- [MAJOR] http: implement body parser
- [BUG] http: typos on several unlikely() around header insertion
- [BUG] stream_sock: wrong max computation on recv
- [MEDIUM] http: rework the buffer alignment logic
- [BUG] buffers: wrong size calculation for displaced data
- [MINOR] stream_sock: prepare for closing when all pending data are sent
- [MEDIUM] http: add two more states for the closing period
- [MEDIUM] http: properly handle "option forceclose"
- [MINOR] stream_sock: add SI_FL_NOLINGER for faster close
- [MEDIUM] http: make forceclose use SI_FL_NOLINGER
- [MEDIUM] session: set SI_FL_NOLINGER when aborting on write timeouts
- [MEDIUM] http: add some SI_FL_NOLINGER around server errors
- [MINOR] config: option forceclose is valid in frontends too
- [BUILD] halog: insufficient include path in makefile
- [MEDIUM] http: make the analyser not rely on msg being initialized anymore
- [MEDIUM] http: make the parsers able to wait for a buffer flush
- [MAJOR] http: add support for option http-server-close
- [BUG] http: ensure we abort data transfer on write error
- [BUG] last fix was overzealous and disabled server-close
- [BUG] http: fix erroneous trailers size computation
- [MINOR] stream_sock: enable MSG_MORE when forwarding finite amount of data
- [OPTIM] http: set MSG_MORE on response when a pipelined request is pending
- [BUG] http: redirects were broken by chunk changes
- [BUG] http: the request URI pointer is relative to the buffer
- [OPTIM] http: don't immediately enable reading on request
- [MINOR] http: move redirect messages to HTTP/1.1 with a content-length
- [BUG] http: take care of errors, timeouts and aborts during the data phase
- [MINOR] http: don't wait for sending requests to the server
- [MINOR] http: make the conditional redirect support keep-alive
- [BUG] http: fix cookie parser to support spaces and commas in values
- [MINOR] config: some options were missing for "redirect"
- [MINOR] redirect: add support for unconditional rules
- [MINOR] config: centralize proxy struct initialization
- [MEDIUM] config: remove the limitation of 10 reqadd/rspadd statements
- [MEDIUM] config: remove the limitation of 10 config files
- [CLEANUP] http: remove a remaining impossible condition
- [OPTIM] http: optimize a bit the construct of the forward loops
This option enables HTTP keep-alive on the client side and close mode
on the server side. This offers the best latency on the slow client
side, and still saves as many resources as possible on the server side
by actively closing connections. Pipelining is supported on both requests
and responses, though there is currently no reason to get pipelined
responses.
This option was disabled for frontends in the configuration because
it was useless in its initial implementation, though it was still
checked in the code. Let's officially enable it now.
The "forceclose" option used to close the output channel to the
server once it started to respond. While this happened to work with
most servers, some of them considered this as a connection abort and
immediately stopped responding.
Now that we're aware of the end of a request and response, we're able
to trivially handle this option and properly close both sides when the
server's response is complete.
During this change it appeared that forwarding could be allowed when
the BF_SHUTW_NOW flag was set on a buffer, which obviously is not
acceptable and was causing some trouble. This has been fixed too and
is the reason for the MEDIUM status on this patch.
Implement decreasing health based on observing communication between
HAProxy and servers.
Changes in this version 2:
- documentation
- close race between a started check and health analysis event
- don't force fastinter if it is not set
- better names for options
- layer4 support
Changes in this version 3:
- add stats
- port to the current 1.4 tree
In some environments it is not possible to rely on any wildcard for a
domain name (eg: .com, .net, .fr...) so it is required to send multiple
domain extensions. (Un)fortunately the syntax check on the domain name
prevented that from being done the dirty way. So let's just build a
domain list when multiple domains are passed on the same line.
(cherry picked from commit 950245ca2b)
To sum up :
- len : it's now the max number of characters for the value, preventing
garbaged results.
- a new option "prefix" is added, this allows to use dynamic cookie
names (e.g. ASPSESSIONIDXXX).
Previously in the thread, I wanted to use the value found with
"capture cookie" but when i started to update the documentation, I
found this solution quite weird. I've made a small rework to not
depend on "capture cookie".
- There's the posssiblity to define the URL parser mode (path parameters
or query string).
