All ACLs which need to process HTTP contents first call this function which
performs all the preliminary tests and also triggers the request parsing if
needed. A macro was written to simplify the code.
As a side effect, it's not required anymore to check for the HTTP ACL before
checking for HTTP contents.
Also add some thoughts about the existing and new design.
Note: an earlier design used the names "head" and "tail" for both sides
of the buffer, but it appears awkward as these words may be understood
in two forms (feed by head, output by tail, or make the newcomers wait
at the tail of the queue). Also there were already a few functions in the
code making use of either terminology. So better avoid this terminology
and use "input" and "output" instead.
- Typo on "dispatch" keyword arguments.
- Reindent some blocks for better parsing by automated tools.
- "option mysql-check" and "option pgsql-check" arguments were not documented
as the others.
This patch is a group commit simplify the parsing of the documenation :
- remove remaining tabulations
- realign some lines
- break lines at 80 columns
- add missing chapters in the summary
- fix chapter numbering format
The Unique ID, is an ID generated with several informations. You can use
a log-format string to customize it, with the "unique-id-format" keyword,
and insert it in the request header, with the "unique-id-header" keyword.
%Fi: Frontend IP
%Fp: Frontend Port
%Si: Server IP
%Sp: Server Port
%Ts: Timestamp
%rt: HTTP request counter
%H: hostname
%pid: PID
+X: Hexadecimal represenation
The +X mode in logformat displays hexadecimal for the following flags
%Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid
rename logformat_write_string() to lf_text()
Optimize size computation
The ACL matches rely on the extract_cookie_value() function as used for
for patterns. This permits ACLs to match cookie values based on the cookie
name instead of having to perform substring matching on the cookie header.
Sometimes it is desirable to forward a particular request to a specific
server without having to declare a dedicated backend for this server. This
can be achieved using the "use-server" rules. These rules are evaluated after
the "redirect" rules and before evaluating cookies, and they have precedence
on them. There may be as many "use-server" rules as desired. All of these
rules are evaluated in their declaration order, and the first one which
matches will assign the server.
Released version 1.5-dev8 with the following main changes :
- MINOR: patch for minor typo (ressources/resources)
- MEDIUM: http: add support for sending the server's name in the outgoing request
- DOC: mention that default checks are TCP connections
- BUG/MINOR: fix options forwardfor if-none when an alternative header name is specified
- CLEANUP: Make check_statuses, analyze_statuses and process_chk static
- CLEANUP: Fix HCHK spelling errors
- BUG/MINOR: fix typo in processing of http-send-name-header
- MEDIUM: log: Use linked lists for loggers
- BUILD: fix declaration inside a scope block
- REORG: log: split send_log function
- MINOR: config: Parse the string of the log-format config keyword
- MINOR: add ultoa, ulltoa, ltoa, lltoa implementations
- MINOR: Date and time fonctions that don't use snprintf
- MEDIUM: log: make http_sess_log use log_format
- DOC: log-format documentation
- MEDIUM: log: use log_format for mode tcplog
- MEDIUM: log-format: backend source address %Bi %Bp
- BUG/MINOR: log-format: fix %o flag
- BUG/MEDIUM: bad length in log_format and __send_log
- MINOR: logformat %st is signed
- BUILD/MINOR: fix the source URL in the spec file
- DOC: acl is http_first_req, not http_req_first
- BUG/MEDIUM: don't trim last spaces from headers consisting only of spaces
- MINOR: acl: add new matches for header/path/url length
- BUILD: halog: make halog build on solaris
- BUG/MINOR: don't use a wrong port when connecting to a server with mapped ports
- MINOR: remove the client/server side distinction in SI addresses
- MINOR: halog: add support for matching queued requests
- DOC: indicate that cookie "prefix" and "indirect" should not be mixed
- OPTIM/MINOR: move struct sockaddr_storage to the tail of structs
- OPTIM/MINOR: make it possible to change pipe size (tune.pipesize)
- BUILD/MINOR: silent a build warning in src/pipe.c (fcntl)
- OPTIM/MINOR: move the hdr_idx pools out of the proxy struct
- MEDIUM: tune.http.maxhdr makes it possible to configure the maximum number of HTTP headers
- BUG/MINOR: fix a segfault when parsing a config with undeclared peers
- CLEANUP: rename possibly confusing struct field "tracked"
- BUG/MEDIUM: checks: fix slowstart behaviour when server tracking is in use
- MINOR: config: tolerate server "cookie" setting in non-HTTP mode
- MEDIUM: buffers: add some new primitives and rework existing ones
- BUG: buffers: don't return a negative value on buffer_total_space_res()
- MINOR: buffers: make buffer_pointer() support negative pointers too
- CLEANUP: kill buffer_replace() and use an inline instead
- BUG: tcp: option nolinger does not work on backends
- CLEANUP: ebtree: remove a few annoying signedness warnings
- CLEANUP: ebtree: clarify licence and update to 6.0.6
- CLEANUP: ebtree: remove 4-year old harmless typo in duplicates insertion code
- CLEANUP: ebtree: remove another typo, a wrong initialization in insertion code
- BUG: ebtree: ebst_lookup() could return the wrong entry
- OPTIM: stream_sock: reduce the amount of in-flight spliced data
- OPTIM: stream_sock: save a failed recv syscall when splice returns EAGAIN
- MINOR: acl: add support for TLS server name matching using SNI
- BUG: http: re-enable TCP quick-ack upon incomplete HTTP requests
- BUG: proto_tcp: don't try to bind to a foreign address if sin_family is unknown
- MINOR: pattern: export the global temporary pattern
- CLEANUP: patterns: get rid of pattern_data_setstring()
- MEDIUM: acl: use temp_pattern to store fetched information in the "method" match
- MINOR: acl: include pattern.h to make pattern migration more transparent
- MEDIUM: pattern: change the pattern data integer from unsigned to signed
- MEDIUM: acl: use temp_pattern to store any integer-type information
- MEDIUM: acl: use temp_pattern to store any address-type information
- CLEANUP: acl: integer part of acl_test is not used anymore
- MEDIUM: acl: use temp_pattern to store any string-type information
- CLEANUP: acl: remove last data fields from the acl_test struct
- MEDIUM: http: replace get_ip_from_hdr2() with http_get_hdr()
- MEDIUM: patterns: the hdr() pattern is now of type string
- DOC: add minimal documentation on how ACLs work internally
- DOC: add a coding-style file
- OPTIM: halog: keep a fast path for the lines-count only
- CLEANUP: silence a warning when building on sparc
- BUG: http: tighten the list of allowed characters in a URI
- MEDIUM: http: block non-ASCII characters in URIs by default
- DOC: add some documentation from RFC3986 about URI format
- BUG/MINOR: cli: correctly remove the whole table on "clear table"
- BUG/MEDIUM: correctly disable servers tracking another disabled servers.
- BUG/MEDIUM: zero-weight servers must not dequeue requests from the backend
- MINOR: halog: add some help on the command line
- BUILD: fix build error on FreeBSD
- BUG: fix double free in peers config error path
- MEDIUM: improve config check return codes
- BUILD: make it possible to look for pcre in the default system paths
- MINOR: config: emit a warning when 'default_backend' masks servers
- MINOR: backend: rework the LC definition to support other connection-based algos
- MEDIUM: backend: add the 'first' balancing algorithm
- BUG: fix httplog trailing LF
- MEDIUM: increase chunk-size limit to 2GB-1
- BUG: queue: fix dequeueing sequence on HTTP keep-alive sessions
- BUG: http: disable TCP delayed ACKs when forwarding content-length data
- BUG: checks: fix server maintenance exit sequence
- BUG/MINOR: stream_sock: don't remove BF_EXPECT_MORE and BF_SEND_DONTWAIT on partial writes
- DOC: enumerate valid status codes for "observe layer7"
- MINOR: buffer: switch a number of buffer args to const
- CLEANUP: silence signedness warning in acl.c
- BUG: stream_sock: si->release was not called upon shutw()
- MINOR: log: use "%ts" to log term status only and "%tsc" to log with cookie
- BUG/CRITICAL: log: fix risk of crash in development snapshot
- BUG/MAJOR: possible crash when using capture headers on TCP frontends
- MINOR: config: disable header captures in TCP mode and complain
The main stats page says "ressources" (French spelling) rather than
"resources" (English spelling).
One little patch attached (against v1.4.20).
Many thanks,
Adrian
The difference could be seen when logging a request in HTTP mode with option
tcplog, as it would keep emitting 4 chars. Better use two distinct flags to
clear the confusion.
%Bi return the backend source IP
%Bp return the backend source port
Add a function pointer in logformat_type to do additional configuration
during the log-format variable parsing.
Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to
log.c
A new field in logformat_type define if you can use a logformat
variable in TCP or HTTP mode.
doc: log-format in tcp mode
Note that due to the way log buffer allocation currently works, trying to
log an HTTP request without "option httplog" is still not possible. This
will change in the near future.
The principle behind this load balancing algorithm was first imagined
and modeled by Steen Larsen then iteratively refined through several
work sessions until it would totally address its original goal.
The purpose of this algorithm is to always use the smallest number of
servers so that extra servers can be powered off during non-intensive
hours. Additional tools may be used to do that work, possibly by
locally monitoring the servers' activity.
The first server with available connection slots receives the connection.
The servers are choosen from the lowest numeric identifier to the highest
(see server parameter "id"), which defaults to the server's position in
the farm. Once a server reaches its maxconn value, the next server is used.
It does not make sense to use this algorithm without setting maxconn. Note
that it can however make sense to use minconn so that servers are not used
at full load before starting new servers, and so that introduction of new
servers requires a progressively increasing load (the number of servers
would more or less follow the square root of the load until maxconn is
reached). This algorithm ignores the server weight, and is more beneficial
to long sessions such as RDP or IMAP than HTTP, though it can be useful
there too.
These ones are invalid and blocked unless "option accept-invalid-http-request"
is specified in the frontend. In any case, the faulty request is logged.
Note that some of the remaining invalid chars are still not checked against,
those are the invalid ones between 32 and 127 :
34 ('"'), 60 ('<'), 62 ('>'), 92 ('\'), 94 ('^'),
96 ('`'), 123 ('{'), 124 ('|'), 125 ('}')
Using a lookup table might be better at some point.
New option "http-send-name-header" specifies the name of a header which
will hold the server name in outgoing requests. This is the name of the
server the connection is really sent to, which means that upon redispatches,
the header's value is updated so that it always matches the server's name.
This pattern previously was limited to type IP. With the new header
extraction function, it becomes possible to extract strings, so that
the header can be returned as a string. This will not change anything
to existing configs, as string will automatically be converted to IP
when needed. However, new configs will be able to use IPv6 addresses
from headers in stick-tables, as well as stick on any non-IP header
(eg: host, user-agent, ...).
Server Name Indication (SNI) is a TLS extension which makes a client
present the name of the server it is connecting to in the client hello.
It allows a transparent proxy to take a decision based on the beginning
of an SSL/TLS stream without deciphering it.
The new ACL "req_ssl_sni" matches the name extracted from the TLS
handshake against a list of names which may be loaded from a file if
needed.
This patch settles the 2 loggers limitation.
Loggers are now stored in linked lists.
Using "global log", the global loggers list content is added at the end
of the current proxy list. Each "log" entries are added at the end of
the proxy list.
"no log" flush a logger list.
For a long time, the max number of headers was taken as a part of the buffer
size. Since the header size can be configured at runtime, it does not make
much sense anymore.
Nothing was making it necessary to have a static value, so let's turn this into
a tunable with a default value of 101 which equals what was previously used.
By default, pipes are the default size for the system. But sometimes when
using TCP splicing, it can improve performance to increase pipe sizes,
especially if it is suspected that pipes are not filled and that many
calls to splice() are performed. This has an impact on the kernel's
memory footprint, so this must not be changed if impacts are not understood.
When prefix and indirect are used together, a client which connects to
a server with a cookie will never get any cookie update from this server,
which will be removed by the "indirect" option.
This patch introduces hdr_len, path_len and url_len for matching these
respective parts lengths against integers. This can be used to detect
abuse or empty headers.
Released version 1.5-dev7 with the following main changes :
- [BUG] fix binary stick-tables
- [MINOR] http: *_dom matching header functions now also split on ":"
- [BUG] checks: fix support of Mysqld >= 5.5 for mysql-check
- [MINOR] acl: add srv_conn acl to count connections on a specific backend server
- [MINOR] check: add redis check support
- [DOC] small fixes to clearly distinguish between keyword and variables
- [MINOR] halog: add support for termination code matching (-tcn/-TCN)
- [DOC] Minor spelling fixes and grammatical enhancements
- [CLEANUP] dumpstats: make symbols static where possible
- [MINOR] Break out dumping table
- [MINOR] Break out processing of clear table
- [MINOR] Allow listing of stick table by key
- [MINOR] Break out all stick table socat command parsing
- [MINOR] More flexible clearing of stick table
- [MINOR] Allow showing and clearing by key of ipv6 stick tables
- [MINOR] Allow showing and clearing by key of integer stick tables
- [MINOR] Allow showing and clearing by key of string stick tables
- [CLEANUP] Remove assigned but unused variables
- [CLEANUP] peers.h: fix declarations
- [CLEANUP] session.c: Make functions static where possible
- [MINOR] Add active connection list to server
- [MINOR] Allow shutdown of sessions when a server becomes unavailable
- [MINOR] Add down termination condition
- [MINOR] Make appsess{,ion}_refresh static
- [MINOR] Add rdp_cookie pattern fetch function
- [CLEANUP] Remove unnecessary casts
- [MINOR] Add non-stick server option
- [MINOR] Consistently use error in tcp_parse_tcp_req()
- [MINOR] Consistently free expr on error in cfg_parse_listen()
- [MINOR] Free rdp_cookie_name on denint()
- [MINOR] Free tcp rules on denint()
- [MINOR] Free stick table pool on denint()
- [MINOR] Free stick rules on denint()
- [MEDIUM] Fix stick-table replication on soft-restart
- [MEDIUM] Correct ipmask() logic
- [MINOR] Correct type in table dump examples
- [MINOR] Fix build error in stream_int_register_handler()
- [MINOR] Use DPRINTF in assign_server()
- [BUG] checks: http-check expect could fail a check on multi-packet responses
- [DOC] fix minor typo in the "dispatch" doc
- [BUG] proto_tcp: fix address binding on remote source
- [MINOR] http: don't report the "haproxy" word on the monitoring response
- [REORG] http: move HTTP error codes back to proto_http.h
- [MINOR] http: make the "HTTP 200" status code configurable.
- [MINOR] http: partially revert the chunking optimization for now
- [MINOR] stream_sock: always clear BF_EXPECT_MORE upon complete transfer
- [CLEANUP] stream_sock: remove unneeded FL_TCP and factor out test
- [MEDIUM] http: add support for "http-no-delay"
- [OPTIM] http: optimize chunking again in non-interactive mode
- [OPTIM] stream_sock: avoid fast-forwarding of partial data
- [OPTIM] stream_sock: don't use splice on too small payloads
- [MINOR] config: make it possible to specify a cookie even without a server
- [BUG] stats: support url-encoded forms
- [MINOR] config: automatically compute a default fullconn value
- [CLEANUP] config: remove some left-over printf debugging code from previous patch
- [DOC] add missing entry or stick store-response
- [MEDIUM] http: add support for 'cookie' and 'set-cookie' patterns
- [BUG] halog: correctly handle truncated last line
- [MINOR] halog: make SKIP_CHAR stop on field delimiters
- [MINOR] halog: add support for HTTP log matching (-H)
- [MINOR] halog: gain back performance before SKIP_CHAR fix
- [OPTIM] halog: cache some common fields positions
- [OPTIM] halog: check once for correct line format and reuse the pointer
- [OPTIM] halog: remove many 'if' by using a function pointer for the filters
- [OPTIM] halog: remove support for tab delimiters in input data
- [BUG] session: risk of crash on out of memory (1.5-dev regression)
- [MINOR] session: try to emit a 500 response on memory allocation errors
- [OPTIM] stream_sock: reduce the default number of accepted connections at once
- [BUG] stream_sock: disable listener when system resources are exhausted
- [MEDIUM] proxy: add a PAUSED state to listeners and move socket tricks out of proxy.c
- [BUG] stream_sock: ensure orphan listeners don't accept too many connections
- [MINOR] listeners: add listen_full() to mark a listener full
- [MINOR] listeners: add support for queueing resource limited listeners
- [MEDIUM] listeners: put listeners in queue upon resource shortage
- [MEDIUM] listeners: queue proxy-bound listeners at the proxy's
- [MEDIUM] listeners: don't stop proxies when global maxconn is reached
- [MEDIUM] listeners: don't change listeners states anymore in maintain_proxies
- [CLEANUP] proxy: rename a few proxy states (PR_STIDLE and PR_STRUN)
- [MINOR] stats: report a "WAITING" state for sockets waiting for resource
- [MINOR] proxy: make session rate-limit more accurate
- [MINOR] sessions: only wake waiting listeners up if rate limit is OK
- [BUG] proxy: peers must only be stopped once, not upon every call to maintain_proxies
- [CLEANUP] proxy: merge maintain_proxies() operation inside a single loop
- [MINOR] task: new function task_schedule() to schedule a wake up
- [MAJOR] proxy: finally get rid of maintain_proxies()
- [BUG] proxy: stats frontend and peers were missing many initializers
- [MEDIUM] listeners: add a global listener management task
- [MINOR] proxy: make findproxy() return proxies from numeric IDs too
- [DOC] fix typos, "#" is a sharp, not a dash
- [MEDIUM] stats: add support for changing frontend's maxconn at runtime
- [MEDIUM] checks: group health checks methods by values and save option bits
- [MINOR] session-counters: add the ability to clear the counters
- [BUG] check: http-check expect + regex would crash in defaults section
- [MEDIUM] http: make x-forwarded-for addition conditional
- [REORG] build: move syscall redefinition to specific places
- [CLEANUP] update the year in the copyright banner
- [BUG] possible crash in 'show table' on stats socket
- [BUG] checks: use the correct destination port for sending checks
- [BUG] backend: risk of picking a wrong port when mapping is used with crossed families
- [MINOR] make use of set_host_port() and get_host_port() to get rid of family mismatches
- [DOC] fixed a few "sensible" -> "sensitive" errors
- [MINOR] make use of addr_to_str() and get_host_port() to replace many inet_ntop()
- [BUG] http: trailing white spaces must also be trimmed after headers
- [MINOR] stats: display "<NONE>" instead of the frontend name when unknown
- [MINOR] http: take a capture of too large requests and responses
- [MINOR] http: take a capture of truncated responses
- [MINOR] http: take a capture of bad content-lengths.
- [DOC] add a few old and uncommitted docs
- [CLEANUP] cfgparse: fix reported options for the "bind" keyword
- [MINOR] halog: add -hs/-HS to filter by HTTP status code range
- [MINOR] halog: support backslash-escaped quotes
- [CLEANUP] remove dirty left-over of a debugging message
- [MEDIUM] stats: disable complex socket reservation for stats socket
- [CLEANUP] remove a useless test in manage_global_listener_queue()
- [MEDIUM] stats: add the "set maxconn" setting to the command line interface
- [MEDIUM] add support for global.maxconnrate to limit the per-process conn rate.
- [MINOR] stats: report the current and max global connection rates
- [MEDIUM] stats: add the ability to adjust the global maxconnrate
- [BUG] peers: don't pre-allocate 65000 connections to each peer
- [MEDIUM] don't limit peers nor stats socket to maxconn nor maxconnrate
- [BUG] peers: the peer frontend must not emit any log
- [CLEANUP] proxy: make pause_proxy() perform the required controls and emit the logs
- [BUG] peers: don't keep a peers section which has a NULL frontend
- [BUG] peers: ensure the peers are resumed if they were paused
- [MEDIUM] stats: add the ability to enable/disable/shutdown a frontend at runtime
- [MEDIUM] session: make session_shutdown() an independant function
- [MEDIUM] stats: offer the possibility to kill a session from the CLI
- [CLEANUP] stats: centralize tests for backend/server inputs on the CLI
- [MEDIUM] stats: offer the possibility to kill sessions by server
- [MINOR] halog: do not consider byte 0x8A as end of line
- [MINOR] frontend: ensure debug message length is always initialized
- [OPTIM] halog: make fgets parse more bytes by blocks
- [OPTIM] halog: add assembly version of the field lookup code
- [MEDIUM] poll: add a measurement of idle vs work time
- [CLEANUP] startup: report only the basename in the usage message
- [MINOR] startup: add an option to change to a new directory
- [OPTIM] task: don't scan the run queue if we know it's empty
- [BUILD] stats: stdint is not present on solaris
- [DOC] update the README file to reflect new naming rules for patches
- [MINOR] stats: report the number of requests intercepted by the frontend
- [DOC] update ROADMAP file
We already had the ability to kill a connection, but it was only
for the checks. Now we can do this for any session, and for this we
add a specific flag "K" to the logs.
The stats socket now allows the admin to disable, enable or shutdown a frontend.
This can be used when a bug is discovered in a configuration and it's desirable
to fix it but the rules in place don't allow to change a running config. Thus it
becomes possible to kill the frontend to release the port and start a new one in
a separate process.
This can also be used to temporarily make haproxy return TCP resets to incoming
requests to pretend the service is not bound. For instance, this may be useful
to quickly flush a very deep SYN backlog.
The frontend check and lookup code was factored with the "set maxconn" usage.
This one enforces a per-process connection rate limit, regardless of what
may be set per frontend. It can be a way to limit the CPU usage of a process
being severely attacked.
The side effect is that the global process connection rate is now measured
for each incoming connection, so it will be possible to report it.
This option permits to change the global maxconn setting within the
limit that was set by the initial value, which is now reported as the
hard maxconn value. This allows to immediately accept more concurrent
connections or to stop accepting new ones until the value passes below
the indicated setting.
The main use of this option is on systems where many haproxy instances
are loaded and admins need to re-adjust resource sharing at run time
to regain a bit of fairness between processes.
If "option forwardfor" has the "if-none" argument, then the header is
only added when the request did not already have one. This option has
security implications, and should not be set blindly.
Manoj Kumar reported a case where haproxy would crash upon start-up. The
cause was an "http-check expect" statement declared in the defaults section,
which caused a NULL regex to be used during the check. This statement is not
allowed in defaults sections precisely because this requires saving a copy
of the regex in the default proxy. But the check was not made to prevent it
from being declared there, hence the issue.
Instead of adding code to detect its abnormal use, we decided to implement
it. It was not that much complex because the expect_str part was not used
with regexes, so it could hold the string form of the regex in order to
compile it again for every backend (there's no way to clone regexes).
This patch has been tested and works. So it's both a bugfix and a minor
feature enhancement.
It should be backported to 1.4 though it's not critical since the config
was not supposed to be supported.
This patch provides a new "option redis-check" statement to enable server health checks based on redis PING request (http://www.redis.io/commands/ping).
The new "set maxconn frontend XXX" statement on the stats socket allows
the admin to change a frontend's maxconn value. If some connections are
queued, they will immediately be accepted up to the new limit. If the
limit is lowered, new connections acceptation might be delayed. This can
be used to temporarily reduce or increase the impact of a specific frontend's
traffic on the whole process.
