We now set msg->col and msg->sov to the first byte of non-header.
They will be used later when parsing chunks. A new macro was added
to perform size additions on an http_msg in order to limit the risks
of copy-paste in the long term.
During this operation, it appeared that the http_msg struct was not
optimal on 64-bit, so it was re-ordered to fill the holes.
Yohan Tordjman at Dstorage found that upgrading haproxy to 1.4-dev4
caused truncated objects to be returned. An strace quickly exhibited
the issue which was 100% reproducible :
4297 epoll_wait(0, {}, 10, 0) = 0
4297 epoll_wait(0, {{EPOLLIN, {u32=7, u64=7}}}, 10, 1000) = 1
4297 splice(0x7, 0, 0x5, 0, 0xffffffffffffffff, 0x3) = -1 EINVAL (Invalid argument)
4297 shutdown(7, 1 /* send */) = 0
4297 close(7) = 0
4297 shutdown(2, 1 /* send */) = 0
4297 close(2) = 0
This is caused by the fact that the forward length is taken from
BUF_INFINITE_FORWARD, which is -1. The problem does not appear
in 32-bit mode because this value is first cast to an unsigned
long, truncating it to 32-bit (4 GB). Setting an upper bound
fixes the issue.
Also, a second error check has been added for splice. If EINVAL
is returned, we fall back to recv().
An HTTP message can be decomposed into several sub-states depending
on the transfer-encoding. We'll have to keep these state information
while parsing chunks, so we must extend the values. In order not to
change everything, we'll now consider that anything >= MSG_BODY is
the body, and that the value indicates the precise state. The
MSG_ERROR status which was greater than MSG_BODY was moved for this.
Right now, an HTTP server cannot track a TCP server and vice-versa.
This patch enables proxy tracking without relying on the proxy's mode
(tcp/http/health). It only requires a matching proxy name to exist. The
original function was renamed to findproxy_mode().
This patch extends and corrects the functionality introduced by
"Collect & provide http response codes received from servers":
- responses are now also accounted for frontends
- backend's and frontend's counters are incremented based
on responses sent to client, not received from servers
This patch adds <a href> html links for proxies, frontends, servers
and backends. Once located, can be clicked. Users no longer have to
manually add #anchor to stat's url.
It's a pain to enable regparm because ebtree is built in its corner
and does not depend on the rest of the config. This causes no problem
except that if the regparm settings are not exactly similar, then we
can get inconsistent function interfaces and crashes.
One solution realized in this patch consists in externalizing all
compiler settings and changing CONFIG_XXX_REGPARM into CONFIG_REGPARM
so that we ensure that any sub-component uses the same setting. Since
ebtree used a value here and not a boolean, haproxy's config has been
set to use a number too. Both haproxy's core and ebtree currently use
the same copy of the compiler.h file. That way we don't have any issue
anymore when one setting changes somewhere.
All files referencing the previous ebtree code were changed to point
to the new one in the ebtree directory. A makefile variable (EBTREE_DIR)
is also available to use files from another directory.
The ability to build the libebtree library temporarily remains disabled
because it can have an impact on some existing toolchains and does not
appear worth it in the medium term if we add support for multi-criteria
stickiness for instance.
We needed to upgrade ebtree to v5.0 to support string indexing,
and it was getting very painful to have it split across 2 dirs
and to have to patch it. Now we just have to copy the .c and .h
files to the right place.
We also check the close status and terminate the server persistent
connection if appropriate. Note that since this change, we'll not
get any "Connection: close" headers added to HTTP/1.0 responses
anymore, which is good.
The code part which waits for an HTTP response has been extracted
from the old function. We now have two analysers and the second one
may re-enable the first one when an 1xx response is encountered.
This has been tested and works.
The calls to stream_int_return() that were remaining in the wait
analyser have been converted to stream_int_retnclose().
Store those elements in the transaction. RFC2616 is strictly followed.
Note that requests containing two different content-length fields are
discarded as invalid.
This patch has 2 goals :
1. I wanted to test the appsession feature with a small PHP code,
using PHPSESSID. The problem is that when PHP gets an unknown session
id, it creates a new one with this ID. So, when sending an unknown
session to PHP, persistance is broken : haproxy won't see any new
cookie in the response and will never attach this session to a
specific server.
This also happens when you restart haproxy : the internal hash becomes
empty and all sessions loose their persistance (load balancing the
requests on all backend servers, creating a new session on each one).
For a user, it's like the service is unusable.
The patch modifies the code to make haproxy also learn the persistance
from the client : if no session is sent from the server, then the
session id found in the client part (using the URI or the client cookie)
is used to associated the server that gave the response.
As it's probably not a feature usable in all cases, I added an option
to enable it (by default it's disabled). The syntax of appsession becomes :
appsession <cookie> len <length> timeout <holdtime> [request-learn]
This helps haproxy repair the persistance (with the risk of losing its
session at the next request, as the user will probably not be load
balanced to the same server the first time).
2. This patch also tries to reduce the memory usage.
Here is a little example to explain the current behaviour :
- Take a Tomcat server where /session.jsp is valid.
- Send a request using a cookie with an unknown value AND a path
parameter with another unknown value :
curl -b "JSESSIONID=12345678901234567890123456789012" http://<haproxy>/session.jsp;jsessionid=00000000000000000000000000000001
(I know, it's unexpected to have a request like that on a live service)
Here, haproxy finds the URI session ID and stores it in its internal
hash (with no server associated). But it also finds the cookie session
ID and stores it again.
- As a result, session.jsp sends a new session ID also stored in the
internal hash, with a server associated.
=> For 1 request, haproxy has stored 3 entries, with only 1 which will be usable
The patch modifies the behaviour to store only 1 entry (maximum).
When processing a GET or HEAD request in close mode, we know we don't
need to read anything anymore on the socket, so we can disable it.
Doing this can save up to 40% of the recv calls, and half of the
epoll_ctl calls.
For this we need a buffer flag indicating that we're not interesting in
reading anymore. Right now, this flag also disables both polled reads.
We might benefit from disabling only speculative reads, but we will need
at least this flag when we want to support keepalive anyway.
Currently we don't disable the flag on completion, but it does not
matter as we close ASAP when performing the shutw().
The fd_list[] used by sepoll was indexed on the fd number and was only
used to store the equivalent of an integer. Changing it to be merged
with fdtab reduces the number of pointer computations, the code size
and some initialization steps. It does not harm other pollers much
either, as only one integer was added to the fdtab array.
Some rarely information are stored in fdtab, making it larger for no
reason (source port ranges, remote address, ...). Such information
lie there because the checks can't find them anywhere else. The goal
will be to move these information to the stream interface once the
checks make use of it.
For now, we move them to an fdinfo array. This simple change might
have improved the cache hit ratio a little bit because a 0.5% of
performance increase has measured.
Till now we would only set SN_CONN_CLOSED after rewriting it. Now we
set it just after checking the Connection header so that we can use
the result later if required.
I'm working on helping Arnaud update haproxy in Debian, and one of the
package build warnings I received was about "hyphen where a minus sign
was intended" in the man page - details:
http://lintian.debian.org/tags/hyphen-used-as-minus-sign.html
Patch included in my 1.3.20 Debian package is attached.
This can ensure that data is readily available on a socket when
we accept it, but a bug in the kernel ignores the timeout so the
socket can remain pending as long as the client does not talk.
Use with care.
This patch makes stats page about 30% smaller and
"CSS 2.1" + "HTML 4.01 Transitional" compliant.
There should be no visible differences.
Changes:
- add DOCTYPE for HTML 4.01 Transitional
- add missing </ul>
- remove cols=, AFAIK no modern browser support this property and
it prevents validation to pass.
- remove "align: center": there is no such property in css. There is
however "text-align: center" but it is definitely not what we would
like to see here.
- by default align .titre to center
- by default align .td to right
- remove all align=right, no longer necessary
- add class=ac (align center): shorter than "align=center" and use it when
necessary
- remove nowrap from td, instead use "white-space: nowrap" in css
Now stats page passes W3C validators for HTML & CSS. We may consider adding
"validated" icons from www.w3.org. ;)
Released version 1.4-dev4 with the following main changes :
- [DOC] add missing rate_lim and rate_max
- [MAJOR] struct chunk rework
- [MEDIUM] Health check reporting code rework + health logging, v3
- [BUG] check if rise/fall has an argument and it is > 0
- [MINOR] health checks logging unification
- [MINOR] add "description", "node" and show-node"/"show-desc", remove "node-name", v2
- [MINOR] Allow dots in show-node & add "white-space: nowrap" in th.pxname.
- [DOC] Add information about http://haproxy.1wt.eu/contrib.html
- [MINOR] Introduce include/types/counters.h
- [CLEANUP] Move counters to dedicated structures
- [MINOR] Add "clear counters" to clear statistics counters
- [MEDIUM] Collect & provide separate statistics for sockets, v2
- [BUG] Fix NULL pointer dereference in stats_check_uri_auth(), v2
- [MINOR] acl: don't report valid acls as potential mistakes
- [MINOR] Add cut_crlf(), ltrim(), rtrim() and alltrim()
- [MINOR] Add chunk_htmlencode and chunk_asciiencode
- [MINOR] Capture & display more data from health checks, v2
- [BUG] task.c: don't assing last_timer to node-less entries
- [BUG] http stats: large outputs sometimes got some parts chopped off
- [MINOR] backend: export some functions to recount servers
- [MINOR] backend: uninline some LB functions
- [MINOR] include time.h from freq_ctr.h as is uses "now".
- [CLEANUP] backend: move LB algos to individual files
- [MINOR] lb_map: reorder code in order to ease integration of new hash functions
- [CLEANUP] proxy: move last lb-specific bits to their respective files
- [MINOR] backend: separate declarations of LB algos from their lookup method
- [MINOR] backend: reorganize the LB algorithm selection
- [MEDIUM] backend: introduce the "static-rr" LB algorithm
- [MINOR] report list of supported pollers with -vv
- [DOC] log-health-checks is an option, not a directive
- [MEDIUM] new option "independant-streams" to stop updating read timeout on writes
- [BUG] stats: don't call buffer_shutw(), but ->shutw() instead
- [MINOR] stats: strip CR and LF from the input command line
- [BUG] don't refresh timeouts late after detected activity
- [MINOR] stats_dump_errors_to_buffer: use buffer_feed_chunk()
- [MINOR] stats_dump_sess_to_buffer: use buffer_feed_chunk()
- [MINOR] stats: make stats_dump_raw_to_buffer() use buffer_feed_chunk
- [MEDIUM] stats: don't use s->ana_state anymore
- [MINOR] remove now obsolete ana_state from the session struct
- [MEDIUM] stats: make HTTP stats use an I/O handler
- [MEDIUM] stream_int: adjust WAIT_ROOM handling
- [BUG] config: look for ID conflicts in all sockets, not only last ones.
- [MINOR] config: reference file and line with any listener/proxy/server declaration
- [MINOR] config: report places of duplicate names or IDs
- [MINOR] config: add pointer to file name in block/redirect/use_backend/monitor rules
- [MINOR] tools: add a new get_next_id() function
- [MEDIUM] config: automatically find unused IDs for proxies, servers and listeners
- [OPTIM] counters: move some max numbers to the counters struct
- [BUG] counters: fix segfault on missing counters for a listener
- [MEDIUM] backend: implement consistent hashing variation
- [MINOR] acl: add fe_conn, be_conn, queue, avg_queue
- [MINOR] stats: use 'clear counters all' to clear all values
- [MEDIUM] add access restrictions to the stats socket
- [MINOR] buffers: add buffer_feed2() and make buffer_feed() measure string length
- [MINOR] proxy: provide function to retrieve backend/server pointers
- [MINOR] add the "initial weight" to the server struct.
- [MEDIUM] stats: add the "get weight" command to report a server's weight
- [MEDIUM] stats: add the "set weight" command
- [BUILD] add a 'make tags' target
- [MINOR] stats: add support for numeric IDs in set weight/get weight
- [MINOR] stats: use a dedicated state to output static data
- [OPTIM] stats: check free space before trying to print
This alone makes a typical HTML stats dump consume 10% CPU less,
because we avoid doing complex printf calls to drop them later.
Only a few common cases have been checked, those which are very
likely to run for nothing.
It is a bit expensive and complex to use to call buffer_feed()
directly from the request parser, and there are risks that some
output messages are lost in case of buffer full. Since most of
these messages are static, let's have a state dedicated to print
these messages and store them in a specific area shared with the
stats in the session. This both reduces code size and risks of
losing output data.
Krzysztof reported that using names only for get weight/set weight
was not enough because it's still possible to have multiple servers
with the same name (and my test config is one of those). He suggested
to be able to designate them by their unique numeric IDs by prefixing
the ID with a dash.
That way we can have :
set weight #120/#2
as well as
get weight static/srv1 10
Capture & display more data from health checks, like
strerror(errno) for L4 failed checks or a first line
from a response for L7 successes/failed checks.
Non ascii or control characters are masked with
chunk_htmlencode() (html stats) or chunk_asciiencode() (logs).
