The run queue scheduler now considers task->nice to queue a task and
to pick a task out of the queue. This makes it possible to boost the
access to statistics (both via HTTP and UNIX socket). The UNIX socket
receives twice as much a boost as the HTTP socket because it is more
sensible.
We now insert tasks in a certain sequence in the run queue.
The sorting key currently is the arrival order. It will now
be possible to apply a "nice" value to any task so that it
goes forwards or backwards in the run queue.
The calls to wake_expired_tasks() and maintain_proxies()
have been moved to the main run_poll_loop(), because they
had nothing to do in process_runnable_tasks().
The task_wakeup() function is not inlined anymore, as it was
only used at one place.
The qlist member of the task structure has been removed now.
The run_queue list has been replaced for an integer indicating
the number of tasks in the run queue.
I got a report that Intel C++ complains about the size of the
base64tab in base64.c. Setting it to 65 chars to allow for the
trailing zero fixes the problem.
The wait queues now rely on 4 trees for past, present and future
timers. The computations are cleaner and more reliable. The
wake_expired_tasks function has become simpler. Also, a bug
previously introduced in task_queue() by the first introduction
of eb_trees has been fixed (the eb->key was never updated).
This new time value will be used to compute timeouts and wait queue
positions. The operation is made once for all when time is retrieved.
A future improvement might consist in having it in ticks of 1/1024
second and to convert all timeouts into ticks.
The following config makes haproxy segfault on exit :
defaults
mode http
balance roundrobin
listen no-stats
bind :8001
listen stats
bind :8002
stats uri /stats
The simple fix is to ensure that p->uri_auth is not NULL
before dereferencing it.
The ultree code has been removed in favor of a simpler and
cleaner ebtree implementation. The eternity queue does not
need to exist anymore, and the pool_tree64 has been removed.
The ebtree node is stored in the task itself. The qlist list
header is still used by the run-queue, but will be able to
disappear once the run-queue uses ebtree too.
The first implementation of the monotonic clock did not verify
forward jumps. The consequence is that a fast changing time may
expire a lot of tasks. While it does seem minor, in fact it is
problematic because most machines which boot with a wrong date
are in the past and suddenly see their time jump by several
years in the future.
The solution is to check if we spent more apparent time in
a poller than allowed (with a margin applied). The margin
is currently set to 1000 ms. It should be large enough for
any poll() to complete.
Tests with randomly jumping clock show that the result is quite
accurate (error less than 1 second at every change of more than
one second).
If the system date is set backwards while haproxy is running,
some scheduled events are delayed by the amount of time the
clock went backwards. This is particularly problematic on
systems where the date is set at boot, because it seldom
happens that health-checks do not get sent for a few hours.
Before switching to use clock_gettime() on systems which
provide it, we can at least ensure that the clock is not
going backwards and maintain two clocks : the "date" which
represents what the user wants to see (mostly for logs),
and an internal date stored in "now", used for scheduled
events.
The dequeuing logic was completely wrong. First, a task was assigned
to all servers to process the queue, but this task was never scheduled
and was only woken up on session free. Second, there was no reservation
of server entries when a task was assigned a server. This means that
as long as the task was not connected to the server, its presence was
not accounted for. This was causing trouble when detecting whether or
not a server had reached maxconn. Third, during a redispatch, a session
could lose its place at the server's and get blocked because another
session at the same moment would have stolen the entry. Fourth, the
redispatch option did not work when maxqueue was reached for a server,
and it was not possible to do so without indefinitely hanging a session.
The root cause of all those problems was the lack of pre-reservation of
connections at the server's, and the lack of tracking of servers during
a redispatch. Everything relied on combinations of flags which could
appear similarly in quite distinct situations.
This patch is a major rework but there was no other solution, as the
internal logic was deeply flawed. The resulting code is cleaner, more
understandable, uses less magics and is overall more robust.
As an added bonus, "option redispatch" now works when maxqueue has
been reached on a server.
The reported queue position in the logs was 0 for the first pending request
in the queue, which is wrong because it means that one request will have to
be completed before the queued one may execute. It caused the undesired side
effect that 0/0 was reported when either 0 or 1 request was pending in the
queue. Thus, we have to increment the queue size before reporting the value.
When a server terminates a connection, the next session in its
own queue was immediately processed. Because of this, if all
server queues are always filled, then no new anonymous request
will be processed. Consider oldest request between global and
server queues to choose from which to pick the request.
An improvement over this will consist in adding a configurable
offset when comparing expiration dates, so that cookie-less
requests can get either less or more priority.
Under some circumstances, a task may already lie in the run queue
(eg: inter-task wakeup). It is disastrous to wait for an event in
this case because some processing gets delayed.
The new TRACE macro is used almost like fprintf, except that a session
has to be passed instead of the file descriptor. It displays infos about
where it is called, session ptr and id, etc...
Reported by Cherife Li : just doing a "make install" fails because it
depends on "all" which is equivalent to "help" if no TARGET was specified.
Make it depend on "haproxy" instead.
A new "redirect" keyword adds the ability to send an HTTP 301/302/303
redirection to either an absolute location or to a prefix followed by
the original URI. The redirection is conditionned by ACL rules, so it
becomes very easy to move parts of a site to another site using this.
This work was almost entirely done at Exceliance by Emeric Brun.
A test-case has been added in the tests/ directory.
- free oldpids
- call free(exp->preg), not only regfree(exp->preg): req_exp, rsp_exp
- build a list of unique uri_auths and eventually free it
- prune_acl_cond/free for switching_rules
- add a callback pointer to free ptr from acl_pattern (used for regexs) and execute it
==1180== malloc/free: in use at exit: 0 bytes in 0 blocks.
==1180== malloc/free: 5,599 allocs, 5,599 frees, 4,220,556 bytes allocated.
==1180== All heap blocks were freed -- no leaks are possible.
New functions implemented:
- deinit_pollers: called at the end of deinit())
- prune_acl: called via list_for_each_entry_safe
Add missing pool_destroy2 calls:
- p->hdr_idx_pool
- pool2_tree64
Implement all task stopping:
- health-check: needs new "struct task" in the struct server
- queue processing: queue_mgt
- appsess_refresh: appsession_refresh
before (idle system):
==6079== LEAK SUMMARY:
==6079== definitely lost: 1,112 bytes in 75 blocks.
==6079== indirectly lost: 53,356 bytes in 2,090 blocks.
==6079== possibly lost: 52 bytes in 1 blocks.
==6079== still reachable: 150,996 bytes in 504 blocks.
==6079== suppressed: 0 bytes in 0 blocks.
after (idle system):
==6945== LEAK SUMMARY:
==6945== definitely lost: 7,644 bytes in 137 blocks.
==6945== indirectly lost: 9,913 bytes in 587 blocks.
==6945== possibly lost: 0 bytes in 0 blocks.
==6945== still reachable: 0 bytes in 0 blocks.
==6945== suppressed: 0 bytes in 0 blocks.
before (running system for ~2m):
==9343== LEAK SUMMARY:
==9343== definitely lost: 1,112 bytes in 75 blocks.
==9343== indirectly lost: 54,199 bytes in 2,122 blocks.
==9343== possibly lost: 52 bytes in 1 blocks.
==9343== still reachable: 151,128 bytes in 509 blocks.
==9343== suppressed: 0 bytes in 0 blocks.
after (running system for ~2m):
==11616== LEAK SUMMARY:
==11616== definitely lost: 7,644 bytes in 137 blocks.
==11616== indirectly lost: 9,981 bytes in 591 blocks.
==11616== possibly lost: 0 bytes in 0 blocks.
==11616== still reachable: 4 bytes in 1 blocks.
==11616== suppressed: 0 bytes in 0 blocks.
Still not perfect but significant improvement.
Add the ability to detect streaming buffers, and set a
flag indicating it. It will later serve us in order to
dynamically resize them, and to prioritize file descriptors
during polls.
If too many events are set for spec I/O, those ones can starve the
polled events. Experiments show that when polled events starve, they
quickly turn into spec I/O, making the situation even worse. While
we can reduce the number of polled events processed at once, we
cannot do this on speculative events because most of them are new
ones (avg 2/3 new - 1/3 old from experiments).
The solution against this problem relies on those two factors :
1) one FD registered as a spec event cannot be polled at the same time
2) even during very high loads, we will almost never be interested in
simultaneous read and write streaming on the same FD.
The first point implies that during starvation, we will not have more than
half of our FDs in the poll list, otherwise it means there is less than that
in the spec list, implying there is no starvation.
The second point implies that we're statically only interested in half of
the maximum number of file descriptors at once, because we will unlikely
have simultaneous read and writes for a same buffer during long periods.
So, if we make it possible to drain maxsock/2/2 during peak loads, then we
can ensure that there will be no starvation effect. This means that we must
always allocate maxsock/4 events for the poller.
Last, sepoll uses an optimization consisting in reducing the number of calls
to epoll_wait() to once every too polls. However, when dealing with many
spec events, we can wait very long and skipping epoll_wait() every second
time increases latency. For this reason, we try to detect if we are beyond
a reasonable limit and stop doing so at this stage.
This patch allows to specify a domain used when inserting a cookie
providing a session stickiness. Usefull for example with wildcard domains.
The patch adds one new variable to the struct proxy: cookiedomain.
When set the domain is appended to a Set-Cookie header.
Domain name is validated using the new invalid_domainchar() function.
It is basically invalid_char() limited to [A-Za-z0-9_.-]. Yes, the test
is too trivial and does not cover all wrong situations, but the main
purpose is to detect most common mistakes, not intentional abuses.
The underscore ("_") character is not RFC-valid but as it is
often (mis)used so I decided to allow it.
This patch adds two optional arguments "len" and "depth" to
"balance uri". They are used to limit the length in characters
of the analysis, as well as the number of directory components
it applies to.
For Fedora 9 gcc 4.3 will be shipping as a feature, and right now haproxy does
not compile with gcc 4.3.
It appears that there is a reordering of headers or something along those lines,
This is the patch that gets haproxy to compile with gcc 4.3. I'm not sure if
this is the correct approach you would want to use, so please correct me.
If this works for you, I'll go ahead and put this patch in the src rpm until a
release of haproxy which compiles with gcc 4.3 is released.
About: [BUG] Flush buffers also where there are exactly 0 bytes left
I'm also attaching a debug patch that helps to trigger this bug.
Without the fix:
# echo -ne "GET /haproxy?stats;csv;norefresh HTTP/1.0\r\n\r\n"|nc 127.0.0.1
801|wc -c
16384
With the fix:
# echo -ne "GET /haproxy?stats;csv;norefresh HTTP/1.0\r\n\r\n"|nc 127.0.0.1
801|wc -c
33089
Best regards,
Krzysztof Oledzki
I noticed it was possible to get truncated http/csv stats. Sometimes.
Usually the problem disappeared as fast as it appeared, but once it
happend that my http-stats page was truncated for about one hour.
It was quite weird as it happened independently for csv and http
output and it took me some time to track & fix this bug.
Both buffer_write & buffer_write_chunk used to return 0 in two
situations: is case of success or where there was exactly 0 bytes
left. The first one is intentional but I believe the second one
is not as it was not possible to distinguish between successful
write and unsuccessful one, which means that if the buffer was 100%
filled, it was never flushed and it was not possible to write
more data.
This patch fixes this problem.
Released version 1.3.15 with the following main changes :
- [BUILD] Added support for 'make install'
- [BUILD] Added 'install-man' make target for installing the man page
- [BUILD] Added 'install-bin' make target
- [BUILD] Added 'install-doc' make target
- [BUILD] Removed "/" after '$(DESTDIR)' in install targets
- [BUILD] Changed 'install' target to install the binaries first
- [BUILD] Replace hardcoded 'LD = gcc' with 'LD = $(CC)'
- [MEDIUM]: Inversion for options
- [MEDIUM]: Count retries and redispatches also for servers, fix redistribute_pending, extend logs, %d->%u cleanup
- [BUG]: Restore clearing t->logs.bytes
- [MEDIUM]: rework checks handling
- [DOC] Update a "contrib" file with a hint about a scheme used for formathing subjects
- [MEDIUM] Implement "track [<backend>/]<server>"
- [MINOR] Implement persistent id for proxies and servers
- [BUG] Don't increment server connections too much + fix retries
- [MEDIUM]: Prevent redispatcher from selecting the same server, version #3
- [MAJOR] proto_uxst rework -> SNMP support
- [BUG] appsession lookup in URL does not work
- [BUG] transparent proxy address was ignored in backend
- [BUG] hot reconfiguration failed because of a wrong error check
- [DOC] big update to the configuration manual
- [DOC] large update to the configuration manual
- [DOC] document more options
- [BUILD] major rework of the GNU Makefile
- [STATS] add support for "show info" on the unix socket
- [DOC] document options forwardfor to logasap
- [MINOR] add support for the "backlog" parameter
- [OPTIM] introduce global parameter "tune.maxaccept"
- [MEDIUM] introduce "timeout http-request" in frontends
- [MINOR] tarpit timeout is also allowed in backends
- [BUG] increment server connections for each connect()
- [MEDIUM] add a turn-around state of one second after a connection failure
- [BUG] fix typo in redispatched connection
- [DOC] document options nolinger to ssl-hello-chk
- [DOC] added documentation for "option tcplog" to "use_backend"
- [BUG] connect_server: server might not exist when sending error report
- [MEDIUM] support fully transparent proxy on Linux (USE_LINUX_TPROXY)
- [MEDIUM] add non-local bind to connect() on Linux
- [MINOR] add transparent proxy support for balabit's Tproxy v4
- [BUG] use backend's source and not server's source with tproxy
- [BUG] fix overlapping server flags
- [MEDIUM] fix server health checks source address selection
- [BUG] build failed on CONFIG_HAP_LINUX_TPROXY without CONFIG_HAP_CTTPROXY
- [DOC] added "server", "source" and "stats" keywords
- [DOC] all server parameters have been documented
- [DOC] document all req* and rsp* keywords.
- [DOC] added documentation about HTTP header manipulations
- [BUG] log response byte count, not request
- [BUILD] code did not build in full debug mode
- [BUG] fix truncated responses with sepoll
- [MINOR] use s->frt_addr as the server's address in transparent proxy
- [MINOR] fix configuration hint about timeouts
- [DOC] minor cleanup of the doc and notice to contributors
- [MINOR] report correct section type for unknown keywords.
- [BUILD] update MacOS Makefile to build on newer versions
- [DOC] fix erroneous "useallbackups" option in the doc
- [DOC] applied small fixes from early readers
- [MINOR] add configuration support for "redir" server keyword
- [MEDIUM] completely implement the server redirection method
- [TESTS] add a test case for the server redirection mechanism
- [DOC] add a configuration entry for "server ... redir <prefix>"
- [BUILD] backend.c and checks.c did not build without tproxy !
- Revert "[BUILD] backend.c and checks.c did not build without tproxy !"
- [BUILD] backend.c and checks.c did not build without tproxy !
- [OPTIM] used unsigned ints for HTTP state and message offsets
- [OPTIM] GCC4's builtin_expect() is suboptimal
- [BUG] failed conns were sometimes incremented in the frontend!
- [BUG] timeout.check was not pre-set to eternity
- [TESTS] add test-pollers.cfg to easily report pollers in use
- [BUG] do not apply timeout.connect in checks if unset
- [BUILD] ensure that makefile understands USE_DLMALLOC=1
- [MINOR] silent gcc for a wrong warning
- [CLEANUP] update .gitignore to ignore more temporary files
- [CLEANUP] report dlmalloc's source path only if explictly specified
- [BUG] str2sun could leak a small buffer in case of error during parsing
- [BUG] option allbackups was not working anymore in roundrobin mode
- [MAJOR] implementation of the "leastconn" load balancing algorithm
- [BUILD] ensure that users don't build without setting the target anymore.
- [DOC] document the leastconn LB algo
- [MEDIUM] fix stats socket limitation to 16 kB
- [DOC] fix unescaped space in httpchk example.
- [BUG] fix double-decrement of server connections
- [TESTS] add a test case for port mapping
- [TESTS] add a benchmark for integer hashing
- [TESTS] add new methods in ip-hash test file
- [MAJOR] implement parameter hashing for POST requests
This patch extends the "url_param" load balancing method by introducing
the "check_post" option. Using this option enables analysis of the beginning
of POST requests to search for the specified URL parameter.
The patch also fixes a few minor typos in comments that were discovered
during code review.
If we want to support netmasks for IP address hashing,
we will need something better than a pure modulus, otherwise
people with even numbers of servers will get surprizes.
Bob Jenkins is known for his works on hashing, and his site
has a lot of very interesting researches and algorithms for
integer hashing. He also points to the work of Thomas Wang
who has similar findings.
The program here tests their algorithms in order to find one
well suited for IP address hashing.
If a client does a sudden dirty close (CL_STCLOSE) during a server
connect turn-around, then the number of server connections is
decremented twice. This causes huge problems on the affected
server because when its connection number becomes negative, it
overflows and prevents the server from accepting new connections
due to an apparent saturation.
The fix consists in not decrementing the counter if the server is
in a turn-around state.
haproxy relies on linking the binary using gcc, so there is no real need to
hardcode both (CC and LD). Setting 'LD = $(CC)' will make the build system
a bit more cross-compile friendly because only the right cross-compiler has
to be passed via make.
Due to the way the stats socket work, it was not possible to
maintain the information related to the command entered, so
after filling a whole buffer, the request was lost and it was
considered that there was nothing to write anymore.
The major reason was that some flags were passed directly
during the first call to stats_dump_raw() instead of being
stored persistently in the session.
To definitely fix this problem, flags were added to the stats
member of the session structure.
A second problem appeared. When the stats were produced, a first
call to client_retnclose() was performed, then one or multiple
subsequent calls to buffer_write_chunks() were done. But once the
stats buffer was full and a reschedule operated, the buffer was
flushed, the write flag cleared from the buffer and nothing was
done to re-arm it.
For this reason, a check was added in the proto_uxst_stats()
function in order to re-call the client FSM when data were added
by stats_dump_raw(). Finally, the whole unix stats dump FSM was
rewritten to avoid all the magics it depended on. It is now
simpler and looks more like the HTTP one.
To be flexible while installing haproxy following variables have been
added to the Makefile:
- DESTDIR useful i.e. while installing in a sandbox (not set by default)
- PREFIX defines the default install prefix (default: /usr/local)
- SBINDIR defines the dir the haproxy binary gets installed
(default: $PREFIX/sbin)
Willy Tarreau
Krzysztof Piotr Oledzki
Marek Majkowski
Jeremy Hinegardner
matt.farnsworth@nokia.com
Christian Wiese