Commit Graph

1392 Commits

Author SHA1 Message Date
Willy Tarreau
6b2e11be1e [MEDIUM] backend: implement consistent hashing variation
Consistent hashing provides some interesting advantages over common
hashing. It avoids full redistribution in case of a server failure,
or when expanding the farm. This has a cost however, the hashing is
far from being perfect, as we associate a server to a request by
searching the server with the closest key in a tree. Since servers
appear multiple times based on their weights, it is recommended to
use weights larger than approximately 10-20 in order to smoothen
the distribution a bit.

In some cases, playing with weights will be the only solution to
make a server appear more often and increase chances of being picked,
so stats are very important with consistent hashing.

In order to indicate the type of hashing, use :

   hash-type map-based      (default, old one)
   hash-type consistent     (new one)

Consistent hashing can make sense in a cache farm, in order not
to redistribute everyone when a cache changes state. It could also
probably be used for long sessions such as terminal sessions, though
that has not be attempted yet.

More details on this method of hashing here :
  http://www.spiteful.com/2008/03/17/programmers-toolbox-part-3-consistent-hashing/
2009-10-09 07:17:58 +02:00
Krzysztof Piotr Oledzki
4cdd8314e9 [MINOR] acl: don't report valid acls as potential mistakes
Commit 404e8ab461 introduced
smart checking for stupid acl typos. However, now haproxy shows
the warning even for valid acls, like this one:
	acl Cookie-X-NoAccel    hdr_reg(cookie) (^|\ |;)X-NoAccel=1(;|$)
2009-10-08 07:00:07 +02:00
Willy Tarreau
ba255bc3c8 [BUG] counters: fix segfault on missing counters for a listener
If a frontend does not set 'option socket-stats', a 'clear counters'
on the stats socket could segfault because li->counters is NULL. The
correct fix is to check for NULL before as this is a valid situation.
2009-10-05 00:45:38 +02:00
Krzysztof Piotr Oledzki
6f61b21524 [BUG] Fix NULL pointer dereference in stats_check_uri_auth(), v2
Recent "struct chunk rework" introduced a NULL pointer dereference
and now haproxy segfaults if auth is required for stats but not found.

The reason is that size_t cannot store negative values, but current
code assumes that "len < 0" == uninitialized.

This patch fixes it.
2009-10-04 23:44:45 +02:00
Willy Tarreau
ac68c5d92c [OPTIM] counters: move some max numbers to the counters struct
There are a few remaining max values that need to move to counters.
Also, the counters are more often used than some config information,
so get them closer to the other useful struct members for better cache
efficiency.
2009-10-04 23:26:19 +02:00
Willy Tarreau
53fb4ae261 [MEDIUM] config: automatically find unused IDs for proxies, servers and listeners
Until now it was required that every custom ID was above 1000 in order to
avoid conflicts. Now we have the list of all assigned IDs and can automatically
pick the first unused one. This means that it is perfectly possible to interleave
automatic IDs with persistent IDs and the parser will automatically allocate
unused values starting with 1.
2009-10-04 23:04:08 +02:00
Willy Tarreau
482b00d1b4 [MINOR] tools: add a new get_next_id() function
This function returns the next unused key in a tree. This will be
used to find spare IDs.
2009-10-04 22:48:42 +02:00
Willy Tarreau
88922354fb [MINOR] config: add pointer to file name in block/redirect/use_backend/monitor rules
Those conditions already referenced the config line, but not the file.
2009-10-04 22:02:50 +02:00
Willy Tarreau
092549fa85 [MINOR] config: report places of duplicate names or IDs
When a name or ID conflict is detected, it is sometimes useful to know
where the other one was declared. Now that we have this information,
report it in error messages.
2009-10-04 21:16:08 +02:00
Willy Tarreau
90a570f025 [MINOR] config: reference file and line with any listener/proxy/server declaration
Those will be used later for cross-references of conflicts or errors.
2009-10-04 21:14:56 +02:00
Willy Tarreau
9d7e335127 [BUG] config: look for ID conflicts in all sockets, not only last ones.
ID conflicts between 'bind' lines were not detected due to this typo.
2009-10-04 21:13:39 +02:00
Krzysztof Piotr Oledzki
aeebf9ba65 [MEDIUM] Collect & provide separate statistics for sockets, v2
This patch allows to collect & provide separate statistics for each socket.
It can be very useful if you would like to distinguish between traffic
generate by local and remote users or between different types of remote
clients (peerings, domestic, foreign).

Currently no "Session rate" is supported, but adding it should be possible
if we found it useful.
2009-10-04 18:56:02 +02:00
Krzysztof Piotr Oledzki
719e726c97 [MINOR] Add "clear counters" to clear statistics counters
Now, when statistics counters are moved to separate
structures, adding "clear counters" is extremely easy.

[wt: updated the help message]
2009-10-04 18:40:08 +02:00
Krzysztof Piotr Oledzki
052d4fd07d [CLEANUP] Move counters to dedicated structures
Move counters from "struct proxy" and "struct server"
to "struct pxcounters" and "struct svcounters".

This patch should make no functional change.
2009-10-04 18:32:39 +02:00
Krzysztof Piotr Oledzki
8d06b8b8db [MINOR] Introduce include/types/counters.h
This patch introduces include/types/counters.h that
will be used to split couters from other structures
and to create statistics for listeners.
2009-10-04 18:32:12 +02:00
Krzysztof Piotr Oledzki
6d45fcd719 [DOC] Add information about http://haproxy.1wt.eu/contrib.html
Add information about http://haproxy.1wt.eu/contrib.html in
the CONTRIB file and remove one useless comment.
2009-10-04 18:31:35 +02:00
Willy Tarreau
96fd4b5f09 [MEDIUM] stream_int: adjust WAIT_ROOM handling
When applets write data, they generall cannot fill the buffer, so as
soon as we find a non-empty buffer, we're sure we're missing some room.
2009-10-04 17:35:08 +02:00
Willy Tarreau
b0c9bc4f95 [MEDIUM] stats: make HTTP stats use an I/O handler
Doing this, we can remove the last BF_HIJACK user and remove
produce_content(). s->data_source could also be removed but
it is currently used to detect if the stats or a server was
used.
2009-10-04 15:56:38 +02:00
Willy Tarreau
65671abd32 [MINOR] remove now obsolete ana_state from the session struct
This one is not used anymore.
2009-10-04 14:24:59 +02:00
Willy Tarreau
f5a885fd28 [MEDIUM] stats: don't use s->ana_state anymore
The stats handler used to store internal states in s->ana_state. Now
we only rely on si->st0 in which we can store as many states as we
have possible outputs. This cleans up the stats code a lot and makes
it more maintainable. It has also reduced code size by a few hundred
bytes.
2009-10-04 14:22:18 +02:00
Willy Tarreau
24955a1000 [MINOR] stats: make stats_dump_raw_to_buffer() use buffer_feed_chunk
Same as previous change. A remaining call to stats_dump_proxy()
still prevents us from completing the update.
2009-10-04 12:17:54 +02:00
Willy Tarreau
7e72a8faf2 [MINOR] stats_dump_sess_to_buffer: use buffer_feed_chunk()
same as previous patch for this function.
2009-10-04 11:00:11 +02:00
Willy Tarreau
61b347342c [MINOR] stats_dump_errors_to_buffer: use buffer_feed_chunk()
We can simplify the code in the stats functions using buffer_feed_chunk()
instead of buffer_write_chunk(). Let's start with this function. This
patch also fixed an issue where we could dump past the end of the capture
buffer if it is shorter than the captured request.
2009-10-04 11:00:11 +02:00
Willy Tarreau
fe8903cc76 [BUG] don't refresh timeouts late after detected activity
In old versions, before 1.3.16, we had to refresh the timeouts after
each call to process_session() because the stream socket handler did
not do it. Now that the sockets can exchange data for a long period
without calling process_session(), we can detect an old activity and
refresh a timeout long after the last activity, causing too late a
detection of some timeouts.

The fix simply consists in not checking for activity anymore in
stream_sock_data_finish() but only set a timeout if it was not
previously set.
2009-10-04 10:56:08 +02:00
Willy Tarreau
816fc22a4a [MINOR] stats: strip CR and LF from the input command line
commands sent from certain tools (eg: telnet) will be terminated
by CRLF and not only LF. So we must suppress CR too.
2009-10-04 09:19:42 +02:00
Willy Tarreau
33b230b34a [BUG] stats: don't call buffer_shutw(), but ->shutw() instead
Calling buffer_shutw() marks the buffer as closed but if it was already
closed in the other direction, the stream interface is not marked as
closed, causing infinite loops.

We took this opportunity to completely remove buffer_shutw() and buffer_shutr()
which have no reason to be used at all and which will always cause trouble
when directly called. The stats occurrence was the last one.
2009-10-04 09:19:36 +02:00
Willy Tarreau
f27b5ea8dc [MEDIUM] new option "independant-streams" to stop updating read timeout on writes
By default, when data is sent over a socket, both the write timeout and the
read timeout for that socket are refreshed, because we consider that there is
activity on that socket, and we have no other means of guessing if we should
receive data or not.

While this default behaviour is desirable for almost all applications, there
exists a situation where it is desirable to disable it, and only refresh the
read timeout if there are incoming data. This happens on sessions with large
timeouts and low amounts of exchanged data such as telnet session. If the
server suddenly disappears, the output data accumulates in the system's
socket buffers, both timeouts are correctly refreshed, and there is no way
to know the server does not receive them, so we don't timeout. However, when
the underlying protocol always echoes sent data, it would be enough by itself
to detect the issue using the read timeout. Note that this problem does not
happen with more verbose protocols because data won't accumulate long in the
socket buffers.

When this option is set on the frontend, it will disable read timeout updates
on data sent to the client. There probably is little use of this case. When
the option is set on the backend, it will disable read timeout updates on
data sent to the server. Doing so will typically break large HTTP posts from
slow lines, so use it with caution.
2009-10-03 22:01:18 +02:00
Willy Tarreau
211ad247ab [DOC] log-health-checks is an option, not a directive
Also fix alphabetical placement of the option in the list.
2009-10-03 21:47:47 +02:00
Willy Tarreau
be5b68584e [MINOR] report list of supported pollers with -vv
During troubleshooting, it's often useful to get the list of supported
pollers but until now it was required to have a working configuration
first. Since the pollers are known before main() is called, let's list
them with the build options.
2009-10-03 18:57:08 +02:00
Willy Tarreau
9757a38feb [MEDIUM] backend: introduce the "static-rr" LB algorithm
The "static-rr" is just the old round-robin algorithm. It is still
in use when a hash algorithm is used and the data to hash is not
present, but it was impossible to configure it explicitly. This one
is cheaper in terms of CPU and supports unlimited numbers of servers,
so it makes sense to be able to use it.
2009-10-03 18:41:19 +02:00
Willy Tarreau
da76f4f711 [MINOR] backend: reorganize the LB algorithm selection
We can now factor out some common tests between all map-based
algorithms. This removes some redundancy and makes the code more
logical.
2009-10-03 18:41:18 +02:00
Willy Tarreau
f3e49f9521 [MINOR] backend: separate declarations of LB algos from their lookup method
LB algo macros were composed of the LB algo by itself without any indication
of the method to use to look up a server (the lb function itself). This
method was implied by the LB algo, which was not very convenient to add
more algorithms. Now we have several fields in the LB macros, some to
describe what to look for in the requests, some to describe how to transform
that (kind of algo) and some to describe what lookup function to use.

The next patch will make it possible to factor out some code for all algos
which rely on a map.
2009-10-03 18:41:18 +02:00
Willy Tarreau
5b4c2b58fe [CLEANUP] proxy: move last lb-specific bits to their respective files
The lbprm structure has moved to backend.h, where it should be, and
all algo-specific types and declarations have moved to their specific
files. The proxy struct is now much more readable.
2009-10-03 18:41:18 +02:00
Krzysztof Piotr Oledzki
619caca740 [MINOR] Allow dots in show-node & add "white-space: nowrap" in th.pxname. 2009-10-03 18:37:06 +02:00
Krzysztof Piotr Oledzki
48cb2aed5a [MINOR] add "description", "node" and show-node"/"show-desc", remove "node-name", v2
This patch implements "description" (proxy and global) and "node" (global)
options, removes "node-name" and adds "show-node" & "show-desc" options
for "stats". It also changes the way the header lines (with proxy name) and
the statistics are displayed, so stats no longer look so clumsy with very
long names.

Instead of "node-name" it is possible to use show-node/show-desc with
an optional parameter that overrides a default node/description.

backend cust-0045
        # report specific values for this customer
        stats show-node Europe
        stats show-desc Master node for Europe, Asia, Africa
2009-10-03 07:10:14 +02:00
Willy Tarreau
39c9ba72a7 [MINOR] lb_map: reorder code in order to ease integration of new hash functions
We need to remove hash map accesses out of backend.c if we want to
later support new hash methods. This patch separates the hash computation
method from the server lookup. It leaves the lookup function to lb_map.c
and calls it with the result of the hash.
2009-10-01 21:11:15 +02:00
Willy Tarreau
f89c1873f8 [CLEANUP] backend: move LB algos to individual files
It was becoming painful to have all the LB algos in backend.c.
Let's move them to their own files. A few hashing functions still
need be broken in two parts, one for the contents and one for the
map position.
2009-10-01 11:19:37 +02:00
Willy Tarreau
78ff5d0a9e [MINOR] include time.h from freq_ctr.h as is uses "now". 2009-10-01 11:05:26 +02:00
Krzysztof Piotr Oledzki
99ab5f800c [MINOR] health checks logging unification
The code was duplicated serveral times, let's use
server_status_printf() instead.

   text    data     bss     dec     hex filename
 263504    5800   64224  333528   516d8 haproxy-old
 262944    5800   64224  332968   514a8 haproxy-new

Depends on "struct chunk rework" and
 "Health check reporting code rework + health logging, v3"
2009-10-01 10:17:38 +02:00
Krzysztof Piotr Oledzki
08ff959c3e [BUG] check if rise/fall has an argument and it is > 0
Check if rise/fall has an argument and it is > 0  or bad things may happen
in the health checks. ;)

Now it is verified and the code no longer allows for such condition:

backend bad
	(...)
        server o-f0 192.168.129.27:80 check inter 4000 source 0.0.0.0 rise 0
        server o-r0 192.168.129.27:80 check inter 4000 source 0.0.0.0 fall 0
        server o-f1 192.168.129.27:80 check inter 4000 source 0.0.0.0 rise
        server o-r1 192.168.129.27:80 check inter 4000 source 0.0.0.0 fall

[ALERT] 269/161830 (24136) : parsing [../git/haproxy.cfg:98]: 'rise' has to be > 0.
[ALERT] 269/161830 (24136) : parsing [../git/haproxy.cfg:99]: 'fall' has to be > 0.
[ALERT] 269/161830 (24136) : parsing [../git/haproxy.cfg:100]: 'rise' expects an integer argument.
[ALERT] 269/161830 (24136) : parsing [../git/haproxy.cfg:101]: 'fall' expects an integer argument.

Also add endline in the custom id checking code.
2009-10-01 10:17:38 +02:00
Krzysztof Piotr Oledzki
213014e587 [MEDIUM] Health check reporting code rework + health logging, v3
This patch adds health logging so it possible to check what
was happening before a crash. Failed healt checks are logged if
server is UP and succeeded healt checks if server is DOWN,
so the amount of additional information is limited.

I also reworked the code a little:

 - check_status_description[] and check_status_info[] is now
   joined into check_statuses[]

 - set_server_check_status updates not only s->check_status and
   s->check_duration but also s->result making the code simpler

Changes in v3:
 - for now calculate and use local versions of health/rise/fall/state,
   it is a slow path, no harm should be done. One day we may centralize
   processing of the checks and remove the duplicated code.
 - also log checks that are restoring current state
 - use "conditionally succeeded" for 404 with disable-on-404
2009-10-01 10:17:37 +02:00
Krzysztof Piotr Oledzki
78abe618a8 [MAJOR] struct chunk rework
Add size to struct chunk and simplify the code as there is
no longer required to pass sizeof in chunk_printf().
2009-10-01 10:17:37 +02:00
Willy Tarreau
ca7d4b98d4 [MINOR] backend: uninline some LB functions
There is no reason to inline functions which are used to grab a server
depending on an LB algo. They are large and used at several places.
Uninlining them saves 400 bytes of code.
2009-10-01 09:21:55 +02:00
Willy Tarreau
c5d9c80182 [MINOR] backend: export some functions to recount servers
Those functions will be used by new LB algorithms.
2009-10-01 09:17:05 +02:00
Willy Tarreau
2f9cc8ab52 [BUG] http stats: large outputs sometimes got some parts chopped off
Due to a misplaced call to stream_int_retnclose(), the stats output
buffer was erased before each call to produce_content(), resulting
in missing pieces in the stats output if the connection was not
fast enough between haproxy and the client.
2009-09-24 22:22:18 +02:00
Krzysztof Piotr Oledzki
db57c6b293 [DOC] add missing rate_lim and rate_max 2009-09-24 02:09:02 +02:00
Willy Tarreau
9f389e0152 [RELEASE] Released version 1.4-dev3
Released version 1.4-dev3 with the following main changes :
    - [BUILD] compilation of haproxy-1.4-dev2 on FreeBSD
    - [MEDIUM] Collect & show information about last health check, v3
    - [MINOR] export the hostname variable so that all the code can access it
    - [MINOR] stats: add a new node-name setting
    - [MEDIUM] remove old experimental tcpsplice option
    - [BUILD] fix build for systems without SOL_TCP
    - [MEDIUM] move connection establishment from backend to the SI.
    - [MEDIUM] make the global stats socket part of a frontend
    - [MEDIUM] session: account per-listener connections
    - [MINOR] session: switch to established state if no connect function
    - [MEDIUM] make the unix stats sockets use the generic session handler
    - [CLEANUP] unix: remove uxst_process_session()
    - [CLEANUP] move remaining stats sockets code to dumpstats
    - [MINOR] move the initial task's nice value to the listener
    - [MINOR] cleanup set_session_backend by using pre-computed analysers
    - [MINOR] set s->srv_error according to the analysers
    - [MEDIUM] set rep->analysers from fe and be analysers
    - [MEDIUM] replace BUFSIZE with buf->size in computations
    - [MEDIUM] make it possible to change the buffer size in the configuration
    - [MEDIUM] report error on buffer writes larger than buffer size
    - [MEDIUM] stream_interface: add and use ->update function to resync
    - [CLEANUP] remove ifdef MSG_NOSIGNAL and define it instead
    - [MEDIUM] remove TCP_CORK and make use of MSG_MORE instead
    - [BUG] tarpit did not work anymore
    - [MINOR] acl: add support for hdr_ip to match IP addresses in headers
    - [MAJOR] buffers: fix misuse of the BF_SHUTW_NOW flag
    - [MINOR] buffers: provide more functions to handle buffer data
    - [MEDIUM] buffers: provide new buffer_feed*() function
    - [MINOR] buffers: add peekchar and peekline functions for stream interfaces
    - [MINOR] buffers: provide buffer_si_putchar() to send a char from a stream interface
    - [BUG] buffer_forward() would not correctly consider data already scheduled
    - [MINOR] buffers: add buffer_cut_tail() to cut only unsent data
    - [MEDIUM] stream_interface: make use of buffer_cut_tail() to report errors
    - [MAJOR] http: add support for HTTP 1xx informational responses
    - [MINOR] buffers: inline buffer_si_putchar()
    - [MAJOR] buffers: split BF_WRITE_ENA into BF_AUTO_CONNECT and BF_AUTO_CLOSE
    - [MAJOR] buffers: fix the BF_EMPTY flag's meaning
    - [BUG] stream_interface: SI_ST_CLO must have buffers SHUT
    - [MINOR] stream_sock: don't set SI_FL_WAIT_DATA if BF_SHUTW_NOW is set
    - [MEDIUM] add support for infinite forwarding
    - [BUILD] stream_interface: fix conflicting declaration
    - [BUG] buffers: buffer_forward() must not always clear BF_OUT_EMPTY
    - [BUG] variable buffer size ignored at initialization time
    - [MINOR] ensure that buffer_feed() and buffer_skip() set BF_*_PARTIAL
    - [BUG] fix buffer_skip() and buffer_si_getline() to correctly handle wrap-arounds
    - [MINOR] stream_interface: add SI_FL_DONT_WAKE flag
    - [MINOR] stream_interface: add iohandler callback
    - [MINOR] stream_interface: add functions to support running as internal/external tasks
    - [MEDIUM] session: call iohandler for embedded tasks (applets)
    - [MINOR] add a ->private member to the stream_interface
    - [MEDIUM] stats: prepare the connection for closing before dumping
    - [MEDIUM] stats: replace the stats socket analyser with an SI applet
2009-09-24 00:12:50 +02:00
Willy Tarreau
9a42c0d771 [MEDIUM] stats: replace the stats socket analyser with an SI applet
We can get rid of the stats analyser by moving all the stats code
to a stream interface applet. Above being cleaner, it provides new
advantages such as the ability to process requests and responses
from the same function and work only with simple state machines.
There's no need for any hijack hack anymore.

The direct advantage for the user are the interactive mode and the
ability to chain several commands delimited by a semi-colon. Now if
the user types "prompt", he gets a prompt from which he can send
as many requests as he wants. All outputs are terminated by a
blank line followed by a new prompt, so this can be used from
external tools too.

The code is not very clean, it needs some rework, but some part
of the dirty parts are due to the remnants of the hijack mode used
in the old functions we call.

The old AN_REQ_STATS_SOCK analyser flag is now unused and has been
removed.
2009-09-23 23:52:17 +02:00
Willy Tarreau
56a560aef4 [MEDIUM] stats: prepare the connection for closing before dumping
We will need to modify the stats dump functions so that they can
be used in interactive mode. For this, we want their caller to
prepare the connection for a close, not themselves to do it.
Let's simply move the stream_int_retnclose() out.
2009-09-23 23:52:16 +02:00
Willy Tarreau
eecc8ee673 [MINOR] add a ->private member to the stream_interface
iohandlers will need to store some form of context and for this will
need a way to find their call context. We add the ->private as well
as ->st0 and ->st1 for that purpose. Most likely ->private will be
initialized to the current session and ->st0 and ->st1 will be used
to maintain any form of internal state between calls.
2009-09-23 23:52:16 +02:00