Commit Graph

344 Commits

Author SHA1 Message Date
Cyril Bonté
70be45dbdf [MEDIUM] enable/disable servers from the stats web interface
Based on a patch provided by Judd Montgomery, it is now possible to
enable/disable servers from the stats web interface. This allows to select
several servers in a backend and apply the action to them at the same time.

Currently, there are 2 known limitations :
- The POST data are limited to one packet
  (don't alter too many servers at a time).
- Expect: 100-continue is not supported.
(cherry picked from commit 7693948766cb5647ac03b48e782cfee2b1f14491)
2010-10-30 19:04:34 +02:00
Willy Tarreau
74b08c9ab7 [MEDIUM] buffers: rework the functions to exchange between SI and buffers
There was no consistency between all the functions used to exchange data
between a buffer and a stream interface. Also, the functions used to send
data to a buffer did not consider the possibility that the buffer was
shutdown for read.

Now the functions are called buffer_{put,get}_{char,block,chunk,string}.

The old buffer_feed* functions have been left available for existing code
but marked deprecated.
2010-09-08 17:04:31 +02:00
Willy Tarreau
d0807c3c60 [MEDIUM] signals: support redistribution of signal zero when stopping
Signal zero is never delivered by the system. However having a signal to
which functions and tasks can subscribe to be notified of a stopping event
is useful. So this patch does two things :
  1) allow signal zero to be delivered from any function of signal handler
  2) make soft_stop() deliver this signal so that tasks can be notified of
     a stopping condition.
2010-08-27 18:26:11 +02:00
Willy Tarreau
24f4efa670 [MEDIUM] signals: add support for registering functions and tasks
The two new functions below make it possible to register any number
of functions or tasks to a system signal. They will be called in the
registration order when the signal is received.

    struct sig_handler *signal_register_fct(int sig, void (*fct)(struct sig_handler *), int arg);
    struct sig_handler *signal_register_task(int sig, struct task *task, int reason);
2010-08-27 18:00:40 +02:00
Willy Tarreau
0a4838cd31 [MEDIUM] session-counters: correctly unbind the counters tracked by the backend
In case of HTTP keepalive processing, we want to release the counters tracked
by the backend. Till now only the second set of counters was released, while
it could have been assigned by the frontend, or the backend could also have
assigned the first set. Now we reuse to unused bits of the session flags to
mark which stick counters were assigned by the backend and to release them as
appropriate.
2010-08-10 18:04:16 +02:00
Willy Tarreau
56123282ef [MINOR] session-counters: use "track-sc{1,2}" instead of "track-{fe,be}-counters"
The assumption that there was a 1:1 relation between tracked counters and
the frontend/backend role was wrong. It is perfectly possible to track the
track-fe-counters from the backend and the track-be-counters from the
frontend. Thus, in order to reduce confusion, let's remove this useless
{fe,be} reference and simply use {1,2} instead. The keywords have also been
renamed in order to limit confusion. The ACL rule action now becomes
"track-sc{1,2}". The ACLs are now "sc{1,2}_*" instead of "trk{fe,be}_*".

That means that we can reasonably document "sc1" and "sc2" (sticky counters
1 and 2) as sort of patterns that are available during the whole session's
life and use them just like any other pattern.
2010-08-10 18:04:15 +02:00
Willy Tarreau
f6efda1189 [MEDIUM] session counters: automatically remove expired entries.
When a ref_cnt goes down to zero and the entry is expired, remove it.
2010-08-10 18:04:15 +02:00
Willy Tarreau
f059a0f63a [MAJOR] session-counters: split FE and BE track counters
Having a single tracking pointer for both frontend and backend counters
does not work. Instead let's have one for each. The keyword has changed
to "track-be-counters" and "track-fe-counters", and the ACL "trk_*"
changed to "trkfe_*" and "trkbe_*".
2010-08-10 18:04:15 +02:00
Willy Tarreau
3b9c6e053e [MEDIUM] stick-table: make use of generic types for stored data
It's a bit cumbersome to have to know all possible storable types
from the stats interface. Instead, let's have generic types for
all data, which will facilitate their manipulation.
2010-08-10 18:04:14 +02:00
Willy Tarreau
69f58c8058 [MEDIUM] stats: add "show table [<name>]" to dump a stick-table
It is now possible to dump a table's contents with keys, expire,
use count, and various data using the command above on the stats
socket.

"show table" only shows main table stats, while "show table <name>"
dumps table contents, only if the socket level is admin.
2010-08-10 18:04:14 +02:00
Willy Tarreau
da7ff64aa9 [MEDIUM] session-counters: add HTTP req/err tracking
This patch adds support for the following session counters :
  - http_req_cnt : HTTP request count
  - http_req_rate: HTTP request rate
  - http_err_cnt : HTTP request error count
  - http_err_rate: HTTP request error rate

The equivalent ACLs have been added to check the tracked counters
for the current session or the counters of the current source.
2010-08-10 18:04:14 +02:00
Willy Tarreau
1f7e925d6a [MINOR] stktable: add a stktable_update_key() function
This function looks up a key, updates its expiration date, or creates
it if it was not found. acl_fetch_src_updt_conn_cnt() was updated to
make use of it.
2010-08-10 18:04:14 +02:00
Willy Tarreau
91c43d7fe4 [MEDIUM] session counters: add conn_rate and sess_rate counters
These counters maintain incoming connection rates and session rates
in a stick-table, over a period which is defined in the configuration
(2 ms to 24 days). They can be used to detect service abuse and
enforce a certain accept rate per source address for instance, and
block if the rate is passed over.

Example :
	# block if more than 50 requests per 5 seconds from a source.
	stick-table type ip size 200k expire 1m store conn_rate(5s),sess_rate(5s)
	tcp-request track-counters src
	tcp-request reject if { trk_conn_rate gt 50 }

	# cause a 3 seconds pause to requests from sources in excess of 20 requests/5s
	tcp-request inspect-delay 3s
	tcp-request content accept if { trk_sess_rate gt 20 } WAIT_END
2010-08-10 18:04:13 +02:00
Willy Tarreau
ac78288eaf [MEDIUM] stick-tables: add stored data argument type checking
We're now able to return errors based on the validity of an argument
passed to a stick-table store data type. We also support ARG_T_DELAY
to pass delays to stored data types (eg: for rate counters).
2010-08-10 18:04:13 +02:00
Willy Tarreau
888617dc3b [MEDIUM] stick-tables: add support for arguments to data_types
Some data types will require arguments (eg: period for a rate counter).
This patch adds support for such arguments between parenthesis in the
"store" directive of the stick-table statement. Right now only integers
are supported.
2010-08-10 18:04:13 +02:00
Willy Tarreau
e348793696 [MEDIUM] session-counters: automatically update tracked connection count
When a session tracks a counter, automatically increase the cumulated
connection count. This makes src_updt_conn_cnt() almost useless. In
fact it might still be used to update different tables.
2010-08-10 18:04:12 +02:00
Willy Tarreau
38285c18f4 [MEDIUM] session: add concurrent connections counter
The new "conn_cur" session counter has been added. It is automatically
updated upon "track XXX" directives, and the entry is touched at the
moment we increment the value so that we don't consider further counter
updates as real updates, otherwise we would end up updating upon completion,
which may not be desired. Probably that some other event counters (eg: HTTP
requests) will have to be updated upon each event though.

This counter can be matched against current session's source address using
the "src_conn_cur" ACL.
2010-08-10 18:04:12 +02:00
Willy Tarreau
4a0347add0 [MINOR] stick-table: provide a table lookup function
We'll often need to lookup a table by its name. This will change
in the future once we can resolve these names on startup.
2010-08-10 18:04:12 +02:00
Willy Tarreau
9ba2dcc86c [MAJOR] session: add track-counters to track counters related to the session
This patch adds the ability to set a pointer in the session to an
entry in a stick table which holds various counters related to a
specific pattern.

Right now the syntax matches the target syntax and only the "src"
pattern can be specified, to track counters related to the session's
IPv4 source address. There is a special function to extract it and
convert it to a key. But the goal is to be able to later support as
many patterns as for the stick rules, and get rid of the specific
function.

The "track-counters" directive may only be set in a "tcp-request"
statement right now. Only the first one applies. Probably that later
we'll support multi-criteria tracking for a single session and that
we'll have to name tracking pointers.

No counter is updated right now, only the refcount is. Some subsequent
patches will have to bring that feature.
2010-08-10 18:04:12 +02:00
Willy Tarreau
591fedc2c3 [MEDIUM] buffer: make buffer_feed* support writing non-contiguous chunks
The buffer_feed* functions that are used to send data to buffers did only
support sending contiguous chunks while they're relying on memcpy(). This
patch improves on this by making them able to write in two chunks if needed.
Thus, the buffer_almost_full() function has been improved to really consider
the remaining space and not just what can be written at once.
2010-08-10 17:48:57 +02:00
Willy Tarreau
2970b0bedf [MINOR] freq_ctr: add new types and functions for periods different from 1s
Some freq counters will have to work on periods different from 1 second.
The original freq counters rely on the period to be exactly one second.
The new ones (freq_ctr_period) let the user define the period in ticks,
and all computations are operated over that period. When reading a value,
it indicates the amount of events over that period too.
2010-08-10 14:01:09 +02:00
Willy Tarreau
cb18364ca7 [MEDIUM] stick_table: separate storage and update of session entries
When an entry already exists, we just need to update its expiration
timer. Let's have a dedicated function for that instead of spreading
open code everywhere.

This change also ensures that an update of an existing sticky session
really leads to an update of its expiration timer, which was apparently
not the case till now. This point needs to be checked in 1.4.
2010-06-14 15:10:26 +02:00
Willy Tarreau
41883e2041 [MINOR] stick_table: export the stick_table_key
This one is huge and will be needed by other portions of code for various
data lookups. Let's not have them allocate it in the stack.
2010-06-14 15:10:25 +02:00
Willy Tarreau
68129b90eb [MINOR] stick_table: provide functions to return stksess data from a type
This function does the indirection job in the table to find the pointer
to the real data matching the requested type.
2010-06-14 15:10:25 +02:00
Willy Tarreau
f16d2b8c1b [MEDIUM] stick_table: don't overwrite data when storing an entry
Till now sticky sessions only held server IDs. Now there are other
data types so it is not acceptable anymore to overwrite the server ID
when writing something. The server ID must then only be written from
the caller when appropriate. Doing this has also led to separate
lookup and storage.
2010-06-14 15:10:24 +02:00
Willy Tarreau
08d5f98294 [MEDIUM] stick_table: add room for extra data types
The stick_tables will now be able to store extra data for a same key.
A limited set of extra data types will be defined and for each of them
an offset in the sticky session will be assigned at startup time. All
of this information will be stored in the stick table.

The extra data types will have to be specified after the new "store"
keyword of the "stick-table" directive, which will reserve some space
for them.
2010-06-14 15:10:24 +02:00
Willy Tarreau
f0b38bfc33 [CLEANUP] stick_table: move pattern to key functions to stick_table.c
pattern.c depended on stick_table while in fact it should be the opposite.
So we move from pattern.c everything related to stick_tables and invert the
dependency. That way the code becomes more logical and intuitive.
2010-06-14 15:10:24 +02:00
Willy Tarreau
393379c3e0 [MINOR] stick_table: add support for variable-sized data
Right now we're only able to store a server ID in a sticky session.
The goal is to be able to store anything whose size is known at startup
time. For this, we store the extra data before the stksess pointer,
using a negative offset. It will then be easy to cumulate multiple
data provided they each have their own offset.
2010-06-14 15:10:23 +02:00
Willy Tarreau
b36b4244a2 [MINOR] session: differenciate between accepted connections and received connections
Now we're able to reject connections very early, so we need to use a
different counter for the connections that are received and the ones
that are accepted and converted into sessions, so that the rate limits
can still apply to the accepted ones. The session rate must still be
used to compute the rate limit, so that we can reject undesired traffic
without affecting the rate.
2010-06-14 10:53:19 +02:00
Willy Tarreau
decd14d298 [MEDIUM] stats: rely on the standard session_accept() function
The stats' accept() function is now ridiculously small. It could
even be reduced by moving some parts to the common accept code.
2010-06-14 10:53:18 +02:00
Willy Tarreau
81f9aa3bf2 [MAJOR] frontend: split accept() into frontend_accept() and session_accept()
A new function session_accept() is now called from the lower layer to
instanciate a new session. Once the session is instanciated, the upper
layer's frontent_accept() is called. This one can be service-dependant.

That way, we have a 3-phase accept() sequence :
  1) protocol-specific, session-less accept(), which is pointed to by
     the listener. It defaults to the generic stream_sock_accept().
  2) session_accept() which relies on a frontend but not necessarily
     for use in a proxy (eg: stats or any future service).
  3) frontend_accept() which performs the accept for the service
     offerred by the frontend. It defaults to frontend_accept() which
     is really what is used by a proxy.

The TCP/HTTP proxies have been moved to this mode so that we can now rely on
frontend_accept() for any type of session initialization relying on a frontend.

The next step will be to convert the stats to use the same system for the stats.
2010-06-14 10:53:17 +02:00
Willy Tarreau
a8f55d5473 [MEDIUM] backend: initialize the server stream_interface upon connect()
It's not normal to initialize the server-side stream interface from the
accept() function, because it may change later. Thus, we introduce a new
stream_sock_prepare_interface() function which is called just before the
connect() and which sets all of the stream_interface's callbacks to the
default ones used for real sockets. The ->connect function is also set
at the same instant so that we can easily add new server-side protocols
soon.
2010-06-14 10:53:15 +02:00
Willy Tarreau
a5c0ab200b [MEDIUM] frontend: check for LI_O_TCP_RULES in the listener
The new LI_O_TCP_RULES listener option indicates that some TCP rules
must be checked upon accept on this listener. It is now checked by
the frontend and the L4 rules are evaluated only in this case. The
flag is only set when at least one tcp-req rule is present in the
frontend.

The L4 rules check function has now been moved to proto_tcp.c where
it ought to be.
2010-06-14 10:53:13 +02:00
Willy Tarreau
eb472685cb [MEDIUM] separate protocol-level accept() from the frontend's
For a long time we had two large accept() functions, one for TCP
sockets instanciating proxies, and another one for UNIX sockets
instanciating the stats interface.

A lot of code was duplicated and both did not work exactly the same way.

Now we have a stream_sock layer accept() called for either TCP or UNIX
sockets, and this function calls the frontend-specific accept() function
which does the rest of the frontend-specific initialisation.

Some code is still duplicated (session & task allocation, stream interface
initialization), and might benefit from having an intermediate session-level
accept() callback to perform such initializations. Still there are some
minor differences that need to be addressed first. For instance, the monitor
nets should only be checked for proxies and not for other connection templates.

Last, we renamed l->private as l->frontend. The "private" pointer in
the listener is only used to store a frontend, so let's rename it to
eliminate this ambiguity. When we later support detached listeners
(eg: FTP), we'll add another field to avoid the confusion.
2010-06-14 10:53:11 +02:00
Willy Tarreau
03fa5df64a [CLEANUP] rename client -> frontend
The 'client.c' file now only contained frontend-specific functions,
so it has naturally be renamed 'frontend.c'. Same for client.h. This
has also been an opportunity to remove some cross references from
files that should not have depended on it.

In the end, this file should contain a protocol-agnostic accept()
code, which would initialize a session, task, etc... based on an
accept() from a lower layer. Right now there are still references
to TCP.
2010-06-14 10:53:10 +02:00
Willy Tarreau
44b90cc4d8 [CLEANUP] tcp: move some non tcp-specific layer6 processing out of proto_tcp
Some functions which act on generic buffer contents without being
tcp-specific were historically in proto_tcp.c. This concerns ACLs
and RDP cookies. Those have been moved away to more appropriate
locations. Ideally we should create some new files for each layer6
protocol parser. Let's do that later.
2010-06-14 10:53:09 +02:00
Willy Tarreau
bce7088275 [MEDIUM] add ability to connect to a server from an IP found in a header
Using get_ip_from_hdr2() we can look for occurrence #X or #-X and
extract the IP it contains. This is typically designed for use with
the X-Forwarded-For header.

Using "usesrc hdr_ip(name,occ)", it becomes possible to use the IP address
found in <name>, and possibly specify occurrence number <occ>, as the
source to connect to a server. This is possible both in a server and in
a backend's source statement. This is typically used to use the source
IP previously set by a upstream proxy.
2010-03-30 10:39:43 +02:00
Willy Tarreau
b1d67749db [MEDIUM] backend: move the transparent proxy address selection to backend
The transparent proxy address selection was set in the TCP connect function
which is not the most appropriate place since this function has limited
access to the amount of parameters which could produce a source address.

Instead, now we determine the source address in backend.c:connect_server(),
right after calling assign_server_address() and we assign this address in
the session and pass it to the TCP connect function. This cannot be performed
in assign_server_address() itself because in some cases (transparent mode,
dispatch mode or http_proxy mode), we assign the address somewhere else.

This change will open the ability to bind to addresses extracted from many
other criteria (eg: from a header).
2010-03-30 09:59:43 +02:00
Willy Tarreau
e45997661b [MEDIUM] session: better fix for connection to servers with closed input
The following patch fixed an issue but brought another one :
  296897 [MEDIUM] connect to servers even when the input has already been closed

The new issue is that when a connection is inspected and aborted using
TCP inspect rules, now it is sent to the server before being closed. So
that test is not satisfying. A probably better way is not to prevent a
connection from establishing if only BF_SHUTW_NOW is set but BF_SHUTW
is not. That way, the BF_SHUTW flag is not set if the request has any
data pending, which still fixes the stats issue, but does not let any
empty connection pass through.

Also, as a safety measure, we extend buffer_abort() to automatically
disable the BF_AUTO_CONNECT flag. While it appears to always be OK,
it is by pure luck, so better safe than sorry.
2010-03-21 23:31:42 +01:00
Willy Tarreau
d9b587f260 [STATS] report HTTP requests (total and rate) in frontends
Now that we support keep-alive, it's important to report a separate
counter for requests. Right now it just appears in the CSV output.
2010-02-26 10:05:55 +01:00
Willy Tarreau
b97f199d4b [MEDIUM] http: don't use trash to realign large buffers
The trash buffer may now be smaller than a buffer because we can tune
it at run time. This causes a risk when we're trying to use it as a
temporary buffer to realign unaligned requests, because we may have to
put up to a full buffer into it.

Instead of doing a double copy, we're now relying on an open-coded
bouncing copy algorithm. The principle is that we move one byte at
a time to its final place, and if that place also holds a byte, then
we move it too, and so on. We finish when we've moved all the buffer.
It limits the number of memory accesses, but since it proceeds one
byte at a time and with random walk, it's not cache friendly and
should be slower than a double copy. However, it's only used in
extreme situations and the difference will not be noticeable.

It has been extensively tested and works reliably.
2010-02-25 23:54:31 +01:00
Willy Tarreau
9cc670f7d9 [CLEANUP] config: use build_acl_cond() to simplify http-request ACL parsing
Now that we have this new function to make your life better, use it.
2010-02-01 10:43:44 +01:00
Cyril Bonté
cd19e51b05 [MEDIUM] add a maintenance mode to servers
This is a first attempt to add a maintenance mode on servers, using
the stat socket (in admin level).

It can be done with the following command :
   - disable server <backend>/<server>
   - enable  server <backend>/<server>

In this mode, no more checks will be performed on the server and it
will be marked as a special DOWN state (MAINT).

If some servers were tracking it, they'll go DOWN until the server
leaves the maintenance mode. The stats page and the CSV export also
display this special state.

This can be used to disable the server in haproxy before doing some
operations on this server itself. This is a good complement to the
"http-check disable-on-404" keyword and works in TCP mode.
2010-01-31 23:33:18 +01:00
Krzysztof Piotr Oledzki
8c8bd4593c [MAJOR] use the new auth framework for http stats
Support the new syntax (http-request allow/deny/auth) in
http stats.

Now it is possible to use the same syntax is the same like in
the frontend/backend http-request access control:
 acl src_nagios src 192.168.66.66
 acl stats_auth_ok http_auth(L1)

 stats http-request allow if src_nagios
 stats http-request allow if stats_auth_ok
 stats http-request auth realm LB

The old syntax is still supported, but now it is emulated
via private acls and an aditional userlist.
2010-01-31 19:14:09 +01:00
Krzysztof Piotr Oledzki
f9423ae43a [MINOR] acl: add http_auth and http_auth_group
Add two acls to match http auth data:
 acl <name> http_auth(userlist)
 acl <name> http_auth_hroup(userlist) group1 group2 (...)
2010-01-31 19:14:09 +01:00
Krzysztof Piotr Oledzki
961050465e [MINOR] generic auth support with groups and encrypted passwords
Add generic authentication & authorization support.

Groups are implemented as bitmaps so the count is limited to
sizeof(int)*8 == 32.

Encrypted passwords are supported with libcrypt and crypt(3), so it is
possible to use any method supported by your system. For example modern
Linux/glibc instalations support MD5/SHA-256/SHA-512 and of course classic,
DES-based encryption.
2010-01-31 19:14:07 +01:00
Willy Tarreau
fdb563c06f [MEDIUM] http: add support for conditional response header rewriting
Just as for the req* rules, we can now condition rsp* rules with ACLs.
ACLs match on response, so volatile request information cannot be used.
A warning is emitted if a configuration contains such an anomaly.
2010-01-31 15:43:27 +01:00
Willy Tarreau
6c123b15cb [MEDIUM] http: make the request filter loop check for optional conditions
From now on, if request filters have ACLs defined, these ACLs will be
evaluated to condition the filter. This will be used to conditionally
remove/rewrite headers based on ACLs.
2010-01-28 20:22:06 +01:00
Willy Tarreau
f1e98b8628 [CLEANUP] config: use warnif_cond_requires_resp() to check for bad ACLs
Factor out some repetitive copy-pasted code to check for request ACLs
validity.
2010-01-28 17:59:39 +01:00
Willy Tarreau
2bbba415d7 [MINOR] acl: add build_acl_cond() to make it easier to add ACLs in config
This function automatically builds a rule, considering the if/unless
statements, and automatically updates the proxy's acl_requires, the
condition's file and line.
2010-01-28 16:48:33 +01:00
Willy Tarreau
68085d8cfb [MINOR] http: add http_remove_header2() to remove a header value.
Calling this function after http_find_header2() automatically deletes
the current value of the header, and removes the header itself if the
value is the only one. The context is automatically adjusted for a
next call to http_find_header2() to return the next header. No other
change nor test should be made on the transient context though.
2010-01-18 19:51:33 +01:00
Emeric Brun
107ca30d54 [MEDIUM] Add pattern fetch management types and functions 2010-01-12 16:01:19 +01:00
Emeric Brun
3bd697e071 [MEDIUM] Add stick table (persistence) management functions and types 2010-01-12 11:23:15 +01:00
Willy Tarreau
148d099406 [BUG] stream_interface: fix retnclose and remove cond_close
The stream_int_cond_close() function was added to preserve the
contents of the response buffer because stream_int_retnclose()
was buggy. It flushed the response instead of flushing the
request. This caused issues with pipelined redirects followed
by error messages which ate the previous response.

This might even have caused object truncation on pipelined
requests followed by an error or by a server redirection.

Now that this is fixed, simply get rid of the now useless
function.
2010-01-10 10:21:21 +01:00
Willy Tarreau
90deb18916 [MEDIUM] http: make safer use of the DONT_READ and AUTO_CLOSE flags
Several HTTP analysers used to set those flags to values that
were useful but without considering the possibility that they
were not called again to clean what they did. First, replace
direct flag manipulation with more explicit macros. Second,
enforce a rule stating that any buffer which changes one of
these flags from the default must restore it after completion,
so that other analysers see correct flags.

With both this fix and the previous one about analyser bits,
we should not see any more stuck sessions.
2010-01-07 00:20:41 +01:00
Krzysztof Piotr Oledzki
15514c21a2 [MINOR]: stats: add show-legends to report additional informations
Supported informations, available via "tr/td title":
  - cap: capabilities (proxy)
  - mode: one of tcp, http or health (proxy)
  - id: SNMP ID (proxy, socket, server)
  - IP (socket, server)
  - cookie (backend, server)
2010-01-06 00:28:06 +01:00
Willy Tarreau
d98cf93395 [MAJOR] http: implement body parser
The body parser will be used in close and keep-alive modes. It follows
the stream to keep in sync with both the request and the response message.
Both chunked transfer-coding and content-length are supported according to
RFC2616.

The multipart/byterange encoding has not yet been implemented and if not
seconded by any of the two other ones, will be forwarded till the close,
as requested by the specification.

Both the request and the response analysers converge into an HTTP_MSG_DONE
state where it will be possible to force a close (option forceclose) or to
restart with a fresh new transaction and maintain keep-alive.

This change is important. All tests are OK but any possible behaviour
change with "option httpclose" might find its root here.
2009-12-27 22:54:55 +01:00
Willy Tarreau
5d881d0f3a [MINOR] new function stream_int_cond_close()
This one will be used to conditionally send a message upon a
close on a stream interface. It will not overwrite any existing
data.
2009-12-27 22:51:06 +01:00
Willy Tarreau
d21e01c63f [MINOR] buffers: add buffer_ignore() to skip some bytes
This simple function will be used to skip blanks at the beginning of a
request or response.
2009-12-27 15:45:38 +01:00
Willy Tarreau
0937bc43cf [MINOR] http: move the http transaction init/cleanup code to proto_http
This code really belongs to the http part since it's transaction-specific.
This will also make it easier to later reinitialize a transaction in order
to support keepalive.
2009-12-22 15:03:09 +01:00
Willy Tarreau
7c3c54177a [MAJOR] buffers: automatically compute the maximum buffer length
We used to apply a limit to each buffer's size in order to leave
some room to rewrite headers, then we used to remove this limit
once the session switched to a data state.

Proceeding that way becomes a problem with keepalive because we
have to know when to stop reading too much data into the buffer
so that we can leave some room again to process next requests.

The principle we adopt here consists in only relying on to_forward+send_max.
Indeed, both of those data define how many bytes will leave the buffer.
So as long as their sum is larger than maxrewrite, we can safely
fill the buffers. If they are smaller, then we refrain from filling
the buffer. This means that we won't risk to fill buffers when
reading last data chunk followed by a POST request and its contents.

The only impact identified so far is that we must ensure that the
BF_FULL flag is correctly dropped when starting to forward. Right
now this is OK because nobody inflates to_forward without using
buffer_forward().
2009-12-22 10:06:34 +01:00
Krzysztof Piotr Oledzki
97f07b832f [MEDIUM] Decrease server health based on http responses / events, version 3
Implement decreasing health based on observing communication between
HAProxy and servers.

Changes in this version 2:
 - documentation
 - close race between a started check and health analysis event
 - don't force fastinter if it is not set
 - better names for options
 - layer4 support

Changes in this version 3:
 - add stats
 - port to the current 1.4 tree
2009-12-16 00:29:27 +01:00
Cyril Bonté
b21570ae0f [MEDIUM] appsession: add "len", "prefix" and "mode" options
To sum up :
- len : it's now the max number of characters for the value, preventing
  garbaged results.
- a new option "prefix" is added, this allows to use dynamic cookie
  names (e.g. ASPSESSIONIDXXX).

Previously in the thread, I wanted to use the value found with
"capture cookie" but when i started to update the documentation, I
found this solution quite weird. I've made a small rework to not
depend on "capture cookie".

- There's the posssiblity to define the URL parser mode (path parameters
  or query string).
2009-11-30 11:31:53 +01:00
Willy Tarreau
fa355d4a51 [MINOR] http: keep pointer to beginning of data
We now set msg->col and msg->sov to the first byte of non-header.
They will be used later when parsing chunks. A new macro was added
to perform size additions on an http_msg in order to limit the risks
of copy-paste in the long term.

During this operation, it appeared that the http_msg struct was not
optimal on 64-bit, so it was re-ordered to fill the holes.
2009-11-29 18:12:29 +01:00
Alex Williams
96532db923 [MINOR] server tracking: don't care about the tracked server's mode
Right now, an HTTP server cannot track a TCP server and vice-versa.
This patch enables proxy tracking without relying on the proxy's mode
(tcp/http/health). It only requires a matching proxy name to exist. The
original function was renamed to findproxy_mode().
2009-11-02 11:08:00 +01:00
Willy Tarreau
45cb4fb640 [MEDIUM] build: switch ebtree users to use new ebtree version
All files referencing the previous ebtree code were changed to point
to the new one in the ebtree directory. A makefile variable (EBTREE_DIR)
is also available to use files from another directory.

The ability to build the libebtree library temporarily remains disabled
because it can have an impact on some existing toolchains and does not
appear worth it in the medium term if we add support for multi-criteria
stickiness for instance.
2009-10-26 21:10:04 +01:00
Willy Tarreau
8e89b84848 [MINOR] http: remove the last call to stream_int_return
And remove the now unused function itself too.
2009-10-18 23:56:35 +02:00
Willy Tarreau
b37c27e28f [MAJOR] http: create the analyser which waits for a response
The code part which waits for an HTTP response has been extracted
from the old function. We now have two analysers and the second one
may re-enable the first one when an 1xx response is encountered.
This has been tested and works.

The calls to stream_int_return() that were remaining in the wait
analyser have been converted to stream_int_retnclose().
2009-10-18 23:15:41 +02:00
Cyril Bonté
bf47aeb946 [MEDIUM] appsession: add the "request-learn" option
This patch has 2 goals :

1. I wanted to test the appsession feature with a small PHP code,
using PHPSESSID. The problem is that when PHP gets an unknown session
id, it creates a new one with this ID. So, when sending an unknown
session to PHP, persistance is broken : haproxy won't see any new
cookie in the response and will never attach this session to a
specific server.

This also happens when you restart haproxy : the internal hash becomes
empty and all sessions loose their persistance (load balancing the
requests on all backend servers, creating a new session on each one).
For a user, it's like the service is unusable.

The patch modifies the code to make haproxy also learn the persistance
from the client : if no session is sent from the server, then the
session id found in the client part (using the URI or the client cookie)
is used to associated the server that gave the response.

As it's probably not a feature usable in all cases, I added an option
to enable it (by default it's disabled). The syntax of appsession becomes :

  appsession <cookie> len <length> timeout <holdtime> [request-learn]

This helps haproxy repair the persistance (with the risk of losing its
session at the next request, as the user will probably not be load
balanced to the same server the first time).

2. This patch also tries to reduce the memory usage.
Here is a little example to explain the current behaviour :
- Take a Tomcat server where /session.jsp is valid.
- Send a request using a cookie with an unknown value AND a path
  parameter with another unknown value :

  curl -b "JSESSIONID=12345678901234567890123456789012" http://<haproxy>/session.jsp;jsessionid=00000000000000000000000000000001

(I know, it's unexpected to have a request like that on a live service)
Here, haproxy finds the URI session ID and stores it in its internal
hash (with no server associated). But it also finds the cookie session
ID and stores it again.

- As a result, session.jsp sends a new session ID also stored in the
  internal hash, with a server associated.

=> For 1 request, haproxy has stored 3 entries, with only 1 which will be usable

The patch modifies the behaviour to store only 1 entry (maximum).
2009-10-18 11:56:26 +02:00
Willy Tarreau
4e33d8677a [OPTIM] stats: check free space before trying to print
This alone makes a typical HTML stats dump consume 10% CPU less,
because we avoid doing complex printf calls to drop them later.
Only a few common cases have been checked, those which are very
likely to run for nothing.
2009-10-11 23:35:10 +02:00
Willy Tarreau
ea1f5fe28a [MINOR] stats: use a dedicated state to output static data
It is a bit expensive and complex to use to call buffer_feed()
directly from the request parser, and there are risks that some
output messages are lost in case of buffer full. Since most of
these messages are static, let's have a state dedicated to print
these messages and store them in a specific area shared with the
stats in the session. This both reduces code size and risks of
losing output data.
2009-10-11 23:12:51 +02:00
Krzysztof Piotr Oledzki
ba8d7d3916 [MINOR] Add chunk_htmlencode and chunk_asciiencode
Add two functions to encode input chunk replacing
non-printable, non ascii or special characters
with:
 "&#%u;"  - chunk_htmlencode
 "<%02X>" - chunk_asciiencode

Above functions should be used when adding strings, received
from possible unsafe sources, to html stats or logs.
2009-10-10 21:51:16 +02:00
Willy Tarreau
f395017227 [MINOR] proxy: provide function to retrieve backend/server pointers
int get_backend_server(const char *bk_name, const char *sv_name,
                       struct proxy **bk, struct server **sv);

This function scans the list of backends and servers to retrieve the first
backend and the first server with the given names, and sets them in both
parameters. It returns zero if either is not found, or non-zero and sets
the ones it did not found to NULL. If a NULL pointer is passed for the
backend, only the pointer to the server will be updated.
2009-10-10 18:36:25 +02:00
Willy Tarreau
9bcc91e80e [MINOR] buffers: add buffer_feed2() and make buffer_feed() measure string length
It's inconvenient to always have to compute string lengths when calling
buffer_feed(), so change that.
2009-10-10 18:01:44 +02:00
Willy Tarreau
6b2e11be1e [MEDIUM] backend: implement consistent hashing variation
Consistent hashing provides some interesting advantages over common
hashing. It avoids full redistribution in case of a server failure,
or when expanding the farm. This has a cost however, the hashing is
far from being perfect, as we associate a server to a request by
searching the server with the closest key in a tree. Since servers
appear multiple times based on their weights, it is recommended to
use weights larger than approximately 10-20 in order to smoothen
the distribution a bit.

In some cases, playing with weights will be the only solution to
make a server appear more often and increase chances of being picked,
so stats are very important with consistent hashing.

In order to indicate the type of hashing, use :

   hash-type map-based      (default, old one)
   hash-type consistent     (new one)

Consistent hashing can make sense in a cache farm, in order not
to redistribute everyone when a cache changes state. It could also
probably be used for long sessions such as terminal sessions, though
that has not be attempted yet.

More details on this method of hashing here :
  http://www.spiteful.com/2008/03/17/programmers-toolbox-part-3-consistent-hashing/
2009-10-09 07:17:58 +02:00
Krzysztof Piotr Oledzki
6f61b21524 [BUG] Fix NULL pointer dereference in stats_check_uri_auth(), v2
Recent "struct chunk rework" introduced a NULL pointer dereference
and now haproxy segfaults if auth is required for stats but not found.

The reason is that size_t cannot store negative values, but current
code assumes that "len < 0" == uninitialized.

This patch fixes it.
2009-10-04 23:44:45 +02:00
Willy Tarreau
ac68c5d92c [OPTIM] counters: move some max numbers to the counters struct
There are a few remaining max values that need to move to counters.
Also, the counters are more often used than some config information,
so get them closer to the other useful struct members for better cache
efficiency.
2009-10-04 23:26:19 +02:00
Krzysztof Piotr Oledzki
aeebf9ba65 [MEDIUM] Collect & provide separate statistics for sockets, v2
This patch allows to collect & provide separate statistics for each socket.
It can be very useful if you would like to distinguish between traffic
generate by local and remote users or between different types of remote
clients (peerings, domestic, foreign).

Currently no "Session rate" is supported, but adding it should be possible
if we found it useful.
2009-10-04 18:56:02 +02:00
Krzysztof Piotr Oledzki
052d4fd07d [CLEANUP] Move counters to dedicated structures
Move counters from "struct proxy" and "struct server"
to "struct pxcounters" and "struct svcounters".

This patch should make no functional change.
2009-10-04 18:32:39 +02:00
Willy Tarreau
b0c9bc4f95 [MEDIUM] stats: make HTTP stats use an I/O handler
Doing this, we can remove the last BF_HIJACK user and remove
produce_content(). s->data_source could also be removed but
it is currently used to detect if the stats or a server was
used.
2009-10-04 15:56:38 +02:00
Willy Tarreau
f5a885fd28 [MEDIUM] stats: don't use s->ana_state anymore
The stats handler used to store internal states in s->ana_state. Now
we only rely on si->st0 in which we can store as many states as we
have possible outputs. This cleans up the stats code a lot and makes
it more maintainable. It has also reduced code size by a few hundred
bytes.
2009-10-04 14:22:18 +02:00
Willy Tarreau
24955a1000 [MINOR] stats: make stats_dump_raw_to_buffer() use buffer_feed_chunk
Same as previous change. A remaining call to stats_dump_proxy()
still prevents us from completing the update.
2009-10-04 12:17:54 +02:00
Willy Tarreau
7e72a8faf2 [MINOR] stats_dump_sess_to_buffer: use buffer_feed_chunk()
same as previous patch for this function.
2009-10-04 11:00:11 +02:00
Willy Tarreau
61b347342c [MINOR] stats_dump_errors_to_buffer: use buffer_feed_chunk()
We can simplify the code in the stats functions using buffer_feed_chunk()
instead of buffer_write_chunk(). Let's start with this function. This
patch also fixed an issue where we could dump past the end of the capture
buffer if it is shorter than the captured request.
2009-10-04 11:00:11 +02:00
Willy Tarreau
33b230b34a [BUG] stats: don't call buffer_shutw(), but ->shutw() instead
Calling buffer_shutw() marks the buffer as closed but if it was already
closed in the other direction, the stream interface is not marked as
closed, causing infinite loops.

We took this opportunity to completely remove buffer_shutw() and buffer_shutr()
which have no reason to be used at all and which will always cause trouble
when directly called. The stats occurrence was the last one.
2009-10-04 09:19:36 +02:00
Willy Tarreau
39c9ba72a7 [MINOR] lb_map: reorder code in order to ease integration of new hash functions
We need to remove hash map accesses out of backend.c if we want to
later support new hash methods. This patch separates the hash computation
method from the server lookup. It leaves the lookup function to lb_map.c
and calls it with the result of the hash.
2009-10-01 21:11:15 +02:00
Willy Tarreau
f89c1873f8 [CLEANUP] backend: move LB algos to individual files
It was becoming painful to have all the LB algos in backend.c.
Let's move them to their own files. A few hashing functions still
need be broken in two parts, one for the contents and one for the
map position.
2009-10-01 11:19:37 +02:00
Willy Tarreau
78ff5d0a9e [MINOR] include time.h from freq_ctr.h as is uses "now". 2009-10-01 11:05:26 +02:00
Krzysztof Piotr Oledzki
78abe618a8 [MAJOR] struct chunk rework
Add size to struct chunk and simplify the code as there is
no longer required to pass sizeof in chunk_printf().
2009-10-01 10:17:37 +02:00
Willy Tarreau
ca7d4b98d4 [MINOR] backend: uninline some LB functions
There is no reason to inline functions which are used to grab a server
depending on an LB algo. They are large and used at several places.
Uninlining them saves 400 bytes of code.
2009-10-01 09:21:55 +02:00
Willy Tarreau
c5d9c80182 [MINOR] backend: export some functions to recount servers
Those functions will be used by new LB algorithms.
2009-10-01 09:17:05 +02:00
Willy Tarreau
9a42c0d771 [MEDIUM] stats: replace the stats socket analyser with an SI applet
We can get rid of the stats analyser by moving all the stats code
to a stream interface applet. Above being cleaner, it provides new
advantages such as the ability to process requests and responses
from the same function and work only with simple state machines.
There's no need for any hijack hack anymore.

The direct advantage for the user are the interactive mode and the
ability to chain several commands delimited by a semi-colon. Now if
the user types "prompt", he gets a prompt from which he can send
as many requests as he wants. All outputs are terminated by a
blank line followed by a new prompt, so this can be used from
external tools too.

The code is not very clean, it needs some rework, but some part
of the dirty parts are due to the remnants of the hijack mode used
in the old functions we call.

The old AN_REQ_STATS_SOCK analyser flag is now unused and has been
removed.
2009-09-23 23:52:17 +02:00
Willy Tarreau
fb90d94d7a [MINOR] stream_interface: add functions to support running as internal/external tasks
It will soon be necessary to have stream interfaces running as part of
the current task, or as independant tasks. For instance when we want to
implement compression or SSL. It will also be used for applets running
as stream interfaces.

These new functions are used to perform exactly that. Note that it's
still not easy to write a simple echo applet and more functions will
likely be needed.
2009-09-23 23:52:15 +02:00
Willy Tarreau
2e1dd3d213 [BUG] fix buffer_skip() and buffer_si_getline() to correctly handle wrap-arounds
Those two functions did not correctly deal with full buffers and/or
buffers that wrapped around. Buffer_skip() was even able to incorrectly
set buf->w further than the end of buffer if its len argument was wrong,
and buffer_si_getline() was able to incorrectly return a length larger
than the effective buffer data available.
2009-09-23 23:52:14 +02:00
Willy Tarreau
fb0e9209a9 [MINOR] ensure that buffer_feed() and buffer_skip() set BF_*_PARTIAL
It's important that these functions set these flags themselves, otherwise
the callers will always have to do this, and there is no valid reason for
not doing it.
2009-09-23 23:50:57 +02:00
Krzysztof Piotr Oledzki
0960541e49 [MEDIUM] Collect & show information about last health check, v3
Collect information about last health check result,
including L7 code if possible (for example http or smtp
return code) and time took to finish last check.

Health check info is provided on both stats pages (html & csv)
and logged when a server is marked UP or DOWN. Currently active
check are marked with an asterisk, but only in html mode.

Currently there are 14 status codes:
  UNK     -> unknown

  INI     -> initializing
  SOCKERR -> socket error

  L4OK    -> check passed on layer 4, no upper layers testing enabled
  L4TOUT  -> layer 1-4 timeout
  L4CON   -> layer 1-4 connection problem, for example "Connection refused"
              (tcp rst) or "No route to host" (icmp)

  L6OK    -> check passed on layer 6
  L6TOUT  -> layer 6 (SSL) timeout
  L6RSP   -> layer 6 invalid response - protocol error

  L7OK    -> check passed on layer 7
  L7OKC   -> check conditionally passed on layer 7, for example
               404 with disable-on-404
  L7TOUT  -> layer 7 (HTTP/SMTP) timeout
  L7RSP   -> layer 7 invalid response - protocol error
  L7STS   -> layer 7 response error, for example HTTP 5xx
2009-09-23 23:15:36 +02:00
Willy Tarreau
2d028db75a [BUG] buffers: buffer_forward() must not always clear BF_OUT_EMPTY
This flag was unconditionally cleared, which is wrong because we
can enable forwarding on an empty buffer.
2009-09-21 06:25:15 +02:00
Willy Tarreau
269358db93 [BUILD] stream_interface: fix conflicting declaration
stream_int_check_timeouts was declared void while it's an int.
2009-09-21 06:24:42 +02:00
Willy Tarreau
31971e536a [MEDIUM] add support for infinite forwarding
In TCP, we don't want to forward chunks of data, we want to forward
indefinitely. This patch introduces a special value for the amount
of data to be forwarded. When buffer_forward() is called with
BUF_INFINITE_FORWARD, it configures the buffer to never stop
forwarding until the end.
2009-09-20 12:07:52 +02:00
Willy Tarreau
ba0b63d2c7 [MAJOR] buffers: fix the BF_EMPTY flag's meaning
The BF_EMPTY flag was once used to indicate an empty buffer. However,
it was used half the time as meaning the buffer is empty for the reader,
and half the time as meaning there is nothing left to send.

"nothing to send" is only indicated by "->send_max=0 && !pipe". Once
we fix this, we discover that the flag is not used anymore. So the
flags has been renamed BF_OUT_EMPTY and means exactly the condition
above, ie, there is nothing to send.

Doing so has allowed us to remove some unused tests for emptiness,
but also to uncover a certain amount of situations where the flag
was not correctly set or tested.
2009-09-20 08:17:45 +02:00