Commit Graph

31 Commits

Author SHA1 Message Date
Frédéric Lécaille
523cc9e858 MEDIUM: peers: Fix a peer stick-tables synchronization issue.
During the stick-table teaching process which occurs at reloading/restart time,
expiration dates of stick-tables entries were not synchronized between peers.

This patch adds two new stick-table messages to provide such a synchronization feature.

As these new messages are not supported by older haproxy peers protocol versions,
this patch increments peers protol version, from 2.0 to 2.1, to help in detecting/supporting
such older peers protocol implementations so that new versions might still be able
to transparently communicate with a newer one.

[wt: technically speaking it would be nice to have this backported into 1.6
 as some people who reload often are affected by this design limitation, but
 it's not a totally transparent change that may make certain users feel
 reluctant to upgrade older versions. Let's let it cook in 1.7 first and
 decide later]
2016-10-17 19:44:35 +02:00
Willy Tarreau
e2dc1fa8ca MEDIUM: stick-table: remove the now duplicate find_stktable() function
Since proxy_tbl_by_name() already does the same job, let's not keep
duplicate functions and use this one only.
2015-05-26 12:08:07 +02:00
Willy Tarreau
7698c9080a REORG: stktable: move the stkctr_* functions from stream to sticktable
These ones are not stream-specific at all and will be needed outside of
stream, so let's move them to stick_tables where struct stkctr is defined.
2015-04-06 11:37:30 +02:00
Willy Tarreau
192252e2d8 MAJOR: sample: pass a pointer to the session to each sample fetch function
Many such function need a session, and till now they used to dereference
the stream. Once we remove the stream from the embryonic session, this
will not be possible anymore.

So as of now, sample fetch functions will be called with this :

   - sess = NULL,  strm = NULL                     : never
   - sess = valid, strm = NULL                     : tcp-req connection
   - sess = valid, strm = valid, strm->txn = NULL  : tcp-req content
   - sess = valid, strm = valid, strm->txn = valid : http-req / http-res
2015-04-06 11:37:25 +02:00
Willy Tarreau
15e91e1b36 MAJOR: sample: don't pass l7 anymore to sample fetch functions
All of them can now retrieve the HTTP transaction *if it exists* from
the stream and be sure to get NULL there when called with an embryonic
session.

The patch is a bit large because many locations were touched (all fetch
functions had to have their prototype adjusted). The opportunity was
taken to also uniformize the call names (the stream is now always "strm"
instead of "l4") and to fix indent where it was broken. This way when
we later introduce the session here there will be less confusion.
2015-04-06 11:35:53 +02:00
Willy Tarreau
87b09668be REORG/MAJOR: session: rename the "session" entity to "stream"
With HTTP/2, we'll have to support multiplexed streams. A stream is in
fact the largest part of what we currently call a session, it has buffers,
logs, etc.

In order to catch any error, this commit removes any reference to the
struct session and tries to rename most "session" occurrences in function
names to "stream" and "sess" to "strm" when that's related to a session.

The files stream.{c,h} were added and session.{c,h} removed.

The session will be reintroduced later and a few parts of the stream
will progressively be moved overthere. It will more or less contain
only what we need in an embryonic session.

Sample fetch functions and converters will have to change a bit so
that they'll use an L5 (session) instead of what's currently called
"L4" which is in fact L6 for now.

Once all changes are completed, we should see approximately this :

   L7 - http_txn
   L6 - stream
   L5 - session
   L4 - connection | applet

There will be at most one http_txn per stream, and a same session will
possibly be referenced by multiple streams. A connection will point to
a session and to a stream. The session will hold all the information
we need to keep even when we don't yet have a stream.

Some more cleanup is needed because some code was already far from
being clean. The server queue management still refers to sessions at
many places while comments talk about connections. This will have to
be cleaned up once we have a server-side connection pool manager.
Stream flags "SN_*" still need to be renamed, it doesn't seem like
any of them will need to move to the session.
2015-04-06 11:23:56 +02:00
Willy Tarreau
edee1d60b7 MEDIUM: stick-table: make it easier to register extra data types
Some users want to add their own data types to stick tables. We don't
want to use a linked list here for performance reasons, so we need to
continue to use an indexed array. This patch allows one to reserve a
compile-time-defined number of extra data types by setting the new
macro STKTABLE_EXTRA_DATA_TYPES to anything greater than zero, keeping
in mind that anything larger will slightly inflate the memory consumed
by stick tables (not per entry though).

Then calling stktable_register_data_store() with the new keyword will
either register a new keyword or fail if the desired entry was already
taken or the keyword already registered.

Note that this patch does not dictate how the data will be used, it only
offers the possibility to create new keywords and have an index to
reference them in the config and in the tables. The caller will not be
able to use stktable_data_cast() and will have to explicitly cast the
stable pointers to the expected types. It can be used for experimentation
as well.
2014-07-15 19:14:52 +02:00
Willy Tarreau
8fed9037cd MEDIUM: stick-table: implement lookup from a sample fetch
Currently we have stktable_fetch_key() which fetches a sample according
to an expression and returns a stick table key, but we also need a function
which does only the second half of it from a known sample. So let's cut the
function in two and introduce smp_to_stkey() to perform this lookup. The
first function was adapted to make use of it in order to avoid code
duplication.
2014-07-10 16:43:44 +02:00
Willy Tarreau
b5975defba MINOR: stick-table: make stktable_fetch_key() indicate why it failed
stktable_fetch_key() does not indicate whether it returns NULL because
the input sample was not found or because it's unstable. It causes trouble
with track-sc* rules. Just like with sample_fetch_string(), we want it to
be able to give more information to the caller about what it found. Thus,
now we use the pointer to a sample passed by the caller, and fill it with
the information we have about the sample. That way, even if we return NULL,
the caller has the ability to check whether a sample was found and if it is
still changing or not.
2014-06-25 17:17:53 +02:00
Willy Tarreau
3a925c155d MEDIUM: stick-tables: flush old entries upon soft-stop
When a process with large stick tables is replaced by a new one and remains
present until the last connection finishes, it keeps these data in memory
for nothing since they will never be used anymore by incoming connections,
except during syncing with the new process. This is especially problematic
when dealing with long session protocols such as WebSocket as it becomes
possible to stack many processes and eat a lot of memory.

So the idea here is to know if a table still needs to be synced or not,
and to purge all unused entries once the sync is complete. This means that
after a few hundred milliseconds when everything has been synchronized with
the new process, only a few entries will remain allocated (only the ones
held by sessions during the restart) and all the remaining memory will be
freed.

Note that we carefully do that only after the grace period is expired so as
not to impact a possible proxy that needs to accept a few more connections
before leaving.

Doing this required to add a sync counter to the stick tables, to know how
many peer sync sessions are still in progress in order not to flush the entries
until all synchronizations are completed.
2013-09-04 17:54:01 +02:00
Willy Tarreau
07115412d3 MEDIUM: stick-table: allocate the table key of size buffer size
Keys are copied from samples to stick_table_key. If a key is larger
than the stick_table_key, we have an overflow. In pratice it does not
happen because it requires :
   1) a configuration with tune.bufsize larger than BUFSIZE (common)
   2) a stick-table configured with keys strictly larger than buffers
   3) extraction of data larger than BUFSIZE (eg: using payload())

Points 2 and 3 don't make any sense for a real world configuration. That
said the issue needs be fixed. The solution consists in allocating it the
same size as the global buffer size, just like the samples. This fixes the
issue.
2012-10-29 21:56:59 +01:00
Willy Tarreau
1278578487 REORG: use the name "sample" instead of "pattern" to designate extracted data
This is mainly a massive renaming in the code to get it in line with the
calling convention. Next patch will rename a few files to complete this
operation.
2012-05-08 20:57:20 +02:00
Willy Tarreau
32a6f2e572 MEDIUM: acl/pattern: use the same direction scheme
Patterns were using a bitmask to indicate if request or response was desired
in fetch functions and keywords. ACLs were using a bitmask in fetch keywords
and a single bit in fetch functions. ACLs were also using an ACL_PARTIAL bit
in fetch functions indicating that a non-final fetch was performed, which was
an abuse of the existing direction flag.

The change now consists in using :
  - a capabilities field for fetch keywords => SMP_CAP_REQ/RES to indicate
    if a keyword supports requests, responses, both, etc...
  - an option field for fetch functions to indicate what the caller expects
    (request/response, final/non-final)

The ACL_PARTIAL bit was reversed to get SMP_OPT_FINAL as it's more explicit
to know we're working on a final buffer than on a non-final one.

ACL_DIR_* were removed, as well as PATTERN_FETCH_*. L4 fetches were improved
to support being called on responses too since they're still available.

The <dir> field of all fetch functions was changed to <opt> which is now
unsigned.

The patch is large but mostly made of cosmetic changes to accomodate this, as
almost no logic change happened.
2012-05-08 20:57:17 +02:00
Emeric Brun
85e77c7f0d [MEDIUM] Create updates tree on stick table to manage sync. 2010-11-11 09:29:08 +01:00
Emeric Brun
485479d8e9 [MEDIUM] Create new protected pattern types CONSTSTRING and CONSTDATA to force memcpy if data from protected areas need to be manipulated.
Enhance pattern convs and fetch argument parsing, now fetchs and convs callbacks used typed args.
Add more details on error messages on parsing pattern expression function.
Update existing pattern convs and fetchs to new proto.
Create stick table key type "binary".
Manage Truncation and padding if pattern's fetch-converted result don't match table key size.
2010-11-11 09:29:07 +01:00
Emeric Brun
c89a57284a [BUG] stick table entries expire on counters updates/read or show table, even if there is no "expire" parameter 2010-11-11 09:28:18 +01:00
Willy Tarreau
f6efda1189 [MEDIUM] session counters: automatically remove expired entries.
When a ref_cnt goes down to zero and the entry is expired, remove it.
2010-08-10 18:04:15 +02:00
Willy Tarreau
3b9c6e053e [MEDIUM] stick-table: make use of generic types for stored data
It's a bit cumbersome to have to know all possible storable types
from the stats interface. Instead, let's have generic types for
all data, which will facilitate their manipulation.
2010-08-10 18:04:14 +02:00
Willy Tarreau
1f7e925d6a [MINOR] stktable: add a stktable_update_key() function
This function looks up a key, updates its expiration date, or creates
it if it was not found. acl_fetch_src_updt_conn_cnt() was updated to
make use of it.
2010-08-10 18:04:14 +02:00
Willy Tarreau
ac78288eaf [MEDIUM] stick-tables: add stored data argument type checking
We're now able to return errors based on the validity of an argument
passed to a stick-table store data type. We also support ARG_T_DELAY
to pass delays to stored data types (eg: for rate counters).
2010-08-10 18:04:13 +02:00
Willy Tarreau
888617dc3b [MEDIUM] stick-tables: add support for arguments to data_types
Some data types will require arguments (eg: period for a rate counter).
This patch adds support for such arguments between parenthesis in the
"store" directive of the stick-table statement. Right now only integers
are supported.
2010-08-10 18:04:13 +02:00
Willy Tarreau
4a0347add0 [MINOR] stick-table: provide a table lookup function
We'll often need to lookup a table by its name. This will change
in the future once we can resolve these names on startup.
2010-08-10 18:04:12 +02:00
Willy Tarreau
9ba2dcc86c [MAJOR] session: add track-counters to track counters related to the session
This patch adds the ability to set a pointer in the session to an
entry in a stick table which holds various counters related to a
specific pattern.

Right now the syntax matches the target syntax and only the "src"
pattern can be specified, to track counters related to the session's
IPv4 source address. There is a special function to extract it and
convert it to a key. But the goal is to be able to later support as
many patterns as for the stick rules, and get rid of the specific
function.

The "track-counters" directive may only be set in a "tcp-request"
statement right now. Only the first one applies. Probably that later
we'll support multi-criteria tracking for a single session and that
we'll have to name tracking pointers.

No counter is updated right now, only the refcount is. Some subsequent
patches will have to bring that feature.
2010-08-10 18:04:12 +02:00
Willy Tarreau
cb18364ca7 [MEDIUM] stick_table: separate storage and update of session entries
When an entry already exists, we just need to update its expiration
timer. Let's have a dedicated function for that instead of spreading
open code everywhere.

This change also ensures that an update of an existing sticky session
really leads to an update of its expiration timer, which was apparently
not the case till now. This point needs to be checked in 1.4.
2010-06-14 15:10:26 +02:00
Willy Tarreau
41883e2041 [MINOR] stick_table: export the stick_table_key
This one is huge and will be needed by other portions of code for various
data lookups. Let's not have them allocate it in the stack.
2010-06-14 15:10:25 +02:00
Willy Tarreau
68129b90eb [MINOR] stick_table: provide functions to return stksess data from a type
This function does the indirection job in the table to find the pointer
to the real data matching the requested type.
2010-06-14 15:10:25 +02:00
Willy Tarreau
f16d2b8c1b [MEDIUM] stick_table: don't overwrite data when storing an entry
Till now sticky sessions only held server IDs. Now there are other
data types so it is not acceptable anymore to overwrite the server ID
when writing something. The server ID must then only be written from
the caller when appropriate. Doing this has also led to separate
lookup and storage.
2010-06-14 15:10:24 +02:00
Willy Tarreau
08d5f98294 [MEDIUM] stick_table: add room for extra data types
The stick_tables will now be able to store extra data for a same key.
A limited set of extra data types will be defined and for each of them
an offset in the sticky session will be assigned at startup time. All
of this information will be stored in the stick table.

The extra data types will have to be specified after the new "store"
keyword of the "stick-table" directive, which will reserve some space
for them.
2010-06-14 15:10:24 +02:00
Willy Tarreau
f0b38bfc33 [CLEANUP] stick_table: move pattern to key functions to stick_table.c
pattern.c depended on stick_table while in fact it should be the opposite.
So we move from pattern.c everything related to stick_tables and invert the
dependency. That way the code becomes more logical and intuitive.
2010-06-14 15:10:24 +02:00
Willy Tarreau
393379c3e0 [MINOR] stick_table: add support for variable-sized data
Right now we're only able to store a server ID in a sticky session.
The goal is to be able to store anything whose size is known at startup
time. For this, we store the extra data before the stksess pointer,
using a negative offset. It will then be easy to cumulate multiple
data provided they each have their own offset.
2010-06-14 15:10:23 +02:00
Emeric Brun
3bd697e071 [MEDIUM] Add stick table (persistence) management functions and types 2010-01-12 11:23:15 +01:00