Commit Graph

5258 Commits

Author SHA1 Message Date
Baptiste Assmann
37bb372ea2 MINOR: DNS counters: increment valid counter
Valid counter was never incremented.
Now it is.
2015-08-08 18:13:59 +02:00
Dragan Dosen
9373fc5811 MINOR: 51d: unable to start haproxy without "51degrees-data-file"
This patch adds a few checks on "global._51degrees.data_file_path" and allows
haproxy to start even when the pattern or trie data file is not specified.

If the "51d" converter is used, a new function "_51d_conv_check" will check
"global._51degrees.data_file_path" and displays a warning if necessary.

In src/haproxy.c, the global 51Degrees "cache_size" has moved outside of the
FIFTYONEDEGREES_H_PATTERN_INCLUDED ifdef block.
2015-08-08 12:15:55 +02:00
Willy Tarreau
067fceffb3 DOC: internals: document next steps for HTTP connection reuse
This is mostly based on the design notes and experiments that were
not turned into final code yet.
2015-08-06 16:40:59 +02:00
Willy Tarreau
30631956d6 DOC: document the new http-reuse directive
This documents the 4 strategies : never, safe, aggressive, always.
2015-08-06 16:35:03 +02:00
Willy Tarreau
449d74a906 MEDIUM: backend: add the "http-reuse aggressive" strategy
This strategy is less extreme than "always", it only dispatches first
requests to validated reused connections, and moves a connection from
the idle list to the safe list once it has seen a second request, thus
proving that it could be reused.
2015-08-06 16:29:01 +02:00
Willy Tarreau
7017cb040c MINOR: server: add a list of safe, already reused idle connections
These ones are considered safe as they have already been reused.
They will be useful in "aggressive" and "always" http-reuse modes
in order to place the first request of a connection with the least
risk.
2015-08-06 16:29:01 +02:00
Willy Tarreau
161d45ffc7 MEDIUM: backend: implement "http-reuse safe"
The "safe" mode consists in picking existing connections only when
processing a request that's not the first one from a connection. This
ensures that in case where the server finally times out and closes, the
client can decide to replay idempotent requests.
2015-08-06 11:50:53 +02:00
Willy Tarreau
efb90f9dd3 MAJOR: backend: improve the connection reuse mechanism
Now instead of closing the existing connection attached to the
stream interface, we first check if the one we pick was attached to
another stream interface, in which case the connections are swapped
if possible (eg: if the current connection is not private). That way
the previous connection remains attached to an existing session and
significantly increases the chances of being reused.
2015-08-06 11:41:06 +02:00
Willy Tarreau
8dff998b91 MAJOR: backend: initial work towards connection reuse
In connect_server(), if we don't have a connection attached to the
stream-int, we first look into the server's idle_conns list and we
pick the first one there, we detach it from its owner if it had one.
If we used to have a connection, we close it.

This mechanism works well but doesn't scale : as servers increase,
the likeliness that the connection attached to the stream interface
doesn't match the server and gets closed increases.
2015-08-06 11:34:21 +02:00
Willy Tarreau
d8fecee291 MINOR: config: add new setting "http-reuse"
For now it only supports "never", meaning that we never want to reuse a
shared connection, and "always", meaning that we can use any connection
that was not marked private. When "never" is set, this also implies that
no idle connection may become a shared one.
2015-08-06 11:14:20 +02:00
Willy Tarreau
387ebf84dd MINOR: connection: add a new flag CO_FL_PRIVATE
This flag is set on an outgoing connection when this connection gets
some properties that must not be shared with other connections, such
as dynamic transparent source binding, SNI or a proxy protocol header,
or an authentication challenge from the server. This will be needed
later to implement connection reuse.
2015-08-06 11:14:17 +02:00
Willy Tarreau
173a1c6b43 MINOR: server: add a list of already used idle connections
There's a difference with the other idle conns in that these new
ones have already been used and may be reused by other streams.
2015-08-06 11:13:47 +02:00
Willy Tarreau
4320eaac62 MINOR: stream-int: make si_idle_conn() only accept valid connections
This function is now dedicated to idle connections only, which means
that it must not be used without any endpoint nor anything not a
connection. The connection remains attached to the stream interface.
2015-08-06 11:11:10 +02:00
Willy Tarreau
323a2d925c MEDIUM: stream-int: queue idle connections at the server
Now we get a per-server list of all idle connections. That way we'll
be able to reclaim them upon shortage later.
2015-08-06 11:06:25 +02:00
Willy Tarreau
d75d40e9a8 MINOR: connection: add a new list member in the connection struct
This list member will be used to attach a connection to a list of
idle, reusable or queued connections. It's unused for now. Given
that it's not expected to be used more than a few times per session,
the member was put after the target, in the area starting at the
second cache line of the structure.
2015-08-06 11:00:34 +02:00
Willy Tarreau
600802aef0 MINOR: server: add a list of private idle connections
For now it's not populated but we have the list entry. It will carry
all idle connections that sessions don't want to share. They may be
used later to reclaim connections upon socket shortage for example.
2015-08-06 10:59:08 +02:00
Willy Tarreau
41b09f4bbc MINOR: stream-int: add new function si_detach_endpoint()
This function only detaches the endpoint from the stream-int and
optionally returns the original pointer. This will be needed to
steal idle connections from other connections.
2015-08-05 21:54:25 +02:00
Willy Tarreau
973a54235f MEDIUM: stream-int: simplify si_alloc_conn()
Since we now always call this function with the reuse parameter cleared,
let's simplify the function's logic as it cannot return the existing
connection anymore. The savings on this inline function are appreciable
(240 bytes) :

$ size haproxy.old haproxy.new
   text    data     bss     dec     hex filename
1020383   40816   36928 1098127  10c18f haproxy.old
1020143   40816   36928 1097887  10c09f haproxy.new
2015-08-05 21:51:09 +02:00
Willy Tarreau
c12b5e663d MEDIUM: backend: don't call si_alloc_conn() when we reuse a valid connection
connect_server() already does most of the check that is done again in
si_alloc_conn(), so let's simply reuse the existing connection instead
of calling the function again. It will also simplify the connection
reuse.

Indeed, for reuse to be set, it also requires srv_conn to be valid. In the
end, the only situation where we have to release the existing connection
and allocate a new one is when reuse == 0.
2015-08-05 21:42:12 +02:00
Willy Tarreau
7b00492ce3 CLEANUP: backend: factor out objt_server() in connect_server()
objt_server() is called multiple times at various places while some
places already make use of srv for this. Let's move the call at the
top of the function and use it all over the place.
2015-08-05 10:12:47 +02:00
Baptiste Assmann
c4aabae0e9 DOC: resolve-prefer default value and default-server update
By the code, resolve-prefer defaults to IPv6 and is also available in
the default-server directive.
These information were missing or wrong in the documentation.
2015-08-05 00:39:32 +02:00
Baptiste Assmann
a332299ad3 DOC: typos in tcp-check expect examples
The match keyword 'string' was missing in a few tcp-check expect example
statements.
2015-08-05 00:39:27 +02:00
Baptiste Assmann
ea849c0cca DOC: typo in 'redirect', 302 code meaning
302 means a temprary move, not a permanent one
2015-08-05 00:39:23 +02:00
Daniel Jakots
17d228be14 DOC/MINOR: fix OpenBSD versions where haproxy works 2015-07-29 17:39:35 +02:00
Remi Gacogne
7fb9de25e2 MINOR: stream: initialize the current_rule field to NULL on stream init
Currently it is possible for the current_rule field to be evaluated before
being set, leading to valgrind complaining:

==16783== Conditional jump or move depends on uninitialised value(s)
==16783==    at 0x44E662: http_res_get_intercept_rule (proto_http.c:3730)
==16783==    by 0x44E662: http_process_res_common (proto_http.c:6528)
==16783==    by 0x4797B7: process_stream (stream.c:1851)
==16783==    by 0x414634: process_runnable_tasks (task.c:238)
==16783==    by 0x40B02F: run_poll_loop (haproxy.c:1528)
==16783==    by 0x407F25: main (haproxy.c:1887)

This was introduced by commit 152b81e7b2565862956af883820d4f79177d0651.
2015-07-26 21:50:50 +02:00
Jrmie Courrges-Anglas
17c3f6284c BUILD: ssl: Allow building against libssl without SSLv3.
If SSLv3 is explicitely requested but not available, warn the user and
bail out.
2015-07-26 08:11:57 +02:00
Vincent Bernat
e192cbb585 BUILD: link with libdl if needed for Lua support
On platforms where the dl*() functions are not part of the libc, a
program linking Lua also needs to link to libdl.

Moreover, on platforms using a gold linker with the --as-needed flag,
the libdl library needs to be linked after linking Lua, otherwise, it
won't be marked as needed and will be discarded and its symbols won't be
present at the end of the linking phase.

Ubuntu enables the --as-needed flag by default. Other distributions may
advertise its use, like Gentoo.
2015-07-23 09:47:02 +02:00
Pieter Baauw
ed35c371dc BUG/MEDIUM: mailer: DATA part must be terminated with <CRLF>.<CRLF>
The dot is send in the wrong place.
As defined in https://www.ietf.org/rfc/rfc2821.txt 'the character sequence "<CRLF>.<CRLF>" ends the mail text'
2015-07-22 22:39:39 +02:00
Willy Tarreau
50bdda6e51 [RELEASE] Released version 1.6-dev3
Released version 1.6-dev3 with the following main changes :
    - CLEANUP: sample: generalize sample_fetch_string() as sample_fetch_as_type()
    - MEDIUM: http: Add new 'set-src' option to http-request
    - DOC usesrc root privileges requirments
    - BUG/MINOR: dns: wrong time unit for some DNS default parameters
    - MINOR: proxy: bit field for proxy_find_best_match diff status
    - MINOR: server: new server flag: SRV_F_FORCED_ID
    - MINOR: server: server_find functions: id, name, best_match
    - DOC: dns: fix chapters syntax
    - BUILD/MINOR: tools: rename popcount to my_popcountl
    - BUILD: add netbsd TARGET
    - MEDIUM: 51Degrees code refactoring and cleanup
    - MEDIUM: 51d: add LRU-based cache on User-Agent string detection
    - DOC: add notes about the "51degrees-cache-size" parameter
    - BUG/MEDIUM: 51d: possible incorrect operations on smp->data.str.str
    - BUG/MAJOR: connection: fix TLV offset calculation for proxy protocol v2 parsing
    - MINOR: Add sample fetch to detect Supported Elliptic Curves Extension
    - BUG/MINOR: payload: Add volatile flag to smp_fetch_req_ssl_ec_ext
    - BUG/MINOR: lua: type error in the arguments wrapper
    - CLEANUP: vars: remove unused struct
    - BUG/MINOR: http/sample: gmtime/localtime can fail
    - MINOR: standard: add 64 bits conversion functions
    - MAJOR: sample: converts uint and sint in 64 bits signed integer
    - MAJOR: arg: converts uint and sint in sint
    - MEDIUM: sample: switch to saturated arithmetic
    - MINOR: vars: returns variable content
    - MEDIUM: vars/sample: operators can use variables as parameter
    - BUG/MINOR: ssl: fix smp_fetch_ssl_fc_session_id
    - BUILD/MINOR: lua: fix a harmless build warning
    - BUILD/MINOR: stats: fix build warning due to condition always true
    - BUG/MAJOR: lru: fix unconditional call to free due to unexpected semi-colon
    - BUG/MEDIUM: logs: fix improper systematic use of quotes with a few tags
    - BUILD/MINOR: lua: ensure that hlua_ctx_destroy is properly defined
    - BUG/MEDIUM: lru: fix possible memory leak when ->free() is used
    - MINOR: vars: make the accounting not depend on the stream
    - MEDIUM: vars: move the session variables to the session, not the stream
    - BUG/MEDIUM: vars: do not freeze the connection when the expression cannot be fetched
    - BUG/MAJOR: buffers: make the buffer_slow_realign() function respect output data
    - BUG/MAJOR: tcp: tcp rulesets were still broken
    - MINOR: stats: improve compression stats reporting
    - MINOR: ssl: make self-generated certs also work with raw IPv6 addresses
    - CLEANUP: ssl: make ssl_sock_generated_cert_serial() take a const
    - CLEANUP: ssl: make ssl_sock_generate_certificate() use ssl_sock_generated_cert_serial()
    - BUG/MINOR: log: missing some ARGC_* entries in fmt_directives()
    - MINOR: args: add new context for servers
    - MINOR: stream: maintain consistence between channel_forward and HTTP forward
    - MINOR: ssl: provide ia function to set the SNI extension on a connection
    - MEDIUM: ssl: add sni support on the server lines
    - CLEANUP: stream: remove a useless call to si_detach()
    - CLEANUP: stream-int: fix a few outdated comments about stream_int_register_handler()
    - CLEANUP: stream-int: remove stream_int_unregister_handler() and si_detach()
    - MINOR: stream-int: only use si_release_endpoint() to release a connection
    - MINOR: standard: provide htonll() and ntohll()
    - CLEANUP/MINOR: dns: dns_str_to_dn_label() only needs a const char
    - BUG/MAJOR: dns: fix the length of the string to be copied
2015-07-22 17:32:56 +02:00
Willy Tarreau
d69d6f3678 BUG/MAJOR: dns: fix the length of the string to be copied
Jan A. Bruder reported that some very specific hostnames on server
lines were causing haproxy to crash on startup. Given that hist
backtrace showed some heap corruption, it was obvious there was an
overflow somewhere. The bug in fact is a typo in dns_str_to_dn_label()
which mistakenly copies one extra byte from the host name into the
output value, thus effectively corrupting the structure.

The bug triggers while parsing the next server of similar length
after the corruption, which generally triggers at config time but
could theorically crash at any moment during runtime depending on
what malloc sizes are needed next. This is why it's tagged major.

No backport is needed, this bug was introduced in 1.6-dev2.
2015-07-22 16:53:22 +02:00
Willy Tarreau
2100b49122 CLEANUP/MINOR: dns: dns_str_to_dn_label() only needs a const char
The string is an input, let's constify it.
2015-07-22 16:42:43 +02:00
Thierry FOURNIER
5d86fae234 MEDIUM: vars/sample: operators can use variables as parameter
This patch allow the existing operators to take a variable as parameter.
This is useful to add the content of two variables. This patch modify
the behavior of operators.
2015-07-22 00:48:24 +02:00
Thierry FOURNIER
fd77e05f65 MINOR: vars: returns variable content
This patch copy the content of a variable in a sample. The function
returns 0 if the variable is not found.
2015-07-22 00:48:24 +02:00
Thierry FOURNIER
00c005c726 MEDIUM: sample: switch to saturated arithmetic
This patch check calculus for overflow and returns capped values.
This permits to protect against integer overflow in certain operations
involving ratios, percentages, limits or anything. That can sometimes
be critically important with some operations (eg: content-length < X).
2015-07-22 00:48:24 +02:00
Thierry FOURNIER
bf65cd4d77 MAJOR: arg: converts uint and sint in sint
This patch removes the 32 bits unsigned integer and the 32 bit signed
integer. It replaces these types by a unique type 64 bit signed.
2015-07-22 00:48:23 +02:00
Thierry FOURNIER
07ee64ef4d MAJOR: sample: converts uint and sint in 64 bits signed integer
This patch removes the 32 bits unsigned integer and the 32 bit signed
integer. It replaces these types by a unique type 64 bit signed.

This makes easy the usage of integer and clarify signed and unsigned use.
With the previous version, signed and unsigned are used ones in place of
others, and sometimes the converter loose the sign. For example, divisions
are processed with "unsigned", if one entry is negative, the result is
wrong.

Note that the integer pattern matching and dotted version pattern matching
are already working with signed 64 bits integer values.

There is one user-visible change : the "uint()" and "sint()" sample fetch
functions which used to return a constant integer have been replaced with
a new more natural, unified "int()" function. These functions were only
introduced in the latest 1.6-dev2 so there's no impact on regular
deployments.
2015-07-22 00:48:23 +02:00
Willy Tarreau
5b4dd683cb MINOR: standard: provide htonll() and ntohll()
These are the 64-bit equivalent of htonl() and ntohl(). They're a bit
tricky in order to avoid expensive operations.

The principle consists in letting the compiler detect we're playing
with a union and simplify most or all operations. The asm-optimized
htonl() version involving bswap (x86) / rev (arm) / other is a single
operation on little endian, or a NOP on big-endian. In both cases,
this lets the compiler "see" that we're rebuilding a 64-bit word from
two 32-bit quantities that fit into a 32-bit register. In big endian,
the whole code is optimized out. In little endian, with a decent compiler,
a few bswap and 2 shifts are left, which is the minimum acceptable.
2015-07-21 23:50:06 +02:00
Thierry FOURNIER
763a5d85f7 MINOR: standard: add 64 bits conversion functions
This patch adds 3 functions for 64 bit integer conversion.

 * lltoa_r : converts signed 64 bit integer to string
 * read_uint64 : converts from string to signed 64 bits integer with capping
 * read_int64 : converts from string to unsigned 64 bits integer with capping
2015-07-21 23:27:10 +02:00
Baptiste Assmann
19a106d24a MINOR: server: server_find functions: id, name, best_match
This patch introduces three new functions which can be used to find a
server in a farm using different server information:
- server unique id (srv->puid)
- server name
- find best match using either name or unique id

When performing best matching, the following applies:
 - use the server name first (if provided)
 - use the server id if provided
 in any case, the function can update the caller about mismatches
 encountered.
2015-07-21 23:24:16 +02:00
Baptiste Assmann
7cc419ae1d MINOR: server: new server flag: SRV_F_FORCED_ID
This flag aims at reporting whether the server unique id (srv->puid) has
been forced by the administrator in HAProxy's configuration.
If not set, it means HAProxy has generated automatically the server's
unique id.
2015-07-21 23:24:16 +02:00
Baptiste Assmann
8a027ccb38 MINOR: proxy: bit field for proxy_find_best_match diff status
function proxy_find_best_match can update the caller by updating an int
provided in argument.
For now, proxy_find_best_match hardcode bit values 0x01, 0x02 and 0x04,
which is not understandable when reading a code exploiting them.

This patch defines 3 macros with a more explicit wording, so further
reading of a code exploiting the magic bit values will be understandable
more easily.
2015-07-21 23:24:16 +02:00
Thierry FOURNIER
fac9ccfb70 BUG/MINOR: http/sample: gmtime/localtime can fail
The man said that gmtime() and localtime() can return a NULL value.
This is not tested. It appears that all the values of a 32 bit integer
are valid, but it is better to check the return of these functions.

However, if the integer move from 32 bits to 64 bits, some 64 values
can be unsupported.
2015-07-20 12:21:35 +02:00
Nenad Merdanovic
8a39a1fcc3 BUG/MINOR: payload: Add volatile flag to smp_fetch_req_ssl_ec_ext
This bug was introduced in 5fc7d7e. No backport to 1.5 needed.

Signed-off-by: Nenad Merdanovic <nmerdan@anine.io>
2015-07-20 10:11:44 +02:00
Willy Tarreau
d7c7475ad6 MINOR: stream-int: only use si_release_endpoint() to release a connection
Change si_alloc_conn() to call si_release_endpoint() instead of
open-coding the connection releasing code when reuse is disabled.
This fuses the code with the one already dealing with applets, makes
it shorter and helps centralizing the connection freeing logic at a
single place.
2015-07-19 19:14:10 +02:00
Willy Tarreau
7a08d3b2d7 CLEANUP: stream-int: remove stream_int_unregister_handler() and si_detach()
The former was not used anymore and the latter was only used by the former.
They were only aliases to other existing functions anyway.
2015-07-19 18:48:20 +02:00
Willy Tarreau
a9ff5e64c1 CLEANUP: stream-int: fix a few outdated comments about stream_int_register_handler()
They were not updated after the infrastructure change.
2015-07-19 18:46:30 +02:00
Willy Tarreau
069b0c8e6f CLEANUP: stream: remove a useless call to si_detach()
It's pointless to call si_detach() after si_reset() since it does
nothing due to si->end being NULL.
2015-07-19 17:57:02 +02:00
Baptiste Assmann
4c5490ab9c BUG/MINOR: dns: wrong time unit for some DNS default parameters
Madison May reported that the timeout applied by the default
configuration is inproperly set up.
This patch fix this:
 - hold valid default to 10s
 - timeout retry default to 1s
2015-07-18 08:22:01 +02:00
Baptiste Assmann
91bd337d90 DOC usesrc root privileges requirments
The "usesrc" parameter of the source statement requires root privileges.
2015-07-18 07:33:40 +02:00
Thierry FOURNIER
25f4e3ef33 CLEANUP: vars: remove unused struct
The commit "MEDIUM: vars: move the session variables to the session, not the stream" (ebcd4844e82a4198ea5d98fe491a46267da1d1ec")
moves the variables from the stream to the session. It forgot to remove
the stream definition of the "vars_sess".
2015-07-10 16:30:08 +02:00