Commit Graph

1264 Commits

Author SHA1 Message Date
Baptiste Assmann
9d8dbbc56b MINOR: dns: Maximum DNS udp payload set to 8192
Following up DNS extension introduction, this patch aims at making the
computation of the maximum number of records in DNS response dynamic.
This computation is based on the announced payload size accepted by
HAProxy.
2017-08-22 11:39:57 +02:00
Baptiste Assmann
747359eeca BUG/MINOR: dns: server set by SRV records stay in "no resolution" status
This patch fixes a bug where some servers managed by SRV record query
types never ever recover from a "no resolution" status.
The problem is due to a wrong function called when breaking the
server/resolution (A/AAAA) relationship: this is performed when a server's SRV
record disappear from the SRV response.
2017-08-22 11:34:49 +02:00
Fr�d�ric L�caille
6ca71a9297 BUG/MINOR: Wrong type used as argument for spoe_decode_buffer().
Contrary to 64-bits libCs where size_t type size is 8, on systems with 32-bits
size of size_t is 4 (the size of a long) which does not equal to size of uint64_t type.
This was revealed by such GCC warnings on 32bits systems:

src/flt_spoe.c:2259:40: warning: passing argument 4 of spoe_decode_buffer from
incompatible pointer type
  if (spoe_decode_buffer(&p, end, &str, &sz) == -1)
                                         ^
As the already existing code using spoe_decode_buffer() already use such pointers to
uint64_t, in place of pointer to size_t ;), most of this code is in contrib directory,
this simple patch modifies the prototype of spoe_decode_buffer() so that to use a
pointer to uint64_t in place of a pointer to size_t, uint64_t type being the type
finally required for decode_varint().
2017-08-22 11:27:20 +02:00
Willy Tarreau
a5480694bf MINOR: http: export some of the HTTP parser macros
The two macros EXPECT_LF_HERE and EAT_AND_JUMP_OR_RETURN were exported
for use outside the HTTP parser. They now take extra arguments to avoid
implicit pointers and jump labels. These will be used to reimplement a
minimalist HTTP/1 parser in the H1->H2 gateway.
2017-08-18 13:38:47 +02:00
Baptiste Assmann
2af08fe3de MINOR: dns: enabled edns0 extension and make accpeted payload size tunable
Edns extensions may be used to negotiate some settings between a DNS
client and a server.
For now we only use it to announce the maximum response payload size accpeted
by HAProxy.
This size can be set through a configuration parameter in the resolvers
section. If not set, it defaults to 512 bytes.
2017-08-18 11:25:56 +02:00
Emmanuel Hocdet
15969297af BUILD: ssl: replace SSL_CTX_get0_privatekey for openssl < 1.0.2
Commit 48a8332a introduce SSL_CTX_get0_privatekey in openssl-compat.h but
SSL_CTX_get0_privatekey access internal structure and can't be a candidate
to openssl-compat.h. The workaround with openssl < 1.0.2 is to use SSL_new
then SSL_get_privatekey.
2017-08-11 11:35:26 +02:00
Olivier Houchard
8da5f98fbe MINOR: dns: Handle SRV records.
Make it so for each server, instead of specifying a hostname, one can use
a SRV label.
When doing so, haproxy will first resolve the SRV label, then use the
resulting hostnames, as well as port and weight (priority is ignored right
now), to each server using the SRV label.
It is resolved periodically, and any server disappearing from the SRV records
will be removed, and any server appearing will be added, assuming there're
free servers in haproxy.
2017-08-09 16:32:49 +02:00
Olivier Houchard
e2c222b12f MINOR: obj: Add a new type of object, OBJ_TYPE_SRVRQ.
dns_srvrq will be objects used for dealing with SRV records.
2017-08-09 16:32:49 +02:00
Olivier Houchard
a8c6db8d2d MINOR: dns: Cache previous DNS answers.
As DNS servers may not return all IPs in one answer, we want to cache the
previous entries. Those entries are removed when considered obsolete, which
happens when the IP hasn't been returned by the DNS server for a time
defined in the "hold obsolete" parameter of the resolver section. The default
is 30s.
2017-08-09 16:32:49 +02:00
Christopher Faulet
48a8332a4a BUG/MEDIUM: ssl: Fix regression about certificates generation
Since the commit f6b37c67 ["BUG/MEDIUM: ssl: in bind line, ssl-options after
'crt' are ignored."], the certificates generation is broken.

To generate a certificate, we retrieved the private key of the default
certificate using the SSL object. But since the commit f6b37c67, the SSL object
is created with a dummy certificate (initial_ctx).

So to fix the bug, we use directly the default certificate in the bind_conf
structure. We use SSL_CTX_get0_privatekey function to do so. Because this
function does not exist for OpenSSL < 1.0.2 and for LibreSSL, it has been added
in openssl-compat.h with the right #ifdef.
2017-07-28 18:25:18 +02:00
Willy Tarreau
6d0d3f6546 MINOR: listener: add a function to return a listener's state as a string
This will be used in debugging output, so it's a short 3-character string.
2017-07-28 17:03:12 +02:00
Willy Tarreau
71d058c288 MINOR: ssl: add a new error codes for wrong server certificates
If a server presents an unexpected certificate to haproxy, that is, a
certificate that doesn't match the expected name as configured in
verifyhost or as requested using SNI, we want to store that precious
information. Fortunately we have access to the connection in the
verification callback so it's possible to store an error code there.

For this purpose we use CO_ER_SSL_MISMATCH_SNI (for when the cert name
didn't match the one requested using SNI) and CO_ER_SSL_MISMATCH for
when it doesn't match verifyhost.
2017-07-28 11:50:16 +02:00
Christopher Faulet
96c7b8dbd2 BUG/MINOR: ssl: Fix check against SNI during server certificate verification
This patch fixes the commit 2ab8867 ("MINOR: ssl: compare server certificate
names to the SNI on outgoing connections")

When we check the certificate sent by a server, in the verify callback, we get
the SNI from the session (SSL_SESSION object). In OpenSSL, tlsext_hostname value
for this session is copied from the ssl connection (SSL object). But the copy is
done only if the "server_name" extension is found in the server hello
message. This means the server has found a certificate matching the client's
SNI.

When the server returns a default certificate not matching the client's SNI, it
doesn't set any "server_name" extension in the server hello message. So no SNI
is set on the SSL session and SSL_SESSION_get0_hostname always returns NULL.

To fix the problemn, we get the SNI directly from the SSL connection. It is
always defined with the value set by the client.

If the commit 2ab8867 is backported in 1.7 and/or 1.6, this one must be
backported too.

Note: it's worth mentionning that by making the SNI check work, we
      introduce another problem by which failed SNI checks can cause
      long connection retries on the server, and in certain cases the
      SNI value used comes from the client. So this patch series must
      not be backported until this issue is resolved.
2017-07-26 19:43:33 +02:00
Willy Tarreau
f42199975c MINOR: task: always preinitialize the task's timeout in task_init()
task_init() is called exclusively by task_new() which is the only way
to create a task. Most callers set t->expire to TICK_ETERNITY, some set
it to another value and a few like Lua don't set it at all as they don't
need a timeout, causing random values to be used in case the task gets
queued.

Let's always set t->expire to TICK_ETERNITY in task_init() so that all
tasks are now initialized in a clean state.

This patch can be backported as it will definitely make the code more
robust (at least the Lua code, possibly other places).
2017-07-24 17:52:58 +02:00
Christopher Faulet
5db105e8b2 MINOR: samples: Handle the type SMP_T_METH in smp_is_safe and smp_is_rw
For all known methods, samples are considered as safe and rewritable. For
unknowns, we handle them like strings (SMP_T_STR).
2017-07-24 17:16:00 +02:00
Emmanuel Hocdet
8c2ddc20de BUILD: ssl: fix compatibility with openssl without TLSEXT_signature_*
In openssl < 1.0.1, TLSEXT_signature_* is undefined. Add TLSEXT signatures
(RFC 5246) when TLSEXT_signature_anonymous is undefined.
2017-07-19 17:19:33 +02:00
Willy Tarreau
106f631280 CLEANUP: hdr_idx: make some function arguments const where possible
Functions hdr_idx_first_idx() and hdr_idx_first_pos() were missing a
"const" qualifier on their arguments which are not modified, causing
a warning in some experimental H2 code.
2017-07-17 21:11:30 +02:00
Willy Tarreau
2ab88675ec MINOR: ssl: compare server certificate names to the SNI on outgoing connections
When support for passing SNI to the server was added in 1.6-dev3, there
was no way to validate that the certificate presented by the server would
really match the name requested in the SNI, which is quite a problem as
it allows other (valid) certificates to be presented instead (when hitting
the wrong server or due to a man in the middle).

This patch adds the missing check against the value passed in the SNI.
The "verifyhost" value keeps precedence if set. If no SNI is used and
no verifyhost directive is specified, then the certificate name is not
checked (this is unchanged).

In order to extract the SNI value, it was necessary to make use of
SSL_SESSION_get0_hostname(), which appeared in openssl 1.1.0. This is
a trivial function which returns the value of s->tlsext_hostname, so
it was provided in the compat layer for older versions. After some
refinements from Emmanuel, it now builds with openssl 1.0.2, openssl
1.1.0 and boringssl. A test file was provided to ease testing all cases.

After some careful observation period it may make sense to backport
this to 1.7 and 1.6 as some users rightfully consider this limitation
as a bug.

Cc: Emmanuel Hocdet <manu@gandi.net>
Signed-off-by: Willy Tarreau <w@1wt.eu>
2017-07-06 15:15:28 +02:00
Emeric Brun
7d27f3c12d BUG/MEDIUM: map/acl: fix unwanted flags inheritance.
The bug: Maps/ACLs using the same file/id can mistakenly inherit
their flags from the last declared one.

i.e.

    $ cat haproxy.conf
    listen mylistener
	mode http
	bind 0.0.0.0:8080

	acl myacl1 url -i -f mine.acl
	acl myacl2 url -f mine.acl
	acl myacl3 url -i -f mine.acl
	redirect location / if myacl2
    $ cat mine.acl
    foobar

Shows an unexpected redirect for request 'GET /FOObAR HTTP/1.0\n\n'.

This fix should be backported on mainline branches v1.6 and v1.7.
2017-07-04 10:45:53 +02:00
Emeric Brun
c730606879 MAJOR: applet: applet scheduler rework.
In order to authorize call of appctx_wakeup on running task:
- from within the task handler itself.
- in futur, from another thread.

The appctx is considered paused as default after running the handler.

The handler should explicitly call appctx_wakeup to be re-called.

When the appctx_free is called on a running handler. The real
free is postponed at the end of the handler process.
2017-06-27 14:38:02 +02:00
Willy Tarreau
a9c1741820 MINOR: connection: add a .get_alpn() method to xprt_ops
This will be used to retrieve the ALPN negociated over SSL (or possibly
via the proxy protocol later). It's likely that this information should
be stored in the connection itself, but it requires adding an extra
pointer and an extra integer. Thus better rely on the transport layer
to pass this info for now.
2017-06-27 14:38:02 +02:00
Christopher Faulet
f3a55dbd22 MINOR: queue: Change pendconn_from_srv/pendconn_from_px into private functions 2017-06-27 14:38:02 +02:00
Christopher Faulet
f0614e8111 MINOR: backends: Change get_server_sh/get_server_uh into private function 2017-06-27 14:38:02 +02:00
Christopher Faulet
87566c923b MINOR: queue: Change pendconn_get_next_strm into private function 2017-06-27 14:38:02 +02:00
Emeric Brun
0194897e54 MAJOR: task: task scheduler rework.
In order to authorize call of task_wakeup on running task:
- from within the task handler itself.
- in futur, from another thread.

The lookups on runqueue and waitqueue are re-worked
to prepare multithread stuff.

If task_wakeup is called on a running task, the woken
message flags are savec in the 'pending_state' attribute of
the state. The real wakeup is postponed at the end of the handler
process and the woken messages are copied from pending_state
to the state attribute of the task.

It's important to note that this change will cause a very minor
(though measurable) performance loss but it is necessary to make
forward progress on a multi-threaded scheduler. Most users won't
ever notice.
2017-06-27 14:38:02 +02:00
Emeric Brun
ff4491726f BUG/MINOR: stream: flag TASK_WOKEN_RES not set if task in runqueue
Under certain circumstances, if a stream's task is first woken up
(eg: I/O event) then notified of the availability of a buffer it
was waiting for via stream_res_wakeup(), this second event is lost
because the flags are only merged after seeing that the task is
running. At the moment it seems that the TASK_WOKEN_RES event is
not explicitly checked for, but better fix this before getting
reports of lost events.

This fix removes this "task running" test which is properly
performed in task_wakeup(), while the flags are properly merged.

It must be backported to 1.7 and 1.6.
2017-06-27 14:37:52 +02:00
Willy Tarreau
2686dcad1e CLEANUP: connection: remove unused CO_FL_WAIT_DATA
Very early in the connection rework process leading to v1.5-dev12, commit
56a77e5 ("MEDIUM: connection: complete the polling cleanups") marked the
end of use for this flag which since was never set anymore, but it continues
to be tested. Let's kill it now.
2017-06-02 15:50:27 +02:00
Baptiste Assmann
201c07f681 MAJOR/REORG: dns: DNS resolution task and requester queues
This patch is a major upgrade of the internal run-time DNS resolver in
HAProxy and it brings the following 2 main changes:

1. DNS resolution task

Up to now, DNS resolution was triggered by the health check task.
From now, DNS resolution task is autonomous. It is started by HAProxy
right after the scheduler is available and it is woken either when a
network IO occurs for one of its nameserver or when a timeout is
matched.

From now, this means we can enable DNS resolution for a server without
enabling health checking.

2. Introduction of a dns_requester structure

Up to now, DNS resolution was purposely made for resolving server
hostnames.
The idea, is to ensure that any HAProxy internal object should be able
to trigger a DNS resolution. For this purpose, 2 things has to be done:
  - clean up the DNS code from the server structure (this was already
    quite clean actually) and clean up the server's callbacks from
    manipulating too much DNS resolution
  - create an agnostic structure which allows linking a DNS resolution
    and a requester of any type (using obj_type enum)

3. Manage requesters through queues

Up to now, there was an uniq relationship between a resolution and it's
owner (aka the requester now). It's a shame, because in some cases,
multiple objects may share the same hostname and may benefit from a
resolution being performed by a third party.
This patch introduces the notion of queues, which are basically lists of
either currently running resolution or waiting ones.

The resolutions are now available as a pool, which belongs to the resolvers.
The pool has has a default size of 64 resolutions per resolvers and is
allocated at configuration parsing.
2017-06-02 11:58:54 +02:00
Baptiste Assmann
fa4a663095 MINOR: dns: implement a LRU cache for DNS resolutions
Introduction of a DNS response LRU cache in HAProxy.

When a positive response is received from a DNS server, HAProxy stores
it in the struct resolution and then also populates a LRU cache with the
response.
For now, the key in the cache is a XXHASH64 of the hostname in the
domain name format concatened to the query type in string format.
2017-06-02 11:40:01 +02:00
Baptiste Assmann
729c901c3f MAJOR: dns: save a copy of the DNS response in struct resolution
Prior this patch, the DNS responses were stored in a pre-allocated
memory area (allocated at HAProxy's startup).
The problem is that this memory is erased for each new DNS responses
received and processed.

This patch removes the global memory allocation (which was not thread
safe by the way) and introduces a storage of the dns response  in the
struct
resolution.
The memory in the struct resolution is also reserved at start up and is
thread safe, since each resolution structure will have its own memory
area.

For now, we simply store the response and use it atomically per
response per server.
2017-06-02 11:30:21 +02:00
Baptiste Assmann
fb7091e213 MINOR: dns: new snr_check_ip_callback function
In the process of breaking links between dns_* functions and other
structures (mainly server and a bit of resolution), the function
dns_get_ip_from_response needs to be reworked: it now can call
"callback" functions based on resolution's owner type to allow modifying
the way the response is processed.

For now, main purpose of the callback function is to check that an IP
address is not already affected to an element of the same type.

For now, only server type has a callback.
2017-06-02 11:28:14 +02:00
Baptiste Assmann
42746373eb REORG: dns: dns_option structure, storage of hostname_dn
This patch introduces a some re-organisation around the DNS code in
HAProxy.

1. make the dns_* functions less dependent on 'struct server' and 'struct resolution'.

With this in mind, the following changes were performed:
- 'struct dns_options' has been removed from 'struct resolution' (well,
  we might need it back at some point later, we'll see)
  ==> we'll use the 'struct dns_options' from the owner of the resolution
- dns_get_ip_from_response(): takes a 'struct dns_options' instead of
  'struct resolution'
  ==> so the caller can pass its own dns options to get the most
      appropriate IP from the response
- dns_process_resolve(): struct dns_option is deduced from new
  resolution->requester_type parameter

2. add hostname_dn and hostname_dn_len into struct server

In order to avoid recomputing a server's hostname into its domain name
format (and use a trash buffer to store the result), it is safer to
compute it once at configuration parsing and to store it into the struct
server.
In the mean time, the struct resolution linked to the server doesn't
need anymore to store the hostname in domain name format. A simple
pointer to the server one will make the trick.

The function srv_alloc_dns_resolution() properly manages everything for
us: memory allocation, pointer updates, etc...

3. move resolvers pointer into struct server

This patch makes the pointer to struct dns_resolvers from struct
dns_resolution obsolete.
Purpose is to make the resolution as "neutral" as possible and since the
requester is already linked to the resolvers, then we don't need this
information anymore in the resolution itself.
2017-06-02 11:26:48 +02:00
Baptiste Assmann
81ed1a0516 MINOR: dns: functions to manage memory for a DNS resolution structure
A couple of new functions to allocate and free memory for a DNS
resolution structure. Main purpose is to to make the code related to DNS
more consistent.
They allocate or free memory for the structure itself. Later, if needed,
they should also allocate / free the buffers, etc, used by this structure.
They don't set/unset any parameters, this is the role of the caller.

This patch also implement calls to these function eveywhere it is
required.
2017-06-02 11:20:29 +02:00
Emeric Brun
3854e0102b MEDIUM: ssl: handle multiple async engines
This patch adds the support of a maximum of 32 engines
in async mode.

Some tests have been done using 2 engines simultaneously.

This patch also removes specific 'async' attribute from the connection
structure. All the code relies only on Openssl functions.
2017-05-27 07:12:27 +02:00
Grant Zhang
fa6c7ee702 MAJOR: ssl: add openssl async mode support
ssl-mode-async is a global configuration parameter which enables
asynchronous processing in OPENSSL for all SSL connections haproxy
handles. With SSL_MODE_ASYNC set, TLS I/O operations may indicate a
retry with SSL_ERROR_WANT_ASYNC with this mode set if an asynchronous
capable engine is used to perform cryptographic operations. Currently
async mode only supports one async-capable engine.

This is the latest version of the patchset which includes Emeric's
updates :
  - improved async fd cleaning when openssl reports an fd to delete
  - prevent conn_fd_handler from calling SSL_{read,write,handshake} until
    the async fd is ready, as these operations are very slow and waste CPU
  - postpone of SSL_free to ensure the async operation can complete and
    does not cause a dereference a released SSL.
  - proper removal of async fd from the fdtab and removal of the unused async
    flag.
2017-05-27 07:05:54 +02:00
Grant Zhang
872f9c2139 MEDIUM: ssl: add basic support for OpenSSL crypto engine
This patch adds the global 'ssl-engine' keyword. First arg is an engine
identifier followed by a list of default_algorithms the engine will
operate.

If the openssl version is too old, an error is reported when the option
is used.
2017-05-27 07:05:00 +02:00
Lukas Tribus
23953686da DOC: update RFC references
A few doc and code comment updates bumping RFC references to the new
ones.
2017-04-28 18:58:11 +02:00
Thierry FOURNIER
6ab2bae084 REORG: spoe: move spoe_encode_varint / spoe_decode_varint from spoe to common
These encoding functions does general stuff and can be used in
other context than spoe. This patch moves the function spoe_encode_varint
and spoe_decode_varint from spoe to common. It also remove the prefix spoe.

These functions will be used for encoding values in new binary sample fetch.
2017-04-27 11:50:41 +02:00
Thierry FOURNIER
f4128a9981 BUG/MINOR: change header-declared function to static inline
When we include the header proto/spoe.h in other files in the same
project, the compilator claim that the symbol have multiple definitions:

   src/flt_spoe.o: In function `spoe_encode_varint':
   ~/git/haproxy/include/proto/spoe.h:45: multiple definition of `spoe_encode_varint'
   src/proto_http.o:~/git/haproxy/include/proto/spoe.h:45: first defined here
2017-04-27 11:50:07 +02:00
Olivier Houchard
1fc0516516 MINOR: proxy: Don't close FDs if not our proxy.
When running with multiple process, if some proxies are just assigned
to some processes, the other processes will just close the file descriptors
for the listening sockets. However, we may still have to provide those
sockets when reloading, so instead we just try hard to pretend those proxies
are dead, while keeping the sockets opened.
A new global option, no-reused-socket", has been added, to restore the old
behavior of closing the sockets not bound to this process.
2017-04-13 19:15:17 +02:00
Olivier Houchard
f73629d23a MINOR: global: Add an option to get the old listening sockets.
Add the "-x" flag, that takes a path to a unix socket as an argument. If
used, haproxy will connect to the socket, and asks to get all the
listening sockets from the old process. Any failure is fatal.
This is needed to get seamless reloads on linux.
2017-04-13 19:15:17 +02:00
Christopher Faulet
533182f1c8 CLEANUP: http: Remove channel_congested function
Not used anymore since last commit.
2017-03-31 14:38:08 +02:00
Emmanuel Hocdet
9490cedb4e BUILD: ssl: fix OPENSSL_NO_SSL_TRACE for boringssl and libressl
"sample-fetch which captures the cipherlist" patch introduce #define
do deal with trace functions only available in openssl > 1.0.2.
Add this #define to libressl and boringssl environment.
Thanks to Piotr Kubaj for postponing and testing with libressl.
2017-03-20 12:01:44 +01:00
Emmanuel Hocdet
a52bb15cc7 BUILD: ssl: simplify SSL_CTX_set_ecdh_auto compatibility
SSL_CTX_set_ecdh_auto is declared (when present) with #define. A simple #ifdef
avoid to list all cases of ssllibs. It's a placebo in new ssllibs. It's ok with
openssl 1.0.1, 1.0.2, 1.1.0, libressl and boringssl.
Thanks to Piotr Kubaj for postponing and testing with libressl.
2017-03-20 12:01:34 +01:00
Willy Tarreau
de40d798de CLEANUP: connection: completely remove CO_FL_WAKE_DATA
Since it's only set and never tested anymore, let's remove it.
2017-03-19 12:18:27 +01:00
Olivier Houchard
614f8d7d56 MINOR: cli: Let configure the dynamic cookies from the cli.
This adds 3 new commands to the cli :
enable dynamic-cookie backend <backend> that enables dynamic cookies for a
specified backend
disable dynamic-cookie backend <backend> that disables dynamic cookies for a
specified backend
set dynamic-cookie-key backend <backend> that lets one change the dynamic
cookie secret key, for a specified backend.
2017-03-15 11:38:29 +01:00
Olivier Houchard
4e694049fa MINOR: server: Add dynamic session cookies.
This adds a new "dynamic" keyword for the cookie option. If set, a cookie
will be generated for each server (assuming one isn't already provided on
the "server" line), from the IP of the server, the TCP port, and a secret
key provided. To provide the secret key, a new keyword as been added,
"dynamic-cookie-key", for backends.

Example :
backend bk_web
  balance roundrobin
  dynamic-cookie-key "bla"
  cookie WEBSRV insert dynamic
  server s1 127.0.0.1:80 check
  server s2 192.168.56.1:80 check

This is a first step to be able to dynamically add and remove servers,
without modifying the configuration file, and still have all the load
balancers redirect the traffic to the right server.

Provide a way to generate session cookies, based on the IP address of the
server, the TCP port, and a secret key provided.
2017-03-15 11:37:30 +01:00
Jarno Huuskonen
9e6906b9ec MEDIUM: http_error_message: txn->status / http_get_status_idx.
This commit removes second argument(msgnum) from http_error_message and
changes http_error_message to use s->txn->status/http_get_status_idx for
mapping status code from 200..504 to HTTP_ERR_200..HTTP_ERR_504(enum).

This is needed for http-request tarpit deny_status commit.
2017-03-14 10:41:41 +01:00
Nenad Merdanovic
b7e7c4720a MINOR: Add nbsrv sample converter
This is like the nbsrv() sample fetch function except that it works as
a converter so it can count the number of available servers of a backend
name retrieved using a sample fetch or an environment variable.

Signed-off-by: Nenad Merdanovic <nmerdan@haproxy.com>
2017-03-13 18:26:05 +01:00
Christopher Faulet
4ff3e574ac REORG: spoe: Move low-level encoding/decoding functions in dedicated header file
So, it will be easier to anyone to develop external services using these
functions.

SPOA example has been updated accordingly.
2017-03-09 15:32:55 +01:00
Christopher Faulet
8ef75251e3 MAJOR: spoe: refactor the filter to clean up the code
The SPOE code is now pretty big and it was the good time to clean it up. It is
not perfect, some parts remains a bit ugly. But it is far better now.
2017-03-09 15:32:55 +01:00
Baptiste Assmann
5cd1b9222e MINOR: dns: give ability to dns_init_resolvers() to close a socket when requested
The function dns_init_resolvers() is used to initialize socket used to
send DNS queries.
This patch gives the function the ability to close a socket before
re-opening it.

[wt: this needs to be backported to 1.7 for next fix]
2017-02-03 07:21:32 +01:00
Willy Tarreau
e3e326d9f0 BUILD: ssl: kill a build warning introduced by BoringSSL compatibility
A recent patch to support BoringSSL caused this warning to appear on
OpenSSL 1.1.0 :
   src/ssl_sock.c:3062:4: warning: statement with no effect [-Wunused-value]

It's caused by SSL_CTX_set_ecdh_auto() which is now only a macro testing
that the last argument is zero, and the result is not used here. Let's
just kill it for both versions.

Tested with 0.9.8, 1.0.0, 1.0.1, 1.0.2, 1.1.0. This fix may be backported
to 1.7 if the boringssl fix is as well.
2017-01-19 17:56:20 +01:00
Willy Tarreau
77d88da7e1 BUILD: ssl: eliminate warning with OpenSSL 1.1.0 regarding RAND_pseudo_bytes()
This function was deprecated in 1.1.0 causing this warning :

  src/ssl_sock.c:551:3: warning: 'RAND_pseudo_bytes' is deprecated (declared at /opt/openssl-1.1.0/include/openssl/rand.h:47) [-Wdeprecated-declarations]

The man suggests to use RAND_bytes() instead. While the return codes
differ, it turns out that the function was already misused and was
relying on RAND_bytes() return code instead.

The patch was tested on 0.9.8, 1.0.0, 1.0.1, 1.0.2 and 1.1.0.

This fix must be backported to 1.7 and the return code check should
be backported to earlier versions if relevant.
2017-01-19 17:28:08 +01:00
Willy Tarreau
a4fb8ed1f2 BUILD: ssl: silence a warning reported for ERR_remove_state()
In 1.0.0, this function was replaced with ERR_remove_thread_state().
As of openssl 1.1.0, both are now deprecated and do nothing at all.
Thus we simply make this call do nothing in 1.1.0 to silence the
warning.

The change was tested with 0.9.8, 1.0.0, 1.0.1, 1.0.2 and 1.1.0.

This kills the following warning on 1.1.0 :
   src/ssl_sock.c:7266:9: warning: 'ERR_remove_state' is deprecated (declared at /dev/shm/openssl-1.1.0b/include/openssl/err.h:247) [-Wdeprecated-declarations]

This fix should be backported to 1.7.
2017-01-19 17:28:08 +01:00
Willy Tarreau
2b3205b5fa BUILD: ssl: fix build on OpenSSL 1.0.0
After the code was ported to support 1.1.0, this one broke on 1.0.0 :

  src/shctx.c:406: undefined reference to `SSL_SESSION_set1_id_context'

The function was indeed introduced only in 1.0.1. The build was validated
with 0.9.8, 1.0.0, 1.0.1, 1.0.2 and 1.1.0.

This fix must be backported to 1.7.
2017-01-19 17:27:57 +01:00
Emmanuel Hocdet
fdec7897fd BUILD: ssl: fix to build (again) with boringssl
Limitations:
. disable force-ssl/tls (need more work)
should be set earlier with SSL_CTX_new (SSL_CTX_set_ssl_version is removed)
. disable generate-certificates (need more work)
introduce SSL_NO_GENERATE_CERTIFICATES to disable generate-certificates.

Cleanup some #ifdef and type related to boringssl env.
2017-01-16 12:40:35 +01:00
Emmanuel Hocdet
98263291cc MAJOR: ssl: bind configuration per certificat
crt-list is extend to support ssl configuration. You can now have
such line in crt-list <file>:
mycert.pem [npn h2,http/1.1]

Support include "npn", "alpn", "verify", "ca_file", "crl_file",
"ecdhe", "ciphers" configuration and ssl options.

"crt-base" is also supported to fetch certificates.
2017-01-13 11:40:34 +01:00
Robin H. Johnson
52f5db2a44 MINOR: http: custom status reason.
The older 'rsprep' directive allows modification of the status reason.

Extend 'http-response set-status' to take an optional string of the new
status reason.

  http-response set-status 418 reason "I'm a coffeepot"

Matching updates in Lua code:
- AppletHTTP.set_status
- HTTP.res_set_status

Signed-off-by: Robin H. Johnson <robbat2@gentoo.org>
2017-01-06 11:57:44 +01:00
Willy Tarreau
d1c5750370 CLEANUP: ssl: move tlskeys_finalize_config() to a post_check callback
tlskeys_finalize_config() was the only reason for haproxy.c to still
require ifdef and includes for ssl_sock. This one fits perfectly well
in the late initializers so it was changed to be registered with
hap_register_post_check().
2016-12-22 23:26:38 +01:00
Willy Tarreau
d9f5cca3d5 CLEANUP: connection: unexport raw_sock and ssl_sock
This way we're sure not to reuse them by accident.
2016-12-22 23:26:38 +01:00
Willy Tarreau
13e1410f8a MINOR: connection: add a minimal transport layer registration system
There are still a lot of #ifdef USE_OPENSSL in the code (still 43
occurences) because we never know if we can directly access ssl_sock
or not. This patch attacks the problem differently by providing a
way for transport layers to register themselves and for users to
retrieve the pointer. Unregistered transport layers will point to NULL
so it will be easy to check if SSL is registered or not. The mechanism
is very inexpensive as it relies on a two-entries array of pointers,
so the performance will not be affected.
2016-12-22 23:26:38 +01:00
Willy Tarreau
55d3791b46 MEDIUM: ssl_sock: implement ssl_sock_prepare_bind_conf()
Instead of hard-coding all SSL preparation in cfgparse.c, we now register
this new function as the transport layer's prepare_bind_conf() and call it
only when definied. This removes some non-obvious SSL-specific code from
cfgparse.c as well as a #ifdef.
2016-12-22 23:26:38 +01:00
Willy Tarreau
0320934f7e MEDIUM: ssl: remote the proxy argument from most functions
Most of the SSL functions used to have a proxy argument which was mostly
used to be able to emit clean errors using Alert(). First, many of them
were converted to memprintf() and don't require this pointer anymore.
Second, the rare which still need it also have either a bind_conf argument
or a server argument, both of which carry a pointer to the relevant proxy.

So let's now get rid of it, it needlessly complicates the API and certain
functions already have many arguments.
2016-12-22 23:26:38 +01:00
Willy Tarreau
c95bad5013 MEDIUM: move listener->frontend to bind_conf->frontend
Historically, all listeners have a pointer to the frontend. But since
the introduction of SSL, we now have an intermediary layer called
bind_conf corresponding to a "bind" line. It makes no sense to have
the frontend on each listener given that it's the same for all
listeners belonging to a same bind_conf. Also certain parts like
SSL can only operate on bind_conf and need the frontend.

This patch fixes this by moving the frontend pointer from the listener
to the bind_conf. The extra indirection is quite cheap given and the
places were this is used are very scarce.
2016-12-22 23:26:38 +01:00
Willy Tarreau
71a8c7c49e MINOR: listener: move the transport layer pointer to the bind_conf
A mistake was made when the socket layer was cut into proto and
transport, the transport was attached to the listener while all
listeners in a single "bind" line always have exactly the same
transport. It doesn't seem obvious but this is the reason why there
are so many #ifdefs USE_OPENSSL in cfgparse : a lot of operations
have to be open-coded because cfgparse only manipulates bind_conf
and we don't have the information of the transport layer here.

Very little code makes use of the transport layer, mainly session
setup and log. These places can afford an extra pointer indirection
(the listener points to the bind_conf). This change is thus very small,
it saves a little bit of memory (8B per listener) and makes the code
more flexible.
2016-12-22 23:26:37 +01:00
Willy Tarreau
800f93f375 CLEANUP: wurfl: register the deinit function via the dedicated list
By registering the deinit function we avoid another #ifdef in haproxy.c.
The ha_wurfl_deinit() function has been made static and unexported. Now
proto/wurfl.h is totally empty, the code being self-contained in wurfl.c,
so the useless .h has been removed.
2016-12-21 21:30:54 +01:00
Willy Tarreau
dc2ed47163 CLEANUP: wurfl: make use of the late init registration
This removes some #ifdefs from the main haproxy code path and enables
error checking. The current code only makes use of warnings even for
some errors that look serious. While this choice is questionnable, it
has been kept as-is, and only the return codes were adapted to ERR_WARN
to at least report that some warnings were emitted. ha_wurfl_init() was
unexported as it's not needed anymore.
2016-12-21 21:30:54 +01:00
Willy Tarreau
64bca599d9 CLEANUP: filters: use the function registration to initialize all proxies
Function flt_init() was called in the main init code path, now we move
it to the list of initializers and we can unexport flt_init().
2016-12-21 21:30:54 +01:00
Willy Tarreau
865c5148e6 CLEANUP: checks: make use of the post-init registration to start checks
Instead of calling the checks directly from the init code, we now
register the start_checks() function to be run at this point. This
also allows to unexport the check init function and to remove one
include from haproxy.c.
2016-12-21 21:30:54 +01:00
Luca Pizzamiglio
578b169dcb BUILD/MEDIUM: Fixing the build using LibreSSL
Fixing the build using LibreSSL as OpenSSL implementation.
Currently, LibreSSL 2.4.4 provides the same API of OpenSSL 1.0.1x,
but it redefine the OpenSSL version number as 2.0.x, breaking all
checks with OpenSSL 1.1.x.
The patch solves the issue checking the definition of the symbol
LIBRESSL_VERSION_NUMBER when Openssl 1.1.x features are requested.
2016-12-12 22:57:04 +01:00
Christopher Faulet
a73e59b690 BUG/MAJOR: Fix how the list of entities waiting for a buffer is handled
When an entity tries to get a buffer, if it cannot be allocted, for example
because the number of buffers which may be allocated per process is limited,
this entity is added in a list (called <buffer_wq>) and wait for an available
buffer.

Historically, the <buffer_wq> list was logically attached to streams because it
were the only entities likely to be added in it. Now, applets can also be
waiting for a free buffer. And with filters, we could imagine to have more other
entities waiting for a buffer. So it make sense to have a generic list.

Anyway, with the current design there is a bug. When an applet failed to get a
buffer, it will wait. But we add the stream attached to the applet in
<buffer_wq>, instead of the applet itself. So when a buffer is available, we
wake up the stream and not the waiting applet. So, it is possible to have
waiting applets and never awakened.

So, now, <buffer_wq> is independant from streams. And we really add the waiting
entity in <buffer_wq>. To be generic, the entity is responsible to define the
callback used to awaken it.

In addition, applets will still request an input buffer when they become
active. But they will not be sleeped anymore if no buffer are available. So this
is the responsibility to the applet I/O handler to check if this buffer is
allocated or not. This way, an applet can decide if this buffer is required or
not and can do additional processing if not.

[wt: backport to 1.7 and 1.6]
2016-12-12 19:11:04 +01:00
Christopher Faulet
34c5cc98da MINOR: task: Rename run_queue and run_queue_cur counters
<run_queue> is used to track the number of task in the run queue and
<run_queue_cur> is a copy used for the reporting purpose. These counters has
been renamed, respectively, <tasks_run_queue> and <tasks_run_queue_cur>. So the
naming is consistent between tasks and applets.

[wt: needed for next fixes, backport to 1.7 and 1.6]
2016-12-12 19:10:54 +01:00
Christopher Faulet
1cbe74cd83 MINOR: applet: Count number of (active) applets
As for tasks, 2 counters has been added to track :
  * the total number of applets : nb_applets
  * the number of active applets : applets_active_queue

[wt: needed for next fixes, to backport to 1.7 and 1.6]
2016-12-12 19:10:46 +01:00
Willy Tarreau
796c5b7997 OPTIM: stream-int: don't disable polling anymore on DONT_READ
Commit 5fddab0 ("OPTIM: stream_interface: disable reading when
CF_READ_DONTWAIT is set") improved the connection layer's efficiency
back in 1.5-dev13 by avoiding successive read attempts on an active
FD. But by disabling this on a polled FD, it causes an unpleasant
side effect which is that the FD that was subscribed to polling is
suddenly stopped and may need to be re-enabled once the kernel
starts to slow down on data eviction (eg: saturated server at the
other end, bursty traffic caused by too large maxpollevents).

This behaviour is observable with persistent connections when there
is a large enough connection count so that there's no data in the
early connection and polling is required, because there are then
up to 4 epoll_ctl() calls per request. It's important that the
server is slower than haproxy to cause some delays when reading
response.

The current connection layer as designed in 1.6 with the FD cache
doesn't require this trick anymore, though it still benefits from
it when it saves an FD from being uselessly polled. But compared
to the increased cost of enabling and disabling poll all the time,
it's still better to disable it. In some cases it's possible to
observe a performance increase as high as 30% by avoiding this
epoll_ctl() dance.

In the end we only want to disable it when the FD is speculatively
read and not when it's polled. For this we introduce a new function
__conn_data_done_recv() which is used to indicate that we're done
with recv() and not interested in new attempts. If/when we later
support event-triggered epoll, this function will have to change
a bit to do the same even in the polled case.

A quick test with keep-alive requests run on a dual-core / dual-
thread Atom shows a significant improvement :

single process, 0 bytes :
before: Requests per second:    12243.20 [#/sec] (mean)
after:  Requests per second:    13354.54 [#/sec] (mean)

single process, 4k :
before: Requests per second:    9639.81 [#/sec] (mean)
after:  Requests per second:    10991.89 [#/sec] (mean)

dual process, 0 bytes (unstable) :
before: Requests per second:    16900-19800 ~ 17600 [#/sec] (mean)
after:  Requests per second:    18600-21400 ~ 20500 [#/sec] (mean)
2016-12-05 13:49:57 +01:00
Christopher Faulet
4aad833924 BUG/MINOR: filters: Protect args in macros HAS_DATA_FILTERS and IS_DATA_FILTER
[wt: backport needed in 1.7]
2016-11-29 17:03:04 +01:00
Willy Tarreau
ddc3e9e55d BUG/MINOR: stats: make field_str() return an empty string on NULL
It already returns an empty string when the field is empty, but as a
preventive measure we should do the same when the string itself is a
NULL. While it is not supposed to happen, it will make the code more
resistant against failed allocations and unexpected results.

This fix should be backported to 1.7.
2016-11-26 15:58:37 +01:00
Willy Tarreau
7d56221d57 REORG: stkctr: move all the stick counters processing to stick-tables.c
Historically we used to have the stick counters processing put into
session.c which became stream.c. But a big part of it is now in
stick-table.c (eg: converters) but despite this we still have all
the sample fetch functions in stream.c

These parts do not depend on the stream anymore, so let's move the
remaining chunks to stick-table.c and have cleaner files.

What remains in stream.c is everything needed to attach/detach
trackers to the stream and to update the counters while the stream
is being processed.
2016-11-25 16:10:05 +01:00
Willy Tarreau
397131093f REORG: tcp-rules: move tcp rules processing to their own file
There's no more reason to keep tcp rules processing inside proto_tcp.c
given that there is nothing in common there except these 3 letters : tcp.
The tcp rules are in fact connection, session and content processing rules.
Let's move them to "tcp-rules" and let them live their life there.
2016-11-25 15:57:38 +01:00
Willy Tarreau
3758581e19 BUG/MINOR: freq-ctr: make swrate_add() support larger values
Reinhard Vicinus reported that the reported average response times cannot
be larger than 16s due to the double multiply being performed by
swrate_add() which causes an overflow very quickly. Indeed, with N=512,
the highest average value is 16448.

One solution proposed by Reinhard is to turn to long long, but this
involves 64x64 multiplies and 64->32 divides, which are extremely
expensive on 32-bit platforms.

There is in fact another way to avoid the overflow without using larger
integers, it consists in avoiding the multiply using the fact that
x*(n-1)/N = x-(x/N).

Now it becomes possible to store average values as large as 8.4 millions,
which is around 2h18mn.

Interestingly, this improvement also makes the code cheaper to execute
both on 32 and on 64 bit platforms :

Before :

00000000 <swrate_add>:
   0:   8b 54 24 04             mov    0x4(%esp),%edx
   4:   8b 0a                   mov    (%edx),%ecx
   6:   89 c8                   mov    %ecx,%eax
   8:   c1 e0 09                shl    $0x9,%eax
   b:   29 c8                   sub    %ecx,%eax
   d:   8b 4c 24 0c             mov    0xc(%esp),%ecx
  11:   c1 e8 09                shr    $0x9,%eax
  14:   01 c8                   add    %ecx,%eax
  16:   89 02                   mov    %eax,(%edx)

After :

00000020 <swrate_add>:
  20:   8b 4c 24 04             mov    0x4(%esp),%ecx
  24:   8b 44 24 0c             mov    0xc(%esp),%eax
  28:   8b 11                   mov    (%ecx),%edx
  2a:   01 d0                   add    %edx,%eax
  2c:   81 c2 ff 01 00 00       add    $0x1ff,%edx
  32:   c1 ea 09                shr    $0x9,%edx
  35:   29 d0                   sub    %edx,%eax
  37:   89 01                   mov    %eax,(%ecx)

This fix may be backported to 1.6.
2016-11-25 11:55:10 +01:00
Thierry FOURNIER / OZON.IO
8a4e4420fb MEDIUM: log-format: Use standard HAProxy log system to report errors
The function log format emit its own error message using Alert(). This
patch replaces this behavior and uses the standard HAProxy error system
(with memprintf).

The benefits are:
 - cleaning the log system

 - the logformat can ignore the caller (actually the caller must set
   a flag designing the caller function).

 - Make the usage of the logformat function easy for future components.
2016-11-25 07:32:58 +01:00
Thierry FOURNIER / OZON.IO
7f3aa8b62f MINOR: lua: add utility function for check boolean argument
Strangely, the Lua API doesn't provides a function like
luaL_checkboolean(). This little function add this one.
2016-11-24 21:35:10 +01:00
Willy Tarreau
80ebacfc82 BUILD: ssl: make the SSL layer build again with openssl 0.9.8
Commit 1866d6d ("MEDIUM: ssl: Add support for OpenSSL 1.1.0")
introduced support for openssl 1.1.0 and temporarily broke 0.9.8.
In the end the port was not very hard given that the only cause of
build failures were functions supposedly absent from 0.9.8 that in
fact did exist.

Thus, adding a new #if to move these functions for versions older
than 0.9.8 was enough to fix the trouble. It received very light
testing, basically only an SSL bridge decrypting and re-encrypting
traffic, and checking that everything looks right. That said, the
functions specific to 0.9.8 here compared to 1.0.x are only
SSL_SESSION_set1_id_context(), EVP_PKEY_base_id(), and
X509_PUBKEY_get0_param().
2016-11-24 20:18:21 +01:00
Thierry FOURNIER / OZON.IO
a2c38d7904 MEDIUM: log-format: strict parsing and enable fail
Until now, the function parse_logformat_string() never fails. It
send warnings when it parses bad format, and returns expression in
best effort.

This patch replaces warnings by alert and returns a fail code.

Maybe the warning mode is designed for a compatibility with old
configuration versions. If it is the case, now this compatibility
is broken.

[wt: no, the reason is that an alert must cause a startup failure,
 but this will be OK with next patch]
2016-11-24 18:54:26 +01:00
Thierry FOURNIER / OZON.IO
6fe0e1b977 CLEANUP: log-format: remove unused arguments
The log-format function parse_logformat_string() takes file and line
for building parsing logs. These two parameters are embedded in the
struct proxy curproxy, which is the current parsing context.

This patch removes these two unused arguments.
2016-11-24 18:54:26 +01:00
Thierry FOURNIER / OZON.IO
eb51e16306 CLEANUP/MINOR: log-format: unexport functions parse_logformat_var_args() and parse_logformat_var()
Remove export of the fucntion parse_logformat_var_args() and
parse_logformat_var(). These functions are a part of the
logformat parser, and this export is useless.
2016-11-24 18:54:25 +01:00
Willy Tarreau
df4399fcb6 BUILD: server: remove a build warning introduced by latest series
We get this when Lua is disabled, just a missing include.

In file included from src/queue.c:18:0:
include/proto/server.h:51:39: warning: 'struct appctx' declared inside parameter list [enabled by default]
2016-11-24 17:32:01 +01:00
Willy Tarreau
2b812e29f6 REORG: cli: move "show stat" to stats.c
Move the "show stat" command to stats.c using the CLI keyword API
to register it on the CLI. The stats_dump_stat_to_buffer() function
is now static again.
2016-11-24 16:59:27 +01:00
Willy Tarreau
960f2cb056 MINOR: proxy: create new function cli_find_frontend() to find a frontend
Several CLI commands require a frontend, so let's have a function to
look this one up and prepare the appropriate error message and the
appctx's state in case of failure.
2016-11-24 16:59:27 +01:00
Willy Tarreau
21b069dca8 MINOR: server: create new function cli_find_server() to find a server
Several CLI commands require a server, so let's have a function to
look this one up and prepare the appropriate error message and the
appctx's state in case of failure.
2016-11-24 16:59:27 +01:00
Willy Tarreau
de57a578ba MINOR: cli: create new function cli_has_level() to validate permissions
This function is used to check that the CLI features the appropriate
level of permissions or to prepare the adequate error message.
2016-11-24 16:59:27 +01:00
William Lallemand
9ed6203aef REORG: cli: split dumpstats.h in stats.h and cli.h
proto/dumpstats.h has been split in 4 files:

  * proto/cli.h  contains protypes for the CLI
  * proto/stats.h contains prototypes for the stats
  * types/cli.h contains definition for the CLI
  * types/stats.h contains definition for the stats
2016-11-24 16:59:27 +01:00
William Lallemand
74c24fb071 REORG: cli: split dumpstats.c in src/cli.c and src/stats.c
dumpstats.c was containing either the stats code and the CLI code.
The cli code has been moved to cli.c and the stats code to stats.c
2016-11-24 16:59:27 +01:00
Willy Tarreau
8e0bb0ae16 MINOR: connection: add names for transport and data layers
This makes debugging easier and avoids having to put ugly checks
against certain well-known internal struct pointers.
2016-11-24 16:58:12 +01:00
Willy Tarreau
2dc770c641 MINOR: connection: add a few functions to report the data and xprt layers' names
These functions will be needed by "show sess" on the CLI, let's make them
globally available. It's important to note that due to the fact that we
still do not set the data and transport layers' names in the structures,
we still have to rely on some exports just to match the pointers. This is
ugly but is preferable to adding many includes since the short-term goal
is to get rid of these tests by having proper names in place.
2016-11-24 16:49:40 +01:00
Christopher Faulet
a00d817aba MINOR: filters: Add check_timeouts callback to handle timers expiration on streams
A filter can now be notified when a stream is woken up because of an expired
timer.

The documentation and the TRACE filter have been updated.
2016-11-21 15:29:58 +01:00
Willy Tarreau
733b1327a6 DEBUG: connection: mark the closed FDs with a value that is easier to detect
Setting an FD to -1 when closed isn't the most easily noticeable thing
to do when we're chasing accidental reuse of a stale file descriptor.
Instead set it to that large a negative value that it will overflow the
fdtab and provide an analysable core at the moment the issue happens.
Care was taken to ensure it doesn't overflow nor change sign on 32-bit
machines when multiplied by fdtab, and that it also remains negative for
the various checks that exist. The value equals 0xFDDEADFD which happens
to be easily spotted in a debugger.
2016-11-18 15:00:42 +01:00
Willy Tarreau
350135cf49 BUG/MEDIUM: connection: check the control layer before stopping polling
The bug described in commit 568743a ("BUG/MEDIUM: stream-int: completely
detach connection on connect error") was not a stream-interface layer bug
but a connection layer bug. There was exactly one place in the code where
we could change a file descriptor's status without first checking whether
it is valid or not, it was in conn_stop_polling(). This one is called when
the polling status is changed after an update, and calls fd_stop_both even
if we had already closed the file descriptor :

1479388298.484240 ->->->->->   conn_fd_handler > conn_cond_update_polling
1479388298.484240 ->->->->->->   conn_cond_update_polling > conn_stop_polling
1479388298.484241 ->->->->->->->   conn_stop_polling > conn_ctrl_ready
1479388298.484241                  conn_stop_polling < conn_ctrl_ready
1479388298.484241 ->->->->->->->   conn_stop_polling > fd_stop_both
1479388298.484242 ->->->->->->->->   fd_stop_both > fd_update_cache
1479388298.484242 ->->->->->->->->->   fd_update_cache > fd_release_cache_entry
1479388298.484242                      fd_update_cache < fd_release_cache_entry
1479388298.484243                    fd_stop_both < fd_update_cache
1479388298.484243                  conn_stop_polling < fd_stop_both
1479388298.484243                conn_cond_update_polling < conn_stop_polling
1479388298.484243              conn_fd_handler < conn_cond_update_polling

The problem with the previous fix above is that it break the http_proxy mode
and possibly even some Lua parts and peers to a certain extent ; all outgoing
connections where the target address is initially copied into the outgoing
connection which experience a retry would use a random outgoing address after
the retry because closing and detaching the connection causes the target
address to be lost. This was attempted to be addressed by commit 0857d7a
("BUG/MAJOR: stream: properly mark the server address as unset on connect
retry") but it used to only solve the most visible effect and not the root
cause.

Prior to this fix, it was possible to cause this config to keep CLOSE_WAIT
for as long as it takes to expire a client or server timeout (note the
missing client timeout) :

   listen test
        mode http
        bind :8002
        server s1 127.0.0.1:8001

   $ tcploop 8001 L0 W N20 A R P100 S:"HTTP/1.1 200 OK\r\nContent-length: 0\r\n\r\n" &
   $ tcploop 8002 N200 C T W S:"GET / HTTP/1.0\r\n\r\n" O P10000 K

With this patch, these CLOSE_WAIT properly vanish when both processes leave.

This commit reverts the two fixes above and replaces them with the proper
fix in connection.h. It must be backported to 1.6 and 1.5. Thanks to
Robson Roberto Souza Peixoto for providing very detailed traces showing
some obvious inconsistencies leading to finding this bug.
2016-11-18 14:48:52 +01:00
Thierry FOURNIER / OZON.IO
6a22dcbe27 MINOR: cli: add private pointer and release function
This pointer will be used for storing private context. With this,
the same executed function can handle more than one keyword. This
will be very useful for creation Lua cli bindings.

The release function is called when the command is terminated (give
back the hand to the prompt) or when the session is broken (timeout
or client closed).
2016-11-18 14:32:03 +01:00
Christopher Faulet
ba7bc164f7 MINOR: spoe/checks: Add support for SPOP health checks
A new "option spop-check" statement has been added to enable server health
checks based on SPOP HELLO handshake. SPOP is the protocol used by SPOE filters
to talk to servers.
2016-11-09 22:57:02 +01:00
Christopher Faulet
85d79c94a9 MINOR: vars: Add 'unset-var' action/converter
It does the opposite of 'set-var' action/converter. It is really useful for
per-process variables. But, it can be used for any scope.

The lua function 'unset_var' has also been added.
2016-11-09 22:57:01 +01:00
Christopher Faulet
09c9df286b MINOR: vars: Add vars_set_by_name_ifexist function
This function, unsurprisingly, sets a variable value only if it already
exists. In other words, this function will succeed only if the variable was
found somewhere in the configuration during HAProxy startup.

It will be used by SPOE filter. So an agent will be able to set a value only for
existing variables. This prevents an agent to create a very large number of
unused variables to flood HAProxy and exhaust the memory reserved to variables..
2016-11-09 22:57:00 +01:00
Christopher Faulet
476e5d0e03 REORG: sample: move code to release a sample expression in sample.c
This code has been moved from haproxy.c to sample.c and the function
release_sample_expr can now be called from anywhere to release a sample
expression. This function will be used by the stream processing offload engine
(SPOE).
2016-11-09 22:57:00 +01:00
Willy Tarreau
25e515235a MEDIUM: server: make use of init-addr
It is now supported. If not set, we default to the legacy methods list
which is "last,libc".
2016-11-09 15:30:47 +01:00
Baptiste Assmann
25938278b7 MEDIUM: server: add a new init-addr server line setting
This new setting supports a comma-delimited list of methods used to
resolve the server's FQDN to an IP address. Currently supported methods
are "libc" (use the regular libc's resolver) and "last" (use the last
known valid address found in the state file).

The list is implemented in a 32-bit integer, because each init-addr
method only requires 3 bits. The last one must always be SRV_IADDR_END
(0), allowing to store up to 10 methods in a single 32 bit integer.

Note: the doc is provided at the end of this series.
2016-11-09 15:30:47 +01:00
Willy Tarreau
8b42848a44 MINOR: server: make srv_set_admin_state() capable of telling why this happens
It will be important to help debugging some DNS resolution issues to
know why a server was marked down, so let's make  the function support
a 3rd argument with an indication of the reason. Passing NULL will keep
the message as-is.
2016-11-09 15:30:47 +01:00
Baptiste Assmann
83cbaa531f MAJOR: server: postpone address resolution
Server addresses are not resolved anymore upon the first pass so that we
don't fail if an address cannot be resolved by the libc. Instead they are
processed all at once after the configuration is fully loaded, by the new
function srv_init_addr(). This function only acts on the server's address
if this address uses an FQDN, which appears in server->hostname.

For now the function does two things, to followup with HAProxy's historical
default behavior:

  1. apply server IP address found in server-state file if runtime DNS
     resolution is enabled for this server

  2. use the DNS resolver provided by the libc

If none of the 2 options above can find an IP address, then an error is
returned.

All of this will be needed to support the new server parameter "init-addr".
For now, the biggest user-visible change is that all server resolution errors
are dumped at once instead of causing a startup failure one by one.
2016-11-09 14:24:20 +01:00
Dirkjan Bussink
1866d6d8f1 MEDIUM: ssl: Add support for OpenSSL 1.1.0
In the last release a lot of the structures have become opaque for an
end user. This means the code using these needs to be changed to use the
proper functions to interact with these structures instead of trying to
manipulate them directly.

This does not fix any deprecations yet that are part of 1.1.0, it only
ensures that it can be compiled against that version and is still
compatible with older ones.

[wt: openssl-0.9.8 doesn't build with it, there are conflicts on certain
     function prototypes which we declare as inline here and which are
     defined differently there. But openssl-0.9.8 is not supported anymore
     so probably it's OK to go without it for now and we'll see later if
     some users still need it. Emeric has reviewed this change and didn't
     spot anything obvious which requires special care. Let's try it for
     real now]
2016-11-08 20:54:41 +01:00
Willy Tarreau
e5d3169e1c CLEANUP: wurfl: reduce exposure in the rest of the code
The only reason wurfl/wurfl.h was needed outside of wurfl.c was to expose
wurfl_handle which is a pointer to a structure, referenced by global.h.
By just storing a void* there instead, we can confine all wurfl code to
wurfl.c, which is really nice.
2016-11-08 18:47:25 +01:00
Willy Tarreau
757478e900 BUG/MEDIUM: servers: properly propagate the maintenance states during startup
Right now there is an issue with the way the maintenance flags are
propagated upon startup. They are not propagate, just copied from the
tracked server. This implies that depending on the server's order, some
tracking servers may not be marked down. For example this configuration
does not work as expected :

        server s1 1.1.1.1:8000 track s2
        server s2 1.1.1.1:8000 track s3
        server s3 1.1.1.1:8000 track s4
        server s4 wtap:8000 check inter 1s disabled

It results in s1/s2 being up, and s3/s4 being down, while all of them
should be down.

The only clean way to process this is to run through all "root" servers
(those not tracking any other server), and to propagate their state down
to all their trackers. This is the same algorithm used to propagate the
state changes. It has to be done both to compute the IDRAIN flag and the
IMAINT flag. However, doing so requires that tracking servers are not
marked as inherited maintenance anymore while parsing the configuration
(and given that it is wrong, better drop it).

This fix also addresses another side effect of the bug above which is
that the IDRAIN/IMAINT flags are stored in the state files, and if
restored while the tracked server doesn't have the equivalent flag,
the servers may end up in a situation where it's impossible to remove
these flags. For example in the configuration above, after removing
"disabled" on server s4, the other servers would have remained down,
and not anymore with this fix. Similarly, the combination of IMAINT
or IDRAIN with their respective forced modes was not accepted on
reload, which is wrong as well.

This bug has been present at least since 1.5, maybe even 1.4 (it came
with tracking support). The fix needs to be backported there, though
the srv-state parts are irrelevant.

This commit relies on previous patch to silence warnings on startup.
2016-11-07 14:31:52 +01:00
Willy Tarreau
2235b261b6 OPTIM: http: move all http character classs tables into a single one
We used to have 7 different character classes, each was 256 bytes long,
resulting in almost 2kB being used in the L1 cache. It's as cheap to
test a bit than to check the byte is not null, so let's store a 7-bit
composite value and check for the respective bits there instead.

The executable is now 4 kB smaller and the performance on small
objects increased by about 1% to 222k requests/second with a config
involving 4 http-request rules including 1 header lookup, one header
replacement, and 2 variable assignments.
2016-11-05 15:58:08 +01:00
Willy Tarreau
a5bc36b31c MINOR: stats: emit dses
This is the number of denied sessions, blocked by "tcp-request session reject".
2016-10-21 18:19:48 +02:00
Willy Tarreau
620408f406 MEDIUM: tcp: add registration and processing of TCP L5 rules
This commit introduces "tcp-request session" rules. These are very
much like "tcp-request connection" rules except that they're processed
after the handshake, so it is possible to consider SSL information and
addresses rewritten by the proxy protocol header in actions. This is
particularly useful to track proxied sources as this was not possible
before, given that tcp-request content rules are processed after each
HTTP request. Similarly it is possible to assign the proxied source
address or the client's cert to a variable.
2016-10-21 18:19:24 +02:00
Willy Tarreau
7d9736fb5d CLEANUP: tcp rules: mention everywhere that tcp-conn rules are L4
This is in order to make integration of tcp-request-session cleaner :
- tcp_exec_req_rules() was renamed tcp_exec_l4_rules()
- LI_O_TCP_RULES was renamed LI_O_TCP_L4_RULES
  (LI_O_*'s horrible indent was also fixed and a provision was left
   for L5 rules).
2016-10-21 18:19:24 +02:00
Willy Tarreau
8a90b8ea19 MINOR: stats: output dcon
These are denied conns. Strangely this wasn't emitted while it used to be
available for a while. It corresponds to the number of connections blocked
by "tcp-request connection reject".
2016-10-21 18:17:56 +02:00
William Lallemand
1e08cd819a MEDIUM: cli: register CLI keywords with cli_register_kw()
To register a new cli keyword, you need to declare a cli_kw_list
structure in your source file:

	static struct cli_kw_list cli_kws = {{ },{
		{ { "test", "list", NULL }, "test list : do some tests on the cli", test_parsing, NULL },
		{ { NULL }, NULL, NULL, NULL, NULL }
	}};

And then register it:

	cli_register_kw(&cli_kws);

The first field is an array of 5 elements, where you declare the
keywords combination which will match, it must be ended by a NULL
element.

The second field is used as a usage message, it will appear in the help
of the cli, you can set it to NULL if you don't want to show it, it's a
good idea if you want to overwrite some existing keywords.

The two last fields are callbacks.

The first one is used at parsing time, you can use it to parse the
arguments of your keywords and print small messages. The function must
return 1 in case of a failure, otherwise 0:

	#include <proto/dumpstats.h>

	static int test_parsing(char **args, struct appctx *appctx)
	{
		struct chunk out;

		if (!*args[2]) {
			appctx->ctx.cli.msg = "Error: the 3rd argument is mandatory !";
			appctx->st0 = STAT_CLI_PRINT;
			return 1;
		}
		chunk_reset(&trash);
		chunk_printf(&trash, "arg[3]: %s\n", args[2]);
		chunk_init(&out, NULL, 0);
		chunk_dup(&out, &trash);
		appctx->ctx.cli.err = out.str;
		appctx->st0 = STAT_CLI_PRINT_FREE; /* print and free in the default cli_io_handler */
		return 0;
	}

The last field is the IO handler callback, it can be set to NULL if you
want to use the default cli_io_handler() otherwise you can write your
own. You can use the private pointer in the appctx if you need to store
a context or some data. stats_dump_sess_to_buffer() is a good example of
IO handler, IO handlers often use the appctx->st2 variable for the state
machine. The handler must return 0 in case it have to be recall later
otherwise 1.
2016-10-19 19:03:40 +02:00
Fr�d�ric L�caille
523cc9e858 MEDIUM: peers: Fix a peer stick-tables synchronization issue.
During the stick-table teaching process which occurs at reloading/restart time,
expiration dates of stick-tables entries were not synchronized between peers.

This patch adds two new stick-table messages to provide such a synchronization feature.

As these new messages are not supported by older haproxy peers protocol versions,
this patch increments peers protol version, from 2.0 to 2.1, to help in detecting/supporting
such older peers protocol implementations so that new versions might still be able
to transparently communicate with a newer one.

[wt: technically speaking it would be nice to have this backported into 1.6
 as some people who reload often are affected by this design limitation, but
 it's not a totally transparent change that may make certain users feel
 reluctant to upgrade older versions. Let's let it cook in 1.7 first and
 decide later]
2016-10-17 19:44:35 +02:00
Baptiste Assmann
c1ce5f358e MEDIUM: dns: new DNS response parser
New DNS response parser function which turn the DNS response from a
network buffer into a DNS structure, much easier for later analysis
by upper layer.

Memory is pre-allocated at start-up in a chunk dedicated to DNS
response store.

New error code to report a wrong number of queries in a DNS response.
2016-09-12 19:54:23 +02:00
Baptiste Assmann
d458adcc52 MINOR: new update_server_addr_port() function to change both server's ADDR and service PORT
This function can replace update_server_addr() where the need to change the
server's port as well as the IP address is required.
It performs some validation before performing each type of change.
2016-09-11 08:13:11 +02:00
Baptiste Assmann
95db2bcfee MAJOR: check: find out which port to use for health check at run time
HAProxy used to deduce port used for health checks when parsing configuration
at startup time.
Because of this way of working, it makes it complicated to change the port at
run time.

The current patch changes this behavior and makes HAProxy to choose the
port used for health checking when preparing the check task itself.

A new type of error is introduced and reported when no port can be found.

There won't be any impact on performance, since the process to find out the
port value is made of a few 'if' statements.

This patch also introduces a new check state CHK_ST_PORT_MISS: this flag is
used to report an error in the case when HAProxy needs to establish a TCP
connection to a server, to perform a health check but no TCP ports can be
found for it.

And last, it also introduces a new stream termination condition:
SF_ERR_CHK_PORT. Purpose of this flag is to report an error in the event when
HAProxy has to run a health check but no port can be found to perform it.
2016-09-11 08:12:13 +02:00
Willy Tarreau
77128f585c MINOR: sample: provide smp_is_rw() and smp_make_rw()
At some places, smp_dup() is inappropriately called to ensure a modification
is possible while in fact we only need to ensure the sample may be modified
in place. Let's provide smp_is_rw() to check for this capability and
smp_make_rw() to perform the smp_dup() when it is not the case.

Note that smp_is_rw() will also try to add the trailing zero on strings when
needed if possible, to avoid a useless duplication.
2016-08-09 14:30:57 +02:00
Willy Tarreau
2c594794dd MINOR: sample: implement smp_is_safe() and smp_make_safe()
These functions ensure that the designated sample is "safe for use",
which means that its size is known, its length is correct regarding its
size, and that strings are properly zero-terminated.

smp_is_safe() only checks (and optionally sets the trailing zero when
needed and possible). smp_make_safe() will call smp_dup() after
smp_is_safe() fails.
2016-08-09 14:03:36 +02:00
Ruoshan Huang
e4edc6b628 MEDIUM: http: implement http-response track-sc* directive
This enables tracking of sticky counters from current response. The only
difference from "http-request track-sc" is the <key> sample expression
can only make use of samples in response (eg. res.*, status etc.) and
samples below Layer 6.
2016-07-26 14:31:14 +02:00
Bertrand Jacquin
93b227db95 MINOR: listener: add the "accept-netscaler-cip" option to the "bind" keyword
When NetScaler application switch is used as L3+ switch, informations
regarding the original IP and TCP headers are lost as a new TCP
connection is created between the NetScaler and the backend server.

NetScaler provides a feature to insert in the TCP data the original data
that can then be consumed by the backend server.

Specifications and documentations from NetScaler:
  https://support.citrix.com/article/CTX205670
  https://www.citrix.com/blogs/2016/04/25/how-to-enable-client-ip-in-tcpip-option-of-netscaler/

When CIP is enabled on the NetScaler, then a TCP packet is inserted just after
the TCP handshake. This is composed as:

  - CIP magic number : 4 bytes
    Both sender and receiver have to agree on a magic number so that
    they both handle the incoming data as a NetScaler Client IP insertion
    packet.

  - Header length : 4 bytes
    Defines the length on the remaining data.

  - IP header : >= 20 bytes if IPv4, 40 bytes if IPv6
    Contains the header of the last IP packet sent by the client during TCP
    handshake.

  - TCP header : >= 20 bytes
    Contains the header of the last TCP packet sent by the client during TCP
    handshake.
2016-06-20 23:02:47 +02:00
Dragan Dosen
e984a0e4fb MINOR: stream: export the function 'smp_create_src_stkctr'
Could be useful outside of this file.
2016-06-13 21:21:51 +02:00
Vincent Bernat
6e61589573 BUG/MAJOR: fix listening IP address storage for frontends
When compiled with GCC 6, the IP address specified for a frontend was
ignored and HAProxy was listening on all addresses instead. This is
caused by an incomplete copy of a "struct sockaddr_storage".

With the GNU Libc, "struct sockaddr_storage" is defined as this:

    struct sockaddr_storage
      {
        sa_family_t ss_family;
        unsigned long int __ss_align;
        char __ss_padding[(128 - (2 * sizeof (unsigned long int)))];
      };

Doing an aggregate copy (ss1 = ss2) is different than using memcpy():
only members of the aggregate have to be copied. Notably, padding can be
or not be copied. In GCC 6, some optimizations use this fact and if a
"struct sockaddr_storage" contains a "struct sockaddr_in", the port and
the address are part of the padding (between sa_family and __ss_align)
and can be not copied over.

Therefore, we replace any aggregate copy by a memcpy(). There is another
place using the same pattern. We also fix a function receiving a "struct
sockaddr_storage" by copy instead of by reference. Since it only needs a
read-only copy, the function is converted to request a reference.
2016-05-19 10:43:24 +02:00
Christopher Faulet
3a394fa7cd MEDIUM: filters: Add pre and post analyzer callbacks
'channel_analyze' callback has been removed. Now, there are 2 callbacks to
surround calls to analyzers:

  * channel_pre_analyze: Called BEFORE all filterable analyzers. it can be
    called many times for the same analyzer, once at each loop until the
    analyzer finishes its processing. This callback is resumable, it returns a
    negative value if an error occurs, 0 if it needs to wait, any other value
    otherwise.

  * channel_post_analyze: Called AFTER all filterable analyzers. Here, AFTER
    means when an analyzer finishes its processing. This callback is NOT
    resumable, it returns a negative value if an error occurs, any other value
    otherwise.

Pre and post analyzer callbacks are not automatically called. 'pre_analyzers'
and 'post_analyzers' bit fields in the filter structure must be set to the right
value using AN_* flags (see include/types/channel.h).

The flag AN_RES_ALL has been added (AN_REQ_ALL already exists) to ease the life
of filter developers. AN_REQ_ALL and AN_RES_ALL include all filterable
analyzers.
2016-05-18 15:11:54 +02:00
Christopher Faulet
a9215b7206 MINOR: filters: Simplify calls to analyzers using 2 new macros
Now, to call an analyzer in 'process_stream' function, we should use
FLT_ANALAYZE or ANALYZE macros, depending if this is a filterable analyzer or
not.
2016-05-18 15:11:54 +02:00
Willy Tarreau
8bf242b764 BUG/MEDIUM: channel: fix inconsistent handling of 4GB-1 transfers
In 1.4-dev3, commit 31971e5 ("[MEDIUM] add support for infinite forwarding")
made it possible to configure the lower layer to forward data indefinitely
by setting the forward size to CHN_INFINITE_FORWARD (4GB-1). By then larger
chunk sizes were not supported so there was no confusion in the usage of the
function.

Since 1.5 we support 64-bit content-lengths and chunk sizes and the function
has grown to support 64-bit arguments, though it still limits a single pass
to 32-bit quantities (what fit in the channel's to_forward field). The issue
now becomes that a 4GB-1 content-length can be confused with infinite
forwarding (in fact it's 4GB-1+what was already in the buffer). It causes a
visible effect when transferring this exact size because the transfer rate
is lower than with other sizes due in part to the disabling of the Nagle
algorithm on the sendto() call.

In theory with keep-alive it should prevent a second request from being
processed after such a transfer, but since the analysers are still present,
the forwarding analyser properly counts down the remaining size to transfer
and ultimately the transaction gets correctly reset so there is no visible
effect.

Since the root cause of the issue is an API problem (lack of distinction
between a real valid length and a magic value), this patch modifies the API
to have a new dedicated function called channel_forward_forever() to program
a permanent forwarding. The existing function __channel_forward() was modified
to properly take care of the requested sizes and ensure it 1) never overflows
and 2) never reaches CHN_INFINITE_FORWARD by accident.

It is worth noting that the function used to have a bug causing a 2GB
forward to be scheduled if it was called with less data than what is present
in buf->i. Fortunately this bug couldn't be triggered with existing code.

This fix should be backported to 1.6 and 1.5. While it also theorically
affects 1.4, it's better not to backport it there, as the risk of breaking
large object transfers due to significant API differences is high, compared
to the fact that the largest supported objects (4GB-1) are just slower to
transfer.
2016-05-04 15:26:37 +02:00
Willy Tarreau
ef907fee12 BUG/MAJOR: channel: fix miscalculation of available buffer space (4th try)
Unfortunately, commit 169c470 ("BUG/MEDIUM: channel: fix miscalculation of
available buffer space (3rd try)") was still not enough to completely
address the issue. It fell into an integer comparison trap. Contrary to
expectations, chn->to_forward may also have the sign bit set when
forwarding regular data having a large content-length, resulting in
an incomplete check of the result and of the reserve because the with
to_forward very large, to_forward+o could become very small and also
the reserve could become positive again and make channel_recv_limit()
return a negative value.

One way to reproduce this situation is to transfer a large file (> 2GB)
with http-keep-alive or http-server-close, without splicing, and ensure
that the server uses content-length instead of chunks. The transfer
should stall very early after the first buffer has been transferred
to the client.

This fix now properly checks 1) for an overflow caused by summing o and
to_forward, and 2) for o+to_forward being smaller or larger than maxrw
before performing the subtract, so that all sensitive operations are
properly performed on 33-bit arithmetics.

The code was subjected again to a series of tests using inject+httpterm
scanning a wide range of object sizes (+10MB after each new request) :

  $ printf "new page 1\nget 127.0.0.1:8002 / s=%%s0m\n" | \
      inject64 -o 1 -u 1 -f /dev/stdin

With previous fix, the transfer would suddenly stop when reaching 2GB :

   hits ^hits hits/s  ^h/s     bytes  kB/s  last  errs  tout htime  sdht ptime
    203     1      2     1 216816173354 2710202 3144892     0     0 685.0 0.0 685.0
    205     2      2     2 219257283186 2706880 2441109     0     0 679.5 6.5 679.5
    205     0      2     0 219257283186 2673836     0     0     0 0.0 0.0 0.0
    205     0      2     0 219257283186 2641622     0     0     0 0.0 0.0 0.0
    205     0      2     0 219257283186 2610174     0     0     0 0.0 0.0 0.0

Now it's fine even past 4 GB.

Many thanks to Vedran Furac for reporting this issue early with a common
access pattern helping to troubleshoot this.

This fix must be backported to 1.6 and 1.5 where the commit above was
already backported.
2016-05-03 17:58:03 +02:00
Willy Tarreau
55e58f2334 MINOR: channel: add new function channel_congested()
This function returns non-zero if the channel is congested with data in
transit waiting for leaving, indicating to the caller that it should wait
for the reserve to be released before starting to process new data in
case it needs the ability to append data. This is meant to be used while
waiting for a clean response buffer before processing a request.
2016-05-02 16:39:22 +02:00
Nenad Merdanovic
174dd37d88 MINOR: Add ability for agent-check to set server maxconn
This is very useful in complex architecture systems where HAproxy
is balancing DB connections for example. We want to keep the maxconn
high in order to avoid issues with queueing on the LB level when
there is slowness on another part of the system. Example is a case of
an architecture where each thread opens multiple DB connections, which
if get stuck in queue cause a snowball effect (old connections aren't
closed, new ones cannot be established). These connections are mostly
idle and the DB server has no problem handling thousands of them.

Allowing us to dynamically set maxconn depending on the backend usage
(LA, CPU, memory, etc.) enables us to have high maxconn for situations
like above, but lowering it in case there are real issues where the
backend servers become overloaded (cache issues, DB gets hit hard).
2016-04-25 17:23:50 +02:00
Willy Tarreau
169c47028a BUG/MEDIUM: channel: fix miscalculation of available buffer space (3rd try)
Latest fix 8a32106 ("BUG/MEDIUM: channel: fix miscalculation of available
buffer space (2nd try)") did happen to fix some observable issues but not
all of them in fact, some corner cases still remained and at least one user
reported a busy loop that appeared possible, though not easily reproducible
under experimental conditions.

The remaining issue is that we still consider min(i, to_fwd) as the number
of bytes in transit, but in fact <i> is not relevant here. Indeed, what
matters is that we can read everything we want at once provided that at
the end, <i> cannot be larger than <size-maxrw> (if it was not already).

This is visible in two cases :
  - let's have i=o=max/2 and to_fwd=0. Then i+o >= max indicates that the
    buffer is already full, while it is not since once <o> is forwarded,
    some space remains.

  - when to_fwd is much larger than i, it's obvious that we can fill the
    buffer.

The only relevant part in fact is o + to_fwd. to_fwd will ensure that at
least this many bytes will be moved from <i> to <o> hence will leave the
buffer, whatever the number of rounds it takes.

Interestingly, the fix applied here ensures that channel_recv_max() will
now equal (size - maxrw - i + to_fwd), which is indeed what remains
available below maxrw after to_fwd bytes are forwarded from i to o and
leave the buffer.

Additionally, the latest fix made it possible to meet an integer overflow
that was not caught by the range test when forwarding in TCP or tunnel
mode due to to_forward being added to an existing value, causing the
buffer size to be limited when it should not have been, resulting in 2
to 3 recv() calls when a single one was enough. The first one was limited
to the unreserved buffer size, the second one to the size of the reserve
minus 1, and the last one to the last byte. Eg with a 2kB buffer :

recvfrom(22, "HTTP/1.1 200\r\nConnection: close\r"..., 1024, 0, NULL, NULL) = 1024
recvfrom(22, "23456789.123456789.123456789.123"..., 1023, 0, NULL, NULL) = 1023
recvfrom(22, "5", 1, 0, NULL, NULL)     = 1

This bug is still present in 1.6 and 1.5 so the fix should be backported
there.
2016-04-21 18:06:08 +02:00
Willy Tarreau
93dc478a04 BUG/MEDIUM: channel: incorrect polling condition may delay event delivery
The condition to poll for receive as implemented in channel_may_recv()
is still incorrect. If buf->o is null and buf->i is slightly larger than
chn->to_forward and at least as large as buf->size - maxrewrite, then
reading will be disabled. It may slightly delay some data delivery by
having first to forward pending bytes, but may also cause some random
issues with analysers that wait for some data before starting to forward
what they correctly parsed. For instance, a body analyser may be prevented
from seeing the data that only fits in the reserve.

This bug may also prevent an applet's chk_rcv() function from being called
when part of a buffer is released. It is possible (though not verified)
that this participated to some peers frozen session issues some people
have been facing.

This fix should be backported to 1.6 and 1.5 to ensure better coherency
with channel_recv_limit().
2016-04-21 17:03:46 +02:00
Willy Tarreau
4b46a3e8cc BUG/MEDIUM: channel: don't allow to overwrite the reserve until connected
Commit 9c06ee4 ("BUG/MEDIUM: channel: don't schedule data in transit for
leaving until connected") took care of an issue involving POST in conjunction
with http-send-name-header, where we absolutely never want to touch the
reserve until we're sure not to touch the buffer contents anymore, which
is indicated by the output stream-interface being connected.

But channel_may_recv() was not equipped with such a test, so in some
situations it might decide that it is possible to poll for reads, and
later channel_recv_limit() will decide it's not possible to read,
causing a loop. So we must add a similar test there.

Since the fix above was backported to 1.6 and 1.5, this fix must as well.
2016-04-21 15:31:22 +02:00
Christopher Faulet
b3f4e14932 MINOR: filters: Print the list of existing filters during HA startup
This is done  in verbose/debug mode and when build options are reported.
2016-04-21 06:58:08 +02:00
Willy Tarreau
7a798e5d6b CLEANUP: fix inconsistency between fd->iocb, proto->accept and accept()
There's quite some inconsistency in the internal API. listener_accept()
which is the main accept() function returns void but is declared as int
in the include file. It's assigned to proto->accept() for all stream
protocols where an int is expected but the result is never checked (nor
is it documented by the way). This proto->accept() is in turn assigned
to fd->iocb() which is supposed to return an int composed of FD_WAIT_*
flags, but which is never checked either.

So let's fix all this mess :
  - nobody checks accept()'s return
  - nobody checks iocb()'s return
  - nobody sets a return value

=> let's mark all these functions void and keep the current ones intact.

Additionally we now include listener.h from listener.c to ensure we won't
silently hide this incoherency in the future.

Note that this patch could/should be backported to 1.6 and even 1.5 to
simplify debugging sessions.
2016-04-14 11:18:22 +02:00
Willy Tarreau
8a32106fff BUG/MEDIUM: channel: fix miscalculation of available buffer space (2nd try)
Commit 999f643 ("BUG/MEDIUM: channel: fix miscalculation of available buffer
space.") introduced a bug which made output data to be ignored when computing
the remaining room in a buffer. The problem is that channel_may_recv()
properly considers them and may declare that the FD may be polled for read
events, but once the even strikes, channel_recv_limit() called before recv()
says the opposite. In 1.6 and later this case is automatically caught by
polling loop detection at the connection level and is harmless. But the
backport in 1.5 ends up with a busy polling loop as soon as it becomes
possible to have a buffer with this conflict. In order to reproduce it, it
is necessary to have less than [maxrewrite] bytes available in a buffer, no
forwarding enabled (end of transfer) and [buf->o >= maxrewrite - free space].

Since this heavily depends on socket buffers, it will randomly strike users.
On 1.5 with 8kB buffers it was possible to reproduce it with httpterm using
the following command line :

   $ (printf "GET /?s=675000 HTTP/1.0\r\n\r\n"; sleep 60) | \
       nc6 --rcvbuf-size 1 --send-only 127.0.0.1 8002

This bug is only medium in 1.6 and later but is major in the 1.5 backport,
so it must be backported there.

Thanks to Nenad Merdanovic and Janusz Dziemidowicz for reporting this issue
with enough elements to help understand it.
2016-04-11 17:13:35 +02:00
Thierry Fournier
d0a56c2953 MINOR: dumpstats: split stats_dump_be_stats() in two parts
This patch splits the function stats_dump_be_stats() in two parts. The
part is called stats_fill_be_stats(), and just fill the stats buffer.
This split allows the usage of preformated stats in other parts of HAProxy
like the Lua.
2016-03-30 17:26:19 +02:00
Thierry Fournier
61fe6c0adb MINOR: dumpstats: split stats_dump_sv_stats() in two parts
This patch splits the function stats_dump_sv_stats() in two parts. The
extracted part is called stats_fill_sv_stats(), and just fill the stats buffer.
This split allows the usage of preformated stats in other parts of HAProxy
like the Lua.
2016-03-30 17:26:09 +02:00
Thierry Fournier
c4456856b0 MINOR: dumpstats: split stats_dump_li_stats() in two parts
This patch splits the function stats_dump_li_stats() in two parts. The
extracted part is called stats_fill_li_stats(), and just fill the stats buffer.
This split allows the usage of preformated stats in other parts of HAProxy
like the Lua.
2016-03-30 17:26:02 +02:00
Thierry Fournier
23d2d64185 MINOR: dumpstats: split stats_dump_fe_stats() in two parts
This patch splits the function stats_dump_fe_stats() in two parts. The
extracted part is called stats_fill_fe_stats(), and just fill the stats buffer.
This split allows the usage of preformated stats in other parts of HAProxy
like the Lua.
2016-03-30 17:21:59 +02:00
Thierry Fournier
cb2c767681 MINOR: dumpstats: split stats_dump_info_to_buffer() in two parts
This patch splits the function stats_dump_info_to_buffer() in two parts. The
extracted part is called stats_fill_info(), and just fill the stats buffer.
This split allows the usage of preformated stats in other parts of HAProxy
like the Lua.
2016-03-30 17:21:37 +02:00
Thierry Fournier
31e64ca301 MINOR: dumpstats: extract stats fields enum and names
These field names can be used outside of the dumpstats file.
This will be useful for exporting stats in Lua.
2016-03-30 17:21:09 +02:00
Thierry Fournier
3d4a675f24 MINOR: lua: post initialization
This patch adds a Lua post initialisation wrapper. It already exists for
pure Lua function, now it executes also C. It is useful for doing things
when the configuration is ready to use. For example we can can browse and
register all the proxies.
2016-03-30 15:44:58 +02:00
Thierry Fournier
45e78d7aa9 MINOR: lua: refactor the Lua object registration
All the HAProxy Lua object are declared with the same pattern:
 - Add the function __tosting which dumps the object name
 - Register the name in the Lua REGISTRY
 - Register the reference ID

These action are refactored in on function. This remove some
lines of code.
2016-03-30 15:43:52 +02:00
Thierry Fournier
ddd8988fe5 MINOR: lua: move class registration facilities
The functions
 - hlua_class_const_int()
 - hlua_class_const_str()
 - hlua_class_function()
are use for common class registration actions.

The function 'hlua_dump_object()' is generic dump name function.

These functions can be used by all the HAProxy objects, so I move
it into the safe functions file.
2016-03-30 15:42:20 +02:00
David Carlier
15073a3393 MINOR: sample: Moves ARGS underlying type from 32 to 64 bits.
ARG# macros allow to create a list up to 7 in theory but 5 in
practice. The change to a guaranteed 64 bits type increase to
up to 12.
2016-03-15 22:11:52 +01:00
Willy Tarreau
cb80912001 MEDIUM: stats: support "show info typed" on the CLI
This emits the field positions, names and types. It is more convenient
than the default output for a parser that doesn't know all the fields. It
simply relies on stats_emit_typed_data_field() and stats_emit_field_tags()
added by previous patch for the output. A new stats format flag was added,
STAT_FMT_TYPED, which is set when the "typed" keyword is specified on the
CLI.
2016-03-11 17:08:06 +01:00
Willy Tarreau
b47785f862 MINOR: stats: add functions to emit typed fields into a chunk
New function stats_emit_typed_data_field() does exactly like
stats_emit_raw_data_field() except that it also prints the data
type after a colon. This will be used to print using the typed
format.

And function stats_emit_field_tags() appends a 3-letter code
describing the origin, nature, and scope, followed by an optional
delimiter. This will be particularly convenient to dump typed
data.
2016-03-11 17:08:05 +01:00
Willy Tarreau
8e62c05af2 MINOR: stats: create fields types suitable for all CSV output data
We're preparing for various data types for each stats field as they
appear in the CSV output. For now we only cover the regular types handled
by printf, so we have 32 and 64 bit ints and counters, strings, and of
course "empty" to indicate that there's nothing in the field and which
guarantees that any accessed entry will return 0.

More types will surely come later so that some fields are properly
represented. For example, we could see limits where only the value 0
doesn't show up, or human time, etc.
2016-03-11 17:08:04 +01:00
Willy Tarreau
6204cd9f27 BUG/MAJOR: vars: always retrieve the stream and session from the sample
This is the continuation of previous patch called "BUG/MAJOR: samples:
check smp->strm before using it".

It happens that variables may have a session-wide scope, and that their
session is retrieved by dereferencing the stream. But nothing prevents them
from being used from a streamless context such as tcp-request connection,
thus crashing the process. Example :

    tcp-request connection accept if { src,set-var(sess.foo) -m found }

In order to fix this, we have to always ensure that variable manipulation
only happens via the sample, which contains the correct owner and context,
and that we never use one from a different source. This results in quite a
large change since a lot of functions are inderctly involved in the call
chain, but the change is easy to follow.

This fix must be backported to 1.6, and requires the last two patches.
2016-03-10 17:28:04 +01:00
Willy Tarreau
1777ea63e0 MINOR: sample: add a new helper to initialize the owner of a sample
Since commit 6879ad3 ("MEDIUM: sample: fill the struct sample with the
session, proxy and stream pointers") merged in 1.6-dev2, the sample
contains the pointer to the stream and sample fetch functions as well
as converters use it heavily. This requires from a lot of call places
to initialize 4 fields, and it was even forgotten at a few places.

This patch provides a convenient helper to initialize all these fields
at once, making it easy to prepare a new sample from a previous one for
example.

A few call places were cleaned up to make use of it. It will be needed
by further fixes.

At one place in the Lua code, it was moved earlier because we used to
call sample casts with a non completely initialized sample, which is
not clean eventhough at the moment there are no consequences.
2016-03-10 16:42:58 +01:00
Thierry Fournier
09a9178311 MINOR: server: generalize the "updater" source
the function server_parse_addr_change_request() contain an hardcoded
updater source "stats command". this function can be called from other
sources than the "stats command", so this patch make this argument
generic.
2016-02-24 23:37:39 +01:00
Thierry Fournier
d35b7a6d93 CLEANUP: server: add "const" to some message strings
"updater" is used in "read only" mode, so I add a const qualifier
to the variable declaration.
2016-02-24 23:37:39 +01:00
Thierry Fournier
9f72555b65 BUG/MINOR: server: some prototypes are renamed
The commit 87b096 renames the functions srv_shutdown_backup_sessions()
and srv_shutdown_sessions() to srv_shutdown_backup_streams() and
srv_shutdown_streams().

The header file <proto/servers.h> does not repport these changes.

This bug should be repported in the 1.6 branch, even if it is useless
because new dev are frozen.
2016-02-23 22:42:47 +01:00
Thierry Fournier
ada348459f MEDIUM: dns: extract options
DNS selection preferences are actually declared inline in the
struct server. There are copied from the server struct to the
dns_resolution struct for each resolution.

Next patchs adds new preferences options, and it is not a good
way to copy all the configuration information before each dns
resolution.

This patch extract the configuration preference from the struct
server and declares a new dedicated struct. Only a pointer to this
new striuict will be copied before each dns resolution.
2016-02-19 14:37:46 +01:00
Dragan Dosen
835b9212f6 MEDIUM: log: add a new log format flag "E"
The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It
mostly makes sense to use it in the RFC5424 structured-data log formats.

Example:

log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 13:36:47 +01:00
Thierry Fournier
9e7e3ea991 MINOR: lua: move common function
This patch moves the function hlua_checkudata which check that
an object contains the expected class_reference as metatable.
This function is commonly used by all the lua functions.
The function hlua_metatype is also moved.
2016-02-12 11:08:53 +01:00
Thierry Fournier
fb0b5467ca MINOR: lua: file dedicated to unsafe functions
When Lua executes functions from its API, these can throws an error.
These function must be executed in a special environment which catch
these error, otherwise a critical error (like segfault) can raise.

This patch add a c file called "hlua_fcn.c" which collect all the
Lua/c function needing safe environment for its execution.
2016-02-12 11:08:53 +01:00
Thierry Fournier
8feaa661b6 MINOR: map: Add regex matching replacement
This patch declares a new map which provides a string based on
a string with back references replaced by the content matched
by the regex.
2016-02-10 23:38:34 +01:00
Christopher Faulet
443ea1a242 MINOR: filters: Extract proxy stuff from the struct filter
Now, filter's configuration (.id, .conf and .ops fields) is stored in the
structure 'flt_conf'. So proxies own a flt_conf list instead of a filter
list. When a filter is attached to a stream, it gets a pointer on its
configuration. This avoids mixing the filter's context (owns by a stream) and
its configuration (owns by a proxy). It also saves 2 pointers per filter
instance.
2016-02-09 14:53:15 +01:00
Christopher Faulet
3e7bc67722 MINOR: filters: Remove unused or useless stuff and do small optimizations 2016-02-09 14:53:15 +01:00
Christopher Faulet
da02e17d42 MAJOR: filters: Require explicit registration to filter HTTP body and TCP data
Before, functions to filter HTTP body (and TCP data) were called from the moment
at least one filter was attached to the stream. If no filter is interested by
these data, this uselessly slows data parsing.
A good example is the HTTP compression filter. Depending of request and response
headers, the response compression can be enabled or not. So it could be really
nice to call it only when enabled.

So, now, to filter HTTP/TCP data, a filter must use the function
register_data_filter. For TCP streams, this function can be called only
once. But for HTTP streams, when needed, it must be called for each HTTP request
or HTTP response.
Only registered filters will be called during data parsing. At any time, a
filter can be unregistered by calling the function unregister_data_filter.
2016-02-09 14:53:15 +01:00
Christopher Faulet
fcf035cb5a MINOR: filters: Add stream_filters structure to hide filters info
From the stream point of view, this new structure is opaque. it hides filters
implementation details. So, impact for future optimizations will be reduced
(well, we hope so...).

Some small improvements has been made in filters.c to avoid useless checks.
2016-02-09 14:53:15 +01:00
Christopher Faulet
309c6418b0 MEDIUM: filters: Replace filter_http_headers callback by an analyzer
This new analyzer will be called for each HTTP request/response, before the
parsing of the body. It is identified by AN_FLT_HTTP_HDRS.

Special care was taken about the following condition :

  * the frontend is a TCP proxy
  * filters are defined in the frontend section
  * the selected backend is a HTTP proxy

So, this patch explicitly add AN_FLT_HTTP_HDRS analyzer on the request and the
response channels when the backend is a HTTP proxy and when there are filters
attatched on the stream.
This patch simplifies http_request_forward_body and http_response_forward_body
functions.
2016-02-09 14:53:15 +01:00
Christopher Faulet
2fb2880caf MEDIUM: filters: remove http_start_chunk, http_last_chunk and http_chunk_end
For Chunked HTTP request/response, the body filtering can be really
expensive. In the worse case (many chunks of 1 bytes), the filters overhead is
of 3 calls per chunk. If http_data callback is useful, others are just
informative.

So these callbacks has been removed. Of course, existing filters (trace and
compression) has beeen updated accordingly. For the HTTP compression filter, the
update is quite huge. Its implementation is closer to the old one.
2016-02-09 14:53:15 +01:00
Christopher Faulet
3e34429515 MEDIUM: filters: Use macros to call filters callbacks to speed-up processing
When no filter is attached to the stream, the CPU footprint due to the calls to
filters_* functions is huge, especially for chunk-encoded messages. Using macros
to check if we have some filters or not is a great improvement.

Furthermore, instead of checking the filter list emptiness, we introduce a flag
to know if filters are attached or not to a stream.
2016-02-09 14:53:15 +01:00
Christopher Faulet
92d3638d2d MAJOR: filters/http: Rewrite the HTTP compression as a filter
HTTP compression has been rewritten to use the filter API. This is more a PoC
than other thing for now. It allocates memory to work. So, if only for that, it
should be rewritten.

In the mean time, the implementation has been refactored to allow its use with
other filters. However, there are limitations that should be respected:

  - No filter placed after the compression one is allowed to change input data
    (in 'http_data' callback).
  - No filter placed before the compression one is allowed to change forwarded
    data (in 'http_forward_data' callback).

For now, these limitations are informal, so you should be careful when you use
several filters.

About the configuration, 'compression' keywords are still supported and must be
used to configure the HTTP compression behavior. In absence of a 'filter' line
for the compression filter, it is added in the filter chain when the first
compression' line is parsed. This is an easy way to do when you do not use other
filters. But another filter exists, an error is reported so that the user must
explicitly declare the filter.

For example:

  listen tst
      ...
      compression algo gzip
      compression offload
      ...
      filter flt_1
      filter compression
      filter flt_2
      ...
2016-02-09 14:53:15 +01:00
Christopher Faulet
3d97c90974 REORG: filters: Prepare creation of the HTTP compression filter
HTTP compression will be moved in a true filter. To prepare the ground, some
functions have been moved in a dedicated file. Idea is to keep everything about
compression algos in compression.c and everything related to the filtering in
flt_http_comp.c.

For now, a header has been added to help during the transition. It will be
removed later.

Unused empty ACL keyword list was removed. The "compression" keyword
parser was moved from cfgparse.c to flt_http_comp.c.
2016-02-09 14:53:15 +01:00
Christopher Faulet
d7c9196ae5 MAJOR: filters: Add filters support
This patch adds the support of filters in HAProxy. The main idea is to have a
way to "easely" extend HAProxy by adding some "modules", called filters, that
will be able to change HAProxy behavior in a programmatic way.

To do so, many entry points has been added in code to let filters to hook up to
different steps of the processing. A filter must define a flt_ops sutrctures
(see include/types/filters.h for details). This structure contains all available
callbacks that a filter can define:

struct flt_ops {
       /*
        * Callbacks to manage the filter lifecycle
        */
       int  (*init)  (struct proxy *p);
       void (*deinit)(struct proxy *p);
       int  (*check) (struct proxy *p);

        /*
         * Stream callbacks
         */
        void (*stream_start)     (struct stream *s);
        void (*stream_accept)    (struct stream *s);
        void (*session_establish)(struct stream *s);
        void (*stream_stop)      (struct stream *s);

       /*
        * HTTP callbacks
        */
       int  (*http_start)         (struct stream *s, struct http_msg *msg);
       int  (*http_start_body)    (struct stream *s, struct http_msg *msg);
       int  (*http_start_chunk)   (struct stream *s, struct http_msg *msg);
       int  (*http_data)          (struct stream *s, struct http_msg *msg);
       int  (*http_last_chunk)    (struct stream *s, struct http_msg *msg);
       int  (*http_end_chunk)     (struct stream *s, struct http_msg *msg);
       int  (*http_chunk_trailers)(struct stream *s, struct http_msg *msg);
       int  (*http_end_body)      (struct stream *s, struct http_msg *msg);
       void (*http_end)           (struct stream *s, struct http_msg *msg);
       void (*http_reset)         (struct stream *s, struct http_msg *msg);
       int  (*http_pre_process)   (struct stream *s, struct http_msg *msg);
       int  (*http_post_process)  (struct stream *s, struct http_msg *msg);
       void (*http_reply)         (struct stream *s, short status,
                                   const struct chunk *msg);
};

To declare and use a filter, in the configuration, the "filter" keyword must be
used in a listener/frontend section:

  frontend test
    ...
    filter <FILTER-NAME> [OPTIONS...]

The filter referenced by the <FILTER-NAME> must declare a configuration parser
on its own name to fill flt_ops and filter_conf field in the proxy's
structure. An exemple will be provided later to make it perfectly clear.

For now, filters cannot be used in backend section. But this is only a matter of
time. Documentation will also be added later. This is the first commit of a long
list about filters.

It is possible to have several filters on the same listener/frontend. These
filters are stored in an array of at most MAX_FILTERS elements (define in
include/types/filters.h). Again, this will be replaced later by a list of
filters.

The filter API has been highly refactored. Main changes are:

* Now, HA supports an infinite number of filters per proxy. To do so, filters
  are stored in list.

* Because filters are stored in list, filters state has been moved from the
  channel structure to the filter structure. This is cleaner because there is no
  more info about filters in channel structure.

* It is possible to defined filters on backends only. For such filters,
  stream_start/stream_stop callbacks are not called. Of course, it is possible
  to mix frontend and backend filters.

* Now, TCP streams are also filtered. All callbacks without the 'http_' prefix
  are called for all kind of streams. In addition, 2 new callbacks were added to
  filter data exchanged through a TCP stream:

    - tcp_data: it is called when new data are available or when old unprocessed
      data are still waiting.

    - tcp_forward_data: it is called when some data can be consumed.

* New callbacks attached to channel were added:

    - channel_start_analyze: it is called when a filter is ready to process data
      exchanged through a channel. 2 new analyzers (a frontend and a backend)
      are attached to channels to call this callback. For a frontend filter, it
      is called before any other analyzer. For a backend filter, it is called
      when a backend is attached to a stream. So some processing cannot be
      filtered in that case.

    - channel_analyze: it is called before each analyzer attached to a channel,
      expects analyzers responsible for data sending.

    - channel_end_analyze: it is called when all other analyzers have finished
      their processing. A new analyzers is attached to channels to call this
      callback. For a TCP stream, this is always the last one called. For a HTTP
      one, the callback is called when a request/response ends, so it is called
      one time for each request/response.

* 'session_established' callback has been removed. Everything that is done in
  this callback can be handled by 'channel_start_analyze' on the response
  channel.

* 'http_pre_process' and 'http_post_process' callbacks have been replaced by
  'channel_analyze'.

* 'http_start' callback has been replaced by 'http_headers'. This new one is
  called just before headers sending and parsing of the body.

* 'http_end' callback has been replaced by 'channel_end_analyze'.

* It is possible to set a forwarder for TCP channels. It was already possible to
  do it for HTTP ones.

* Forwarders can partially consumed forwardable data. For this reason a new
  HTTP message state was added before HTTP_MSG_DONE : HTTP_MSG_ENDING.

Now all filters can define corresponding callbacks (http_forward_data
and tcp_forward_data). Each filter owns 2 offsets relative to buf->p, next and
forward, to track, respectively, input data already parsed but not forwarded yet
by the filter and parsed data considered as forwarded by the filter. A any time,
we have the warranty that a filter cannot parse or forward more input than
previous ones. And, of course, it cannot forward more input than it has
parsed. 2 macros has been added to retrieve these offets: FLT_NXT and FLT_FWD.

In addition, 2 functions has been added to change the 'next size' and the
'forward size' of a filter. When a filter parses input data, it can alter these
data, so the size of these data can vary. This action has an effet on all
previous filters that must be handled. To do so, the function
'filter_change_next_size' must be called, passing the size variation. In the
same spirit, if a filter alter forwarded data, it must call the function
'filter_change_forward_size'. 'filter_change_next_size' can be called in
'http_data' and 'tcp_data' callbacks and only these ones. And
'filter_change_forward_size' can be called in 'http_forward_data' and
'tcp_forward_data' callbacks and only these ones. The data changes are the
filter responsability, but with some limitation. It must not change already
parsed/forwarded data or data that previous filters have not parsed/forwarded
yet.

Because filters can be used on backends, when we the backend is set for a
stream, we add filters defined for this backend in the filter list of the
stream. But we must only do that when the backend and the frontend of the stream
are not the same. Else same filters are added a second time leading to undefined
behavior.

The HTTP compression code had to be moved.

So it simplifies http_response_forward_body function. To do so, the way the data
are forwarded has changed. Now, a filter (and only one) can forward data. In a
commit to come, this limitation will be removed to let all filters take part to
data forwarding. There are 2 new functions that filters should use to deal with
this feature:

 * flt_set_http_data_forwarder: This function sets the filter (using its id)
   that will forward data for the specified HTTP message. It is possible if it
   was not already set by another filter _AND_ if no data was yet forwarded
   (msg->msg_state <= HTTP_MSG_BODY). It returns -1 if an error occurs.

 * flt_http_data_forwarder: This function returns the filter id that will
   forward data for the specified HTTP message. If there is no forwarder set, it
   returns -1.

When an HTTP data forwarder is set for the response, the HTTP compression is
disabled. Of course, this is not definitive.
2016-02-09 14:53:15 +01:00
Christopher Faulet
635c0adec2 BUG/MINOR: ssl: Be sure to use unique serial for regenerated certificates
The serial number for a generated certificate was computed using the requested
servername, without any variable/random part. It is not a problem from the
moment it is not regenerated.

But if the cache is disabled or when the certificate is evicted from the cache,
we may need to regenerate it. It is important to not reuse the same serial
number for the new certificate. Else clients (especially browsers) trigger a
warning because 2 certificates issued by the same CA have the same serial
number.

So now, the serial is a static variable initialized with now_ms (internal date
in milliseconds) and incremented at each new certificate generation.

(Ref MPS-2031)
2016-02-09 09:04:53 +01:00
Christopher Faulet
c34d19fc3c BUG: stream_interface: Reuse connection even if the output channel is empty
in function 'si_connect', an existing connection is reused (and considered as
established) only when there are some pending data in the output channel.

This can be problem when filters are used, because a filter can choose to not
forward data immediatly. So when we try to initiate a connection to a server,
the output channel can be empty. In this situation, if the connection already
exists, it is not considered as established and nothing happens. If the stream
interface is in the state SI_ST_ASS, this leads to an infinite loop in
process_stream because it remains in this state.

This patch fixes this problem. Now, in 'si_connect', we always reuse an existing
connection, whether or not there are pending data in the output channel.
2016-02-03 14:22:55 +01:00
Willy Tarreau
999f643ed2 BUG/MEDIUM: channel: fix miscalculation of available buffer space.
The function channel_recv_limit() relies on channel_reserved() which
itself relies on channel_in_transit(). Individually they're OK but
combined they're doing the wrong thing.

The problem is that we refrain from filling buffers while to_forward
is even much larger than the buffer because of a semantic issue along
the call chain. This is particularly visible when offloading SSL on
moderately large files (1 MB), though it is also visible on clear text.
Twice the number of recv() calls are made compared to what is needed,
and the typical performance drops by 15-20% in SSL in 1.6 and later,
and no directly measurable drop in 1.5 except when using strace.

There's no need for all these intermediate functions, so let's get
rid of them and reimplement channel_recv_limit() from scratch in a
safer way.

This fix needs to be backported to 1.6 and 1.5 (at least). Note that in
1.5 the function is called buffer_recv_limit() and it may differ a bit.
2016-01-25 02:31:18 +01:00
Thiago Farina
b1af23ebea MINOR: fix the return type for dns_response_get_query_id() function
This function should return a 16-bit type as that is the type for
dns header id.
Also because it is doing an uint16 unpack big-endian operation.

Backport: can be backported to 1.6

Signed-off-by: Thiago Farina <tfarina@chromium.org>
Signed-off-by: Baptiste Assmann <bedis9@gmail.com>
2016-01-20 23:51:24 +01:00
Christopher Faulet
a94e5a548c MINOR: filters/http: Use a wrapper function instead of stream_int_retnclose
The function http_reply_and_close has been added in proto_http.c to wrap calls
to stream_int_retnclose. This functions will be modified when the filters will
be added.
2015-12-28 16:49:36 +01:00
Baptiste Assmann
e9544935e8 BUG/MINOR: http rule: http capture 'id' rule points to a non existing id
It is possible to create a http capture rule which points to a capture slot
id which does not exist.

Current patch prevent this when parsing configuration and prevent running
configuration which contains such rules.

This configuration is now invalid:

  frontend f
   bind :8080
   http-request capture req.hdr(User-Agent) id 0
   default_backend b

this one as well:

  frontend f
   bind :8080
   declare capture request len 32 # implicit id is 0 here
   http-request capture req.hdr(User-Agent) id 1
   default_backend b

It applies of course to both http-request and http-response rules.
2015-11-04 08:47:55 +01:00
Christopher Faulet
7969a33a01 MINOR: ssl: Add support for EC for the CA used to sign generated certificates
This is done by adding EVP_PKEY_EC type in supported types for the CA private
key when we get the message digest used to sign a generated X509 certificate.
So now, we support DSA, RSA and EC private keys.

And to be sure, when the type of the private key is not directly supported, we
get its default message digest using the function
'EVP_PKEY_get_default_digest_nid'.

We also use the key of the default certificate instead of generated it. So we
are sure to use the same key type instead of always using a RSA key.
2015-10-09 12:13:12 +02:00
Christopher Faulet
77fe80c0b4 MINOR: ssl: Release Servers SSL context when HAProxy is shut down
[wt: could be backported to 1.5 as well]
2015-10-09 10:33:00 +02:00
Christopher Faulet
d2cab92e75 BUG/MINOR: ssl: fix management of the cache where forged certificates are stored
First, the LRU cache must be initialized after the configuration parsing to
correctly set its size.
Next, the function 'ssl_sock_set_generated_cert' returns -1 when an error occurs
(0 if success). In that case, the caller is responsible to free the memory
allocated for the certificate.
Finally, when a SSL certificate is generated by HAProxy but cannot be inserted
in the cache, it must be freed when the SSL connection is closed. This happens
when 'tune.ssl.ssl-ctx-cache-size' is set to 0.
2015-10-09 10:20:53 +02:00
Thierry FOURNIER
ab95e656ea MINOR: http/tcp: fill the avalaible actions
This patch adds a function that generates the list of avalaible actions
for the error message.
2015-10-02 22:56:11 +02:00
Dragan Dosen
43885c728e BUG/MEDIUM: logs: segfault writing to log from Lua
Michael Ezzell reported a bug causing haproxy to segfault during startup
when trying to send syslog message from Lua. The function __send_log() can
be called with *p that is NULL and/or when the configuration is not fully
parsed, as is the case with Lua.

This patch fixes this problem by using individual vectors instead of the
pre-generated strings log_htp and log_htp_rfc5424.

Also, this patch fixes a problem causing haproxy to write the wrong pid in
the logs -- the log_htp(_rfc5424) strings were generated at the haproxy
start, but "pid" value would be changed after haproxy is started in
daemon/systemd mode.
2015-10-02 00:57:45 +02:00
Dragan Dosen
c8cfa7b4f3 MEDIUM: logs: have global.log_send_hostname not contain the trailing space
This patch unifies global.log_send_hostname addition in the log header
processing.
2015-09-28 18:27:45 +02:00
David Carlier
4686f792b4 MINOR: proto_http: Externalisation of previously internal functions
Needs to expose the HTTP headers 'iterator' and the client's cookie
value extraction functions.
2015-09-28 14:01:27 +02:00
Dragan Dosen
0b85ecee53 MEDIUM: logs: add a new RFC5424 log-format for the structured-data
This patch adds a new RFC5424-specific log-format for the structured-data
that is automatically send by __send_log() when the sender is in RFC5424
mode.

A new statement "log-format-sd" should be used in order to set log-format
for the structured-data part in RFC5424 formatted syslog messages.
Example:

    log-format-sd [exampleSDID@1234\ bytes=\"%B\"\ status=\"%ST\"]
2015-09-28 14:01:27 +02:00
Dragan Dosen
1322d09a6f MEDIUM: logs: add support for RFC5424 header format per logger
The function __send_log() iterates over senders and passes the header as
the first vector to sendmsg(), thus it can send a logger-specific header
in each message.

A new logger arguments "format rfc5424" should be used in order to enable
RFC5424 header format. For example:

    log 10.2.3.4:1234 len 2048 format rfc5424 local2 info
2015-09-28 14:01:27 +02:00
Dragan Dosen
68d2e3a742 MEDIUM: logs: remove the hostname, tag and pid part from the logheader
At the moment we have to call snprintf() for every log line just to
rebuild a constant. Thanks to sendmsg(), we send the message in 3 parts:
time-based header, proxy-specific hostname+log-tag+pid, session-specific
message.
2015-09-28 14:01:27 +02:00
Dragan Dosen
59cee973cd MEDIUM: log: use a separate buffer for the header and for the message
Make sendmsg() use two vectors, one for the message header that is updated
by update_log_hdr() and one for the message buffer.
2015-09-28 14:01:27 +02:00
Thierry FOURNIER
5a363e71b2 MINOR: stream/applet: add use-service action
This new target can be called from the frontend or the backend. It
is evaluated just before the backend choice and just before the server
choice. So, the input stream or HTTP request can be forwarded to a
server or to an internal service.
2015-09-28 01:03:48 +02:00
Thierry FOURNIER
fd50f0bcc8 MINOR: http: split initialization
The goal is to export the http txn initialisation functions for
using it in the Lua code.
2015-09-25 23:39:48 +02:00
Thierry FOURNIER
8d16de0ad0 MINOR: http: export function http_msg_analyzer()
Exports the function http_msg_analyzer().
2015-09-25 23:39:43 +02:00
Thierry FOURNIER
127169e7f5 MINOR: http: export the get_reason() function
This patch exports the get_reason() function.
2015-09-25 23:39:36 +02:00
Thierry FOURNIER
3c3317849f MINOR: http: export http_get_path() function
This patch simply exports the http_get_path() function from the proto_http.c file.
2015-09-25 23:39:27 +02:00
Thierry FOURNIER
27929fbfd7 MINOR: channel: rename function chn_sess to chn_strm
The name of the function chn_sess is no longer appropriate.
This patch renames it to chn_strm.
2015-09-25 23:27:33 +02:00
Willy Tarreau
958f0742a2 BUG/MEDIUM: stream-int: avoid double-call to applet->release
While the SI_ST_DIS state is set *after* doing the close on a connection,
it was set *before* calling release on an applet. Applets have no internal
flags contrary to connections, so they have no way to detect they were
already released. Because of this it happened that applets were closed
twice, once via si_applet_release() and once via si_release_endpoint() at
the end of a transaction. The CLI applet could perform a double free in
this case, though the situation to cause it is quite hard because it
requires that the applet is stuck on output in states that produce very
few data.

In order to solve this, we now assign the SI_ST_DIS state *after* calling
->release, and we refrain from doing so if the state is already assigned.
This makes applets work much more like connections and definitely avoids
this double release.

In the future it might be worth making applets have their own flags like
connections to carry their own state regardless of the stream interface's
state, especially when dealing with connection reuse.

No backport is needed since this issue was caused by the rearchitecture
in 1.6.
2015-09-25 21:16:03 +02:00
Willy Tarreau
aa977ba205 MINOR: stream-int: rename si_applet_done() to si_applet_wake_cb()
This function is a callback made only for calls from the applet handler.
Rename it to remove confusion. It's currently called from the Lua code
but that's not correct, we should call the notify and update functions
instead otherwise it will not enable the applet again.
2015-09-25 21:16:02 +02:00
Willy Tarreau
335520305c MEDIUM: stream-int: completely remove stream_int_update_embedded()
This one is not needed anymore as what it used to do is either
completely covered by the new stream_int_notify() function, or undesired
and inherited from the past as a side effect of introducing the
connections.

This update is theorically never called since it's assigned only when
nothing is connected to the stream interface. However a test has been
added to si_update() to stay safe if some foreign code decides to call
si_update() in unsafe situations.
2015-09-25 21:16:02 +02:00
Willy Tarreau
615f28bec1 MINOR: stream-int: implement the stream_int_notify() function
stream_int_notify() was taken from the common part between si_conn_wake_cb()
and si_applet_done(). It is designed to report activity to a stream from
outside its handler. It'll generally be used by lower layers to report I/O
completion but may also be used by remote streams if the buffer processing
is shared.
2015-09-25 21:16:02 +02:00
Willy Tarreau
cbc32601a6 MINOR: stream-int: export stream_int_update_*
Not only these functions were not static, but we'll also want to export
them.
2015-09-25 18:07:16 +02:00
Willy Tarreau
5d5b2fecac MEDIUM: stream-int: call stream_int_update() from si_update()
Now the call to stream_int_update() is moved to si_update(), which
is exclusively called from the stream, so that the socket layer may
be updated without updating the stream layer. This will later permit
to call it individually from other places (other tasks or applets for
example).
2015-09-25 18:07:16 +02:00