Commit Graph

1148 Commits

Author SHA1 Message Date
Willy Tarreau
a1b1ed53e7 MINOR: cli: make "show stat" support a proxy name
Till now it was needed to know the proxy's ID while we do have the
ability to look up a proxy by its name now.
2016-11-25 08:55:25 +01:00
Thierry FOURNIER / OZON.IO
8a1027aa45 MINOR: lua: Add tokenize function.
For tokenizing a string, standard Lua recommends to use regexes.
The followinf example splits words:

   for i in string.gmatch(example, "%S+") do
      print(i)
   end

This is a little bit overkill for simply split words. This patch
adds a tokenize function which quick and do not use regexes.
2016-11-24 21:35:34 +01:00
Simon Horman
1084a3631f MINOR: stats: correct documentation of process ID for typed output
The process ID appears at the end of the first column rather than
the line.
2016-11-21 22:54:00 +01:00
Christopher Faulet
985532d1d8 MINOR: spoe: Add "option set-on-error" statement
It defines the variable to set when an error occurred during an event
processing. It will only be set when an error occurred in the scope of the
transaction. As for all other variables define by the SPOE, it will be
prefixed. So, if your variable name is "error" and your prefix is "my_spoe_pfx",
the variable will be "txn.my_spoe_pfx.error".

When set, the variable is the boolean "true". Note that if "option
continue-on-error" is set, the variable is not automatically removed between
events processing.
2016-11-21 15:29:59 +01:00
Christopher Faulet
4802672274 MINOR: spoe: Add "maxconnrate" and "maxerrrate" statements
"maxconnrate" is the maximum number of connections per second. The SPOE will
stop to open new connections if the maximum is reached and will wait to acquire
an existing one.

"maxerrrate" is the maximum number of errors per second. The SPOE will stop its
processing if the maximum is reached.

These options replace hardcoded macros MAX_NEW_SPOE_APPLETS and
MAX_NEW_SPOE_APPLET_ERRS. We use it to limit SPOE activity, especially when
servers are down..
2016-11-21 15:29:59 +01:00
Christopher Faulet
ea62c2a345 MINOR: spoe: Add 'option continue-on-error' statement in spoe-agent section
By default, for a specific stream, when an abnormal/unexpected error occurs, the
SPOE is disabled for all the transaction. So if you have several events
configured, such error on an event will disabled all followings. For TCP
streams, this will disable the SPOE for the whole session. For HTTP streams,
this will disable it for the transaction (request and response).

To bypass this behaviour, you can set 'continue-on-error' option in 'spoe-agent'
section. With this option, only the current event will be ignored.
2016-11-21 15:29:59 +01:00
Christopher Faulet
03a3449e1a MINOR: spoe: Remove useless 'timeout ack' option
To limit the time to process an event, you should set 'timeout processing'
option. So 'timeout ack' option is redundant and useless.
2016-11-21 15:29:59 +01:00
Christopher Faulet
f7a3092512 MINOR: spoe: Add 'timeout processing' option to limit time to process an event
It is a way to set the maximum time to wait for a stream to process an event,
i.e to acquire a stream to talk with an agent, to encode all messages, to send
the NOTIFY frame, to receive the corrsponding acknowledgement and to process all
actions. It is applied on the stream that handle the client and the server
sessions.
2016-11-21 15:29:59 +01:00
Christopher Faulet
a00d817aba MINOR: filters: Add check_timeouts callback to handle timers expiration on streams
A filter can now be notified when a stream is woken up because of an expired
timer.

The documentation and the TRACE filter have been updated.
2016-11-21 15:29:58 +01:00
Thierry FOURNIER / OZON.IO
a44fdd95f9 MEDIUM: lua: Add cli handler for Lua
Now, HAProxy allows to register some keys in the "cli". This patch allows
to handle these keys with Lua code.
2016-11-18 14:32:03 +01:00
Bertrand Jacquin
d4d0a23ca4 DOC: ssl: Use correct wording for ca-sign-pass
Doc references ca-sign-passphrase but the source code is referring
ca-sign-pass. Align doc to reality.
2016-11-14 18:15:20 +01:00
Thierry FOURNIER / OZON.IO
62fec75183 MINOR: lua: add ip addresses and network manipulation function
Add two functions core.parse_addr() and core.match_addr() where are used
for matching networks.
2016-11-12 10:42:30 +01:00
Thierry FOURNIER / OZON.IO
65192f35d2 MINOR: lua: add function which return true if the channel is full.
Add function which return true if the channel is full. It is
useful for triggering some process when the buffer is full.
2016-11-12 10:42:25 +01:00
Willy Tarreau
d5d890be21 [RELEASE] Released version 1.7-dev6
Released version 1.7-dev6 with the following main changes :
    - DOC: fix the entry for hash-balance-factor config option
    - DOC: Fix typo in description of `-st` parameter in man page
    - CLEANUP: cfgparse: Very minor spelling correction
    - MINOR: examples: Update haproxy.spec URLs to haproxy.org
    - BUG/MEDIUM: peers: on shutdown, wake up the appctx, not the stream
    - BUG/MEDIUM: peers: fix use after free in peer_session_create()
    - MINOR: peers: make peer_session_forceshutdown() use the appctx and not the stream
    - MINOR: peers: remove the pointer to the stream
    - BUG/MEDIUM: systemd-wrapper: return correct exit codes
    - DOC: stats: provide state details for show servers state
    - MEDIUM: tools: make str2ip2() preserve existing ports
    - CLEANUP: tools: make ipcpy() preserve the original port
    - OPTIM: http: move all http character classs tables into a single one
    - OPTIM: http: improve parsing performance of long header lines
    - OPTIM: http: improve parsing performance of long URIs
    - OPTIM: http: optimize lookup of comma and quote in header values
    - BUG/MEDIUM: srv-state: properly restore the DRAIN state
    - BUG/MINOR: srv-state: allow to have both CMAINT and FDRAIN flags
    - MINOR: server: do not emit warnings/logs/alerts on server state changes at boot
    - BUG/MEDIUM: servers: properly propagate the maintenance states during startup
    - MEDIUM: wurfl: add Scientiamobile WURFL device detection module
    - DOC: move the device detection modules documentation to their own files
    - CLEANUP: wurfl: reduce exposure in the rest of the code
    - MEDIUM: ssl: Add support for OpenSSL 1.1.0
    - MINOR: stream: make option contstats usable again
    - MEDIUM: tools: make str2sa_range() return the FQDN even when not resolving
    - MINOR: init: move apply_server_state in haproxy.c before MODE_CHECK
    - MAJOR: server: postpone address resolution
    - MINOR: new srv_admin flag: SRV_ADMF_RMAINT
    - MINOR: server: indicate in the logs when RMAINT is cleared
    - MINOR: stats: indicate it when a server is down due to resolution
    - MINOR: server: make srv_set_admin_state() capable of telling why this happens
    - MINOR: dns: implement extra 'hold' timers.
    - MAJOR: dns: runtime resolution can change server admin state
    - MEDIUM: cli: leave the RMAINT state when setting an IP address on the CLI
    - MEDIUM: server: add a new init-addr server line setting
    - MEDIUM: server: make use of init-addr
    - MINOR: server: implement init-addr none
    - MEDIUM: server: make libc resolution failure non-fatal
    - MINOR: server: add support for explicit numeric address in init-addr
    - DOC: add some documentation for the "init-addr" server keyword
    - MINOR: init: add -dr to ignore server address resolution failures
    - MEDIUM: server: do not restrict anymore usage of IP address from the state file
    - BUG: vars: Fix 'set-var' converter because of a typo
    - CLEANUP: remove last references to 'ruleset' section
    - MEDIUM: filters: Add attch/detach and stream_set_backend callbacks
    - MINOR: filters: Update filters documentation accordingly to recent changes
    - MINOR: filters: Call stream_set_backend callbacks before updating backend stats
    - MINOR: filters: Remove backend filters attached to a stream only for HTTP streams
    - MINOR: flt_trace: Add hexdump option to dump forwarded data
    - MINOR: cfgparse: Add functions to backup and restore registered sections
    - MINOR: cfgparse: Parse scope lines and save the last one parsed
    - REORG: sample: move code to release a sample expression in sample.c
    - MINOR: vars: Allow '.' in variable names
    - MINOR: vars: Add vars_set_by_name_ifexist function
    - MEDIUM: vars: Add a per-process scope for variables
    - MINOR: vars: Add 'unset-var' action/converter
    - MAJOR: spoe: Add an experimental Stream Processing Offload Engine
    - MINOR: spoe: add random ip-reputation service as SPOA example
    - MINOR: spoe/checks: Add support for SPOP health checks
    - DOC: update ROADMAP file
2016-11-09 23:18:17 +01:00
Christopher Faulet
ba7bc164f7 MINOR: spoe/checks: Add support for SPOP health checks
A new "option spop-check" statement has been added to enable server health
checks based on SPOP HELLO handshake. SPOP is the protocol used by SPOE filters
to talk to servers.
2016-11-09 22:57:02 +01:00
Christopher Faulet
f7e4e7e096 MAJOR: spoe: Add an experimental Stream Processing Offload Engine
SPOE makes possible the communication with external components to retrieve some
info using an in-house binary protocol, the Stream Processing Offload Protocol
(SPOP). In the long term, its aim is to allow any kind of offloading on the
streams. This first version, besides being experimental, won't do lot of
things. The most important today is to validate the protocol design and lay the
foundations of what will, one day, be a full offload engine for the stream
processing.

So, for now, the SPOE can offload the stream processing before "tcp-request
content", "tcp-response content", "http-request" and "http-response" rules. And
it only supports variables creation/suppression. But, in spite of these limited
features, we can easily imagine to implement a SSO solution, an ip reputation
service or an ip geolocation service.

Internally, the SPOE is implemented as a filter. So, to use it, you must use
following line in a proxy proxy section:

  frontend my-front
      ...
      filter spoe [engine <name>] config <file>
      ...

It uses its own configuration file to keep the HAProxy configuration clean. It
is also a easy way to disable it by commenting out the filter line.

See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-11-09 22:57:01 +01:00
Christopher Faulet
85d79c94a9 MINOR: vars: Add 'unset-var' action/converter
It does the opposite of 'set-var' action/converter. It is really useful for
per-process variables. But, it can be used for any scope.

The lua function 'unset_var' has also been added.
2016-11-09 22:57:01 +01:00
Christopher Faulet
ff2613ed7a MEDIUM: vars: Add a per-process scope for variables
Now it is possible to use variables attached to a process. The scope name is
'proc'. These variables are released only when HAProxy is stopped.

'tune.vars.proc-max-size' directive has been added to confiure the maximum
amount of memory used by "proc" variables. And because memory accounting is
hierachical for variables, memory for "proc" vars includes memory for "sess"
vars.
2016-11-09 22:57:00 +01:00
Christopher Faulet
b71557a98b MINOR: vars: Allow '.' in variable names
This is required to have implicit prefix or scope. SPOE filter will use it to
keep variables set by an agent in its own namespace.
2016-11-09 22:57:00 +01:00
Christopher Faulet
9adb0a5458 MINOR: filters: Update filters documentation accordingly to recent changes 2016-11-09 22:50:55 +01:00
Willy Tarreau
e5a60688a4 MEDIUM: server: do not restrict anymore usage of IP address from the state file
Now that it is possible to decide whether we prefer to use libc or the
state file to resolve the server's IP address and it is possible to change
a server's IP address at run time on the CLI, let's not restrict the reuse
of the address from the state file anymore to the DNS only.

The impact is that by default the state file will be considered first
(which matches its purpose) and only then the libc. This way any address
change performed at run time over the CLI will be preserved regardless
of DNS usage or not.
2016-11-09 15:33:52 +01:00
Willy Tarreau
3eed10e54b MINOR: init: add -dr to ignore server address resolution failures
It is very common when validating a configuration out of production not to
have access to the same resolvers and to fail on server address resolution,
making it difficult to test a configuration. This option simply appends the
"none" method to the list of address resolution methods for all servers,
ensuring that even if the libc fails to resolve an address, the startup
sequence is not interrupted.
2016-11-09 15:33:52 +01:00
Willy Tarreau
6a031d1c90 DOC: add some documentation for the "init-addr" server keyword
All 4 new values were documented at once.
2016-11-09 15:33:52 +01:00
Baptiste Assmann
987e16d6f4 MINOR: dns: implement extra 'hold' timers.
This adds new "hold" timers : nx, refused, timeout, other. This timers
will be used to tell HAProxy to keep an erroneous response as valid for
the corresponding period. For now they're only configured, not enforced.
2016-11-09 15:30:47 +01:00
Willy Tarreau
b96dd28477 MINOR: stats: indicate it when a server is down due to resolution
The server's state is now "MAINT (resolution)" just like we also have
"MAINT (via x/y)" when servers are tracked. The HTML stats page reports
"resolution" in the checks field similarly to what is done for the "via"
entry.
2016-11-09 15:30:47 +01:00
Baptiste Assmann
89aa7f30da MINOR: new srv_admin flag: SRV_ADMF_RMAINT
This flag has to be set when an IP address resolution fails (either
using libc at start up or using HAProxy's runtime resolver). This will
automatically trigger the administrative status "MAINT", through the
global mask SRV_ADMF_MAINT.
2016-11-09 15:23:16 +01:00
Willy Tarreau
def0d22cc5 MINOR: stream: make option contstats usable again
Quite a lot of people have been complaining about option contstats not
working correctly anymore since about 1.4. The reason was that one reason
for the significant performance boost between 1.3 and 1.4 was the ability
to forward data between a server and a client without waking up the stream
manager. And we couldn't afford to force sessions to constantly wake it
up given that most of the people interested in contstats are also those
interested in high performance transmission.

An idea was experimented with in the past, consisting in limiting the
amount of transmissible data before waking it up, but it was not usable
on slow connections (eg: FTP over modem lines, RDP, SSH) as stats would
be updated too rarely if at all, so that idea was dropped.

During a discussion today another idea came up : ensure that stats are
updated once in a while, since it's the only thing that matters. It
happens that we have the request channel's analyse_exp timeout that is
used to wake the stream up after a configured delay, and that by
definition this timeout is not used when there's no more analyser
(otherwise the stream would wake up and the stats would be updated).

Thus here the idea is to reuse this timeout when there's no analyser
and set it to now+5 seconds so that a stream wakes up at least once
every 5 seconds to update its stats. It should be short enough to
provide smooth traffic graphs and to allow to debug outputs of "show
sess" more easily without inflicting too much load even for very large
number of concurrent connections.

This patch is simple enough and safe enough to be backportable to 1.6
if there is some demand.
2016-11-08 22:03:00 +01:00
Willy Tarreau
29b25317fc DOC: move the device detection modules documentation to their own files
Both DeviceAtlas and 51Degrees used to put their building instructions
in the README, representing more than 1/3 of it. It's better to let the
README focus on generic stuff and building procedure and move the DD
docs to their own files.
2016-11-08 15:06:21 +01:00
scientiamobile
d0027ed5b1 MEDIUM: wurfl: add Scientiamobile WURFL device detection module
WURFL is a high-performance and low-memory footprint mobile device
detection software component that can quickly and accurately detect
over 500 capabilities of visiting devices. It can differentiate between
portable mobile devices, desktop devices, SmartTVs and any other types
of devices on which a web browser can be installed.

In order to add WURFL device detection support, you would need to
download Scientiamobile InFuze C API and install it on your system.
Refer to www.scientiamobile.com to obtain a valid InFuze license.

Any useful information on how to configure HAProxy working with WURFL
may be found in:

  doc/WURFL-device-detection.txt
  doc/configuration.txt
  examples/wurfl-example.cfg

Please find more information about WURFL device detection API detection
at https://docs.scientiamobile.com/documentation/infuze/infuze-c-api-user-guide
2016-11-08 14:21:43 +01:00
Cyril Bont
5b2ce8a10f DOC: stats: provide state details for show servers state
Add the state values to the documentation instead of adding a reference to the
source code.
2016-11-05 13:55:23 +01:00
Jorrit Schippers
1458fdbe97 DOC: Fix typo in description of -st parameter in man page
extra "wait".
2016-10-26 08:36:37 +02:00
Andrew Rodland
17be45ebfa DOC: fix the entry for hash-balance-factor config option
It was accidentally added as "balance-factor". Fix it and
re-alphabetize.

Signed-off-by: Andrew Rodland <andrewr@vimeo.com>
2016-10-25 23:30:01 +02:00
Willy Tarreau
608efa173c [RELEASE] Released version 1.7-dev5
Released version 1.7-dev5 with the following main changes :
    - MINOR: cfgparse: few memory leaks fixes.
    - MEDIUM: log: Decompose %Tq in %Th %Ti %TR
    - CLEANUP: logs: remove unused log format field definitions
    - BUILD/MAJOR:updated 51d Trie implementation to incorperate latest update to 51Degrees.c
    - BUG/MAJOR: stream: properly mark the server address as unset on connect retry
    - CLEANUP: proto_http: Removing useless variable assignation
    - CLEANUP: dumpstats: Removing useless variables allocation
    - CLEANUP: dns: Removing usless variable & assignation
    - BUG/MINOR: payload: fix SSLv2 version parser
    - MINOR: cli: allow the semi-colon to be escaped on the CLI
    - MINOR: cli: change a server health check port through the stats socket
    - BUG/MINOR: Fix OSX compilation errors
    - MAJOR: check: find out which port to use for health check at run time
    - MINOR: server: introduction of 3 new server flags
    - MINOR: new update_server_addr_port() function to change both server's ADDR and service PORT
    - MINOR: cli: ability to change a server's port
    - CLEANUP/MINOR dns: comment do not follow up code update
    - MINOR: chunk: new strncat function
    - MINOR: dns: wrong DNS_MAX_UDP_MESSAGE value
    - MINOR: dns: new MAX values
    - MINOR: dns: new macro to compute DNS header size
    - MINOR: dns: new DNS structures to store received packets
    - MEDIUM: dns: new DNS response parser
    - MINOR: dns: query type change when last record is a CNAME
    - MINOR: dns: proper domain name validation when receiving DNS response
    - MINOR: dns: comments in types/dns.h about structures endianness
    - BUG/MINOR: displayed PCRE version is running release
    - MINOR: show Built with PCRE version
    - MINOR: show Running on zlib version
    - MEDIUM: make SO_REUSEPORT configurable
    - MINOR: enable IP_BIND_ADDRESS_NO_PORT on backend connections
    - BUG/MEDIUM: http/compression: Fix how chunked data are copied during the HTTP body parsing
    - BUG/MINOR: stats: report the correct conn_time in backend's html output
    - BUG/MEDIUM: dns: don't randomly crash on out-of-memory
    - MINOR: Add fe_req_rate sample fetch
    - MEDIUM: peers: Fix a peer stick-tables synchronization issue.
    - MEDIUM: cli: register CLI keywords with cli_register_kw()
    - BUILD: Make use of accept4() on OpenBSD.
    - MINOR: tcp: make set-src/set-src-port and set-dst/set-dst-port commutative
    - DOC: fix missed entry for "set-{src,dst}{,-port}"
    - BUG/MINOR: vars: use sess and not s->sess in action_store()
    - BUG/MINOR: vars: make smp_fetch_var() more robust against misuses
    - BUG/MINOR: vars: smp_fetch_var() doesn't depend on HTTP but on the session
    - MINOR: stats: output dcon
    - CLEANUP: tcp rules: mention everywhere that tcp-conn rules are L4
    - MINOR: counters: add new fields for denied_sess
    - MEDIUM: tcp: add registration and processing of TCP L5 rules
    - MINOR: stats: emit dses
    - DOC: document tcp-request session
    - MINOR: ssl: add debug traces
    - BUILD/CLEANUP: ssl: Check BIO_reset() return code
    - BUG/MINOR: ssl: Check malloc return code
    - BUG/MINOR: ssl: prevent multiple entries for the same certificate
    - BUG/MINOR: systemd: make the wrapper return a non-null status code on error
    - BUG/MINOR: systemd: always restore signals before execve()
    - BUG/MINOR: systemd: check return value of calloc()
    - MINOR: systemd: report it when execve() fails
    - BUG/MEDIUM: systemd: let the wrapper know that haproxy has completed or failed
    - MINOR: proxy: add 'served' field to proxy, equal to total of all servers'
    - MINOR: backend: add hash-balance-factor option for hash-type consistent
    - MINOR: server: compute a "cumulative weight" to allow chash balancing to hit its target
    - MEDIUM: server: Implement bounded-load hash algorithm
    - SCRIPTS: make git-show-backports also dump a "git show" command
    - MINOR: build: Allow linking to device-atlas library file
    - MINOR: stats: Escape equals sign on socket dump
2016-10-25 22:22:00 +02:00
Andrew Rodland
b1f48e3161 MINOR: backend: add hash-balance-factor option for hash-type consistent
0 will mean no balancing occurs; otherwise it represents the ratio
between the highest-loaded server and the average load, times 100 (i.e.
a value of 150 means a 1.5x ratio), assuming equal weights.

Signed-off-by: Andrew Rodland <andrewr@vimeo.com>
2016-10-25 20:21:32 +02:00
Willy Tarreau
4f614290db DOC: document tcp-request session
This is mostly a copy-paste-edit of the section about "tcp-request connection".
Maybe all these sections should be merged.
2016-10-21 18:19:48 +02:00
Willy Tarreau
a5bc36b31c MINOR: stats: emit dses
This is the number of denied sessions, blocked by "tcp-request session reject".
2016-10-21 18:19:48 +02:00
Willy Tarreau
8a90b8ea19 MINOR: stats: output dcon
These are denied conns. Strangely this wasn't emitted while it used to be
available for a while. It corresponds to the number of connections blocked
by "tcp-request connection reject".
2016-10-21 18:17:56 +02:00
Willy Tarreau
0c63053b30 DOC: fix missed entry for "set-{src,dst}{,-port}"
There was the same explanation for tcp-request connection that I missed
in previous patch.
2016-10-21 17:52:58 +02:00
Willy Tarreau
00005ce5a1 MINOR: tcp: make set-src/set-src-port and set-dst/set-dst-port commutative
When the tcp/http actions above were introduced in 1.7-dev4, we used to
proceed like this :

  - set-src/set-dst would force the port to zero
  - set-src-port/set-dst-port would not do anything if the address family is
    neither AF_INET nor AF_INET6.

It was a stupid idea of mine to request this behaviour because it ensures
that these functions cannot be used in a wide number of situations. Because
of the first rule, it is necessary to save the source port one way or
another if only the address has to be changed (so you have to use an
variable). Due to the second rule, there's no way to set the source port
on a unix socket without first overwriting the address. And sometimes it's
really not convenient, especially when there's no way to guarantee that all
fields will properly be set.

In order to fix all this, this small change does the following :
  - set-src/set-dst always preserve the original port even if the address
    family changes. If the previous address family didn't have a port (eg:
    AF_UNIX), then the port is set to zero ;

  - set-src-port/set-dst-port always preserve the original address. If the
    address doesn't have a port, then the family is forced to IPv4 and the
    address to "0.0.0.0".

Thanks to this it now becomes possible to perform one action, the other or
both in any order.
2016-10-21 15:15:20 +02:00
Nenad Merdanovic
ad9a7e9770 MINOR: Add fe_req_rate sample fetch
The fe_req_rate is similar to fe_sess_rate, but fetches the number
of HTTP requests per second instead of connections/sessions per second.

Signed-off-by: Nenad Merdanovic <nmerdan@anine.io>
2016-10-03 16:08:09 +02:00
Lukas Tribus
7d56c6d347 MINOR: enable IP_BIND_ADDRESS_NO_PORT on backend connections
Enable IP_BIND_ADDRESS_NO_PORT on backend connections when the source
address is specified without port or port ranges. This is supported
since Linux 4.2/libc 2.23.

If the kernel supports it but the libc doesn't, we can define it at
build time:
make [...] DEFINE=-DIP_BIND_ADDRESS_NO_PORT=24

For more informations about this feature, see Linux commit 90c337da
2016-09-13 15:22:54 +02:00
Lukas Tribus
a0bcbdcb04 MEDIUM: make SO_REUSEPORT configurable
With Linux officially introducing SO_REUSEPORT support in 3.9 and
its mainstream adoption we have seen more people running into strange
SO_REUSEPORT related issues (a process management issue turning into
hard to diagnose problems because the kernel load-balances between the
new and an obsolete haproxy instance).

Also some people simply want the guarantee that the bind fails when
the old process is still bound.

This change makes SO_REUSEPORT configurable, introducing the command
line argument "-dR" and the noreuseport configuration directive.

A backport to 1.6 should be considered.
2016-09-13 07:56:03 +02:00
Baptiste Assmann
3749ebf6fc MINOR: cli: ability to change a server's port
Enrichment of the 'set server <b>/<s> addr' cli directive to allow changing
now a server's port.
The new syntax looks like:
  set server <b>/<s> addr [port <port>]
2016-09-11 08:13:31 +02:00
Baptiste Assmann
5094656a67 MINOR: cli: change a server health check port through the stats socket
Introduction of a new CLI command "set server <srv> check-port <port>' to
allow admins to change a server's health check port at run time.

This changes the equivalent of the configuration server parameter
called 'port'.
2016-09-06 07:39:16 +02:00
Chad Lavoie
e3f5031b51 MINOR: cli: allow the semi-colon to be escaped on the CLI
Today I was working on an auto-update script for some ACLs, and found
that I couldn't load ACL entries with a semi-colon in them no matter
how I tried to escape it.

As such, I wrote this patch (this one is for 1.7dev, but it applies to
1.5 the same with just line numbers changed), which seems to allow me
to execute a command such as "add acl /etc/foo.lst foo\;bar" over the
socket. It's worth noting that stats_sock_parse_request() already uses
the backslash to escape spaces in words so it makes sense to use it as
well to escape the semi-colon.
2016-08-30 22:32:23 +02:00
Thierry FOURNIER / OZON.IO
4cac359a39 MEDIUM: log: Decompose %Tq in %Th %Ti %TR
Tq is the time between the instant the connection is accepted and a
complete valid request is received. This time includes the handshake
(SSL / Proxy-Protocol), the idle when the browser does preconnect and
the request reception.

This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR
which returns respectively the handshake time, the idle time and the
duration of valid request reception. It also adds %Ta which reports
the request's active time, which is the total time without %Th nor %Ti.
It replaces %Tt as the total time, reporting accurate measurements for
HTTP persistent connections.

%Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only
avalaible for HTTP connections.

In addition to this, we have new timestamps %tr, %trg and %trl, which
log the date of start of receipt of the request, respectively in the
default format, in GMT time and in local time (by analogy with %t, %T
and %Tl). All of them are obviously only available for HTTP. These values
are more relevant as they more accurately represent the request date
without being skewed by a browser's preconnect nor a keep-alive idle
time.

The HTTP log format and the CLF log format have been modified to
use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This
way the default log formats now produce the expected output for users
who don't want to manually fiddle with the log-format directive.

Example with the following log-format :

   log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

The request was sent by hand using "openssl s_client -connect" :

   Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

=> 6 ms of SSL handshake, 2375 waiting before sending the first char (in
fact the time to type the first line), 261 ms before the end of the request,
no time spent in queue, 1 ms spend connecting to the server, immediate
response, total active time for this request = 262ms. Total time from accept
to close : 2643 ms.

The timing now decomposes like this :

                 first request               2nd request
      |<-------------------------------->|<-------------- ...
      t         tr                       t    tr ...
   ---|----|----|----|----|----|----|----|----|--
      : Th   Ti   TR   Tw   Tc   Tr   Td : Ti   ...
      :<---- Tq ---->:                   :
      :<-------------- Tt -------------->:
                :<--------- Ta --------->:
2016-08-23 15:18:08 +02:00
Willy Tarreau
41d5e3a610 [RELEASE] Released version 1.7-dev4
Released version 1.7-dev4 with the following main changes :
    - MINOR: add list_append_word function
    - MEDIUM: init: use list_append_word in haproxy.c
    - MEDIUM: init: allow directory as argument of -f
    - CLEANUP: config: detect double registration of a config section
    - MINOR: log: add the %Td log-format specifier
    - MEDIUM: filters: Move HTTP headers filtering in its own callback
    - MINOR: filters: Simplify calls to analyzers using 2 new macros
    - MEDIUM: filters: Add pre and post analyzer callbacks
    - DOC: filters: Update the filters documentation accordingly to recent changes
    - BUG/MEDIUM: init: don't use environment locale
    - SCRIPTS: teach git-show-backports how to report upstream commits
    - SCRIPTS: make git-show-backports capable of limiting its history
    - BUG/MAJOR: fix listening IP address storage for frontends
    - BUG/MINOR: fix listening IP address storage for frontends (cont)
    - DOC: Fix typo so fetch is properly parsed by Cyril's converter
    - BUG/MAJOR: http: fix breakage of "reqdeny" causing random crashes
    - BUG/MEDIUM: stick-tables: fix breakage in table converters
    - MINOR: stick-table: change all stick-table converters' inputs to SMP_T_ANY
    - BUG/MEDIUM: dns: unbreak DNS resolver after header fix
    - BUILD: fix build on Solaris 11
    - BUG/MEDIUM: config: fix multiple declaration of section parsers
    - BUG/MEDIUM: stats: show servers state may show an servers from another backend
    - BUG/MEDIUM: fix risk of segfault with "show tls-keys"
    - MEDIUM: dumpstats: 'show tls-keys' is now able to show secrets
    - DOC: update doc about tls-tickets-keys dump
    - MEDIUM: tcp: add 'set-src' to 'tcp-request connection'
    - MINOR: set the CO_FL_ADDR_FROM_SET flags with 'set-src'
    - MEDIUM: tcp/http: add 'set-src-port' action
    - MEDIUM: tcp/http: new set-dst/set-dst-port actions
    - BUG/MEDIUM: sticktables: segfault in some configuration error cases
    - BUILD/MEDIUM: rebuild everything when an include file is changed
    - BUILD/MEDIUM: force a full rebuild if some build options change
    - BUG/MEDIUM: lua: converters doesn't work
    - BUG/MINOR: http: add-header: header name copied twice
    - BUG/MEDIUM: http: add-header: buffer overwritten
    - BUG/MINOR: ssl: fix potential memory leak in ssl_sock_load_dh_params()
    - MINOR: stream: export the function 'smp_create_src_stkctr'
    - BUG/MEDIUM: dumpstats: undefined behavior in stats_tlskeys_list()
    - MEDIUM: dumpstats: make stats_tlskeys_list() yield-aware during tls-keys dump
    - BUG/MINOR: http: url32+src should use the big endian version of url32
    - BUG/MINOR: http: url32+src should check cli_conn before using it
    - DOC: http: add documentation for url32 and url32+src
    - BUG/MINOR: fix http-response set-log-level parsing error
    - MINOR: systemd: Use variable for config and pidfile paths
    - MINOR: systemd: Perform sanity check on config before reload
    - MEDIUM: ssl: support SNI filters with multicerts
    - MINOR: ssl: crt-list parsing factor
    - BUILD: ssl: fix typo causing a build failure in the multicert patch
    - MINOR: listener: add the "accept-netscaler-cip" option to the "bind" keyword
    - MINOR: tcp: add "tcp-request connection expect-netscaler-cip layer4"
    - BUG/MINOR: init: always ensure that global.rlimit_nofile matches actual limits
    - BUG/MINOR: init: ensure that FD limit is raised to the max allowed
    - BUG/MEDIUM: external-checks: close all FDs right after the fork()
    - BUG/MAJOR: external-checks: use asynchronous signal delivery
    - BUG/MINOR: external-checks: do not unblock undesired signals
    - CLEANUP: external-check: don't block/unblock SIGCHLD when manipulating the list
    - BUG/MEDIUM: filters: Fix data filtering when data are modified
    - BUG/MINOR: filters: Fix HTTP parsing when a filter loops on data forwarding
    - BUG/MINOR: srv-state: fix incorrect output of state file
    - BUG/MINOR: ssl: close ssl key file on error
    - BUG/MINOR: http: fix misleading error message for response captures
    - BUG/BUILD: don't automatically run "make" on "make install"
    - DOC: add missing doc for http-request deny [deny_status <status>]
    - CLEANUP: dumpstats: u64 field is an unsigned type.
    - BUG/MEDIUM: http: unbreak uri/header/url_param hashing
    - BUG/MINOR: Rework slightly commit 9962f8fc to clean code and avoid mistakes
    - MINOR: new function my_realloc2 = realloc + free upon failure
    - CLEANUP: fixed some usages of realloc leading to memory leak
    - Revert "BUG/MINOR: ssl: fix potential memory leak in ssl_sock_load_dh_params()"
    - CLEANUP: connection: using internal struct to hold source and dest port.
    - DOC: spelling fixes
    - BUG/MINOR: ssl: fix potential memory leak in ssl_sock_load_dh_params()
    - BUG/MEDIUM: dns: fix alignment issues in the DNS response parser
    - BUG/MINOR: Fix endiness issue in DNS header creation code
    - BUG/MEDIUM: lua: the function txn_done() from sample fetches can crash
    - BUG/MEDIUM: lua: the function txn_done() from action wrapper can crash
    - MEDIUM: http: implement http-response track-sc* directive
    - BUG/MINOR: peers: Fix peers data decoding issue
    - BUG/MINOR: peers: don't count track-sc multiple times on errors
    - MINOR: standard: add function "escape_string"
    - BUG/MEDIUM: log: use function "escape_string" instead of "escape_chunk"
    - MINOR: tcp: Return TCP statistics like RTT and RTT variance
    - DOC: lua: remove old functions
    - BUG/MEDIUM: lua: somme HTTP manipulation functions are called without valid requests
    - DOC: fix json converter example and error message
    - BUG/MEDIUM: stream-int: completely detach connection on connect error
    - DOC: minor typo fixes to improve HTML parsing by haproxy-dconv
    - BUILD: make proto_tcp.c compatible with musl library
    - BUG/MAJOR: compression: initialize avail_in/next_in even during flush
    - BUG/MEDIUM: samples: make smp_dup() always duplicate the sample
    - MINOR: sample: implement smp_is_safe() and smp_make_safe()
    - MINOR: sample: provide smp_is_rw() and smp_make_rw()
    - BUG/MAJOR: server: the "sni" directive could randomly cause trouble
    - BUG/MEDIUM: stick-tables: do not fail on string keys with no allocated size
    - BUG/MEDIUM: stick-table: properly convert binary samples to keys
    - MINOR: sample: use smp_make_rw() in upper/lower converters
    - MINOR: tcp: add dst_is_local and src_is_local
    - BUG/MINOR: peers: some updates are pushed twice after a resync.
    - BUILD: protocol: fix some build errors on OpenBSD
    - BUILD: log: iovec requires to include sys/uio.h on OpenBSD
    - BUILD: tcp: do not include netinet/ip.h for IP_TTL
    - BUILD: connection: fix build breakage on openbsd due to missing in_systm.h
    - BUILD: checks: remove the last strcat and eliminate a warning on OpenBSD
    - BUILD: tcp: define SOL_TCP when only IPPROTO_TCP exists
    - BUILD: compression: remove a warning when no compression lib is used
    - BUILD: poll: remove unused hap_fd_isset() which causes a warning with clang
    - MINOR: tcp: add further tcp info fetchers
    - BUG/MINOR: peers: empty chunks after a resync.
    - BUG/MAJOR: stick-counters: possible crash when using sc_trackers with wrong table
    - MINOR: standard.c: ipcmp() function to compare 2 IP addresses stored in 2 struct sockaddr_storage
    - MINOR: standard.c: ipcpy() function to copy an IP address from a struct sockaddr_storage into an other one
    - MAJOR: listen section: don't use first bind port anymore when no server ports are provided
2016-08-14 12:25:21 +02:00
Joe Williams
30fcd39f35 MINOR: tcp: add further tcp info fetchers
Adding on to Thierry's work (http://git.haproxy.org/?p=haproxy.git;h=6310bef5)
I have added a few more fetchers for counters based on the tcp_info struct
maintained by the kernel :

  fc_unacked, fc_sacked, fc_retrans, fc_fackets, fc_lost,
  fc_reordering

Two fields were not added because they're version-dependant :
  fc_rcv_rtt, fc_total_retrans

The fields name depend on the operating system. FreeBSD and NetBSD prefix
all the field names with "__" so we have to rely on a few #ifdef for
portability.
2016-08-10 23:02:46 +02:00
Willy Tarreau
16e015635c MINOR: tcp: add dst_is_local and src_is_local
It is sometimes needed in application server environments to easily tell
if a source is local to the machine or a remote one, without necessarily
knowing all the local addresses (dhcp, vrrp, etc). Similarly in transparent
proxy configurations it is sometimes desired to tell the difference between
local and remote destination addresses.

This patch adds two new sample fetch functions for this :

dst_is_local : boolean
  Returns true if the destination address of the incoming connection is local
  to the system, or false if the address doesn't exist on the system, meaning
  that it was intercepted in transparent mode. It can be useful to apply
  certain rules by default to forwarded traffic and other rules to the traffic
  targetting the real address of the machine. For example the stats page could
  be delivered only on this address, or SSH access could be locally redirected.
  Please note that the check involves a few system calls, so it's better to do
  it only once per connection.

src_is_local : boolean
  Returns true if the source address of the incoming connection is local to the
  system, or false if the address doesn't exist on the system, meaning that it
  comes from a remote machine. Note that UNIX addresses are considered local.
  It can be useful to apply certain access restrictions based on where the
  client comes from (eg: require auth or https for remote machines). Please
  note that the check involves a few system calls, so it's better to do it only
  once per connection.
2016-08-09 16:50:08 +02:00
Olivier Doucet
aa1ea8a9ee DOC: minor typo fixes to improve HTML parsing by haproxy-dconv
This must be backported to 1.6 and 1.5
2016-08-07 09:29:20 +02:00
Herve COMMOWICK
8dfe863fbf DOC: fix json converter example and error message 2016-08-07 08:08:18 +02:00
Thierry FOURNIER / OZON.IO
53e381c3a0 DOC: lua: remove old functions
The functions "req_replace_value()" and "res_replace_value()"
doesn't exists in the 1.6 version. There inherited from the 1.6dev.

This patch must be backported in 1.6 version
2016-08-03 00:05:59 +02:00
Thierry Fournier / OZON.IO
6310bef511 MINOR: tcp: Return TCP statistics like RTT and RTT variance
This patch adds 4 new sample fetches which returns the RTT of the
established connexion and the RTT variance. The established connection
can be between the client and HAProxy, and between HAProxy and the
server. This is very useful for statistics. A great use case is the
estimation of the TCP connection time of the client. Note that the
RTT of the server side is not so interesting because we already have
the connect() time.
2016-07-27 13:47:09 +02:00
Ruoshan Huang
e4edc6b628 MEDIUM: http: implement http-response track-sc* directive
This enables tracking of sticky counters from current response. The only
difference from "http-request track-sc" is the <key> sample expression
can only make use of samples in response (eg. res.*, status etc.) and
samples below Layer 6.
2016-07-26 14:31:14 +02:00
Thierry FOURNIER
ab00df6cf6 BUG/MEDIUM: lua: the function txn_done() from sample fetches can crash
The function txn_done() ends a transaction. It does not make
sense to call this function from a lua sample-fetch wrapper,
because the role of a sample-fetch is not to terminate a
transaction.

This patch modify the role of the fucntion txn_done() if it
is called from a sample-fetch wrapper, now it just ends the
execution of the Lua code like the done() function.

Must be backported in 1.6
2016-07-14 16:14:24 +02:00
Dan Lloyd
8e48b8745e DOC: spelling fixes
[wt: most of them are probably valid for 1.6 and 1.5 as well]
2016-07-05 19:11:02 +02:00
Willy Tarreau
be1d34ded2 DOC: add missing doc for http-request deny [deny_status <status>]
The feature was introduced in 1.6-dev2 by commit 108b1dd ("MEDIUM:
http: configurable http result codes for http-request deny") but the
doc was missing. Thanks to Cyril for noticing.

This must be backported into 1.6.
2016-06-26 19:37:59 +02:00
Bertrand Jacquin
9075968356 MINOR: tcp: add "tcp-request connection expect-netscaler-cip layer4"
This configures the client-facing connection to receive a NetScaler
Client IP insertion protocol header before any byte is read from the
socket. This is equivalent to having the "accept-netscaler-cip" keyword
on the "bind" line, except that using the TCP rule allows the PROXY
protocol to be accepted only for certain IP address ranges using an ACL.
This is convenient when multiple layers of load balancers are passed
through by traffic coming from public hosts.
2016-06-20 23:02:47 +02:00
Bertrand Jacquin
93b227db95 MINOR: listener: add the "accept-netscaler-cip" option to the "bind" keyword
When NetScaler application switch is used as L3+ switch, informations
regarding the original IP and TCP headers are lost as a new TCP
connection is created between the NetScaler and the backend server.

NetScaler provides a feature to insert in the TCP data the original data
that can then be consumed by the backend server.

Specifications and documentations from NetScaler:
  https://support.citrix.com/article/CTX205670
  https://www.citrix.com/blogs/2016/04/25/how-to-enable-client-ip-in-tcpip-option-of-netscaler/

When CIP is enabled on the NetScaler, then a TCP packet is inserted just after
the TCP handshake. This is composed as:

  - CIP magic number : 4 bytes
    Both sender and receiver have to agree on a magic number so that
    they both handle the incoming data as a NetScaler Client IP insertion
    packet.

  - Header length : 4 bytes
    Defines the length on the remaining data.

  - IP header : >= 20 bytes if IPv4, 40 bytes if IPv6
    Contains the header of the last IP packet sent by the client during TCP
    handshake.

  - TCP header : >= 20 bytes
    Contains the header of the last TCP packet sent by the client during TCP
    handshake.
2016-06-20 23:02:47 +02:00
Emmanuel Hocdet
d294aea605 MEDIUM: ssl: support SNI filters with multicerts
SNI filters used to be ignored with multicerts (eg: those providing
ECDSA and RSA at the same time). This patch makes them work like
other certs.

Note: most of the changes in this patch are due to an extra level of
      indent, read it with "git show -b".
2016-06-20 17:15:17 +02:00
Dragan Dosen
0070cd52c8 DOC: http: add documentation for url32 and url32+src
This patch needs to be backported to 1.6 and 1.5.
2016-06-16 12:53:25 +02:00
William Lallemand
13e9b0c9ed MEDIUM: tcp/http: new set-dst/set-dst-port actions
Like 'set-src' and 'set-src-port' but for destination address and port.
It's available in 'tcp-request connection' and 'http-request' actions.
2016-06-01 11:44:11 +02:00
William Lallemand
44be6405a1 MEDIUM: tcp/http: add 'set-src-port' action
set-src-port works the same way as 'set-src' but for the source port.
It's available in 'tcp-request connection' and 'http-request' actions.
2016-06-01 11:44:11 +02:00
William Lallemand
2e785f23cb MEDIUM: tcp: add 'set-src' to 'tcp-request connection'
The 'set-src' action was not available for tcp actions The action code
has been converted into a function in proto_tcp.c to be used for both
'http-request' and 'tcp-request connection' actions.

Both http and tcp keywords are registered in proto_tcp.c
2016-06-01 11:44:11 +02:00
William Lallemand
bb933468b4 DOC: update doc about tls-tickets-keys dump
The unix socket can now dump the tls-tickets-keys.
2016-06-01 10:50:50 +02:00
Nenad Merdanovic
1516fe31dd DOC: Fix typo so fetch is properly parsed by Cyril's converter
Signed-off-by: Nenad Merdanovic <nmerdan@anine.io>
2016-05-19 22:10:33 +02:00
Maxime de Roucy
e3841395ad BUG/MEDIUM: init: don't use environment locale
This patch removes setlocale from the main function. It was introduced
by commit 379d9c7 ("MEDIUM: init: allow directory as argument of -f")
in 1.7-dev a few commits ago after a discussion on the mailing list.

Some regex may have different behaviours depending on the
locale. Some LUA scripts may change their behaviour too
(http://lua-users.org/wiki/LuaLocales).

Without this patch (haproxy is using setlocale) :

	$ cat locale.cfg
	defaults
	  mode http

	frontend test
	  bind :9000
	  mode http
	  use_backend testbk if { hdr_reg(X-Test) ^\w+$ }

	backend testbk
	  mode http
	  server s 127.0.0.1:80

	$ LANG=fr_FR.UTF-8 ./haproxy -f locale.cfg
	$ curl -i -H "X-Test: chec" localhost:9000
	HTTP/1.1 200 OK
	...

	$ LANG=C ./haproxy -f locale.cfg
	$ curl -i -H "X-Test: chec" localhost:9000
	HTTP/1.0 503 Service Unavailable
	...
2016-05-19 07:19:19 +02:00
Christopher Faulet
f34b28aa6e DOC: filters: Update the filters documentation accordingly to recent changes 2016-05-18 15:11:54 +02:00
Willy Tarreau
27b639d37f MINOR: log: add the %Td log-format specifier
As suggested by Pavlos, it's too bad that we didn't have a %Td log
format tag given that there are a few mentions of Td corresponding
to the data transmission time already in the doc, so this is now done.
Just like the other specifiers, we report -1 if the connection failed
before reaching the data transmission state.
2016-05-17 18:04:30 +02:00
Maxime de Roucy
379d9c7c14 MEDIUM: init: allow directory as argument of -f
If -f argument is a directory add all the files (and only files) it
containes to the config files list.
These files are added in lexical order (respecting LC_COLLATE).
Only files with ".cfg" extension are added.
Only non hidden files (not prefixed with ".") are added.
Symlink are followed.
The -f order is still respected:

        $ tree -a rootdir
        rootdir
        |-- dir1
        ||-- .6.cfg
        ||-- 1.cfg
        ||-- 2
        ||-- 3.cfg
        ||-- 4.cfg -> 1.cfg
        ||-- 5 -> 1.cfg
        ||-- 7.cfg -> .
        |`-- dir4
        |`-- 8.cfg
        |-- dir2
        ||-- 10.cfg
        |`-- 9.cfg
        |-- dir3
        |`-- 11.cfg
        |-- link -> dir3/
        |-- root1
        |-- root2
        `-- root3

        $ ./haproxy -C rootdir -f root2 -f dir2 -f root3 -f dir1 \
                               -f link -f root1
        root2
        dir2/10.cfg
        dir2/9.cfg
        root3
        dir1/1.cfg
        dir1/3.cfg
        dir1/4.cfg
        link/11.cfg
        root1

This can be useful on systemd where you can't change the haproxy
commande line options on service reload.
2016-05-14 07:09:33 +02:00
Willy Tarreau
7d1b48fae0 [RELEASE] Released version 1.7-dev3
Released version 1.7-dev3 with the following main changes :
    - MINOR: sample: Moves ARGS underlying type from 32 to 64 bits.
    - BUG/MINOR: log: Don't use strftime() which can clobber timezone if chrooted
    - BUILD: namespaces: fix a potential build warning in namespaces.c
    - MINOR: da: Using ARG12 macro for the sample fetch and the convertor.
    - DOC: add encoding to json converter example
    - BUG/MINOR: conf: "listener id" expects integer, but its not checked
    - DOC: Clarify tunes.vars.xxx-max-size settings
    - CLEANUP: chunk: adding NULL check to chunk_dup allocation.
    - CLEANUP: connection: fix double negation on memcmp()
    - BUG/MEDIUM: peers: fix incorrect age in frequency counters
    - BUG/MEDIUM: Fix RFC5077 resumption when more than TLS_TICKETS_NO are present
    - BUG/MAJOR: Fix crash in http_get_fhdr with exactly MAX_HDR_HISTORY headers
    - BUG/MINOR: lua: can't load external libraries
    - BUG/MINOR: prevent the dump of uninitialized vars
    - CLEANUP: map: it seems that the map were planed to be chained
    - MINOR: lua: move class registration facilities
    - MINOR: lua: remove some useless checks
    - CLEANUP: lua: Remove two same functions
    - MINOR: lua: refactor the Lua object registration
    - MINOR: lua: precise message when a critical error is catched
    - MINOR: lua: post initialization
    - MINOR: lua: Add internal function which strip spaces
    - MINOR: lua: convert field to lua type
    - DOC: "addr" parameter applies to both health and agent checks
    - DOC: timeout client: pointers to timeout http-request
    - DOC: typo on stick-store response
    - DOC: stick-table: amend paragraph blaming the loss of table upon reload
    - DOC: typo: ACL subdir match
    - DOC: typo: maxconn paragraph is wrong due to a wrong buffer size
    - DOC: regsub: parser limitation about the inability to use closing square brackets
    - DOC: typo: req.uri is now replaced by capture.req.uri
    - DOC: name set-gpt0 mismatch with the expected keyword
    - MINOR: http: sample fetch which returns unique-id
    - MINOR: dumpstats: extract stats fields enum and names
    - MINOR: dumpstats: split stats_dump_info_to_buffer() in two parts
    - MINOR: dumpstats: split stats_dump_fe_stats() in two parts
    - MINOR: dumpstats: split stats_dump_li_stats() in two parts
    - MINOR: dumpstats: split stats_dump_sv_stats() in two parts
    - MINOR: dumpstats: split stats_dump_be_stats() in two parts
    - MINOR: lua: dump general info
    - MINOR: lua: add class proxy
    - MINOR: lua: add class server
    - MINOR: lua: add class listener
    - BUG/MEDIUM: stick-tables: some sample-fetch doesn't work in the connection state.
    - MEDIUM: proxy: use dynamic allocation for error dumps
    - CLEANUP: remove unneeded casts
    - CLEANUP: uniformize last argument of malloc/calloc
    - DOC: fix "needed" typo
    - BUG/MINOR: dumpstats: fix write to global chunk
    - BUG/MINOR: dns: inapropriate way out after a resolution timeout
    - BUG/MINOR: dns: trigger a DNS query type change on resolution timeout
    - CLEANUP: proto_http: few corrections for gcc warnings.
    - BUG/MINOR: DNS: resolution structure change
    - BUG/MINOR : allow to log cookie for tarpit and denied request
    - BUG/MEDIUM: ssl: rewind the BIO when reading certificates
    - OPTIM/MINOR: session: abort if possible before connecting to the backend
    - DOC: http: rename the unique-id sample and add the documentation
    - BUG/MEDIUM: trace.c: rdtsc() is defined in two files
    - BUG/MEDIUM: channel: fix miscalculation of available buffer space (2nd try)
    - BUG/MINOR: server: risk of over reading the pref_net array.
    - BUG/MINOR: cfgparse: couple of small memory leaks.
    - BUG/MEDIUM: sample: initialize the pointer before parse_binary call.
    - DOC: fix discrepancy in the example for http-request redirect
    - MINOR: acl: Add predefined METH_DELETE, METH_PUT
    - CLEANUP: .gitignore cleanup
    - DOC: Clarify IPv4 address / mask notation rules
    - CLEANUP: fix inconsistency between fd->iocb, proto->accept and accept()
    - BUG/MEDIUM: fix maxaccept computation on per-process listeners
    - BUG/MINOR: listener: stop unbound listeners on startup
    - BUG/MINOR: fix maxaccept computation according to the frontend process range
    - TESTS: add blocksig.c to run tests with all signals blocked
    - MEDIUM: unblock signals on startup.
    - MINOR: filters: Print the list of existing filters during HA startup
    - MINOR: filters: Typo in an error message
    - MINOR: filters: Filters must define the callbacks struct during config parsing
    - DOC: filters: Add filters documentation
    - BUG/MEDIUM: channel: don't allow to overwrite the reserve until connected
    - BUG/MEDIUM: channel: incorrect polling condition may delay event delivery
    - BUG/MEDIUM: channel: fix miscalculation of available buffer space (3rd try)
    - BUG/MEDIUM: log: fix risk of segfault when logging HTTP fields in TCP mode
    - MINOR: Add ability for agent-check to set server maxconn
    - CLEANUP: Use server_parse_maxconn_change_request for maxconn CLI updates
    - MINOR: filters: add opaque data
    - BUG/MEDIUM: lua: protects the upper boundary of the argument list for converters/fetches.
    - MINOR: lua: migrate the argument mask to 64 bits type.
    - BUG/MINOR: dumpstats: Fix the "Total bytes saved" counter in backends stats
    - BUG/MINOR: log: fix a typo that would cause %HP to log <BADREQ>
    - BUG/MEDIUM: http: fix incorrect reporting of server errors
    - MINOR: channel: add new function channel_congested()
    - BUG/MEDIUM: http: fix risk of CPU spikes with pipelined requests from dead client
    - BUG/MAJOR: channel: fix miscalculation of available buffer space (4th try)
    - BUG/MEDIUM: stream: ensure the SI_FL_DONT_WAKE flag is properly cleared
    - BUG/MEDIUM: channel: fix inconsistent handling of 4GB-1 transfers
    - BUG/MEDIUM: stats: show servers state may show an empty or incomplete result
    - BUG/MEDIUM: stats: show backend may show an empty or incomplete result
    - MINOR: stats: fix typo in help messages
    - MINOR: stats: show stat resolvers missing in the help message
    - BUG/MINOR: dns: fix DNS header definition
    - BUG/MEDIUM: dns: fix alignment issue when building DNS queries
    - CLEANUP: don't ignore scripts in .gitignore
    - BUILD: add a few release and backport scripts in scripts/
2016-05-10 15:36:58 +02:00
Nenad Merdanovic
174dd37d88 MINOR: Add ability for agent-check to set server maxconn
This is very useful in complex architecture systems where HAproxy
is balancing DB connections for example. We want to keep the maxconn
high in order to avoid issues with queueing on the LB level when
there is slowness on another part of the system. Example is a case of
an architecture where each thread opens multiple DB connections, which
if get stuck in queue cause a snowball effect (old connections aren't
closed, new ones cannot be established). These connections are mostly
idle and the DB server has no problem handling thousands of them.

Allowing us to dynamically set maxconn depending on the backend usage
(LA, CPU, memory, etc.) enables us to have high maxconn for situations
like above, but lowering it in case there are real issues where the
backend servers become overloaded (cache issues, DB gets hit hard).
2016-04-25 17:23:50 +02:00
Christopher Faulet
c3fe5330be DOC: filters: Add filters documentation
The configuration documention has been updated. Doc about the filter line has
been added and a new chapter (. 9) has been created to list and document
supported filters (for now, flt_trace and flt_http_comp).

The developer documentation about filters has also been added. The is a "pre"
version. Incoming changes in the filter API will require an update.
This documentation requires a deeper review and some TODO need to be complete.
2016-04-21 07:01:41 +02:00
Daniel Schneller
eba5634f2c DOC: Clarify IPv4 address / mask notation rules
Adds some examples regarding shorthand IPv4 address notation which might
be confused with RFC 4632 CIDR notation, leading to different than
expected results.
2016-04-13 07:20:46 +02:00
Daniel Schneller
9ff96c7a62 MINOR: acl: Add predefined METH_DELETE, METH_PUT
Adds the missing HTTP verbs DELETE and PUT as predefined ACLs, similar
to GET, POST etc.
2016-04-12 11:44:09 +02:00
Coen Rosdorff
596659b023 DOC: fix discrepancy in the example for http-request redirect
Commit c8f0e78 ("DOC: typo: req.uri is now replaced by capture.req.uri")
fixed a discrepancy in the doc but the scheme is still missing, resulting
in a redirect loop. Let's fix this as well. This should be backported to
1.5.
2016-04-12 11:26:22 +02:00
Thierry Fournier
0e00dca58b DOC: http: rename the unique-id sample and add the documentation
This patch renames the ssample fetch from "uniqueid" to "unique-id".
It also adds the documentation associated with this sample fetch.
2016-04-07 19:14:58 +02:00
Thierry Fournier
ff480424ab MINOR: lua: add class listener
This class provides the access to the listener struct, it allows
some manipulations and retrieve informations.
2016-03-30 18:43:47 +02:00
Thierry Fournier
f2fdc9dc39 MINOR: lua: add class server
This class provides the access to the server struct, it allows
some manipulations and retrieve informations.
2016-03-30 18:43:47 +02:00
Thierry Fournier
f61aa6356e MINOR: lua: add class proxy
This class provides the access to the proxy struct, it allows
some manipulations and retrieve informations.
2016-03-30 18:43:42 +02:00
Thierry Fournier
eea77c0e17 MINOR: lua: dump general info
This patch adds function able to dump general haproxy information.
2016-03-30 17:27:40 +02:00
Thierry Fournier
b912567020 DOC: name set-gpt0 mismatch with the expected keyword
replace set-gpt0 by sc-set-gpt0

must be backported in 1.6
2016-03-30 17:16:46 +02:00
Baptiste Assmann
c8f0e78b25 DOC: typo: req.uri is now replaced by capture.req.uri
A configuration example was not updated after the switch from req.uri to
capture.req.uri.

backport: 1.5 and above
2016-03-30 17:13:44 +02:00
Baptiste Assmann
66025d856c DOC: regsub: parser limitation about the inability to use closing square brackets
We can't match range in regsub, since the closing bracket is evaluated
by the configuration parser.

backport: 1.6 and above
2016-03-30 17:13:24 +02:00
Baptiste Assmann
79fb45d10b DOC: typo: maxconn paragraph is wrong due to a wrong buffer size
HAProxy allocates 2 tune.bufsize, which is by default 16kB.

backport: 1.4 and above
2016-03-30 17:12:40 +02:00
Baptiste Assmann
33db600442 DOC: typo: ACL subdir match
ACL subdir match is "-m dir"

backport: 1.5 and above
2016-03-30 17:12:04 +02:00
Baptiste Assmann
123ff0463e DOC: stick-table: amend paragraph blaming the loss of table upon reload
This statement is not true anymore since we have the peers in HAProxy.

backport: 1.6 and above
2016-03-30 17:11:45 +02:00
Baptiste Assmann
2f2d2ec164 DOC: typo on stick-store response
It is used to store responses, and not requests.

backport: 1.5 and above
2016-03-30 16:53:39 +02:00
Baptiste Assmann
2e1941ec6e DOC: timeout client: pointers to timeout http-request
It worth mentionning "timeout http-request" in the "timeout client"
documentation paragraph, to ensure nobody misses this important setting.

backport: 1.5 and above
2016-03-30 16:53:34 +02:00
Baptiste Assmann
13f835380f DOC: "addr" parameter applies to both health and agent checks
It was not obvious in the documentation that the server's "addr"
parameter applies to both the agent and the health check.

backport: 1.5 and above
2016-03-30 15:49:52 +02:00
Daniel Schneller
0b547059ec DOC: Clarify tunes.vars.xxx-max-size settings
Adds a little more clarity to the description of the maximum sizes of
the different variable scopes and adds a note about what happens when
the space allocated for variables is too small.

Also fixes some typos and grammar/spelling issues re/ variables and
their naming conventions, copied throughout the document.
2016-03-24 07:50:47 +01:00
Mac Browning
e83345df1b DOC: add encoding to json converter example
Without the encoding `log-format` will issue a warning like:

$ haproxy -c -f /etc/haproxy/haproxy.cfg
[WARNING] 073/180933 (179) : parsing [/etc/haproxy/haproxy.cfg:46] :
'log-format' : sample fetch <capture.req.hdr(1),json> failed with : missing
args for conv method 'json'
2016-03-17 06:14:41 +01:00
David Carlier
840b0240bc MINOR: da: Using ARG12 macro for the sample fetch and the convertor.
Regarding the minor update introduced in the
cd6c3c7cb4 commit, the DeviceAtlas
module is now able to use up to 12 device properties via the
new ARG12 macro.
2016-03-17 05:44:33 +01:00
Willy Tarreau
8234f6dae8 [RELEASE] Released version 1.7-dev2
Released version 1.7-dev2 with the following main changes :
    - DOC: lua: fix lua API
    - DOC: mailers: typo in 'hostname' description
    - DOC: compression: missing mention of libslz for compression algorithm
    - BUILD/MINOR: regex: missing header
    - BUG/MINOR: stream: bad return code
    - DOC: lua: fix somme errors and add implicit types
    - MINOR: lua: add set/get priv for applets
    - BUG/MINOR: http: fix several off-by-one errors in the url_param parser
    - BUG/MINOR: http: Be sure to process all the data received from a server
    - MINOR: filters/http: Use a wrapper function instead of stream_int_retnclose
    - BUG/MINOR: chunk: make chunk_dup() always check and set dst->size
    - DOC: ssl: fixed some formatting errors in crt tag
    - MINOR: chunks: ensure that chunk_strcpy() adds a trailing zero
    - MINOR: chunks: add chunk_strcat() and chunk_newstr()
    - MINOR: chunk: make chunk_initstr() take a const string
    - MEDIUM: tools: add csv_enc_append() to preserve the original chunk
    - MINOR: tools: make csv_enc_append() always start at the first byte of the chunk
    - MINOR: lru: new function to delete <nb> least recently used keys
    - DOC: add Ben Shillito as the maintainer of 51d
    - BUG/MINOR: 51d: Ensures a unique domain for each configuration
    - BUG/MINOR: 51d: Aligns Pattern cache implementation with HAProxy best practices.
    - BUG/MINOR: 51d: Releases workset back to pool.
    - BUG/MINOR: 51d: Aligned const pointers to changes in 51Degrees.
    - CLEANUP: 51d: Aligned if statements with HAProxy best practices and removed casts from malloc.
    - MINOR: rename master process name in -Ds (systemd mode)
    - DOC: fix a few spelling mistakes
    - DOC: fix "workaround" spelling
    - BUG/MINOR: examples: Fixing haproxy.spec to remove references to .cfg files
    - MINOR: fix the return type for dns_response_get_query_id() function
    - MINOR: server state: missing LF (\n) on error message printed when parsing server state file
    - BUG/MEDIUM: dns: no DNS resolution happens if no ports provided to the nameserver
    - BUG/MAJOR: servers state: server port is erased when dns resolution is enabled on a server
    - BUG/MEDIUM: servers state: server port is used uninitialized
    - BUG/MEDIUM: config: Adding validation to stick-table expire value.
    - BUG/MEDIUM: sample: http_date() doesn't provide the right day of the week
    - BUG/MEDIUM: channel: fix miscalculation of available buffer space.
    - MEDIUM: pools: add a new flag to avoid rounding pool size up
    - BUG/MEDIUM: buffers: do not round up buffer size during allocation
    - BUG/MINOR: stream: don't force retries if the server is DOWN
    - BUG/MINOR: counters: make the sc-inc-gpc0 and sc-set-gpt0 touch the table
    - MINOR: unix: don't mention free ports on EAGAIN
    - BUG/CLEANUP: CLI: report the proper field states in "show sess"
    - MINOR: stats: send content-length with the redirect to allow keep-alive
    - BUG: stream_interface: Reuse connection even if the output channel is empty
    - DOC: remove old tunnel mode assumptions
    - BUG/MAJOR: http-reuse: fix risk of orphaned connections
    - BUG/MEDIUM: http-reuse: do not share private connections across backends
    - BUG/MINOR: ssl: Be sure to use unique serial for regenerated certificates
    - BUG/MINOR: stats: fix missing comma in stats on agent drain
    - MAJOR: filters: Add filters support
    - MINOR: filters: Do not reset stream analyzers if the client is gone
    - REORG: filters: Prepare creation of the HTTP compression filter
    - MAJOR: filters/http: Rewrite the HTTP compression as a filter
    - MEDIUM: filters: Use macros to call filters callbacks to speed-up processing
    - MEDIUM: filters: remove http_start_chunk, http_last_chunk and http_chunk_end
    - MEDIUM: filters: Replace filter_http_headers callback by an analyzer
    - MEDIUM: filters/http: Move body parsing of HTTP messages in dedicated functions
    - MINOR: filters: Add stream_filters structure to hide filters info
    - MAJOR: filters: Require explicit registration to filter HTTP body and TCP data
    - MINOR: filters: Remove unused or useless stuff and do small optimizations
    - MEDIUM: filters: Optimize the HTTP compression for chunk encoded response
    - MINOR: filters/http: Slightly update the parsing of chunks
    - MINOR: filters/http: Forward remaining data when a channel has no "data" filters
    - MINOR: filters: Add an filter example
    - MINOR: filters: Extract proxy stuff from the struct filter
    - MINOR: map: Add regex matching replacement
    - BUG/MINOR: lua: unsafe initialization
    - DOC: lua: fix somme errors
    - MINOR: lua: file dedicated to unsafe functions
    - MINOR: lua: add "now" time function
    - MINOR: standard: add RFC HTTP date parser
    - MINOR: lua: Add date functions
    - MINOR: lua: move common function
    - MINOR: lua: merge function
    - MINOR: lua: Add concat class
    - MINOR: standard: add function "escape_chunk"
    - MEDIUM: log: add a new log format flag "E"
    - DOC: add server name at rate-limit sessions example
    - BUG/MEDIUM: ssl: fix off-by-one in ALPN list allocation
    - BUG/MEDIUM: ssl: fix off-by-one in NPN list allocation
    - DOC: LUA: fix some typos and syntax errors
    - MINOR: cli: add a new "show env" command
    - MEDIUM: config: allow to manipulate environment variables in the global section
    - MEDIUM: cfgparse: reject incorrect 'timeout retry' keyword spelling in resolvers
    - MINOR: mailers: increase default timeout to 10 seconds
    - MINOR: mailers: use <CRLF> for all line endings
    - BUG/MAJOR: lua: segfault using Concat object
    - DOC: lua: copyrights
    - MINOR: common: mask conversion
    - MEDIUM: dns: extract options
    - MEDIUM: dns: add a "resolve-net" option which allow to prefer an ip in a network
    - MINOR: mailers: make it possible to configure the connection timeout
    - BUG/MAJOR: lua: applets can't sleep.
    - BUG/MINOR: server: some prototypes are renamed
    - BUG/MINOR: lua: Useless copy
    - BUG/MEDIUM: stats: stats bind-process doesn't propagate the process mask correctly
    - BUG/MINOR: server: fix the format of the warning on address change
    - CLEANUP: server: add "const" to some message strings
    - MINOR: server: generalize the "updater" source
    - BUG/MEDIUM: chunks: always reject negative-length chunks
    - BUG/MINOR: systemd: ensure we don't miss signals
    - BUG/MINOR: systemd: report the correct signal in debug message output
    - BUG/MINOR: systemd: propagate the correct signal to haproxy
    - MINOR: systemd: ensure a reload doesn't mask a stop
    - BUG/MEDIUM: cfgparse: wrong argument offset after parsing server "sni" keyword
    - CLEANUP: stats: Avoid computation with uninitialized bits.
    - CLEANUP: pattern: Ignore unknown samples in pat_match_ip().
    - CLEANUP: map: Avoid memory leak in out-of-memory condition.
    - BUG/MINOR: tcpcheck: fix incorrect list usage resulting in failure to load certain configs
    - BUG/MAJOR: samples: check smp->strm before using it
    - MINOR: sample: add a new helper to initialize the owner of a sample
    - MINOR: sample: always set a new sample's owner before evaluating it
    - BUG/MAJOR: vars: always retrieve the stream and session from the sample
    - CLEANUP: payload: remove useless and confusing nullity checks for channel buffer
    - BUG/MINOR: ssl: fix usage of the various sample fetch functions
    - MINOR: stats: create fields types suitable for all CSV output data
    - MINOR: stats: add all the "show info" fields in a table
    - MEDIUM: stats: fill all the show info elements prior to displaying them
    - MINOR: stats: add a function to emit fields into a chunk
    - MINOR: stats: add stats_dump_info_fields() to dump one field per line
    - MEDIUM: stats: make use of stats_dump_info_fields() for "show info"
    - MINOR: stats: add a declaration of all stats fields
    - MINOR: stats: don't hard-code the CSV fields list anymore
    - MINOR: stats: create stats fields storage and CSV dump function
    - MEDIUM: stats: convert stats_dump_fe_stats() to use stats_dump_fields_csv()
    - MEDIUM: stats: make stats_dump_fe_stats() use stats fields for HTML dump
    - MEDIUM: stats: convert stats_dump_li_stats() to use stats_dump_fields_csv()
    - MEDIUM: stats: make stats_dump_li_stats() use stats fields for HTML dump
    - MEDIUM: stats: convert stats_dump_be_stats() to use stats_dump_fields_csv()
    - MEDIUM: stats: make stats_dump_be_stats() use stats fields for HTML dump
    - MEDIUM: stats: convert stats_dump_sv_stats() to use stats_dump_fields_csv()
    - MEDIUM: stats: make stats_dump_sv_stats() use the stats field for HTML
    - MEDIUM: stats: move the server state coloring logic to the server dump function
    - MINOR: stats: do not use srv->admin & STATS_ADMF_MAINT in HTML dumps
    - MINOR: stats: do not check srv->state for SRV_ST_STOPPED in HTML dumps
    - MINOR: stats: make CSV report server check status only when enabled
    - MINOR: stats: only report backend's down time if it has servers
    - MINOR: stats: prepend '*' in front of the check status when in progress
    - MINOR: stats: make HTML stats dump rely on the table for the check status
    - MINOR: stats: add agent_status, agent_code, agent_duration to output
    - MINOR: stats: add check_desc and agent_desc to the output fields
    - MINOR: stats: add check and agent's health values in the output
    - MEDIUM: stats: make the HTML server state dump use the CSV states
    - MEDIUM: stats: only report observe errors when observe is set
    - MEDIUM: stats: expose the same flags for CLI and HTTP accesses
    - MEDIUM: stats: report server's address in the CSV output
    - MEDIUM: stats: report the cookie value in the server & backend CSV dumps
    - MEDIUM: stats: compute the color code only in the HTML form
    - MEDIUM: stats: report the listeners' address in the CSV output
    - MEDIUM: stats: make it possible to report the WAITING state for listeners
    - REORG: stats: dump the frontend's HTML stats via a generic function
    - REORG: stats: dump the socket stats via the generic function
    - REORG: stats: dump the server stats via the generic function
    - REORG: stats: dump the backend stats via the generic function
    - MEDIUM: stats: add a new "mode" column to report the proxy mode
    - MINOR: stats: report the load balancing algorithm in CSV output
    - MINOR: stats: add 3 fields to report the frontend-specific connection stats
    - MINOR: stats: report number of intercepted requests for frontend and backends
    - MINOR: stats: introduce stats_dump_one_line() to dump one stats line
    - CLEANUP: stats: make stats_dump_fields_html() not rely on proxy anymore
    - MINOR: stats: add ST_SHOWADMIN to pass the admin info in the regular flags
    - MINOR: stats: make stats_dump_fields_html() not use &trash by default
    - MINOR: stats: add functions to emit typed fields into a chunk
    - MEDIUM: stats: support "show info typed" on the CLI
    - MEDIUM: stats: implement a typed output format for stats
    - DOC: document the "show info typed" and "show stat typed" output formats
    - MINOR: cfgparse: warn when uid parameter is not a number
    - MINOR: cfgparse: warn when gid parameter is not a number
    - BUG/MINOR: standard: Avoid free of non-allocated pointer
    - BUG/MINOR: pattern: Avoid memory leak on out-of-memory condition
    - CLEANUP: http: fix a build warning introduced by a recent fix
    - BUG/MINOR: log: GMT offset not updated when entering/leaving DST
2016-03-14 00:10:05 +01:00
Willy Tarreau
5d8b979e68 DOC: document the "show info typed" and "show stat typed" output formats
These formats are more complex and are only usable if properly
documented. Let's hope it will be enough.
2016-03-11 18:05:41 +01:00
Willy Tarreau
5b9bdff007 MINOR: stats: report number of intercepted requests for frontend and backends
This was reported in HTML dumps already but not CSV. It reports the
number of monitor and stats requests. Ideally use-service and redirs
should be accounted for as well.
2016-03-11 17:08:05 +01:00
Willy Tarreau
c73810f94f MINOR: stats: add 3 fields to report the frontend-specific connection stats
Frontends have extra information compared to other entities, they can
report some statistics at the connection level while the other ones
are limited to the session level. This patch adds 3 more fields for
this :
 - conn_rate
 - conn_rate_max
 - conn_tot

It's worth noting that listeners theorically have such statistics, except
that the distinction between connections and sessions is not clearly made
in the code, so that will have to be improved later.
2016-03-11 17:08:05 +01:00
Willy Tarreau
f1516d9840 MINOR: stats: report the load balancing algorithm in CSV output
It was already present in the HTML output, let's add it to CSV now,
but only when SHLGNDS is set.
2016-03-11 17:08:05 +01:00
Willy Tarreau
f8211dff89 MEDIUM: stats: add a new "mode" column to report the proxy mode
Now even CSV stats will see the proxy mode, and we can save the
HTML stats dumps from accessing px->mode directly.
2016-03-11 17:08:05 +01:00
Willy Tarreau
a6f5a73202 MEDIUM: stats: report the listeners' address in the CSV output
It's the same principle as for the server dump, and we use this field
for the HTML dump of course.
2016-03-11 17:08:05 +01:00