Commit Graph

1099 Commits

Author SHA1 Message Date
Willy Tarreau
234ba2d8eb MINOR: cli: make "show errors" support a proxy name
Till now it was needed to know the proxy's ID while we do have the
ability to look up a proxy by its name now.
2016-11-25 08:56:55 +01:00
Willy Tarreau
a1b1ed53e7 MINOR: cli: make "show stat" support a proxy name
Till now it was needed to know the proxy's ID while we do have the
ability to look up a proxy by its name now.
2016-11-25 08:55:25 +01:00
Thierry FOURNIER / OZON.IO
8a1027aa45 MINOR: lua: Add tokenize function.
For tokenizing a string, standard Lua recommends to use regexes.
The followinf example splits words:

   for i in string.gmatch(example, "%S+") do
      print(i)
   end

This is a little bit overkill for simply split words. This patch
adds a tokenize function which quick and do not use regexes.
2016-11-24 21:35:34 +01:00
Simon Horman
1084a3631f MINOR: stats: correct documentation of process ID for typed output
The process ID appears at the end of the first column rather than
the line.
2016-11-21 22:54:00 +01:00
Christopher Faulet
985532d1d8 MINOR: spoe: Add "option set-on-error" statement
It defines the variable to set when an error occurred during an event
processing. It will only be set when an error occurred in the scope of the
transaction. As for all other variables define by the SPOE, it will be
prefixed. So, if your variable name is "error" and your prefix is "my_spoe_pfx",
the variable will be "txn.my_spoe_pfx.error".

When set, the variable is the boolean "true". Note that if "option
continue-on-error" is set, the variable is not automatically removed between
events processing.
2016-11-21 15:29:59 +01:00
Christopher Faulet
4802672274 MINOR: spoe: Add "maxconnrate" and "maxerrrate" statements
"maxconnrate" is the maximum number of connections per second. The SPOE will
stop to open new connections if the maximum is reached and will wait to acquire
an existing one.

"maxerrrate" is the maximum number of errors per second. The SPOE will stop its
processing if the maximum is reached.

These options replace hardcoded macros MAX_NEW_SPOE_APPLETS and
MAX_NEW_SPOE_APPLET_ERRS. We use it to limit SPOE activity, especially when
servers are down..
2016-11-21 15:29:59 +01:00
Christopher Faulet
ea62c2a345 MINOR: spoe: Add 'option continue-on-error' statement in spoe-agent section
By default, for a specific stream, when an abnormal/unexpected error occurs, the
SPOE is disabled for all the transaction. So if you have several events
configured, such error on an event will disabled all followings. For TCP
streams, this will disable the SPOE for the whole session. For HTTP streams,
this will disable it for the transaction (request and response).

To bypass this behaviour, you can set 'continue-on-error' option in 'spoe-agent'
section. With this option, only the current event will be ignored.
2016-11-21 15:29:59 +01:00
Christopher Faulet
03a3449e1a MINOR: spoe: Remove useless 'timeout ack' option
To limit the time to process an event, you should set 'timeout processing'
option. So 'timeout ack' option is redundant and useless.
2016-11-21 15:29:59 +01:00
Christopher Faulet
f7a3092512 MINOR: spoe: Add 'timeout processing' option to limit time to process an event
It is a way to set the maximum time to wait for a stream to process an event,
i.e to acquire a stream to talk with an agent, to encode all messages, to send
the NOTIFY frame, to receive the corrsponding acknowledgement and to process all
actions. It is applied on the stream that handle the client and the server
sessions.
2016-11-21 15:29:59 +01:00
Christopher Faulet
a00d817aba MINOR: filters: Add check_timeouts callback to handle timers expiration on streams
A filter can now be notified when a stream is woken up because of an expired
timer.

The documentation and the TRACE filter have been updated.
2016-11-21 15:29:58 +01:00
Thierry FOURNIER / OZON.IO
a44fdd95f9 MEDIUM: lua: Add cli handler for Lua
Now, HAProxy allows to register some keys in the "cli". This patch allows
to handle these keys with Lua code.
2016-11-18 14:32:03 +01:00
Bertrand Jacquin
d4d0a23ca4 DOC: ssl: Use correct wording for ca-sign-pass
Doc references ca-sign-passphrase but the source code is referring
ca-sign-pass. Align doc to reality.
2016-11-14 18:15:20 +01:00
Thierry FOURNIER / OZON.IO
62fec75183 MINOR: lua: add ip addresses and network manipulation function
Add two functions core.parse_addr() and core.match_addr() where are used
for matching networks.
2016-11-12 10:42:30 +01:00
Thierry FOURNIER / OZON.IO
65192f35d2 MINOR: lua: add function which return true if the channel is full.
Add function which return true if the channel is full. It is
useful for triggering some process when the buffer is full.
2016-11-12 10:42:25 +01:00
Willy Tarreau
d5d890be21 [RELEASE] Released version 1.7-dev6
Released version 1.7-dev6 with the following main changes :
    - DOC: fix the entry for hash-balance-factor config option
    - DOC: Fix typo in description of `-st` parameter in man page
    - CLEANUP: cfgparse: Very minor spelling correction
    - MINOR: examples: Update haproxy.spec URLs to haproxy.org
    - BUG/MEDIUM: peers: on shutdown, wake up the appctx, not the stream
    - BUG/MEDIUM: peers: fix use after free in peer_session_create()
    - MINOR: peers: make peer_session_forceshutdown() use the appctx and not the stream
    - MINOR: peers: remove the pointer to the stream
    - BUG/MEDIUM: systemd-wrapper: return correct exit codes
    - DOC: stats: provide state details for show servers state
    - MEDIUM: tools: make str2ip2() preserve existing ports
    - CLEANUP: tools: make ipcpy() preserve the original port
    - OPTIM: http: move all http character classs tables into a single one
    - OPTIM: http: improve parsing performance of long header lines
    - OPTIM: http: improve parsing performance of long URIs
    - OPTIM: http: optimize lookup of comma and quote in header values
    - BUG/MEDIUM: srv-state: properly restore the DRAIN state
    - BUG/MINOR: srv-state: allow to have both CMAINT and FDRAIN flags
    - MINOR: server: do not emit warnings/logs/alerts on server state changes at boot
    - BUG/MEDIUM: servers: properly propagate the maintenance states during startup
    - MEDIUM: wurfl: add Scientiamobile WURFL device detection module
    - DOC: move the device detection modules documentation to their own files
    - CLEANUP: wurfl: reduce exposure in the rest of the code
    - MEDIUM: ssl: Add support for OpenSSL 1.1.0
    - MINOR: stream: make option contstats usable again
    - MEDIUM: tools: make str2sa_range() return the FQDN even when not resolving
    - MINOR: init: move apply_server_state in haproxy.c before MODE_CHECK
    - MAJOR: server: postpone address resolution
    - MINOR: new srv_admin flag: SRV_ADMF_RMAINT
    - MINOR: server: indicate in the logs when RMAINT is cleared
    - MINOR: stats: indicate it when a server is down due to resolution
    - MINOR: server: make srv_set_admin_state() capable of telling why this happens
    - MINOR: dns: implement extra 'hold' timers.
    - MAJOR: dns: runtime resolution can change server admin state
    - MEDIUM: cli: leave the RMAINT state when setting an IP address on the CLI
    - MEDIUM: server: add a new init-addr server line setting
    - MEDIUM: server: make use of init-addr
    - MINOR: server: implement init-addr none
    - MEDIUM: server: make libc resolution failure non-fatal
    - MINOR: server: add support for explicit numeric address in init-addr
    - DOC: add some documentation for the "init-addr" server keyword
    - MINOR: init: add -dr to ignore server address resolution failures
    - MEDIUM: server: do not restrict anymore usage of IP address from the state file
    - BUG: vars: Fix 'set-var' converter because of a typo
    - CLEANUP: remove last references to 'ruleset' section
    - MEDIUM: filters: Add attch/detach and stream_set_backend callbacks
    - MINOR: filters: Update filters documentation accordingly to recent changes
    - MINOR: filters: Call stream_set_backend callbacks before updating backend stats
    - MINOR: filters: Remove backend filters attached to a stream only for HTTP streams
    - MINOR: flt_trace: Add hexdump option to dump forwarded data
    - MINOR: cfgparse: Add functions to backup and restore registered sections
    - MINOR: cfgparse: Parse scope lines and save the last one parsed
    - REORG: sample: move code to release a sample expression in sample.c
    - MINOR: vars: Allow '.' in variable names
    - MINOR: vars: Add vars_set_by_name_ifexist function
    - MEDIUM: vars: Add a per-process scope for variables
    - MINOR: vars: Add 'unset-var' action/converter
    - MAJOR: spoe: Add an experimental Stream Processing Offload Engine
    - MINOR: spoe: add random ip-reputation service as SPOA example
    - MINOR: spoe/checks: Add support for SPOP health checks
    - DOC: update ROADMAP file
2016-11-09 23:18:17 +01:00
Christopher Faulet
ba7bc164f7 MINOR: spoe/checks: Add support for SPOP health checks
A new "option spop-check" statement has been added to enable server health
checks based on SPOP HELLO handshake. SPOP is the protocol used by SPOE filters
to talk to servers.
2016-11-09 22:57:02 +01:00
Christopher Faulet
f7e4e7e096 MAJOR: spoe: Add an experimental Stream Processing Offload Engine
SPOE makes possible the communication with external components to retrieve some
info using an in-house binary protocol, the Stream Processing Offload Protocol
(SPOP). In the long term, its aim is to allow any kind of offloading on the
streams. This first version, besides being experimental, won't do lot of
things. The most important today is to validate the protocol design and lay the
foundations of what will, one day, be a full offload engine for the stream
processing.

So, for now, the SPOE can offload the stream processing before "tcp-request
content", "tcp-response content", "http-request" and "http-response" rules. And
it only supports variables creation/suppression. But, in spite of these limited
features, we can easily imagine to implement a SSO solution, an ip reputation
service or an ip geolocation service.

Internally, the SPOE is implemented as a filter. So, to use it, you must use
following line in a proxy proxy section:

  frontend my-front
      ...
      filter spoe [engine <name>] config <file>
      ...

It uses its own configuration file to keep the HAProxy configuration clean. It
is also a easy way to disable it by commenting out the filter line.

See "doc/SPOE.txt" for all details about the SPOE configuration.
2016-11-09 22:57:01 +01:00
Christopher Faulet
85d79c94a9 MINOR: vars: Add 'unset-var' action/converter
It does the opposite of 'set-var' action/converter. It is really useful for
per-process variables. But, it can be used for any scope.

The lua function 'unset_var' has also been added.
2016-11-09 22:57:01 +01:00
Christopher Faulet
ff2613ed7a MEDIUM: vars: Add a per-process scope for variables
Now it is possible to use variables attached to a process. The scope name is
'proc'. These variables are released only when HAProxy is stopped.

'tune.vars.proc-max-size' directive has been added to confiure the maximum
amount of memory used by "proc" variables. And because memory accounting is
hierachical for variables, memory for "proc" vars includes memory for "sess"
vars.
2016-11-09 22:57:00 +01:00
Christopher Faulet
b71557a98b MINOR: vars: Allow '.' in variable names
This is required to have implicit prefix or scope. SPOE filter will use it to
keep variables set by an agent in its own namespace.
2016-11-09 22:57:00 +01:00
Christopher Faulet
9adb0a5458 MINOR: filters: Update filters documentation accordingly to recent changes 2016-11-09 22:50:55 +01:00
Willy Tarreau
e5a60688a4 MEDIUM: server: do not restrict anymore usage of IP address from the state file
Now that it is possible to decide whether we prefer to use libc or the
state file to resolve the server's IP address and it is possible to change
a server's IP address at run time on the CLI, let's not restrict the reuse
of the address from the state file anymore to the DNS only.

The impact is that by default the state file will be considered first
(which matches its purpose) and only then the libc. This way any address
change performed at run time over the CLI will be preserved regardless
of DNS usage or not.
2016-11-09 15:33:52 +01:00
Willy Tarreau
3eed10e54b MINOR: init: add -dr to ignore server address resolution failures
It is very common when validating a configuration out of production not to
have access to the same resolvers and to fail on server address resolution,
making it difficult to test a configuration. This option simply appends the
"none" method to the list of address resolution methods for all servers,
ensuring that even if the libc fails to resolve an address, the startup
sequence is not interrupted.
2016-11-09 15:33:52 +01:00
Willy Tarreau
6a031d1c90 DOC: add some documentation for the "init-addr" server keyword
All 4 new values were documented at once.
2016-11-09 15:33:52 +01:00
Baptiste Assmann
987e16d6f4 MINOR: dns: implement extra 'hold' timers.
This adds new "hold" timers : nx, refused, timeout, other. This timers
will be used to tell HAProxy to keep an erroneous response as valid for
the corresponding period. For now they're only configured, not enforced.
2016-11-09 15:30:47 +01:00
Willy Tarreau
b96dd28477 MINOR: stats: indicate it when a server is down due to resolution
The server's state is now "MAINT (resolution)" just like we also have
"MAINT (via x/y)" when servers are tracked. The HTML stats page reports
"resolution" in the checks field similarly to what is done for the "via"
entry.
2016-11-09 15:30:47 +01:00
Baptiste Assmann
89aa7f30da MINOR: new srv_admin flag: SRV_ADMF_RMAINT
This flag has to be set when an IP address resolution fails (either
using libc at start up or using HAProxy's runtime resolver). This will
automatically trigger the administrative status "MAINT", through the
global mask SRV_ADMF_MAINT.
2016-11-09 15:23:16 +01:00
Willy Tarreau
def0d22cc5 MINOR: stream: make option contstats usable again
Quite a lot of people have been complaining about option contstats not
working correctly anymore since about 1.4. The reason was that one reason
for the significant performance boost between 1.3 and 1.4 was the ability
to forward data between a server and a client without waking up the stream
manager. And we couldn't afford to force sessions to constantly wake it
up given that most of the people interested in contstats are also those
interested in high performance transmission.

An idea was experimented with in the past, consisting in limiting the
amount of transmissible data before waking it up, but it was not usable
on slow connections (eg: FTP over modem lines, RDP, SSH) as stats would
be updated too rarely if at all, so that idea was dropped.

During a discussion today another idea came up : ensure that stats are
updated once in a while, since it's the only thing that matters. It
happens that we have the request channel's analyse_exp timeout that is
used to wake the stream up after a configured delay, and that by
definition this timeout is not used when there's no more analyser
(otherwise the stream would wake up and the stats would be updated).

Thus here the idea is to reuse this timeout when there's no analyser
and set it to now+5 seconds so that a stream wakes up at least once
every 5 seconds to update its stats. It should be short enough to
provide smooth traffic graphs and to allow to debug outputs of "show
sess" more easily without inflicting too much load even for very large
number of concurrent connections.

This patch is simple enough and safe enough to be backportable to 1.6
if there is some demand.
2016-11-08 22:03:00 +01:00
Willy Tarreau
29b25317fc DOC: move the device detection modules documentation to their own files
Both DeviceAtlas and 51Degrees used to put their building instructions
in the README, representing more than 1/3 of it. It's better to let the
README focus on generic stuff and building procedure and move the DD
docs to their own files.
2016-11-08 15:06:21 +01:00
scientiamobile
d0027ed5b1 MEDIUM: wurfl: add Scientiamobile WURFL device detection module
WURFL is a high-performance and low-memory footprint mobile device
detection software component that can quickly and accurately detect
over 500 capabilities of visiting devices. It can differentiate between
portable mobile devices, desktop devices, SmartTVs and any other types
of devices on which a web browser can be installed.

In order to add WURFL device detection support, you would need to
download Scientiamobile InFuze C API and install it on your system.
Refer to www.scientiamobile.com to obtain a valid InFuze license.

Any useful information on how to configure HAProxy working with WURFL
may be found in:

  doc/WURFL-device-detection.txt
  doc/configuration.txt
  examples/wurfl-example.cfg

Please find more information about WURFL device detection API detection
at https://docs.scientiamobile.com/documentation/infuze/infuze-c-api-user-guide
2016-11-08 14:21:43 +01:00
Cyril Bont
5b2ce8a10f DOC: stats: provide state details for show servers state
Add the state values to the documentation instead of adding a reference to the
source code.
2016-11-05 13:55:23 +01:00
Jorrit Schippers
1458fdbe97 DOC: Fix typo in description of -st parameter in man page
extra "wait".
2016-10-26 08:36:37 +02:00
Andrew Rodland
17be45ebfa DOC: fix the entry for hash-balance-factor config option
It was accidentally added as "balance-factor". Fix it and
re-alphabetize.

Signed-off-by: Andrew Rodland <andrewr@vimeo.com>
2016-10-25 23:30:01 +02:00
Willy Tarreau
608efa173c [RELEASE] Released version 1.7-dev5
Released version 1.7-dev5 with the following main changes :
    - MINOR: cfgparse: few memory leaks fixes.
    - MEDIUM: log: Decompose %Tq in %Th %Ti %TR
    - CLEANUP: logs: remove unused log format field definitions
    - BUILD/MAJOR:updated 51d Trie implementation to incorperate latest update to 51Degrees.c
    - BUG/MAJOR: stream: properly mark the server address as unset on connect retry
    - CLEANUP: proto_http: Removing useless variable assignation
    - CLEANUP: dumpstats: Removing useless variables allocation
    - CLEANUP: dns: Removing usless variable & assignation
    - BUG/MINOR: payload: fix SSLv2 version parser
    - MINOR: cli: allow the semi-colon to be escaped on the CLI
    - MINOR: cli: change a server health check port through the stats socket
    - BUG/MINOR: Fix OSX compilation errors
    - MAJOR: check: find out which port to use for health check at run time
    - MINOR: server: introduction of 3 new server flags
    - MINOR: new update_server_addr_port() function to change both server's ADDR and service PORT
    - MINOR: cli: ability to change a server's port
    - CLEANUP/MINOR dns: comment do not follow up code update
    - MINOR: chunk: new strncat function
    - MINOR: dns: wrong DNS_MAX_UDP_MESSAGE value
    - MINOR: dns: new MAX values
    - MINOR: dns: new macro to compute DNS header size
    - MINOR: dns: new DNS structures to store received packets
    - MEDIUM: dns: new DNS response parser
    - MINOR: dns: query type change when last record is a CNAME
    - MINOR: dns: proper domain name validation when receiving DNS response
    - MINOR: dns: comments in types/dns.h about structures endianness
    - BUG/MINOR: displayed PCRE version is running release
    - MINOR: show Built with PCRE version
    - MINOR: show Running on zlib version
    - MEDIUM: make SO_REUSEPORT configurable
    - MINOR: enable IP_BIND_ADDRESS_NO_PORT on backend connections
    - BUG/MEDIUM: http/compression: Fix how chunked data are copied during the HTTP body parsing
    - BUG/MINOR: stats: report the correct conn_time in backend's html output
    - BUG/MEDIUM: dns: don't randomly crash on out-of-memory
    - MINOR: Add fe_req_rate sample fetch
    - MEDIUM: peers: Fix a peer stick-tables synchronization issue.
    - MEDIUM: cli: register CLI keywords with cli_register_kw()
    - BUILD: Make use of accept4() on OpenBSD.
    - MINOR: tcp: make set-src/set-src-port and set-dst/set-dst-port commutative
    - DOC: fix missed entry for "set-{src,dst}{,-port}"
    - BUG/MINOR: vars: use sess and not s->sess in action_store()
    - BUG/MINOR: vars: make smp_fetch_var() more robust against misuses
    - BUG/MINOR: vars: smp_fetch_var() doesn't depend on HTTP but on the session
    - MINOR: stats: output dcon
    - CLEANUP: tcp rules: mention everywhere that tcp-conn rules are L4
    - MINOR: counters: add new fields for denied_sess
    - MEDIUM: tcp: add registration and processing of TCP L5 rules
    - MINOR: stats: emit dses
    - DOC: document tcp-request session
    - MINOR: ssl: add debug traces
    - BUILD/CLEANUP: ssl: Check BIO_reset() return code
    - BUG/MINOR: ssl: Check malloc return code
    - BUG/MINOR: ssl: prevent multiple entries for the same certificate
    - BUG/MINOR: systemd: make the wrapper return a non-null status code on error
    - BUG/MINOR: systemd: always restore signals before execve()
    - BUG/MINOR: systemd: check return value of calloc()
    - MINOR: systemd: report it when execve() fails
    - BUG/MEDIUM: systemd: let the wrapper know that haproxy has completed or failed
    - MINOR: proxy: add 'served' field to proxy, equal to total of all servers'
    - MINOR: backend: add hash-balance-factor option for hash-type consistent
    - MINOR: server: compute a "cumulative weight" to allow chash balancing to hit its target
    - MEDIUM: server: Implement bounded-load hash algorithm
    - SCRIPTS: make git-show-backports also dump a "git show" command
    - MINOR: build: Allow linking to device-atlas library file
    - MINOR: stats: Escape equals sign on socket dump
2016-10-25 22:22:00 +02:00
Andrew Rodland
b1f48e3161 MINOR: backend: add hash-balance-factor option for hash-type consistent
0 will mean no balancing occurs; otherwise it represents the ratio
between the highest-loaded server and the average load, times 100 (i.e.
a value of 150 means a 1.5x ratio), assuming equal weights.

Signed-off-by: Andrew Rodland <andrewr@vimeo.com>
2016-10-25 20:21:32 +02:00
Willy Tarreau
4f614290db DOC: document tcp-request session
This is mostly a copy-paste-edit of the section about "tcp-request connection".
Maybe all these sections should be merged.
2016-10-21 18:19:48 +02:00
Willy Tarreau
a5bc36b31c MINOR: stats: emit dses
This is the number of denied sessions, blocked by "tcp-request session reject".
2016-10-21 18:19:48 +02:00
Willy Tarreau
8a90b8ea19 MINOR: stats: output dcon
These are denied conns. Strangely this wasn't emitted while it used to be
available for a while. It corresponds to the number of connections blocked
by "tcp-request connection reject".
2016-10-21 18:17:56 +02:00
Willy Tarreau
0c63053b30 DOC: fix missed entry for "set-{src,dst}{,-port}"
There was the same explanation for tcp-request connection that I missed
in previous patch.
2016-10-21 17:52:58 +02:00
Willy Tarreau
00005ce5a1 MINOR: tcp: make set-src/set-src-port and set-dst/set-dst-port commutative
When the tcp/http actions above were introduced in 1.7-dev4, we used to
proceed like this :

  - set-src/set-dst would force the port to zero
  - set-src-port/set-dst-port would not do anything if the address family is
    neither AF_INET nor AF_INET6.

It was a stupid idea of mine to request this behaviour because it ensures
that these functions cannot be used in a wide number of situations. Because
of the first rule, it is necessary to save the source port one way or
another if only the address has to be changed (so you have to use an
variable). Due to the second rule, there's no way to set the source port
on a unix socket without first overwriting the address. And sometimes it's
really not convenient, especially when there's no way to guarantee that all
fields will properly be set.

In order to fix all this, this small change does the following :
  - set-src/set-dst always preserve the original port even if the address
    family changes. If the previous address family didn't have a port (eg:
    AF_UNIX), then the port is set to zero ;

  - set-src-port/set-dst-port always preserve the original address. If the
    address doesn't have a port, then the family is forced to IPv4 and the
    address to "0.0.0.0".

Thanks to this it now becomes possible to perform one action, the other or
both in any order.
2016-10-21 15:15:20 +02:00
Nenad Merdanovic
ad9a7e9770 MINOR: Add fe_req_rate sample fetch
The fe_req_rate is similar to fe_sess_rate, but fetches the number
of HTTP requests per second instead of connections/sessions per second.

Signed-off-by: Nenad Merdanovic <nmerdan@anine.io>
2016-10-03 16:08:09 +02:00
Lukas Tribus
7d56c6d347 MINOR: enable IP_BIND_ADDRESS_NO_PORT on backend connections
Enable IP_BIND_ADDRESS_NO_PORT on backend connections when the source
address is specified without port or port ranges. This is supported
since Linux 4.2/libc 2.23.

If the kernel supports it but the libc doesn't, we can define it at
build time:
make [...] DEFINE=-DIP_BIND_ADDRESS_NO_PORT=24

For more informations about this feature, see Linux commit 90c337da
2016-09-13 15:22:54 +02:00
Lukas Tribus
a0bcbdcb04 MEDIUM: make SO_REUSEPORT configurable
With Linux officially introducing SO_REUSEPORT support in 3.9 and
its mainstream adoption we have seen more people running into strange
SO_REUSEPORT related issues (a process management issue turning into
hard to diagnose problems because the kernel load-balances between the
new and an obsolete haproxy instance).

Also some people simply want the guarantee that the bind fails when
the old process is still bound.

This change makes SO_REUSEPORT configurable, introducing the command
line argument "-dR" and the noreuseport configuration directive.

A backport to 1.6 should be considered.
2016-09-13 07:56:03 +02:00
Baptiste Assmann
3749ebf6fc MINOR: cli: ability to change a server's port
Enrichment of the 'set server <b>/<s> addr' cli directive to allow changing
now a server's port.
The new syntax looks like:
  set server <b>/<s> addr [port <port>]
2016-09-11 08:13:31 +02:00
Baptiste Assmann
5094656a67 MINOR: cli: change a server health check port through the stats socket
Introduction of a new CLI command "set server <srv> check-port <port>' to
allow admins to change a server's health check port at run time.

This changes the equivalent of the configuration server parameter
called 'port'.
2016-09-06 07:39:16 +02:00
Chad Lavoie
e3f5031b51 MINOR: cli: allow the semi-colon to be escaped on the CLI
Today I was working on an auto-update script for some ACLs, and found
that I couldn't load ACL entries with a semi-colon in them no matter
how I tried to escape it.

As such, I wrote this patch (this one is for 1.7dev, but it applies to
1.5 the same with just line numbers changed), which seems to allow me
to execute a command such as "add acl /etc/foo.lst foo\;bar" over the
socket. It's worth noting that stats_sock_parse_request() already uses
the backslash to escape spaces in words so it makes sense to use it as
well to escape the semi-colon.
2016-08-30 22:32:23 +02:00
Thierry FOURNIER / OZON.IO
4cac359a39 MEDIUM: log: Decompose %Tq in %Th %Ti %TR
Tq is the time between the instant the connection is accepted and a
complete valid request is received. This time includes the handshake
(SSL / Proxy-Protocol), the idle when the browser does preconnect and
the request reception.

This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR
which returns respectively the handshake time, the idle time and the
duration of valid request reception. It also adds %Ta which reports
the request's active time, which is the total time without %Th nor %Ti.
It replaces %Tt as the total time, reporting accurate measurements for
HTTP persistent connections.

%Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only
avalaible for HTTP connections.

In addition to this, we have new timestamps %tr, %trg and %trl, which
log the date of start of receipt of the request, respectively in the
default format, in GMT time and in local time (by analogy with %t, %T
and %Tl). All of them are obviously only available for HTTP. These values
are more relevant as they more accurately represent the request date
without being skewed by a browser's preconnect nor a keep-alive idle
time.

The HTTP log format and the CLF log format have been modified to
use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This
way the default log formats now produce the expected output for users
who don't want to manually fiddle with the log-format directive.

Example with the following log-format :

   log-format "%ci:%cp [%tr] %ft %b/%s h=%Th/i=%Ti/R=%TR/w=%Tw/c=%Tc/r=%Tr/a=%Ta/t=%Tt %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

The request was sent by hand using "openssl s_client -connect" :

   Aug 23 14:43:20 haproxy[25446]: 127.0.0.1:45636 [23/Aug/2016:14:43:20.221] test~ test/test h=6/i=2375/R=261/w=0/c=1/r=0/a=262/t=2643 200 145 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

=> 6 ms of SSL handshake, 2375 waiting before sending the first char (in
fact the time to type the first line), 261 ms before the end of the request,
no time spent in queue, 1 ms spend connecting to the server, immediate
response, total active time for this request = 262ms. Total time from accept
to close : 2643 ms.

The timing now decomposes like this :

                 first request               2nd request
      |<-------------------------------->|<-------------- ...
      t         tr                       t    tr ...
   ---|----|----|----|----|----|----|----|----|--
      : Th   Ti   TR   Tw   Tc   Tr   Td : Ti   ...
      :<---- Tq ---->:                   :
      :<-------------- Tt -------------->:
                :<--------- Ta --------->:
2016-08-23 15:18:08 +02:00
Willy Tarreau
41d5e3a610 [RELEASE] Released version 1.7-dev4
Released version 1.7-dev4 with the following main changes :
    - MINOR: add list_append_word function
    - MEDIUM: init: use list_append_word in haproxy.c
    - MEDIUM: init: allow directory as argument of -f
    - CLEANUP: config: detect double registration of a config section
    - MINOR: log: add the %Td log-format specifier
    - MEDIUM: filters: Move HTTP headers filtering in its own callback
    - MINOR: filters: Simplify calls to analyzers using 2 new macros
    - MEDIUM: filters: Add pre and post analyzer callbacks
    - DOC: filters: Update the filters documentation accordingly to recent changes
    - BUG/MEDIUM: init: don't use environment locale
    - SCRIPTS: teach git-show-backports how to report upstream commits
    - SCRIPTS: make git-show-backports capable of limiting its history
    - BUG/MAJOR: fix listening IP address storage for frontends
    - BUG/MINOR: fix listening IP address storage for frontends (cont)
    - DOC: Fix typo so fetch is properly parsed by Cyril's converter
    - BUG/MAJOR: http: fix breakage of "reqdeny" causing random crashes
    - BUG/MEDIUM: stick-tables: fix breakage in table converters
    - MINOR: stick-table: change all stick-table converters' inputs to SMP_T_ANY
    - BUG/MEDIUM: dns: unbreak DNS resolver after header fix
    - BUILD: fix build on Solaris 11
    - BUG/MEDIUM: config: fix multiple declaration of section parsers
    - BUG/MEDIUM: stats: show servers state may show an servers from another backend
    - BUG/MEDIUM: fix risk of segfault with "show tls-keys"
    - MEDIUM: dumpstats: 'show tls-keys' is now able to show secrets
    - DOC: update doc about tls-tickets-keys dump
    - MEDIUM: tcp: add 'set-src' to 'tcp-request connection'
    - MINOR: set the CO_FL_ADDR_FROM_SET flags with 'set-src'
    - MEDIUM: tcp/http: add 'set-src-port' action
    - MEDIUM: tcp/http: new set-dst/set-dst-port actions
    - BUG/MEDIUM: sticktables: segfault in some configuration error cases
    - BUILD/MEDIUM: rebuild everything when an include file is changed
    - BUILD/MEDIUM: force a full rebuild if some build options change
    - BUG/MEDIUM: lua: converters doesn't work
    - BUG/MINOR: http: add-header: header name copied twice
    - BUG/MEDIUM: http: add-header: buffer overwritten
    - BUG/MINOR: ssl: fix potential memory leak in ssl_sock_load_dh_params()
    - MINOR: stream: export the function 'smp_create_src_stkctr'
    - BUG/MEDIUM: dumpstats: undefined behavior in stats_tlskeys_list()
    - MEDIUM: dumpstats: make stats_tlskeys_list() yield-aware during tls-keys dump
    - BUG/MINOR: http: url32+src should use the big endian version of url32
    - BUG/MINOR: http: url32+src should check cli_conn before using it
    - DOC: http: add documentation for url32 and url32+src
    - BUG/MINOR: fix http-response set-log-level parsing error
    - MINOR: systemd: Use variable for config and pidfile paths
    - MINOR: systemd: Perform sanity check on config before reload
    - MEDIUM: ssl: support SNI filters with multicerts
    - MINOR: ssl: crt-list parsing factor
    - BUILD: ssl: fix typo causing a build failure in the multicert patch
    - MINOR: listener: add the "accept-netscaler-cip" option to the "bind" keyword
    - MINOR: tcp: add "tcp-request connection expect-netscaler-cip layer4"
    - BUG/MINOR: init: always ensure that global.rlimit_nofile matches actual limits
    - BUG/MINOR: init: ensure that FD limit is raised to the max allowed
    - BUG/MEDIUM: external-checks: close all FDs right after the fork()
    - BUG/MAJOR: external-checks: use asynchronous signal delivery
    - BUG/MINOR: external-checks: do not unblock undesired signals
    - CLEANUP: external-check: don't block/unblock SIGCHLD when manipulating the list
    - BUG/MEDIUM: filters: Fix data filtering when data are modified
    - BUG/MINOR: filters: Fix HTTP parsing when a filter loops on data forwarding
    - BUG/MINOR: srv-state: fix incorrect output of state file
    - BUG/MINOR: ssl: close ssl key file on error
    - BUG/MINOR: http: fix misleading error message for response captures
    - BUG/BUILD: don't automatically run "make" on "make install"
    - DOC: add missing doc for http-request deny [deny_status <status>]
    - CLEANUP: dumpstats: u64 field is an unsigned type.
    - BUG/MEDIUM: http: unbreak uri/header/url_param hashing
    - BUG/MINOR: Rework slightly commit 9962f8fc to clean code and avoid mistakes
    - MINOR: new function my_realloc2 = realloc + free upon failure
    - CLEANUP: fixed some usages of realloc leading to memory leak
    - Revert "BUG/MINOR: ssl: fix potential memory leak in ssl_sock_load_dh_params()"
    - CLEANUP: connection: using internal struct to hold source and dest port.
    - DOC: spelling fixes
    - BUG/MINOR: ssl: fix potential memory leak in ssl_sock_load_dh_params()
    - BUG/MEDIUM: dns: fix alignment issues in the DNS response parser
    - BUG/MINOR: Fix endiness issue in DNS header creation code
    - BUG/MEDIUM: lua: the function txn_done() from sample fetches can crash
    - BUG/MEDIUM: lua: the function txn_done() from action wrapper can crash
    - MEDIUM: http: implement http-response track-sc* directive
    - BUG/MINOR: peers: Fix peers data decoding issue
    - BUG/MINOR: peers: don't count track-sc multiple times on errors
    - MINOR: standard: add function "escape_string"
    - BUG/MEDIUM: log: use function "escape_string" instead of "escape_chunk"
    - MINOR: tcp: Return TCP statistics like RTT and RTT variance
    - DOC: lua: remove old functions
    - BUG/MEDIUM: lua: somme HTTP manipulation functions are called without valid requests
    - DOC: fix json converter example and error message
    - BUG/MEDIUM: stream-int: completely detach connection on connect error
    - DOC: minor typo fixes to improve HTML parsing by haproxy-dconv
    - BUILD: make proto_tcp.c compatible with musl library
    - BUG/MAJOR: compression: initialize avail_in/next_in even during flush
    - BUG/MEDIUM: samples: make smp_dup() always duplicate the sample
    - MINOR: sample: implement smp_is_safe() and smp_make_safe()
    - MINOR: sample: provide smp_is_rw() and smp_make_rw()
    - BUG/MAJOR: server: the "sni" directive could randomly cause trouble
    - BUG/MEDIUM: stick-tables: do not fail on string keys with no allocated size
    - BUG/MEDIUM: stick-table: properly convert binary samples to keys
    - MINOR: sample: use smp_make_rw() in upper/lower converters
    - MINOR: tcp: add dst_is_local and src_is_local
    - BUG/MINOR: peers: some updates are pushed twice after a resync.
    - BUILD: protocol: fix some build errors on OpenBSD
    - BUILD: log: iovec requires to include sys/uio.h on OpenBSD
    - BUILD: tcp: do not include netinet/ip.h for IP_TTL
    - BUILD: connection: fix build breakage on openbsd due to missing in_systm.h
    - BUILD: checks: remove the last strcat and eliminate a warning on OpenBSD
    - BUILD: tcp: define SOL_TCP when only IPPROTO_TCP exists
    - BUILD: compression: remove a warning when no compression lib is used
    - BUILD: poll: remove unused hap_fd_isset() which causes a warning with clang
    - MINOR: tcp: add further tcp info fetchers
    - BUG/MINOR: peers: empty chunks after a resync.
    - BUG/MAJOR: stick-counters: possible crash when using sc_trackers with wrong table
    - MINOR: standard.c: ipcmp() function to compare 2 IP addresses stored in 2 struct sockaddr_storage
    - MINOR: standard.c: ipcpy() function to copy an IP address from a struct sockaddr_storage into an other one
    - MAJOR: listen section: don't use first bind port anymore when no server ports are provided
2016-08-14 12:25:21 +02:00
Joe Williams
30fcd39f35 MINOR: tcp: add further tcp info fetchers
Adding on to Thierry's work (http://git.haproxy.org/?p=haproxy.git;h=6310bef5)
I have added a few more fetchers for counters based on the tcp_info struct
maintained by the kernel :

  fc_unacked, fc_sacked, fc_retrans, fc_fackets, fc_lost,
  fc_reordering

Two fields were not added because they're version-dependant :
  fc_rcv_rtt, fc_total_retrans

The fields name depend on the operating system. FreeBSD and NetBSD prefix
all the field names with "__" so we have to rely on a few #ifdef for
portability.
2016-08-10 23:02:46 +02:00
Willy Tarreau
16e015635c MINOR: tcp: add dst_is_local and src_is_local
It is sometimes needed in application server environments to easily tell
if a source is local to the machine or a remote one, without necessarily
knowing all the local addresses (dhcp, vrrp, etc). Similarly in transparent
proxy configurations it is sometimes desired to tell the difference between
local and remote destination addresses.

This patch adds two new sample fetch functions for this :

dst_is_local : boolean
  Returns true if the destination address of the incoming connection is local
  to the system, or false if the address doesn't exist on the system, meaning
  that it was intercepted in transparent mode. It can be useful to apply
  certain rules by default to forwarded traffic and other rules to the traffic
  targetting the real address of the machine. For example the stats page could
  be delivered only on this address, or SSH access could be locally redirected.
  Please note that the check involves a few system calls, so it's better to do
  it only once per connection.

src_is_local : boolean
  Returns true if the source address of the incoming connection is local to the
  system, or false if the address doesn't exist on the system, meaning that it
  comes from a remote machine. Note that UNIX addresses are considered local.
  It can be useful to apply certain access restrictions based on where the
  client comes from (eg: require auth or https for remote machines). Please
  note that the check involves a few system calls, so it's better to do it only
  once per connection.
2016-08-09 16:50:08 +02:00