Commit Graph

1411 Commits

Author SHA1 Message Date
Willy Tarreau 7a0139e2b6 DOC: clarify the agent-check status line syntax
Nick Ramirez reported that the wording is confusing and lets one think
that the CR or LF are both optional, which is not the case (either is
optional). Let's reformulate this.
2018-12-16 08:42:56 +01:00
Olivier Houchard a4d4fdfaa3 MEDIUM: sessions: Don't keep an infinite number of idling connections.
In session, don't keep an infinite number of connection that can idle.
Add a new frontend parameter, "max-session-srv-conns" to set a max number,
with a default value of 5.
2018-12-15 23:50:10 +01:00
Olivier Houchard b7b3faa79c MEDIUM: servers: Replace idle-timeout with pool-purge-delay.
Instead of the old "idle-timeout" mechanism, add a new option,
"pool-purge-delay", that sets the delay before purging idle connections.
Each time the delay happens, we destroy half of the idle connections.
2018-12-15 23:50:09 +01:00
Olivier Houchard 006e3101f9 MEDIUM: servers: Add a command to limit the number of idling connections.
Add a new command, "pool-max-conn" that sets the maximum number of connections
waiting in the orphan idling connections list (as activated with idle-timeout).
Using "-1" means unlimited. Using pools is now dependant on this.
2018-12-15 23:50:08 +01:00
Olivier Houchard 86006a561e MEDIUM: proxy: Set http-reuse safe as default.
Change the default for http-reuse from "never" to "safe", as it has been
the recommended setting for a few versions now and backend H2 makes little
sense without it.

Some warnings were removed from the config parser since it can dynamically
be disabled depending on the server's configuration, so there's no need to
disable it on a whole backend just for one server.
2018-12-15 23:50:08 +01:00
Christopher Faulet 27d93c3f94 BUG/MAJOR: compression/cache: Make it really works with these both filters
Caching the response with the compression enabled was totally broken. To fix the
problem, the compression must be done after caching the response. Otherwise it
needs to change the cache to store compressed and uncompressed objects for the
same ressource. So, because it is not possible for now, it is forbidden to
declare the compression filter before the cache one. To ease the configuration,
both can be implicitly declared (without "filter" keyword). The compression will
automatically be inserted after the cache.

Then, to make it works this way, the compression filter has been slighly
modified. Now, the response headers are updated after http-response rules
evaluations, instead of before. So, if the response contains a "Content-length"
header, it will be kept with the response stored in the cache. So this cached
response will be able to be served to clients not supporting the compression at
all.
2018-12-15 23:50:07 +01:00
William Lallemand a57b7e33ef MINOR: cli: implements 'reload' on master CLI
The reload command reload the haproxy master like it is done with a kill
-USR2 on the master process.
2018-12-15 13:33:49 +01:00
Willy Tarreau 52880f97b9 MINOR: mworker/cli: indicate in the master prompt when a reload failed
If a reload was issued to the master process and failed, it is critical
that the admin sees it because it means that the saved configuration
does not work anymore and might not be usable after a full restart. For
this reason in this case we modify the "master" prompt to explicitly
indicate that a reload failed.
2018-12-15 13:30:03 +01:00
William Lallemand b7ea141cbb MEDIUM: cli: handle CLI level from the master CLI
Handle the CLI level in the master CLI. In order to do this, the master
CLI stores the level in the stream. Each command are prefixed by a
"user" or "operator" command before they are forwarded to the target
CLI.

The level can be configured in the haproxy program arguments with the
level keyword: -S /tmp/sock,level,admin -S /tmp/sock2,level,user.
2018-12-13 09:45:16 +01:00
William Lallemand 67a234f399 MEDIUM: cli: show and change CLI permissions
Implement "show cli level" which show the level of the current CLI
session.

Implement "operator" and "user" which lower the permissions of the
current CLI session.
2018-12-13 09:45:16 +01:00
William Lallemand 256bf0d37b MINOR: cli: change 'show proc' output of old processes
Change the output of the relative pid for the old processes, displays
"[was: X]" instead of just "X" which was confusing if you want to
connect to the CLI of an old PID.
2018-12-12 13:54:17 +01:00
William Lallemand 142db37043 DOC: master CLI documentation in management.txt
Documentation which explains the basic usage of the master CLI.
2018-12-12 11:47:24 +01:00
Willy Tarreau c77d364905 MINOR: config: round up global.tune.bufsize to the next multiple of 2 void*
Since HTX casts the buffer to a struct and stores relative pointers at the
end, it is mandatory that its end is properly aligned. This patch enforces
a buffer size rounding up to the next multiple of two void*, thus 8 on
32-bit and 16 on 64-bit, to match what malloc() already does on the beginning
of the buffer. In pratice it will never be really noticeable since default
sizes already are such multiples.
2018-12-12 06:19:42 +01:00
Christopher Faulet 99a17a2d91 MEDIUM: cache: Require an explicit filter declaration if other filters are used
As for the compression filter, the cache filter must be explicitly declared
(using the filter keyword) if other filters than cache are used. It is mandatory
to explicitly define the filters order.

Documentation has been updated accordingly.
2018-12-11 17:09:31 +01:00
Jérôme Magnin b36a6d21db DOC: refer to check-sni in the documentation of sni
Make it obvious in the description of the sni directive that it can
not be used for health checks, and refer to the appropriate directive.

This can be backported to 1.8 as check-sni appeared in 1.8.
2018-12-10 05:21:02 +01:00
Jérôme Magnin ae9bb76001 DOC: clarify that check-sni needs an argument.
Make it more obvious that check-sni requires an argument, and that
it can only be a string. Also refer to sni for proxied traffic.

This can be backported to 1.8 as check-sni appeared in 1.8.
2018-12-10 05:20:00 +01:00
Willy Tarreau 72e9227385 [RELEASE] Released version 1.9-dev10
Released version 1.9-dev10 with the following main changes :
    - MINOR: htx: Rename functions htx_*_to_str() to be H1 specific
    - BUG/MINOR: htx: Force HTTP/1.1 on H1 formatting when version is 1.1 or above
    - BUG/MINOR: fix ssl_fc_alpn and actually add ssl_bc_alpn
    - BUG/MEDIUM: mworker: stop proxies which have no listener in the master
    - BUG/MEDIUM: h1: Destroy a connection after detach if it has no owner.
    - BUG/MEDIUM: h2: Don't forget to wake the tasklet after shutr/shutw.
    - BUG/MINOR: flt_trace/compression: Use the right flag to add the HTX support
    - BUG/MEDIUM: stream_interface: Make REALLY sure we read all the data.
    - MEDIUM: mux-h1: Revamp the way subscriptions are handled.
    - BUG/MEDIUM: mux-h1: Always set CS_FL_RCV_MORE when data are received in h1_recv()
    - MINOR: mux-h1: Make sure to return 1 in h1_recv() when needed
    - BUG/MEDIUM: mux-h1: Release the mux H1 in h1_process() if there is no h1s
    - BUG/MINOR: proto_htx: Truncate the request when an error is detected
    - BUG/MEDIUM: h2: When sending in HTX, make sure the caller knows we sent all.
    - BUG/MEDIUM: mux-h2: properly update the window size in HTX mode
    - BUG/MEDIUM: mux-h2: make sure to always report HTX EOM when consumed by headers
    - BUG/MEDIUM: mux-h2: stop sending HTX once the mux is blocked
    - BUG/MEDIUM: mux-h2: don't send more HTX data than requested
    - MINOR: mux-h2: stop on non-DATA and non-EOM HTX blocks
    - BUG/MEDIUM: h1: Correctly report used data with no len.
    - MEDIUM: h1: Realign the ibuf before calling rcv_buf if needed.
    - BUG/MEDIUM: mux_pt: Always set CS_FL_RCV_MORE.
    - MINOR: htx: make htx_from_buf() adjust the size only on new buffers
    - MINOR: htx: add buf_room_for_htx_data() to help optimize buffer transfers
    - MEDIUM: mux-h1: make use of buf_room_for_htx_data() instead of b_room()
    - MEDIUM: mux-h1: attempt to zero-copy Rx DATA transfers
    - MEDIUM: mux-h1: avoid a double copy on the Tx path whenever possible
    - BUG/MEDIUM: stream-int: don't mark as blocked an empty buffer on Rx
    - BUG/MINOR: mux-h1: Check h1m flags to set the server conn_mode on request path
    - MEDIUM: htx: Rework conversion from a buffer to an htx structure
    - MEDIUM: channel/htx: Add functions for forward HTX data
    - MINOR: mux-h1: Don't adjust anymore the amount of data sent in h1_snd_buf()
    - CLEANUP: htx: Fix indentation here and there in HTX files
    - MINOR: mux-h1: Allow partial data consumption during outgoing data processing
    - BUG/MEDIUM: mux-h2: use the correct offset for the HTX start line
    - BUG/MEDIUM: mux-h2: stop sending using HTX on errors
    - MINOR: mux-h1: Drain obuf if the output is closed after sending data
    - BUG/MEDIUM: mworker: stop every tasks in the master
    - BUG/MEDIUM: htx: Set the right start-line offset after a defrag
    - BUG/MEDIUM: stream: Don't dereference s->txn when it is not there yet.
    - BUG/MEDIUM: connections: Reuse an already attached conn_stream.
    - MINOR: stream-int: add a new blocking condition on the remote connection
    - BUG/MEDIUM: stream-int: don't attempt to receive if the connection is not established
    - BUG/MEDIUM: lua: block on remote connection establishment
    - BUG/MEDIUM: mworker: fix several typos in mworker_cleantasks()
    - SCRIPTS/REGTEST: merge grep+sed into sed in run-regtests
    - BUG/MEDIUM: connections: Split CS_FL_RCV_MORE into 2 flags.
    - BUG/MEDIUM: h1: Don't free the connection if it's an outgoing connection.
    - BUG/MEDIUM: h1: Set CS_FL_REOS if we had a read0.
    - BUG/MEDIUM: mux-h1: Be sure to have a conn_stream to set CS_FL_REOS in h1_recv
    - REGTEST: Move LUA reg test 4 to level 1.
    - MINOR: ist: add functions to copy/uppercase/lowercase into a buffer or string
    - MEDIUM: ist: always turn header names to lower case
    - MINOR: h2: don't turn HTX header names to lower case anymore
    - MEDIUM: ist: use local conversion arrays to case conversion
    - MINOR: htx: switch to case sensitive search of lower case header names
    - MINOR: mux-h1: Set CS_FL_EOS when read0 is detected and no data are pending
    - BUG/MINOR: stream-int: Process read0 even if no data was received in si_cs_recv
    - REGTEST: fix the Lua test file name in test lua/h00002 :-)
    - REGTEST: add a basic test for HTTP rules manipulating headers
    - BUG/MEDIUM: sample: Don't treat SMP_T_METH as SMP_T_STR.
    - MINOR: sample: add bc_http_major
    - BUG/MEDIUM: htx: fix typo in htx_replace_stline() making it fail all the time
    - REGTEST: make the HTTP rules test compatible with HTTP/2 as well
    - BUG/MEDIUM: h2: Don't try to chunk data when using HTX.
    - MINOR: compiler: add a new macro ALREADY_CHECKED()
    - BUILD: h2: mark the start line already checked to avoid warnings
    - BUG/MINOR: mux-h1: Remove the connection header when it is useless
2018-12-08 16:20:55 +01:00
Jérôme Magnin 8657742092 MINOR: sample: add bc_http_major
This adds the sample fetch bc_http_major. It returns the backend connection's HTTP
version encoding, which may be 1 for HTTP/0.9 to HTTP/1.1 or 2 for HTTP/2.0. It is
based on the on-wire encoding, and not the version present in the request header.
2018-12-07 15:34:39 +01:00
Willy Tarreau da7e3be36f [RELEASE] Released version 1.9-dev9
Released version 1.9-dev9 with the following main changes :
    - BUILD/MINOR: ssl: fix build with non-alpn/non-npn libssl
    - BUG/MINOR: mworker: Do not attempt to close(2) fd -1
    - BUILD: compression: fix build error with DEFAULT_MAXZLIBMEM
    - MINOR: compression: always create the compression pool
    - BUG/MEDIUM: mworker: fix FD leak upon reload
    - BUILD: htx: fix fprintf format inconsistency on 32-bit platforms
    - BUILD: buffers: buf.h requires unistd to get ssize_t on libmusl
    - MINOR: initcall: introduce a way to register init functions to call at boot
    - MINOR: init: process all initcalls in order at boot time
    - MEDIUM: init: convert all trivial registration calls to initcalls
    - MINOR: thread: provide a set of lock initialisers
    - MINOR: threads: add new macros to declare self-initializing locks
    - MEDIUM: init: use self-initializing spinlocks and rwlocks
    - MINOR: initcall: apply initcall to all register_build_opts() calls
    - MINOR: initcall: use initcalls for most post_{check,deinit} and per_thread*
    - MINOR: initcall: use initcalls for section parsers
    - MINOR: memory: add a callback function to create a pool
    - MEDIUM: init: use initcall for all fixed size pool creations
    - MEDIUM: memory: use pool_destroy_all() to destroy all pools on deinit()
    - MEDIUM: initcall: use initcalls for a few initialization functions
    - MEDIUM: memory: make the pool cache an array and not a thread_local
    - MINOR: ssl: free ctx when libssl doesn't support NPN
    - BUG/MINOR: proto_htx: only mark connections private if NTLM is detected
    - MINOR: h2: make struct h2_ops static
    - BUG/MEDIUM: mworker: avoid leak of client socket
    - REORG: mworker: declare master variable in global.h
    - BUG/MEDIUM: listeners: CLOEXEC flag is not correctly set
    - CLEANUP: http: Fix typo in init_http's comment
    - BUILD: Makefile: Disable -Wcast-function-type if it exists.
    - BUG/MEDIUM: h2: Don't bogusly error if the previous stream was closed.
    - REGTEST/MINOR: script: add run-regtests.sh script
    - REGTEST: Add a basic test for the cache.
    - BUG/MEDIUM: mux_pt: Don't forget to unsubscribe() on attach.
    - BUG/MINOR: ssl: ssl_sock_parse_clienthello ignores session id
    - BUG/MEDIUM: connections: Wake the stream once the mux is chosen.
    - BUG/MEDIUM: connections: Don't forget to detach the connection from the SI.
    - BUG/MEDIUM: stream_interface: Don't check if the handshake is done.
    - BUG/MEDIUM: stream_interface: Make sure we read all the data available.
    - BUG/MEDIUM: h2: Call h2_process() if there's an error on the connection.
    - REGTEST: Fix several issues.
    - REGTEST: lua: check socket functionality from a lua-task
    - BUG/MEDIUM: session: Remove the session from the session_list in session_free.
    - BUG/MEDIUM: streams: Don't assume we have a CS in sess_update_st_con_tcp.
    - BUG/MEDIUM: connections: Don't assume we have a mux in connect_server().
    - BUG/MEDIUM: connections: Remove the connection from the idle list before destroy.
    - BUG/MEDIUM: session: properly clean the outgoing connection before freeing.
    - BUG/MEDIUM: mux_pt: Don't try to send if handshake is not done.
    - MEDIUM: connections: Put H2 connections in the idle list if http-reuse always.
    - MEDIUM: h2: Destroy a connection with no stream if it has no owner.
    - MAJOR: sessions: Store multiple outgoing connections in the session.
    - MEDIUM: session: Steal owner-less connections on end of transaction.
    - MEDIUM: server: Be smarter about deciding to reuse the last server.
    - BUG/MEDIUM: Special-case http_proxy when dealing with outgoing connections.
    - BUG/MINOR: cfgparse: Fix transition between 2 sections with the same name
    - BUG/MINOR: http: Use out buffer instead of trash to display error snapshot
    - BUG/MINOR: htx: Fix block size calculation when a start-line is added/replaced
    - BUG/MINOR: mux-h1: Fix processing of "Connection: " header on outgoing messages
    - BUG/MEDIUM: mux-h1: Reset the H1 parser when an outgoing message is processed
    - BUG/MINOR: proto_htx: Send outgoing data to client to start response processing
    - BUG/MINOR: htx: Stop a header or a start line lookup on the first EOH or EOM
    - BUG/MINOR: connection: report mux modes when HTX is supported
    - MINOR: htx: add a function to cut the beginning of a DATA block
    - MEDIUM: conn_stream: Add a way to get mux's info on a CS from the upper layer
    - MINOR: mux-h1: Implement get_cs_info() callback
    - MINOR: stream: Rely on CS's info if it exists and fallback on session's ones
    - MINOR: proto_htx: Use conn_stream's info to set t_idle duration when possible
    - MINOR: mux-h1: Don't rely on the stream anymore in h1_set_srv_conn_mode()
    - MINOR: mux-h1: Write last chunk and trailers if not found in the HTX message
    - MINOR: mux-h1: Be prepare to fail when EOM is added during trailers parsing
    - MINOR: mux-h1: Subscribe to send in h1_snd_buf() when not all data have been sent
    - MINOR: mux-h1: Consume channel's data in a loop in h1_snd_buf()
    - MEDIUM: mux-h1: Add keep-alive outgoing connections in connections list
    - MINOR: htx: Add function to add an HTX block just before another one
    - MINOR: htx: Add function to iterate on an HTX message using HTX blocks
    - MINOR: htx: Add a function to find the HTX block corresponding to a data offset
    - MINOR: stats: Don't add end-of-data marker and trailers in the HTX response
    - MEDIUM: htx: Change htx_sl to be a struct instead of an union
    - MINOR: htx: Add the start-line offset for the HTX message in the HTX structure
    - MEDIUM: htx: Don't rely on h1_sl anymore except during H1 header parsing
    - MINOR: proto-htx: Use the start-line flags to set the HTTP messsage ones
    - MINOR: htx: Add BODYLESS flags on the HTX start-line and the HTTP message
    - MINOR: proto_htx: Use full HTX messages to send 100-Continue responses
    - MINOR: proto_htx: Use full HTX messages to send 103-Early-Hints responses
    - MINOR: proto_htx: Use full HTX messages to send 401 and 407 responses
    - MINOR: proto_htx: Send valid HTX message when redir mode is enabled on a server
    - MINOR: proto_htx: Send valid HTX message to send 30x responses
    - MEDIUM: proto_htx: Convert all HTTP error messages into HTX
    - MINOR: mux-h1: Process conn_mode on the EOH when no connection header is found
    - MINOR: mux-h1: Change client conn_mode on an explicit close for the response
    - MINOR: mux-h1: Capture bad H1 messages
    - MAJOR: filters: Adapt filters API to be compatible with the HTX represenation
    - MEDIUM: proto_htx/filters: Add data filtering during the forwarding
    - MINOR: flt_trace: Adapt to be compatible with the HTX representation
    - MEDIUM: compression: Adapt to be compatible with the HTX representation
    - MINOR: h2: implement H2->HTX request header frame transcoding
    - MEDIUM: mux-h2: register mux for both HTTP and HTX modes
    - MEDIUM: mux-h2: make h2_rcv_buf() support HTX transfers
    - MEDIUM: mux-h2: make h2_snd_buf() HTX-aware
    - MEDIUM: mux-h2: add basic H2->HTX transcoding support for headers
    - MEDIUM: mux-h2: implement emission of H2 headers frames from HTX blocks
    - MEDIUM: mux-h2: implement the emission of DATA frames from HTX DATA blocks
    - MEDIUM: mux-h2: support passing H2 DATA frames to HTX blocks
    - BUG/MINOR: cfgparse: Fix the call to post parser of the last sections parsed
    - BUG/MEDIUM: mux-h2: don't lose the first response header in HTX mode
    - BUG/MEDIUM: mux-h2: remove the HTX EOM block on H2 response headers
    - MINOR: listener: the mux_proto entry in the bind_conf is const
    - MINOR: connection: create conn_get_best_mux_entry()
    - MINOR: server: the mux_proto entry in the server is const
    - MINOR: config: make sure to associate the proper mux to bind and servers
    - MINOR: hpack: add ":path" to the list of common header fields
    - MINOR: h2: add new functions to produce an HTX message from an H2 response
    - MINOR: mux-h2: mention that the mux is compatible with both sides
    - MINOR: mux-h2: implement an outgoing stream allocator : h2c_bck_stream_new()
    - MEDIUM: mux-h2: start to create the outgoing mux
    - MEDIUM: mux-h2: implement encoding of H2 request on the backend side
    - MEDIUM: mux-h2: make h2_frt_decode_headers() direction-agnostic
    - MEDIUM: mux-h2: make h2_process_demux() capable of processing responses as well
    - MEDIUM: mux-h2: Implement h2_attach().
    - MEDIUM: mux-h2: Don't bother flagging outgoing connections as TOOMANY.
    - REGTEST: Fix LEVEL 4 script 0 of "connection" module.
    - MINOR: connection: Fix a comment.
    - MINOR: mux: add a "max_streams" method.
    - MEDIUM: servers: Add a way to keep idle connections alive.
    - CLEANUP: fix typos in the htx subsystem
    - CLEANUP: Fix typo in the chunk headers file
    - CLEANUP: Fix typos in the h1 subsystem
    - CLEANUP: Fix typos in the h2 subsystem
    - CLEANUP: Fix a typo in the mini-clist header
    - CLEANUP: Fix a typo in the proto_htx subsystem
    - CLEANUP: Fix typos in the proto_tcp subsystem
    - CLEANUP: Fix a typo in the signal subsystem
    - CLEANUP: Fix a typo in the session subsystem
    - CLEANUP: Fix a typo in the queue subsystem
    - CLEANUP: Fix typos in the shctx subsystem
    - CLEANUP: Fix typos in the socket pair protocol subsystem
    - CLEANUP: Fix typos in the map management functions
    - CLEANUP: Fix typo in the fwrr subsystem
    - CLEANUP: Fix typos in the cli subsystem
    - CLEANUP: Fix typo in the 51d subsystem
    - CLEANUP: Fix a typo in the base64 subsystem
    - CLEANUP: Fix a typo in the connection subsystem
    - CLEANUP: Fix a typo in the protocol header file
    - CLEANUP: Fix a typo in the checks header file
    - CLEANUP: Fix typos in the file descriptor subsystem
    - CLEANUP: Fix a typo in the listener subsystem
    - BUG/MINOR: lb-map: fix unprotected update to server's score
    - BUILD: threads: fix minor build warnings when threads are disabled
2018-12-02 19:31:37 +01:00
Olivier Houchard 0c18a6fe34 MEDIUM: servers: Add a way to keep idle connections alive.
Add a new keyword for servers, "idle-timeout". If set, unused connections are
kept alive until the timeout happens, and will be picked for reuse if no
other connection is available.
2018-12-02 18:16:53 +01:00
Willy Tarreau 0108d90c6c MEDIUM: init: convert all trivial registration calls to initcalls
This switches explicit calls to various trivial registration methods for
keywords, muxes or protocols from constructors to INITCALL1 at stage
STG_REGISTER. All these calls have in common to consume a single pointer
and return void. Doing this removes 26 constructors. The following calls
were addressed :

- acl_register_keywords
- bind_register_keywords
- cfg_register_keywords
- cli_register_kw
- flt_register_keywords
- http_req_keywords_register
- http_res_keywords_register
- protocol_register
- register_mux_proto
- sample_register_convs
- sample_register_fetches
- srv_register_keywords
- tcp_req_conn_keywords_register
- tcp_req_cont_keywords_register
- tcp_req_sess_keywords_register
- tcp_res_cont_keywords_register
- flt_register_keywords
2018-11-26 19:50:32 +01:00
Willy Tarreau 0b936ad946 [RELEASE] Released version 1.9-dev8
Released version 1.9-dev8 with the following main changes :
    - REORG: config: extract the global section parser into cfgparse-global
    - REORG: config: extract the proxy parser into cfgparse-listen.c
    - BUILD: update the list of supported targets and compilers in makefile and readme
    - BUILD: reorder the objects in the makefile
    - BUILD: Makefile: make "V=1" show some of the commands that are executed
    - BUILD: Makefile: add the quiet mode to a few more targets
    - BUILD: Makefile: add "$(Q)" to clean, tags and cscope targets
    - BUILD: Makefile: switch to quiet mode by default for CC/LD/AR
    - MINOR: cli: format `show proc` to be more readable
    - MINOR: cli: displays uptime in `show proc`
    - MINOR: cli: show master information in 'show proc'
    - BUG/MEDIUM: hpack: fix encoding of "accept-ranges" field
    - MAJOR: mux-h1: Remove the rxbuf and decode HTTP messages in channel's buffer
    - BUG/MINOR: mux-h1: Enable keep-alive on server side
    - BUG/MEDIUM: mux-h1: Fix freeze when the kernel splicing is used
    - BUG/MEDIUM: mux-h1: Don't set the flag CS_FL_RCV_MORE when nothing was parsed
    - BUG/MINOR: stats/htx: Remove channel's output when the request is eaten
    - BUG/MINOR: proto_htx: Fix request/response synchronisation on error
    - MINOR: stream-int: Notify caller when an error is reported after a rcv_pipe()
    - MINOR: stream-int: Notify caller when an error is reported after a rcv_buf()
    - BUG/MINOR: stream-int: Don't call snd_buf() if there are still data in the pipe
    - MINOR: stream-int: remove useless checks on CS and conn flags in si_cs_send()
    - BUG/MINOR: config: Be aware of the HTX during the check of mux protocols
    - BUG/MINOR: mux-htx: Fix bad test on h1c flags in h1_recv_allowed()
    - MEDIUM: mworker: wait mode use standard init code path
    - MINOR: log: introduce ha_notice()
    - MINOR: mworker: use ha_notice to announce a new worker
    - BUG/MEDIUM: http_fetch: Make sure name is initialized before http_find_header.
    - MINOR: cli: add mworker_accept_wrapper to 'show fd'
    - MEDIUM: signal: signal_unregister() removes every handlers
    - BUG/MEDIUM: mworker: unregister the signals of main()
    - MINOR: cli: add a few missing includes in proto/cli.h
    - REORG: time/activity: move activity measurements to activity.{c,h}
    - MINOR: activity: report the average loop time in "show activity"
    - MINOR: activity: add configuration and CLI support for "profiling.tasks"
    - MEDIUM: tasks: collect per-task CPU time and latency
    - MINOR: sample: add cpu_calls, cpu_ns_avg, cpu_ns_tot, lat_ns_avg, lat_ns_tot
    - MINOR: cli/activity: rename the stolen CPU time fields to mention milliseconds
    - BUG/MINOR: cli: Fix memory leak
    - BUG/MINOR: mworker: fix FD leak and memory leak in error path
    - MINOR: poller: move the call of tv_update_date() back to the pollers
    - MINOR: polling: add an option to support busy polling
    - MINOR: server: Add "alpn" and "npn" keywords.
    - MEDIUM: connection: Don't bother reactivating polling after connection retry.
    - MAJOR: connections: Defer mux creation for outgoing connection if alpn is set.
    - MEDIUM: ssl: Add ssl_bc_alpn and ssl_bc_npn sample fetches.
    - MINOR: servers: Free [idle|safe|priv]_conns on exit.
    - REGTEST: add the option to test only a specific set of files
    - REGTEST: add a test for connections to a "dispatch" address
    - BUG/MEDIUM: connections: Don't reset the conn flags in *connect_server().
    - MINOR: server: Only defined conn_complete_server if USE_OPENSSL is set.
    - BUG/MEDIUM: servers: Don't check if we have a conn_stream too soon.
    - BUG/MEDIUM: sessions: Set sess->origin to NULL if the origin was destroyed.
    - MEDIUM: servers: Store the connection in the SI until we have a mux.
    - BUG/MEDIUM: h2: wake the processing task up after demuxing
    - BUG/MEDIUM: h2: restart demuxing after releasing buffer space
2018-11-25 09:16:46 +01:00
Olivier Houchard 6b77f49e78 MEDIUM: ssl: Add ssl_bc_alpn and ssl_bc_npn sample fetches.
Add 2 new sample fetches, ssl_bc_alpn and ssl_bc_npn, that provides the
ALPN and the NPN for an outgoing connection.
2018-11-22 19:52:44 +01:00
Olivier Houchard c756600103 MINOR: server: Add "alpn" and "npn" keywords.
Add new keywords to "server" lines, alpn and npn.
If set, when connecting through SSL, those alpn/npn will be negociated
during the SSL handshake.
2018-11-22 19:50:08 +01:00
Willy Tarreau beb859abce MINOR: polling: add an option to support busy polling
In some situations, especially when dealing with low latency on processors
supporting a variable frequency or when running inside virtual machines,
each time the process waits for an I/O using the poller, the processor
goes back to sleep or is offered to another VM for a long time, and it
causes excessively high latencies.

A solution to this provided by this patch is to enable busy polling using
a global option. When busy polling is enabled, the pollers never sleep and
loop over themselves waiting for an I/O event to happen or for a timeout
to occur. On multi-processor machines it can significantly overheat the
processor but it usually results in much lower latencies.

A typical test consisting in injecting traffic over a single connection at
a time over the loopback shows a bump from 4640 to 8540 connections per
second on forwarded connections, indicating a latency reduction of 98
microseconds for each connection, and a bump from 12500 to 21250 for
locally terminated connections (redirects), indicating a reduction of
33 microseconds.

It is only usable with epoll and kqueue because select() and poll()'s
API is not convenient for such usages, and the level of performance they
are used in doesn't benefit from this anyway.

The option, which obviously remains disabled by default, can be turned
on using "busy-polling" in the global section, and turned off later
using "no busy-polling". Its status is reported in "show info" to help
troubleshooting suspicious CPU spikes.
2018-11-22 19:47:30 +01:00
Willy Tarreau 70fe94419c MINOR: sample: add cpu_calls, cpu_ns_avg, cpu_ns_tot, lat_ns_avg, lat_ns_tot
These sample fetch keywords report performance metrics about the task calling
them. They are useful to report in logs which requests consume too much CPU
time and what negative performane impact it has on other requests. Typically
logging cpu_ns_avg and lat_ns_avg will show culprits and victims.
2018-11-22 16:07:39 +01:00
Willy Tarreau 75c62c2793 MINOR: activity: add configuration and CLI support for "profiling.tasks"
This is a new global setting which enables or disables CPU profiling
per task. For now it only sets/resets the variable based on the global
option "profiling.tasks" and supports showing it as well as setting it
from the CLI using "show profiling" and "set profiling". The option will
be used by a future commit. It was done in a way which should ease future
addition of profiling options.
2018-11-22 11:48:51 +01:00
Willy Tarreau 5c0e41b7cb [RELEASE] Released version 1.9-dev7
Released version 1.9-dev7 with the following main changes :
    - BUILD: cache: fix a build warning regarding too large an integer for the age
    - CLEANUP: fix typos in the comments of the Makefile
    - CLEANUP: fix a typo in a comment for the contrib/halog subsystem
    - CLEANUP: fix typos in comments for the contrib/modsecurity subsystem
    - CLEANUP: fix typos in comments for contrib/spoa_example
    - CLEANUP: fix typos in comments for contrib/wireshark-dissectors
    - DOC: Fix typos in README and CONTRIBUTING
    - MINOR: log: slightly improve error message syntax on log failure
    - DOC: logs: the format directive was missing from the second log part
    - MINOR: log: report the number of dropped logs in the stats
    - MEDIUM: log: add support for logging to existing file descriptors
    - MEDIUM: log: support a new "short" format
    - MEDIUM: log: add a new "raw" format
    - BUG/MEDIUM: stream-int: change the way buffer room is requested by a stream-int
    - BUG/MEDIUM: stream-int: convert some co_data() checks to channel_is_empty()
    - MINOR: namespaces: don't build namespace.c if disabled
    - BUILD/MEDIUM: threads/affinity: DragonFly build fix
    - MINOR: http: Add new "early-hint" http-request action.
    - MINOR: http: Make new "early-hint" http-request action really be parsed.
    - MINOR: http: Implement "early-hint" http request rules.
    - MINOR: doc: Add information about "early-hint" http-request action.
    - DOC: early-hints: fix truncated line.
    - MINOR: mworker: only close std{in,out,err} in daemon mode
    - BUG/MEDIUM: log: don't CLOEXEC the inherited FDs
    - BUG/MEDIUM: Make sure stksess is properly aligned.
    - BUG/MEDIUM: stream-int: make failed splice_in always subscribe to recv
    - BUG/MEDIUM: stream-int: clear CO_FL_WAIT_ROOM after splicing data in
    - BUG/MINOR: stream-int: make sure not to go through the rcv_buf path after splice()
    - CONTRIB: debug: fix build related to conn_stream flags change
    - REGTEST: fix scripts 1 and 3 to accept development version
    - BUG/MINOR: http_fetch: Remove the version part when capturing the request uri
    - MINOR: http: Regroup return statements of http_req_get_intercept_rule at the end
    - MINOR: http: Regroup return statements of http_res_get_intercept_rule at the end
    - BUG/MINOR: http: Be sure to sent fully formed HTTP 103 responses
    - MEDIUM: jobs: support unstoppable jobs for soft stop
    - MEDIUM: listeners: support unstoppable listener
    - MEDIUM: cli: worker socketpair is unstoppable
    - BUG/MINOR: stream-int: set SI_FL_WANT_PUT in sess_establish()
    - MINOR: stream: move the conn_stream specific calls to the stream-int
    - BUG/MINOR: config: Copy default error messages when parsing of a backend starts
    - CLEANUP: h2: minimum documentation for recent API changes
    - MINOR: mux: implement a get_first_cs() method
    - MINOR: stream-int: make conn_si_send_proxy() use cs_get_first()
    - MINOR: stream-int: relax the forwarding rules in stream_int_notify()
    - MINOR: stream-int: expand the flags to 32-bit
    - MINOR: stream-int: rename SI_FL_WAIT_ROOM to SI_FL_RXBLK_ROOM
    - MINOR: stream-int: introduce new SI_FL_RXBLK flags
    - MINOR: stream-int: add new functions si_{rx,tx}_{blocked,endp_ready}()
    - MINOR: stream-int: replace SI_FL_WANT_PUT with !SI_FL_RX_WAIT_EP
    - MINOR: stream-int: use si_rx_blocked()/si_tx_blocked() to check readiness
    - MEDIUM: stream-int: use si_rx_buff_{rdy,blk} to report buffer readiness
    - MINOR: stream-int: replace si_{want,stop}_put() with si_rx_endp_{more,done}()
    - MEDIUM: stream-int: update the endp polling status only at the end of si_cs_recv()
    - MINOR: stream-int: make si_sync_recv() simply check ENDP before si_cs_recv()
    - MINOR: stream-int: automatically mark applets as ready if they block on the channel
    - MEDIUM: stream-int: fix the si_cant_put() calls used for end point readiness
    - MEDIUM: stream-int: fix the si_cant_put() calls used for buffer readiness
    - MEDIUM: stream-int: use si_rx_shut_blk() to indicate the SI is closed
    - MEDIUM: stream-int: unconditionally call si_chk_rcv() in update and notify
    - MEDIUM: stream-int: make use of si_rx_chan_{rdy,blk} to control the stream-int from the channel
    - MINOR: stream-int: replace si_cant_put() with si_rx_room_{blk,rdy}()
    - MEDIUM: connections: Wait until the connection is established to try to recv.
    - MEDIUM: mux: Teach the mux_pt how to deal with idle connections.
    - MINOR: mux: Add a new "avail_streams" method.
    - MINOR: mux: Add a destroy() method.
    - MINOR: sessions: Start to store the outgoing connection in sessions.
    - MAJOR: connections: Detach connections from streams.
    - MINOR: conn_stream: Add a flag to notify the mux it should flush its buffers
    - MINOR: htx: Add proto_htx.c file
    - MINOR: conn_stream: Add a flag to notify the mux it must respect the reserve
    - MINOR: http: Add standalone functions to parse a start-line or a header
    - MINOR: http: Call http_send_name_header with the stream instead of the txn
    - MINOR: conn_stream: Add a flag to notify the SI some data were received
    - MINOR: http: Add macros to check if a stream uses the HTX representation
    - MEDIUM: proto_htx: Add HTX analyzers and use it when the mux H1 is used
    - MEDIUM: mux-h1: Add dummy mux to handle HTTP/1.1 connections
    - MEDIUM: mux-h1: Add parsing of incoming and ougoing HTTP messages
    - MAJOR: mux-h1/proto_htx: Handle keep-alive connections in the mux
    - MEDIUM: mux-h1: Add support of the kernel TCP splicing to forward data
    - MEDIUM: htx: Add API to deal with the internal representation of HTTP messages
    - MINOR: http_htx: Add functions to manipulate HTX messages in http_htx.c
    - MINOR: proto_htx: Add some functions to handle HTX messages
    - MAJOR: mux-h1/proto_htx: Switch mux-h1 and HTX analyzers on the HTX representation
    - MINOR: http_htx: Add functions to replace part of the start-line
    - MINOR: http_htx: Add functions to retrieve a specific occurrence of a header
    - MINOR: proto_htx: Rewrite htx_apply_redirect_rule to handle HTX messages
    - MINOR: proto_htx: Add the internal function htx_del_hdr_value
    - MINOR: proto_htx: Add the internal function htx_fmt_res_line
    - MINOR: proto_htx: Add functions htx_transform_header and htx_transform_header_str
    - MINOR: proto_htx: Add functions htx_req_replace_stline and htx_res_set_status
    - MINOR: proto_htx: Add function to build and send HTTP 103 responses
    - MINOR: proto_htx: Add functions htx_req_get_intercept_rule and htx_res_get_intercept_rule
    - MINOR: proto_htx: Add functions to apply req* and rsp* rules on HTX messages
    - MINOR: proto_htx: Add functions to manage cookies on HTX messages
    - MINOR: proto_htx: Add functions to check the cacheability of HTX messages
    - MINOR: proto_htx: Add functions htx_send_name_header
    - MINOR: proto_htx: Add functions htx_perform_server_redirect
    - MINOR: proto_htx: Add functions to handle the stats applet
    - MEDIUM: proto_htx: Adapt htx_process_req_common to handle HTX messages
    - MEDIUM: proto_htx: Adapt htx_process_request to handle HTX messages
    - MINOR: proto_htx: Adapt htx_process_tarpit to handle HTX messages
    - MEDIUM: proto_htx: Adapt htx_wait_for_request_body to handle HTX messages
    - MEDIUM: proto_htx: Adapt htx_process_res_common to handle HTX messages
    - MINOR: http_fetch: Add smp_prefetch_htx
    - MEDIUM: http_fetch: Adapt all fetches to handle HTX messages
    - MEDIUM: mux-h1: Wait for connection establishment before consuming channel's data
    - MINOR: stats/htx: Adapt the stats applet to handle HTX messages
    - MINOR: stream: Don't reset sov value with HTX messages
    - MEDIUM: mux-h1: Handle errors and timeouts in the stream
    - MINOR: filters/htx: Forbid filters when the HTX is enabled on a proxy
    - MINOR: lua/htx: Forbid lua usage when the HTX is enabled on a proxy
    - CLEANUP: Fix some typos in the haproxy subsystem
    - CLEANUP: Fix typos in the dns subsystem
    - CLEANUP: Fix typos in the pattern subsystem
    - CLEANUP: fix 2 typos in the xxhash subsystem
    - CLEANUP: fix a few typos in the comments of the server subsystem
    - CLEANUP: fix a misspell in tests/filltab25.c
    - CLEANUP: fix a typo found in the stream subsystem
    - CLEANUP: fix typos in comments in ebtree
    - CLEANUP: fix typos in reg-tests
    - CLEANUP: fix typos in the comments of the vars subsystem
    - CLEANUP: fix typos in the hlua_fcn subsystem
    - CLEANUP: fix typos in the proto_http subsystem
    - CLEANUP: fix typos in the proxy subsystem
    - CLEANUP: fix typos in the ssl_sock subsystem
    - DOC: Fix typos in different subsections of the documentation
    - DOC: fix a few typos in the documentation
    - MINOR: Fix an error message thrown when we run out of memory
    - MINOR: Fix typos in error messages in the proxy subsystem
    - MINOR: fix typos in the examples files
    - CLEANUP: Fix a typo in the stats subsystem
    - CLEANUP: Fix typos in the acl subsystem
    - CLEANUP: Fix typos in the cache subsystem
    - CLEANUP: Fix typos in the cfgparse subsystem
    - CLEANUP: Fix typos in the filters subsystem
    - CLEANUP: Fix typos in the http subsystem
    - CLEANUP: Fix typos in the log subsystem
    - CLEANUP: Fix typos in the peers subsystem
    - CLEANUP: Fix typos in the regex subsystem
    - CLEANUP: Fix typos in the sample subsystem
    - CLEANUP: Fix typos in the spoe subsystem
    - CLEANUP: Fix typos in the standard subsystem
    - CLEANUP: Fix typos in the stick_table subsystem
    - CLEANUP: Fix typos in the task subsystem
    - MINOR: Fix typo in error message in the standard subsystem
    - CLEANUP: fix typos in the comments of hlua
    - MINOR: Fix typo in the error 500 output of hlua
    - MINOR: Fix a typo in a warning message in the spoe subsystem
2018-11-18 22:33:00 +01:00
Joseph Herlant 71b4b150f2 DOC: fix a few typos in the documentation
This commit deals with a few misspells in the documentation.
2018-11-18 22:23:15 +01:00
Joseph Herlant 02cedc48d3 DOC: Fix typos in different subsections of the documentation
Fix typos found in the design-thoughts, internals and lua-api
subsections of the documentation.
2018-11-18 22:23:15 +01:00
Frédéric Lécaille 3aac1068f7 DOC: early-hints: fix truncated line. 2018-11-13 10:14:02 +01:00
Frédéric Lécaille 06f5b6435b MINOR: doc: Add information about "early-hint" http-request action. 2018-11-12 21:08:55 +01:00
Willy Tarreau c1b0645dac MEDIUM: log: add a new "raw" format
This format is pretty similar to the previous "short" format except
that it also removes the severity level. Thus only the raw message is
sent. This is suitable for use in containers, where only the raw
information is expected and where the severity is supposed to come
from the file descriptor used.
2018-11-12 18:37:55 +01:00
Willy Tarreau e8746a08b2 MEDIUM: log: support a new "short" format
This format is meant to be used with local file descriptors. It emits
messages only prefixed with a level, removing all the process name,
system name, date and so on. It is similar to the printk() format used
on Linux. It's suitable to be sent to a local logger compatible with
systemd's output format.

Note that the facility is still required but not used, hence it is
suggested to use "daemon" to remind that it's a local logger.
Example :

    log stdout format short daemon          # send everything to stdout
    log stderr format short daemon notice   # send important events to stderr
2018-11-12 18:37:55 +01:00
Willy Tarreau 5a32ecc6cf MEDIUM: log: add support for logging to existing file descriptors
In certain situations it would be desirable to log to an existing file
descriptor, the most common case being a pipe between containers or
processes. The main issue with pipes is that using write() on them will
randomly truncate messages. But there is a trick. By using writev(), we
can atomically deliver or drop a message, which perfectly fits the
purpose. The only caveat is that large messages (4096 bytes on modern
operating systems) may be interleaved with messages from other processes
if using nbproc for example. In practice such messages are rare and most
of the time when users need such type of logging, the load is low enough
for a single process to be running so this is not really a problem.

This logging method thus uses unbuffered writev() calls and is uses more
CPU than if it used its own buffer with large writes at once, though this
is not a problem for moderate loads.

Logging to a file descriptor attached to a file also works with the side
effect that the process is significantly slowed down during disk accesses
and that it's not possible to rotate the file without restarting the
process. For this reason this option is not offered as a configuration
option, since it would confuse most users, but one could decide to
redirect haproxy's output to a file during debugging sessions. Two aliases
"stdout" and "stderr" are provided, but keep in mind that these are closed
by default in daemon mode.

When logging to a pipe or socket at a high enough rate, some logs will be
dropped and the number of dropped messages is reported in "show info".
2018-11-12 18:37:55 +01:00
Willy Tarreau adb345d485 DOC: logs: the format directive was missing from the second log part
The "log" statement appears both in the global section and in listeners.
The "format" directive was only documented in the first one. Maybe we
should think about moving this definition to the log section by now.
2018-11-12 18:37:55 +01:00
Willy Tarreau 96079492e0 [RELEASE] Released version 1.9-dev6
Released version 1.9-dev6 with the following main changes :
    - BUG/MEDIUM: tools: fix direction of my_ffsl()
    - BUG/MINOR: cli: forward the whole command on master CLI
    - BUG/MEDIUM: auth/threads: use of crypt() is not thread-safe
    - MINOR: compat: automatically detect support for crypt_r()
    - MEDIUM: auth/threads: make use of crypt_r() on systems supporting it
    - DOC: split the http-request actions in their own section
    - DOC: split the http-response actions in their own section
    - BUG/MAJOR: stream-int: don't call si_cs_recv() in stream_int_chk_rcv_conn()
    - BUG/MINOR: tasks: make sure wakeup events are properly reported to subscribers
    - MINOR: stats: report the number of active jobs and listeners in "show info"
    - MINOR: stats: report the number of active peers in "show info"
    - MINOR: stats: report the number of currently connected peers
    - MINOR: cli: show the number of reload in 'show proc'
    - MINOR: cli: can't connect to the target CLI
    - MEDIUM: mworker: does not create the CLI proxy when no listener
    - MINOR: mworker: displays more information when leaving
    - MEDIUM: mworker: exit with the incriminated exit code
    - MINOR: mworker: displays a message when a worker is forked
    - MEDIUM: mworker: leave when the master die
    - CLEANUP: stream-int: retro-document si_cs_io_cb()
    - BUG/MEDIUM: mworker: does not abort() in mworker_pipe_register()
    - BUG/MEDIUM: stream-int: don't wake up for nothing during SI_ST_CON
    - BUG/MEDIUM: cli: crash when trying to access a worker
    - DOC: restore note about "independant" typo
    - MEDIUM: stream: implement stream_buf_available()
    - MEDIUM: appctx: check for allocation attempts in buffer allocation callbacks
    - MINOR: stream-int: rename si_applet_{want|stop|cant}_{get|put}
    - MINOR: stream-int: add si_done_{get,put} to indicate that we won't do it anymore
    - MINOR: stream-int: use si_cant_put() instead of setting SI_FL_WAIT_ROOM
    - MINOR: stream-int: make use of si_done_{get,put}() in shut{w,r}
    - MINOR: stream-int: make it clear that si_ops cannot be null
    - MEDIUM: stream-int: temporarily make si_chk_rcv() take care of SI_FL_WAIT_ROOM
    - MINOR: stream-int: factor the SI_ST_EST state test into si_chk_rcv()
    - MEDIUM: stream-int: make SI_FL_WANT_PUT reflect CF_DONT_READ
    - MEDIUM: stream-int: always call si_chk_rcv() when we make room in the buffer
    - MEDIUM: stream-int: make si_chk_rcv() check that SI_FL_WAIT_ROOM is cleared
    - MINOR: stream-int: replace si_update() with si_update_both()
    - MEDIUM: stream-int: make stream_int_update() aware of the lower layers
    - CLEANUP: stream-int: remove the now unused si->update() function
    - MEDIUM: stream-int: Rely only on SI_FL_WAIT_ROOM to stop data receipt
    - MEDIUM: stream-int: Try to read data even if channel's buffer seems to be full
    - BUG/MINOR: config: better detect the presence of the h2 pattern in npn/alpn
2018-11-11 10:43:39 +01:00
Lukas Tribus 745f15eba9 DOC: restore note about "independant" typo
The independant -> independent error was fixed in 801a0a35 ("DOC: fix
name for "option independant-streams"), but the note about the wrong
name was erroneously fixed in 0e82b92a ("DOC: fix a few config typos").

Restore the "wrong" name so that when reasearching this option people
can actually find it.

Could be backported to 1.8.
2018-11-08 14:48:56 +01:00
Cyril Bonté 6c81d5f41a DOC: split the http-response actions in their own section
Similarly to the "http-request" actions, this is an attempt to make the
documentation easier to read.
2018-10-30 04:16:32 +01:00
Cyril Bonté c6ad23bbe7 DOC: split the http-request actions in their own section
Since http-request was first introduced, more and more actions have been
added over time. This makes the "http-request" difficult to read and some
actions were forgotten in the list.

This is an attempt to make the documenation cleaner. In future steps, it
would be great to provide at least one example for each action.
2018-10-30 04:15:29 +01:00
Willy Tarreau bddf292cbd [RELEASE] Released version 1.9-dev5
Released version 1.9-dev5 with the following main changes :
    - BUILD: Makefile: add the new ERR variable to force -Werror
    - MINOR: freq_ctr: add swrate_add_scaled() to work with large samples
    - MINOR: stream_interface: Avoid calling si_cs_send/recv if not needed.
    - CLEANUP: http: Remove the unused function http_find_header
    - MINOR: h1: Export some functions parsing the value of some HTTP headers
    - BUG/MEDIUM: stream-int: don't set SI_FL_WAIT_ROOM on CF_READ_DONTWAIT
    - MINOR: proxy: add a new option "http-use-htx"
    - BUG/MEDIUM: pools: fix the minimum allocation size
    - MINOR: shctx: Shared objects block by block allocation.
    - MINOR: cache: Larger HTTP objects caching.
    - MINOR: shctx: Add a maximum object size parameter.
    - MINOR: cache: Add "max-object-size" option.
    - DOC: Update about the cache support for big objects.
    - BUG/MINOR: cache: Crashes with "total-max-size" > 2047(MB).
    - BUG/MINOR: cache: Wrong usage of shctx_init().
    - BUG/MINOR: ssl: Wrong usage of shctx_init().
    - MINOR: cache: Avoid usage of atoi() when parsing "max-object-size".
    - MINOR: shctx: Change max. object size type to unsigned int.
    - DOC: cache: Missing information about "total-max-size" and "max-object-size"
    - CLEANUP: tools: fix misleading comment above function LIM2A
    - MEDIUM: channel: merge back flags CF_WRITE_PARTIAL and CF_WRITE_EVENT
    - BUG/MINOR: only mark connections private if NTLM is detected
    - BUG/MINOR: only auto-prefer last server if lb-alg is non-deterministic
    - MINOR: stream: don't prune variables if the list is empty
    - MINOR: stream-int: add si_alloc_ibuf() to ease input buffer allocation
    - MEDIUM: stream-int: replace channel_alloc_buffer() with si_alloc_ibuf() everywhere
    - MEDIUM: stream: always call si_cs_recv() after a failed buffer allocation
    - MEDIUM: stream: don't try to send first in process_stream()
    - MEDIUM: stream-int: make si_update() synchronize flag changes before the I/O
    - MEDIUM: stream-int: call si_cs_process() in stream_int_update_conn
    - MINOR: stream-int: don't needlessly call tasklet_wakeup() in stream_int_chk_snd_conn()
    - MINOR: stream-int: make stream_int_notify() not wake the tasklet up
    - MINOR: stream-int: don't needlessly call si_cs_send() in si_cs_process()
    - MINOR: mworker: number of reload in the life of a worker
    - MEDIUM: mworker: each worker socketpair is a CLI listener
    - REORG: mworker: move struct mworker_proc to global.h
    - MINOR: server: export new_server() function
    - MEDIUM: mworker: move proc_list gen before proxies startup
    - MEDIUM: mworker: add proc_list in global.h
    - MEDIUM: mworker: proxy for the master CLI
    - MEDIUM: mworker: create CLI listeners from argv[]
    - MEDIUM: cli: disable some keywords in the master
    - MEDIUM: mworker: find the server ptr using a CLI prefix
    - MEDIUM: cli: 'show proc' displays processus
    - MEDIUM: cli: implement 'mode cli' proxy analyzers
    - MINOR: cli: displays sockpair@ in "show cli sockets"
    - MEDIUM: cli: enable "show cli sockets" for the master
    - MINOR: cli: put @master @<relative pid> @!<pid> in the help
    - MEDIUM: listeners: set O_CLOEXEC on the accepted FDs
    - MEDIUM: mworker: stop the master proxy in the workers
    - MEDIUM: channel: reorder the channel analyzers for the cli
    - MEDIUM: cli: write a prompt for the CLI proxy of the master
    - MINOR: cli: helper to write an response message and close
    - MINOR: cache: Add "Age" header.
    - REGTEST: make the IP+port logging test more reliable
    - BUG/MINOR: memory: make the thread-local cache allocator set the debugging link
    - BUG/MAJOR: http: http_txn_get_path() may deference an inexisting buffer
    - BUG/MINOR: backend: assign the wait list after the error check
2018-10-28 20:39:31 +01:00
Lukas Tribus 80512b186f BUG/MINOR: only auto-prefer last server if lb-alg is non-deterministic
While "option prefer-last-server" only applies to non-deterministic load
balancing algorithms, 401/407 responses actually caused haproxy to prefer
the last server unconditionally.

As this breaks deterministic load balancing algorithms like uri, this
patch applies the same condition here.

Should be backported to 1.8 (together with "BUG/MINOR: only mark
connections private if NTLM is detected").
2018-10-27 22:10:32 +02:00
Lukas Tribus fd9b68c48e BUG/MINOR: only mark connections private if NTLM is detected
Instead of marking all connections that see a 401/407 response private
(for connection reuse), this patch detects a RFC4559/NTLM authentication
scheme and restricts the private setting to those connections.

This is so we can reuse connections with 401/407 responses with
deterministic load balancing algorithms later (which requires another fix).

This fixes the problem reported here by Elliot Barlas :

  https://discourse.haproxy.org/t/unable-to-configure-load-balancing-per-request-over-persistent-connection/3144

Should be backported to 1.8.
2018-10-27 22:10:29 +02:00
Frédéric Lécaille e3c83d80e3 DOC: cache: Missing information about "total-max-size" and "max-object-size" 2018-10-26 04:54:41 +02:00
Frédéric Lécaille 5f8bea6488 DOC: Update about the cache support for big objects. 2018-10-24 04:41:28 +02:00
Willy Tarreau 68ad3a42f7 MINOR: proxy: add a new option "http-use-htx"
This option makes a proxy use only HTX-compatible muxes instead of the
HTTP-compatible ones for HTTP modes. It must be set on both ends, this
is checked at parsing time.
2018-10-23 10:22:36 +02:00
Willy Tarreau 01fbe74516 [RELEASE] Released version 1.9-dev4
Released version 1.9-dev4 with the following main changes :
    - BUILD: Allow configuration of pcre-config path
    - DOC: clarify force-private-cache is an option
    - BUG/MINOR: connection: avoid null pointer dereference in send-proxy-v2
    - REORG: http: move the code to different files
    - REORG: http: move HTTP rules parsing to http_rules.c
    - CLEANUP: http: remove some leftovers from recent cleanups
    - BUILD: Makefile: add a "make opts" target to simply show the build options
    - BUILD: Makefile: speed up compiler options detection
    - BUG/MINOR: backend: check that the mux installed properly
    - BUG/MEDIUM: h2: check that the connection is still valid at the end of init()
    - BUG/MEDIUM: h2: make h2_stream_new() return an error on memory allocation failure
    - REGTEST/MINOR: compatibility: use unix@ instead of abns@ sockets
    - MINOR: ssl: cleanup old openssl API call
    - MINOR: ssl: generate-certificates for BoringSSL
    - BUG/MEDIUM: buffers: Make sure we don't wrap in ci_insert_line2/b_rep_blk.
    - MEDIUM: ssl: add support for ciphersuites option for TLSv1.3
    - CLEANUP: haproxy: Remove unused variable
    - CLEANUP: h1: Fix debug warnings for h1 headers
    - CLEANUP: stick-tables: Remove unneeded double (()) around conditional clause
    - MEDIUM: task: perform a single tree lookup per run queue batch
    - BUG/MEDIUM: Cur/CumSslConns counters not threadsafe.
    - BUG/MINOR: threads: move declaration of capabilities to config.h
    - OPTIM: tools: optimize my_ffsl() for x86_64
    - BUG/MINOR: h2: null-deref
    - BUG/MINOR: checks: queues null-deref
    - MINOR: connections: Introduce an unsubscribe method.
    - MEDIUM: connections: Change struct wait_list to wait_event.
    - BUG/MEDIUM: h2: Make sure we're not in the send list on flow control.
    - BUG/MEDIUM: mworker: segfault receiving SIGUSR1 followed by SIGTERM.
    - BUG/MEDIUM: stream: Make sure to unsubscribe before si_release_endpoint.
    - MINOR: http: Move comment about some HTTP macros in the right header file
    - MINOR: stats: Add missing include
    - MINOR: http: Export some functions and do cleanup to prepare HTTP refactoring
    - MEDIUM: http: Ignore http-pretend-keepalive option on frontend
    - MEDIUM: http: Ignore http-tunnel option on backend
    - MINOR: http: Use same flag for httpclose and forceclose options
    - MINOR: h1: Add EOH marker during headers parsing
    - MINOR: conn-stream: Add CL_FL_NOT_FIRST flag
    - MINOR: h1: Change the union h1_sl to use indirect strings to store infos
    - MINOR: h1: Add the flag H1_MF_NO_PHDR to not add pseudo-headers during parsing
    - MINOR: log: make sess_log() support sess=NULL
    - MINOR: chunk: add chunk_cpy() and chunk_cat()
    - MEDIUM: h2: stop relying on H2_SS_IDLE / H2_SS_CLOSED
    - CLEANUP: h2: rename h2c_snd_settings() to h2c_send_settings()
    - MINOR: h2: don't try to send data before preface
    - MINOR: h2: unify the mux init function
    - MINOR: h2: retrieve the front proxy from the caller instead of the session
    - MINOR: h2: split h2c_stream_new() into h2s_new() + h2c_frt_stream_new()
    - MINOR: h2: add a new flag to quickly distinguish front vs back connection
    - BUG/MEDIUM: mworker: don't poll on LI_O_INHERITED listeners
    - BUG/MEDIUM: stream: don't crash on out-of-memory
    - BUILD: compiler: add a new statement "__unreachable()"
    - BUILD: lua: silence some compiler warnings about potential null derefs
    - BUILD: ssl: fix null-deref warning in ssl_fc_cipherlist_str sample fetch
    - BUILD: ssl: fix another null-deref warning in ssl_sock_switchctx_cbk()
    - BUILD: stick-table: make sure not to fail on task_new() during initialization
    - BUILD: peers: check allocation error during peers_init_sync()
    - MINOR: tools: add a new function atleast2() to test masks for more than 1 bit
    - MINOR: config: use atleast2() instead of my_popcountl() where relevant
    - MEDIUM: fd/threads: only grab the fd's lock if the FD has more than one thread
    - MAJOR: tasks: create per-thread wait queues
    - OPTIM: tasks: group all tree roots per cache line
    - DOC: Fix a few typos
    - MINOR: pools: allocate most memory pools from an array
    - MINOR: pools: split pool_free() in the lockfree variant
    - MEDIUM: pools: implement a thread-local cache for pool entries
    - BUG/MEDIUM: threads: fix thread_release() at the end of the rendez-vous point
    - Revert "BUILD: lua: silence some compiler warnings about potential null derefs"
    - BUILD: lua: silence some compiler warnings about potential null derefs (#2)
    - MINOR: lua: all functions calling lua_yieldk() may return
    - BUILD: lua: silence some compiler warnings after WILL_LJMP
    - BUILD: Makefile: silence an option conflict warning with clang
    - MINOR: server: Use memcpy() instead of strncpy().
    - CLEANUP: state-file: make the path concatenation code a bit more consistent
    - MINOR: build: Disable -Wstringop-overflow.
    - MINOR: cfgparse: Write 130 as 128 as 0x82 and 0x80.
    - MINOR: peers: use defines instead of enums to appease clang.
    - DOC: fix reference to map files in MAINTAINERS
    - MINOR: fd: centralize poll timeout computation in compute_poll_timeout()
    - MINOR: poller: move time and date computation out of the pollers
    - BUILD: memory: fix pointer declaration for atomic CAS
    - BUILD: Makefile: add USE_RT to pass -lrt for clock_gettime() and friends
    - MINOR: time: add now_mono_time() and now_cpu_time()
    - MEDIUM: time: measure the time stolen by other threads
    - BUILD: memory: fix free_list pointer declaration again for atomic CAS
    - BUILD: compiler: rename __unreachable() to my_unreachable()
    - BUG/MEDIUM: pools: Fix the usage of mmap()) with DEBUG_UAF.
    - BUILD: memory: fix free_list pointer declaration again for atomic CAS
    - BUG/MEDIUM: h2: Close connection if no stream is left an GOAWAY was sent.
    - BUG/MEDIUM: connections: Remove subscription if going in idle mode.
    - BUG/MEDIUM: stream: Make sure polling is right on retry.
    - MINOR: h2: Make sure to return 1 in h2_recv() when needed.
    - MEDIUM: connections: Don't directly mess with the polling from the upper layers.
    - MINOR: streams: Call tasklet_free() after si_release_endpoint().
    - MINOR: connection: Add a SUB_CALL_UNSUBSCRIBE event.
    - MINOR: h2: Don't run tasks that are waiting to send if mux in full.
    - MINOR: ebtree: save 8 bytes in struct eb32sc_node
2018-10-21 20:28:30 +02:00
Willy Tarreau 0a93b6413f MINOR: pools: allocate most memory pools from an array
For caching it will be convenient to have indexes associated with pools,
without having to dereference the pool itself. One solution could consist
in replacing all pool pointers with integers but this would limit the
number of allocatable pools. Instead here we allocate the 32 first pools
from a pre-allocated array whose base address is known so that it's trivial
to convert a pool to an index in this array. Pools that cannot fit there
will be allocated normally.
2018-10-16 10:29:26 +02:00
Christopher Faulet 315b39c391 MINOR: http: Use same flag for httpclose and forceclose options
Since keep-alive mode is the default mode, the passive close has disappeared,
and in the code, httpclose and forceclose options are handled the same way:
connections with the client and the server are closed as soon as the request and
the response are received and missing "Connection: close" header is added in
each direction.

So to make things clearer, forceclose is now an alias for httpclose. And
httpclose is explicitly an active close. So the old passive close does not exist
anymore. Internally, the flag PR_O_HTTP_PCL has been removed and PR_O_HTTP_FCL
has been replaced by PR_O_HTTP_CLO. In HTTP analyzers, the checks done to find
the right mode to use, depending on proxies options and "Connection: " header
value, have been simplified.

This should only be a cleanup and no changes are expected.
2018-10-12 16:07:56 +02:00
Christopher Faulet 4212a30ad1 MEDIUM: http: Ignore http-tunnel option on backend
This option is frontends specific, so there is no reason to support it on
backends. So now, it is ignored if it is set on a backend and a warning is
emitted during the startup. The change is quite trivial, but the commit is
tagged as MEDIUM because it is a small breakage with previous versions and
configurations using this options could emit a warning now.
2018-10-12 16:05:53 +02:00