Commit Graph

206 Commits

Author SHA1 Message Date
Ben51Degrees
f4a82fb26b BUILD/MINOR: 51d: Updated build registration output to indicate thatif the library is a dummy one or not.
When built with the dummy 51Degrees library for testing, the output will
include "(dummy library)" to ensure it is clear that this is this is not
the API.
2019-06-13 18:00:54 +02:00
Willy Tarreau
5e4c5003c5 CLEANUP: 51d: move the 51d dummy lib to contrib/51d/src to match the real lib
This way the directory structure remains the same as with the real lib and
one can apply the same build options regardless of where the lib is stored,
removing any possible confusion.
2019-06-13 15:56:10 +02:00
Ben51Degrees
31c3d51a18 MINOR: 51d: Added dummy libraries for the 51Degrees module for testing.
These are intended for use by HAProxy developers to ensure any changes
did not affect the 51Degrees implementation. The 51Degrees module can be
enabled and used by using the source in contrib/51d. This will run
without breaking, but will not return any meaningful information.

This is ideal for testing HAProxy core code, and other modules alongside
51Degrees, but should never be used as an actual module as it does
nothing.
2019-06-12 18:06:59 +02:00
Daniel Corbett
c802921721 DOC/MINOR: contrib/spoa_server: Fix typo in README
Fix typo in README ps_pyhton.py -> ps_python.py
2019-06-11 19:27:42 +02:00
Daniel Corbett
061766859c MINOR: contrib/spoa_server: Add random IP score
The example configuration uses sess.ip_score however this variable
is not referenced within the example scripts.  This patch adds support
for sess.ip_score to the python + lua scripts and generates a
random number between 1 and 100.
2019-06-11 19:27:42 +02:00
Daniel Corbett
4e0fa55dcd BUG/MEDIUM: contrib/spoa_server: Set FIN flag on agent frames
When communicating over SPOP the AGENT-HELLO, AGENT-DISCONNECT,
and ACK frames must have the FIN flag set.
2019-06-11 19:27:41 +02:00
Daniel Corbett
5897867ac5 MINOR: contrib/spoa_server: Upgrade SPOP to 2.0
Upgrade SPOP version to 2.0
2019-06-11 19:27:41 +02:00
Christopher Faulet
54b5e214b0 MINOR: htx: Don't use end-of-data blocks anymore
This type of blocks is useless because transition between data and trailers is
obvious. And when there is no trailers, the end-of-message is still there to
know when data end for chunked messages.
2019-06-05 10:12:11 +02:00
Christopher Faulet
142b5df778 BUG/MINOR: contrib/prometheus-exporter: Add HTX data block in one time
Since recent changes on the way HTX data blocks are added in an HTX message, we
must now be sure the prometheus service add its own blocks in one time. Indeed,
the function htx_add_data() may now decide to only copy a part of data. So
instead, we must call htx_add_data_atonce() instead.
2019-06-05 10:12:11 +02:00
Willy Tarreau
db20750fe3 CONTRIB: debug: add 4 missing connection/conn_stream flags
The following flags were missing thus added :
- CO_FL_SOCKS4_RECV
- CO_FL_SOCKS4_SEND
- CS_FL_READ_PARTIAL
- CS_FL_KILL_CONN
2019-06-03 14:26:54 +02:00
Willy Tarreau
7bb39d7cd6 CLEANUP: connection: remove the now unused CS_FL_REOS flag
Let's remove it before it gets uesd again. It was mostly replaced with
CS_FL_EOI and by mux-specific states or flags.
2019-06-03 14:23:33 +02:00
Willy Tarreau
76824a8c99 BUILD: contrib/prometheus: fix build breakage caused by move of idle_pct
The idle_pct thread-local variable was moved to struct thread_info by
commit 81036f2 ("MINOR: time: move the cpu, mono, and idle time to
thread_info") but not updated in service-prometheus.c, thus breaking
it.

No backport is needed. This fixes GH issue #110.
2019-06-02 10:38:48 +02:00
Olivier Houchard
250031e444 MEDIUM: sessions: Introduce session flags.
Add session flags, and add a new flag, SESS_FL_PREFER_LAST, to be set when
we use NTLM authentication, and we should reuse the last connection. This
should fix using NTLM with HTX. This totally replaces TX_PREFER_LAST.

This should be backported to 1.9.
2019-05-29 15:41:47 +02:00
Willy Tarreau
ce1f7a16ab CONTRIB: wurfl: address 3 build issues in the wurfl dummy library
Two of them were functions returning an string missing the return
statement and one is just the usual "set but not used".
2019-05-22 14:59:08 +02:00
mbellomi
e30e3001d9 MINOR: WURFL: call header_retireve_callback() in dummy library
The current coverage of the dummy library was limited because the callbacks
passed to wurfl_lookup() were not called. Now we do call them with one existing
and one non-existing headers to make sure that ha_wurfl_retrieve_header() is
covered by the tests as well.
2019-05-22 12:06:05 +02:00
Thierry FOURNIER
4d7bfa1cd1 MINOR/DOC: spoe-server: Add documentation
This is the documentation and examples.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
00a0225636 MINOR: spoa-server: Add python
This commit adds the Python support for the server.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
6908c95804 MINOR: spoa-server: Add Lua processing
Use the defined binding for registering Lua engine.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
a09df3f5d0 MINOR: spoa-server: Execute registered callbacks
Call the right function with the right engine for each received message.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
fbd3824868 MINOR: spoa-server: Prepare responses
This patch adds SPOP responses managament. It provides SPOP
encoding primitives. It also move the example function
ip_reputation to this new behavior.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
8b9a73bac0 MINOR: spoa-server: Load files
Declare files to be executed at the begining and execute it. The binding
between the engine and the file is done throught the extension.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
892f6647c1 MINOR: spoa-server: Allow registering message processors
This function register processor executed by any language for processing
an SPOP message.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
64eaa33214 MINOR: spoa-server: Allow registering external processes
Add struct for declaring an reistrering external processing resource.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
786e9e684b MINOR: spoa-server: With debug mode, start only one process
Because debug with processes is simpler if only one process is started.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
7de6fc6ae1 MINOR: spoa-server: Replace the thread init system by processes
I will replace thread by processes. Note that, I keep the pthread_key
system for identifiying process in the same way that threads. Note
also that I keep commented out the original thread code because I hope
to reactivate it.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
5301ed1589 MINOR: spoe-server: rename "worker" functions
"worker" name is a little bit generic and it is used in many
places, so it is hard to find the expected symbol.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
880d7e15ba MINOR: spoa-server: Externalise debug functions
Make external LOG and DEBUG function. Other process can use this ones
and later these functions will be replaced by another log system
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
4aec0a4a9a MINOR: spoa-server: move some definition from spoa_server.c to spoa_server.h
This will allow to add some other files to the project
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
d8b5c77b55 MINOR: spoa-server: Clone the v1.7 spoa-example project
This is a working base.
2019-05-13 17:43:47 +02:00
William Lallemand
eaa4555003 MINOR: systemd: support /etc/sysconfig/ for redhat based distrib
The patch "MINOR: systemd: Make use of master socket in systemd unit"
introduces an environment file in /etc/default.

Unfortunatly this is not supported on redhat-based system, so we add
/etc/sysconfig/haproxy for that.
2019-05-07 14:11:55 +02:00
Tim Duesterhus
eaa4b8a968 MINOR: systemd: Make use of master socket in systemd unit
Unless the EXTRAOPTS variable is overriden in /etc/default/haproxy
the unit file will use the master socket by default.

This patch may be backported to 1.9 and depends on
MINOR: systemd: Use the variables from /etc/default/haproxy.
2019-05-07 14:11:55 +02:00
Apollon Oikonomopoulos
b06014869e MINOR: systemd: Use the variables from /etc/default/haproxy
This will allow seamless upgrades from the sysvinit system while respecting
any changes the users may have made. It will also make local configuration
easier than overriding the systemd unit file.

Note by Tim:

This GPL-2 licensed patch was taken from the Debian project at [1].

It was slightly modified to cleanly apply, because HAProxy's default unit
file does not include rsyslog.service as an 'After' dependency. Also the
subject line was modified to include the proper subsystem and severity.

This patch may be backported to 1.9.

[1] https://salsa.debian.org/haproxy-team/haproxy/blob/master/debian/patches/haproxy.service-use-environment-variables.patch

Co-authored-by: Tim Duesterhus <tim@bastelstu.be>
2019-05-07 14:11:55 +02:00
Yann Cézard
bf60f6b803 BUG/MEDIUM: contrib/modsecurity: If host header is NULL, don't try to strdup it
I discovered this bug when running OWASP regression tests against HAProxy +
modsecurity-spoa (it's a POC to evaluate how it is working).  I found out that
modsecurity spoa will crash when the request doesn't have any Host header.

See the pull request #86 on github for details.

This patch must be backported to 1.9 and 1.8.
2019-04-29 16:26:05 +02:00
Yann Cézard
494ddbff47 DOC: contrib/modsecurity: Typos and fix the reject example
Thanks to https://www.mail-archive.com/haproxy@formilux.org/msg30056.html

This patch may be backported to 1.9 and 1.8.
2019-04-29 16:25:49 +02:00
David Carlier
0470d704a7 BUILD/MEDIUM: contrib: Dummy DeviceAtlas API.
Creating a "mocked" version mainly for testing purposes.
2019-04-26 17:06:32 +02:00
paulborile
cd9b9bd3e4 MINOR: contrib: dummy wurfl library
This is dummy version of the Scientiamobile WURFL C API that can be used
to successfully build/run haproxy compiled with USE_WURFL=1.
It is marked as version 1.11.2.100 to distinguish it from any real version
of the lib. It has no external dependencies so it should work out of the
box by building it like this :

   $ make -C contrib/wurfl

In order to use it, simply reference this directory as the WURFL include
and library paths :

   $ make TARGET=<target> USE_WURFL=1 WURFL_INC=$PWD/contrib/wurfl WURFL_LIB=$PWD/contrib/wurfl
2019-04-23 11:00:23 +02:00
Christopher Faulet
769a92d86d MINOR: contrib/prometheus-exporter: Follow best practices about metrics type
In short, _total metrics are now counters and others are gauges.

No backport needed. See issue #81 on github.
2019-04-18 10:27:16 +02:00
Christopher Faulet
8c8e4b1263 MINOR: contrib/prometheus-exporter: Rename some metrics to be more usable
Some metrics have been renamed and their type adapted to be more usable in
Prometheus:

  * haproxy_process_uptime_seconds -> haproxy_process_start_time_seconds
  * haproxy_process_max_memory -> haproxy_process_max_memory_bytes
  * haproxy_process_pool_allocated_total -> haproxy_process_pool_allocated_bytes
  * haproxy_process_pool_used_total -> haproxy_process_pool_used_bytes
  * haproxy_process_ssl_cache_lookups -> haproxy_process_ssl_cache_lookups_total
  * haproxy_process_ssl_cache_misses -> haproxy_process_ssl_cache_misses_total

No backport needed. See issue #81 on github.
2019-04-18 10:27:16 +02:00
Christopher Faulet
c58fc0dec9 MINOR: contrib/prometheus-exporter: Remove usless rate metrics
Following metrics have been removed:

  * haproxy_frontend_connections_rate_current (ST_F_CONN_RATE)
  * haproxy_frontend_http_requests_rate_current (ST_F_REQ_RATE)
  * haproxy_*_current_session_rate (ST_F_RATE)

These rates can be deduced using the total value with this kind of formula:

  rate(haproxy_frontend_connections_total[1m])

No backport needed. See issue #81 on github.
2019-04-18 10:27:16 +02:00
Christopher Faulet
f782c23ec6 BUG/MINOR: contrib/prometheus-exporter: Fix a typo in the run-queue metric type
No backport needed.
2019-04-18 10:27:16 +02:00
Willy Tarreau
a1bd1faeeb BUILD: use inttypes.h instead of stdint.h
I found on an (old) AIX 5.1 machine that stdint.h didn't exist while
inttypes.h which is expected to include it does exist and provides the
desired functionalities.

As explained here, stdint being just a subset of inttypes for use in
freestanding environments, it's probably always OK to switch to inttypes
instead:

  https://pubs.opengroup.org/onlinepubs/009696799/basedefs/stdint.h.html

Also it's even clearer here in the autoconf doc :

  https://www.gnu.org/software/autoconf/manual/autoconf-2.61/html_node/Header-Portability.html

  "The C99 standard says that inttypes.h includes stdint.h, so there's
   no need to include stdint.h separately in a standard environment.
   Some implementations have inttypes.h but not stdint.h (e.g., Solaris
   7), but we don't know of any implementation that has stdint.h but not
   inttypes.h"
2019-04-01 07:44:56 +02:00
Christopher Faulet
9744f7c126 BUG/MINOR: contrib/prometheus-exporter: Fix applet accordingly to recent changes
Since the flag EOI was added on channels, some hidden bugs in the prometheus
exporter now leads to error. the visible effect is that responses are
truncated.

So first of all, channel_add_input() must be called when the response headers
and the EOM block are added. To be sure to correctly update the response channel
(especially to_forward value). Then the request must really be fully
consumed. And finally, the return clause in the switch has been replaced by a
break. It was totally wrong to skip the end of the function in the states
PROMEX_DONE and PROMEX_ERROR. (Note that PROMEX_ERROR was never used, so it was
replaced by PROMEX_END just to ease reading the code).

No need to backport this patch, the Prometheus exporter does not exist in early
versions.
2019-03-27 16:08:32 +01:00
Willy Tarreau
f882a85da4 CONTRIB: debug: report the CS and CF's EOI flags
These ones indicate an end of input.
2019-03-25 18:34:28 +01:00
Christopher Faulet
f959d0809e CONTRIB: contrib/prometheus-exporter: Add a Prometheus exporter for HAProxy
It has been developped as a service applet. Internally, it is called
"promex". To build HAProxy with the promex service, you should use the Makefile
variable "EXTRA_OBJS". To be used, it must be enabled in the configuration with
an "http-request" rule and the corresponding HTTP proxy must enable the HTX
support. For instance:

    frontend test
        mode http
        ...
	option http-use-htx
	http-request use-service prometheus-exporter if { path /metrics }
        ...

See contrib/prometheus-exporter/README for details.
2019-02-08 13:55:21 +01:00
Willy Tarreau
f2ddd87362 CONTRIB: debug: report stream-int's flag SI_FL_CLEAN_ABRT
It was missing.
2018-12-18 11:03:11 +01:00
Willy Tarreau
121c4e0bca CONTRIB: debug: teach the "flags" utility about new conn_stream flags
CS_FL_ERR_PENDING and CS_FL_WANT_ROOM were not added.
2018-12-17 21:58:03 +01:00
Willy Tarreau
0007d0afbc CLEANUP: stream: remove SF_TUNNEL, SF_INITIALIZED, SF_CONN_TAR
These flags haven't been used for a while. SF_TUNNEL was reintroduced
by commit d62b98c6e ("MINOR: stream: don't set backend's nor response
analysers on SF_TUNNEL") to handle the two-level streams needed to
deal with the first model for H2, and was not removed after this model
was abandonned. SF_INITIALIZED was only set. SF_CONN_TAR was never
referenced at all.
2018-12-11 18:01:38 +01:00
Willy Tarreau
841bc7d471 CONTRIB: hpack: add a compressed stream generator for the encoder
This generates the tables and indexes which will be used by the HPACK
encoder. The headers are sorted by length, then by statistical frequency,
then by direction (preference for responses), then by name, then by index.
The purpose is to speed up their lookup.
2018-12-11 09:06:46 +01:00
Willy Tarreau
05b9b64afb MINOR: stream-int: replace SI_FL_WANT_PUT with !SI_FL_RX_WAIT_EP
The SI_FL_WANT_PUT flag is used in an awkward way, sometimes it's
set by the stream-interface to mean "I have something to deliver",
sometimes it's cleared by the channel to say "I don't want you to
send what you have", and it has to be set back once CF_DONT_READ
is cleared. This will have to be split between SI_FL_RX_WAIT_EP
and SI_FL_RXBLK_CHAN. This patch only replaces all uses of the
flag with its natural (but negated) replacement SI_FL_RX_WAIT_EP.
The code is expected to be strictly equivalent. The now unused flag
was completely removed.
2018-11-18 21:41:46 +01:00
Willy Tarreau
94f7907d65 MINOR: stream-int: introduce new SI_FL_RXBLK flags
The plan is to have the following flags to describe why a stream interface
doesn't produce data :

    - SI_FL_RXBLK_CHAN : the channel doesn't want it to receive
    - SI_FL_RXBLK_BUFF : waiting for a buffer allocation to complete
    - SI_FL_RXBLK_ROOM : more room is required in the channel to receive
    - SI_FL_RXBLK_SHUT : input now closed, nothing new will come
    - SI_FL_RX_WAIT_EP : waiting for the endpoint to produce more data

Applets like the CLI which consume complete commands at once and produce
large chunks of responses will for example be able to stop being woken up
by clearing SI_FL_WANT_GET and setting SI_FL_RXBLK_ROOM when the rx buffer
is full. Once called they will unblock WANT_GET. The flags were moved
together in readable form with the Rx bits using 2 hex digits and still
have some room to do a similar operation on the Tx path later, with the
WAIT_EP flag being represented alone on a digit.
2018-11-18 21:41:45 +01:00