Commit Graph

213 Commits

Author SHA1 Message Date
Aleksandar Lazic
a71447539d DOC: contrib: spoa_server Add some hints for building spoa_server 2019-07-05 16:31:50 +02:00
Christopher Faulet
0c55a15ce1 BUG/MINOR: contrib/prometheus-exporter: Don't try to add empty data blocks
When the response buffer is full and nothing more can be inserted, it is
important to not try to insert an empty data block. Otherwise, when the function
channel_add_input() is called, the flag CF_READ_PARTIAL is set on the response
channel while nothing was read and the stream is uselessly woken up. Finally, we
have loop while the response buffer is full.

This patch must be backported to 2.0.
2019-07-05 14:26:14 +02:00
Christopher Faulet
11921e6819 BUG/MINOR: contrib/prometheus-exporter: Respect the reserve when data are sent
The previous commit e6cdfe574 ("BUG/MINOR: contrib/prometheus-exporter: Don't
use channel_htx_recv_max()") is buggy. The buffer's reserve must be respected.

This patch must be backported to 2.0 and 1.9.
2019-07-03 11:47:20 +02:00
Christopher Faulet
e6cdfe574e BUG/MINOR: contrib/prometheus-exporter: Don't use channel_htx_recv_max()
The function htx_free_data_space() must be used intead. Otherwise, if there are
some output data not already forwarded, the maximum amount of data that may be
inserted into the buffer may be greater than what we can really insert.

This patch must be backported to 2.0.
2019-07-02 21:08:26 +02:00
Tim Duesterhus
86e6b6ebf8 MEDIUM: Make '(cli|con|srv)timeout' directive fatal
They were deprecated with HAProxy 1.5. Time to remove them.
2019-06-17 13:35:54 +02:00
Willy Tarreau
a8ee4b199f CLEANUP: removed obsolete examples an move a few to better places
The following example files awere removed as irrelevant by this
time :
  auth.cfg check.conf ssl.cfg haproxy.spec

The following scripts were removed as having been unused for more
than a decade :
  debug2ansi debug2html debugfind check init.haproxy stats_haproxy.sh

seemless_reload.txt was moved to doc/ where it's more suitable.

haproxy.vim was moved to contrib/syntax-highlight/

scripts/create-release was updated not to try to update haproxy.spec
anymore.
2019-06-15 21:25:06 +02:00
Willy Tarreau
d254aa8139 DOC: update few references to the linux* targets and change them to linux-glibc
The INSTALL guide, the Lua doc and the Prometheus exporter's README all
used to reference "linux2628", "linux26" or even "linux". These were all
updated to consistently reflect "linux-glibc" instead. The default options
were updated there as well so that it should build cleanly on most distros.
2019-06-15 18:03:48 +02:00
Ben51Degrees
f4a82fb26b BUILD/MINOR: 51d: Updated build registration output to indicate thatif the library is a dummy one or not.
When built with the dummy 51Degrees library for testing, the output will
include "(dummy library)" to ensure it is clear that this is this is not
the API.
2019-06-13 18:00:54 +02:00
Willy Tarreau
5e4c5003c5 CLEANUP: 51d: move the 51d dummy lib to contrib/51d/src to match the real lib
This way the directory structure remains the same as with the real lib and
one can apply the same build options regardless of where the lib is stored,
removing any possible confusion.
2019-06-13 15:56:10 +02:00
Ben51Degrees
31c3d51a18 MINOR: 51d: Added dummy libraries for the 51Degrees module for testing.
These are intended for use by HAProxy developers to ensure any changes
did not affect the 51Degrees implementation. The 51Degrees module can be
enabled and used by using the source in contrib/51d. This will run
without breaking, but will not return any meaningful information.

This is ideal for testing HAProxy core code, and other modules alongside
51Degrees, but should never be used as an actual module as it does
nothing.
2019-06-12 18:06:59 +02:00
Daniel Corbett
c802921721 DOC/MINOR: contrib/spoa_server: Fix typo in README
Fix typo in README ps_pyhton.py -> ps_python.py
2019-06-11 19:27:42 +02:00
Daniel Corbett
061766859c MINOR: contrib/spoa_server: Add random IP score
The example configuration uses sess.ip_score however this variable
is not referenced within the example scripts.  This patch adds support
for sess.ip_score to the python + lua scripts and generates a
random number between 1 and 100.
2019-06-11 19:27:42 +02:00
Daniel Corbett
4e0fa55dcd BUG/MEDIUM: contrib/spoa_server: Set FIN flag on agent frames
When communicating over SPOP the AGENT-HELLO, AGENT-DISCONNECT,
and ACK frames must have the FIN flag set.
2019-06-11 19:27:41 +02:00
Daniel Corbett
5897867ac5 MINOR: contrib/spoa_server: Upgrade SPOP to 2.0
Upgrade SPOP version to 2.0
2019-06-11 19:27:41 +02:00
Christopher Faulet
54b5e214b0 MINOR: htx: Don't use end-of-data blocks anymore
This type of blocks is useless because transition between data and trailers is
obvious. And when there is no trailers, the end-of-message is still there to
know when data end for chunked messages.
2019-06-05 10:12:11 +02:00
Christopher Faulet
142b5df778 BUG/MINOR: contrib/prometheus-exporter: Add HTX data block in one time
Since recent changes on the way HTX data blocks are added in an HTX message, we
must now be sure the prometheus service add its own blocks in one time. Indeed,
the function htx_add_data() may now decide to only copy a part of data. So
instead, we must call htx_add_data_atonce() instead.
2019-06-05 10:12:11 +02:00
Willy Tarreau
db20750fe3 CONTRIB: debug: add 4 missing connection/conn_stream flags
The following flags were missing thus added :
- CO_FL_SOCKS4_RECV
- CO_FL_SOCKS4_SEND
- CS_FL_READ_PARTIAL
- CS_FL_KILL_CONN
2019-06-03 14:26:54 +02:00
Willy Tarreau
7bb39d7cd6 CLEANUP: connection: remove the now unused CS_FL_REOS flag
Let's remove it before it gets uesd again. It was mostly replaced with
CS_FL_EOI and by mux-specific states or flags.
2019-06-03 14:23:33 +02:00
Willy Tarreau
76824a8c99 BUILD: contrib/prometheus: fix build breakage caused by move of idle_pct
The idle_pct thread-local variable was moved to struct thread_info by
commit 81036f2 ("MINOR: time: move the cpu, mono, and idle time to
thread_info") but not updated in service-prometheus.c, thus breaking
it.

No backport is needed. This fixes GH issue #110.
2019-06-02 10:38:48 +02:00
Olivier Houchard
250031e444 MEDIUM: sessions: Introduce session flags.
Add session flags, and add a new flag, SESS_FL_PREFER_LAST, to be set when
we use NTLM authentication, and we should reuse the last connection. This
should fix using NTLM with HTX. This totally replaces TX_PREFER_LAST.

This should be backported to 1.9.
2019-05-29 15:41:47 +02:00
Willy Tarreau
ce1f7a16ab CONTRIB: wurfl: address 3 build issues in the wurfl dummy library
Two of them were functions returning an string missing the return
statement and one is just the usual "set but not used".
2019-05-22 14:59:08 +02:00
mbellomi
e30e3001d9 MINOR: WURFL: call header_retireve_callback() in dummy library
The current coverage of the dummy library was limited because the callbacks
passed to wurfl_lookup() were not called. Now we do call them with one existing
and one non-existing headers to make sure that ha_wurfl_retrieve_header() is
covered by the tests as well.
2019-05-22 12:06:05 +02:00
Thierry FOURNIER
4d7bfa1cd1 MINOR/DOC: spoe-server: Add documentation
This is the documentation and examples.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
00a0225636 MINOR: spoa-server: Add python
This commit adds the Python support for the server.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
6908c95804 MINOR: spoa-server: Add Lua processing
Use the defined binding for registering Lua engine.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
a09df3f5d0 MINOR: spoa-server: Execute registered callbacks
Call the right function with the right engine for each received message.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
fbd3824868 MINOR: spoa-server: Prepare responses
This patch adds SPOP responses managament. It provides SPOP
encoding primitives. It also move the example function
ip_reputation to this new behavior.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
8b9a73bac0 MINOR: spoa-server: Load files
Declare files to be executed at the begining and execute it. The binding
between the engine and the file is done throught the extension.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
892f6647c1 MINOR: spoa-server: Allow registering message processors
This function register processor executed by any language for processing
an SPOP message.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
64eaa33214 MINOR: spoa-server: Allow registering external processes
Add struct for declaring an reistrering external processing resource.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
786e9e684b MINOR: spoa-server: With debug mode, start only one process
Because debug with processes is simpler if only one process is started.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
7de6fc6ae1 MINOR: spoa-server: Replace the thread init system by processes
I will replace thread by processes. Note that, I keep the pthread_key
system for identifiying process in the same way that threads. Note
also that I keep commented out the original thread code because I hope
to reactivate it.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
5301ed1589 MINOR: spoe-server: rename "worker" functions
"worker" name is a little bit generic and it is used in many
places, so it is hard to find the expected symbol.
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
880d7e15ba MINOR: spoa-server: Externalise debug functions
Make external LOG and DEBUG function. Other process can use this ones
and later these functions will be replaced by another log system
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
4aec0a4a9a MINOR: spoa-server: move some definition from spoa_server.c to spoa_server.h
This will allow to add some other files to the project
2019-05-13 17:43:47 +02:00
Thierry FOURNIER
d8b5c77b55 MINOR: spoa-server: Clone the v1.7 spoa-example project
This is a working base.
2019-05-13 17:43:47 +02:00
William Lallemand
eaa4555003 MINOR: systemd: support /etc/sysconfig/ for redhat based distrib
The patch "MINOR: systemd: Make use of master socket in systemd unit"
introduces an environment file in /etc/default.

Unfortunatly this is not supported on redhat-based system, so we add
/etc/sysconfig/haproxy for that.
2019-05-07 14:11:55 +02:00
Tim Duesterhus
eaa4b8a968 MINOR: systemd: Make use of master socket in systemd unit
Unless the EXTRAOPTS variable is overriden in /etc/default/haproxy
the unit file will use the master socket by default.

This patch may be backported to 1.9 and depends on
MINOR: systemd: Use the variables from /etc/default/haproxy.
2019-05-07 14:11:55 +02:00
Apollon Oikonomopoulos
b06014869e MINOR: systemd: Use the variables from /etc/default/haproxy
This will allow seamless upgrades from the sysvinit system while respecting
any changes the users may have made. It will also make local configuration
easier than overriding the systemd unit file.

Note by Tim:

This GPL-2 licensed patch was taken from the Debian project at [1].

It was slightly modified to cleanly apply, because HAProxy's default unit
file does not include rsyslog.service as an 'After' dependency. Also the
subject line was modified to include the proper subsystem and severity.

This patch may be backported to 1.9.

[1] https://salsa.debian.org/haproxy-team/haproxy/blob/master/debian/patches/haproxy.service-use-environment-variables.patch

Co-authored-by: Tim Duesterhus <tim@bastelstu.be>
2019-05-07 14:11:55 +02:00
Yann Cézard
bf60f6b803 BUG/MEDIUM: contrib/modsecurity: If host header is NULL, don't try to strdup it
I discovered this bug when running OWASP regression tests against HAProxy +
modsecurity-spoa (it's a POC to evaluate how it is working).  I found out that
modsecurity spoa will crash when the request doesn't have any Host header.

See the pull request #86 on github for details.

This patch must be backported to 1.9 and 1.8.
2019-04-29 16:26:05 +02:00
Yann Cézard
494ddbff47 DOC: contrib/modsecurity: Typos and fix the reject example
Thanks to https://www.mail-archive.com/haproxy@formilux.org/msg30056.html

This patch may be backported to 1.9 and 1.8.
2019-04-29 16:25:49 +02:00
David Carlier
0470d704a7 BUILD/MEDIUM: contrib: Dummy DeviceAtlas API.
Creating a "mocked" version mainly for testing purposes.
2019-04-26 17:06:32 +02:00
paulborile
cd9b9bd3e4 MINOR: contrib: dummy wurfl library
This is dummy version of the Scientiamobile WURFL C API that can be used
to successfully build/run haproxy compiled with USE_WURFL=1.
It is marked as version 1.11.2.100 to distinguish it from any real version
of the lib. It has no external dependencies so it should work out of the
box by building it like this :

   $ make -C contrib/wurfl

In order to use it, simply reference this directory as the WURFL include
and library paths :

   $ make TARGET=<target> USE_WURFL=1 WURFL_INC=$PWD/contrib/wurfl WURFL_LIB=$PWD/contrib/wurfl
2019-04-23 11:00:23 +02:00
Christopher Faulet
769a92d86d MINOR: contrib/prometheus-exporter: Follow best practices about metrics type
In short, _total metrics are now counters and others are gauges.

No backport needed. See issue #81 on github.
2019-04-18 10:27:16 +02:00
Christopher Faulet
8c8e4b1263 MINOR: contrib/prometheus-exporter: Rename some metrics to be more usable
Some metrics have been renamed and their type adapted to be more usable in
Prometheus:

  * haproxy_process_uptime_seconds -> haproxy_process_start_time_seconds
  * haproxy_process_max_memory -> haproxy_process_max_memory_bytes
  * haproxy_process_pool_allocated_total -> haproxy_process_pool_allocated_bytes
  * haproxy_process_pool_used_total -> haproxy_process_pool_used_bytes
  * haproxy_process_ssl_cache_lookups -> haproxy_process_ssl_cache_lookups_total
  * haproxy_process_ssl_cache_misses -> haproxy_process_ssl_cache_misses_total

No backport needed. See issue #81 on github.
2019-04-18 10:27:16 +02:00
Christopher Faulet
c58fc0dec9 MINOR: contrib/prometheus-exporter: Remove usless rate metrics
Following metrics have been removed:

  * haproxy_frontend_connections_rate_current (ST_F_CONN_RATE)
  * haproxy_frontend_http_requests_rate_current (ST_F_REQ_RATE)
  * haproxy_*_current_session_rate (ST_F_RATE)

These rates can be deduced using the total value with this kind of formula:

  rate(haproxy_frontend_connections_total[1m])

No backport needed. See issue #81 on github.
2019-04-18 10:27:16 +02:00
Christopher Faulet
f782c23ec6 BUG/MINOR: contrib/prometheus-exporter: Fix a typo in the run-queue metric type
No backport needed.
2019-04-18 10:27:16 +02:00
Willy Tarreau
a1bd1faeeb BUILD: use inttypes.h instead of stdint.h
I found on an (old) AIX 5.1 machine that stdint.h didn't exist while
inttypes.h which is expected to include it does exist and provides the
desired functionalities.

As explained here, stdint being just a subset of inttypes for use in
freestanding environments, it's probably always OK to switch to inttypes
instead:

  https://pubs.opengroup.org/onlinepubs/009696799/basedefs/stdint.h.html

Also it's even clearer here in the autoconf doc :

  https://www.gnu.org/software/autoconf/manual/autoconf-2.61/html_node/Header-Portability.html

  "The C99 standard says that inttypes.h includes stdint.h, so there's
   no need to include stdint.h separately in a standard environment.
   Some implementations have inttypes.h but not stdint.h (e.g., Solaris
   7), but we don't know of any implementation that has stdint.h but not
   inttypes.h"
2019-04-01 07:44:56 +02:00
Christopher Faulet
9744f7c126 BUG/MINOR: contrib/prometheus-exporter: Fix applet accordingly to recent changes
Since the flag EOI was added on channels, some hidden bugs in the prometheus
exporter now leads to error. the visible effect is that responses are
truncated.

So first of all, channel_add_input() must be called when the response headers
and the EOM block are added. To be sure to correctly update the response channel
(especially to_forward value). Then the request must really be fully
consumed. And finally, the return clause in the switch has been replaced by a
break. It was totally wrong to skip the end of the function in the states
PROMEX_DONE and PROMEX_ERROR. (Note that PROMEX_ERROR was never used, so it was
replaced by PROMEX_END just to ease reading the code).

No need to backport this patch, the Prometheus exporter does not exist in early
versions.
2019-03-27 16:08:32 +01:00
Willy Tarreau
f882a85da4 CONTRIB: debug: report the CS and CF's EOI flags
These ones indicate an end of input.
2019-03-25 18:34:28 +01:00