Commit Graph

145 Commits

Author SHA1 Message Date
Willy Tarreau
843b7cbe9d MEDIUM: chunks: make the chunk struct's fields match the buffer struct
Chunks are only a subset of a buffer (a non-wrapping version with no head
offset). Despite this we still carry a lot of duplicated code between
buffers and chunks. Replacing chunks with buffers would significantly
reduce the maintenance efforts. This first patch renames the chunk's
fields to match the name and types used by struct buffers, with the goal
of isolating the code changes from the declaration changes.

Most of the changes were made with spatch using this coccinelle script :

  @rule_d1@
  typedef chunk;
  struct chunk chunk;
  @@
  - chunk.str
  + chunk.area

  @rule_d2@
  typedef chunk;
  struct chunk chunk;
  @@
  - chunk.len
  + chunk.data

  @rule_i1@
  typedef chunk;
  struct chunk *chunk;
  @@
  - chunk->str
  + chunk->area

  @rule_i2@
  typedef chunk;
  struct chunk *chunk;
  @@
  - chunk->len
  + chunk->data

Some minor updates to 3 http functions had to be performed to take size_t
ints instead of ints in order to match the unsigned length here.
2018-07-19 16:23:43 +02:00
Vincent Bernat
3b479bd5f5 MINOR: systemd: consider exit status 143 as successful
The master process will exit with the status of the last worker. When
the worker is killed with SIGTERM, it is expected to get 143 as an
exit status. Therefore, we consider this exit status as normal from a
systemd point of view. If it happens when not stopping, the systemd
unit is configured to always restart, so it has no adverse effect.

This has mostly a cosmetic effect. Without the patch, stopping HAProxy
leads to the following status:

    ● haproxy.service - HAProxy Load Balancer
       Loaded: loaded (/lib/systemd/system/haproxy.service; disabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Fri 2018-06-22 20:35:42 CEST; 8min ago
         Docs: man:haproxy(1)
               file:/usr/share/doc/haproxy/configuration.txt.gz
      Process: 32715 ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS (code=exited, status=143)
      Process: 32714 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
     Main PID: 32715 (code=exited, status=143)

After the patch:

    ● haproxy.service - HAProxy Load Balancer
       Loaded: loaded (/lib/systemd/system/haproxy.service; disabled; vendor preset: enabled)
       Active: inactive (dead)
         Docs: man:haproxy(1)
               file:/usr/share/doc/haproxy/configuration.txt.gz
2018-07-12 17:44:51 +02:00
Dragan Dosen
ee07dbeca5 BUG/MINOR: contrib/modsecurity: update pointer on the end of the frame
Similar to commit 94bb4c6 ("BUG/MINOR: spoa: Update pointer on the end of
the frame when a reply is encoded").

This patch should be backported to 1.8.
2018-06-04 17:41:25 +02:00
Dragan Dosen
f21185f236 BUG/MINOR: contrib/mod_defender: update pointer on the end of the frame
Similar to commit 94bb4c6 ("BUG/MINOR: spoa: Update pointer on the end of
the frame when a reply is encoded").

This patch should be backported to 1.8.
2018-06-04 17:41:25 +02:00
Christopher Faulet
58b8e492a3 BUG/MINOR: contrib/modsecurity: Don't reset the status code during disconnect
When the connection is closed by HAProxy, the status code provided in the
DISCONNECT frame is lost. By retransmitting it in the agent's reply, we are sure
to have it in the SPOE logs.

This patch may be backported in 1.8.
2018-06-04 17:34:51 +02:00
Christopher Faulet
72de8eb7aa BUG/MINOR: contrib/mod_defender: Don't reset the status code during disconnect
When the connection is closed by HAProxy, the status code provided in the
DISCONNECT frame is lost. By retransmitting it in the agent's reply, we are sure
to have it in the SPOE logs.

This patch may be backported in 1.8.
2018-06-04 17:34:50 +02:00
Christopher Faulet
b47e438593 BUG/MINOR: contrib/spoa_example: Don't reset the status code during disconnect
When the connection is closed by HAProxy, the status code provided in the
DISCONNECT frame is lost. By retransmitting it in the agent's reply, we are sure
to have it in the SPOE logs.

This patch may be backported in 1.8.
2018-06-04 17:34:50 +02:00
Christopher Faulet
6381650516 MAJOR: spoe: upgrade the SPOP version to 2.0 and remove the support for 1.0
The commit c4dcaff3 ("BUG/MEDIUM: spoe: Flags are not encoded in network order")
introduced an incompatibility with older agents. So the major version of the
SPOP is increased to make the situation unambiguous. And because before the fix,
the protocol is buggy, the support of the version 1.0 is removed to be sure to
not continue to support buggy agents.

The agents in the contrib folder (spoa_example, modsecurity and mod_defender)
are also updated to announce the SPOP version 2.0.

So, to be clear, from the patch, connections to agents announcing the SPOP
version 1.0 will be rejected.

This patch must be backported in 1.8.
2018-06-04 17:33:48 +02:00
David Carlier
53d31594bf DOC: contrib/modsecurity: few typo fixes
Few typo fixes.
2018-05-31 20:47:39 +02:00
Christopher Faulet
633f3bffed BUG/MEDIUM: contrib/modsecurity: Use network order to encode/decode flags
A recent fix on the SPOE revealed a mismatch between the SPOE specification and
the modsecurity implementation on the way flags are encoded or decoded. They
must be exchanged using the network bytes order and not the host one.

Be careful though, this patch breaks the compatiblity with HAProxy SPOE before
commit c4dcaff3 ("BUG/MEDIUM: spoe: Flags are not encoded in network order").
2018-05-18 15:06:31 +02:00
Christopher Faulet
48d02d0d21 BUG/MEDIUM: contrib/mod_defender: Use network order to encode/decode flags
A recent fix on the SPOE revealed a mismatch between the SPOE specification and
the mod_defender implementation on the way flags are encoded or decoded. They
must be exchanged using the network bytes order and not the host one.

Be careful though, this patch breaks the compatiblity with HAProxy SPOE before
commit c4dcaff3 ("BUG/MEDIUM: spoe: Flags are not encoded in network order").
2018-05-18 15:06:18 +02:00
Thierry FOURNIER
c4dcaff3f0 BUG/MEDIUM: spoe: Flags are not encoded in network order
The flags are direct copy of the "unsigned int" in the network stream,
so the stream contains a 32 bits field encoded with the host endian.
 - This is not reliable for stream betwen different architecture host
 - For x86, the bits doesn't correspond to the documentation.

This patch add some precision in the documentation and put the bitfield
in the stream usig network butes order.

Warning: this patch can break compatibility with existing agents.

This patch should be backported in all version supporing SPOE

Original network capture:

   12:28:16.181343 IP 127.0.0.1.46782 > 127.0.0.1.12345: Flags [P.], seq 134:168, ack 59, win 342, options [nop,nop,TS val 2855241281 ecr 2855241281], length 34
           0x0000:  4500 0056 6b94 4000 4006 d10b 7f00 0001  E..Vk.@.@.......
           0x0010:  7f00 0001 b6be 3039 a3d1 ee54 7d61 d6f7  ......09...T}a..
           0x0020:  8018 0156 fe4a 0000 0101 080a aa2f 8641  ...V.J......./.A
           0x0030:  aa2f 8641 0000 001e 0301 0000 0000 010f  ./.A............
                                          ^^^^^^^^^^
           0x0040:  6368 6563 6b2d 636c 6965 6e74 2d69 7001  check-client-ip.
           0x0050:  0006 7f00 0001                           ......

Fixed network capture:

   12:24:26.948165 IP 127.0.0.1.46706 > 127.0.0.1.12345: Flags [P.], seq 4066280627:4066280661, ack 3148908096, win 342, options [nop,nop,TS val 2855183972 ecr 2855177690], length 34
           0x0000:  4500 0056 0538 4000 4006 3768 7f00 0001  E..V.8@.@.7h....
           0x0010:  7f00 0001 b672 3039 f25e 84b3 bbb0 8640  .....r09.^.....@
           0x0020:  8018 0156 fe4a 0000 0101 080a aa2e a664  ...V.J.........d
           0x0030:  aa2e 8dda 0000 001e 0300 0000 0114 010f  ................
                                          ^^^^^^^^^^
           0x0040:  6368 6563 6b2d 636c 6965 6e74 2d69 7001  check-client-ip.
           0x0050:  0006 7f00 0001                           ......
2018-05-18 13:50:53 +02:00
Thierry FOURNIER
29a05c13d1 BUG/MINOR: spoa-example: unexpected behavior for more than 127 args
Buf is unsigned, so nbargs will be negative for more then 127 args.

Note that I cant test this bug because I cant put sufficient args
on the configuration line. It is just detected reading code.

[wt: this can be backported to 1.8 & 1.7]
2018-03-19 12:59:10 +01:00
Tim Duesterhus
2788a39c07 MINOR: systemd: Add SystemD's SystemCallFilter option to the unit file
This option takes away system calls that are unneeded for haproxy's
operation and thus is a good defense in depth measure.
2018-03-01 15:57:15 +01:00
Tim Duesterhus
8a9659212e MINOR: systemd: Add SystemD's Protect*= options to the unit file
While the haproxy workers usually are running chrooted the master
process is not. This patch is a pretty safe defense in depth measure
to ensure haproxy cannot touch sensitive parts of the file system.

ProtectSystem takes non-boolean arguments in newer SystemD versions,
but setting those would leave older systems such as Ubuntu Xenial
unprotected. Distro maintainers and system administrators could
adapt the ProtectSystem value to the SystemD version they ship.
2018-03-01 15:57:15 +01:00
Tim Duesterhus
1ce8de2d93 MINOR: systemd: Add section for SystemD sandboxing to unit file
This commit adds a warning for settings that possibly provide better
sandboxing and explains their tradeoffs.
2018-03-01 15:57:15 +01:00
Christopher Faulet
0b89f72e88 MINOR: spoa_example: Count the number of frames processed by each worker
This is done for debug purpose. This way, it is easy to know if the load is
equally distributed between workers.
2018-02-02 16:00:32 +01:00
Willy Tarreau
260bf5c106 CONTRIB: debug: fix a few flags definitions
Commit f4cfcf9 ("MINOR: debug/flags: Add missing flags") added a number
of missing flags but a few of them were incorrect, hiding real values.
This can be backported to 1.8.
2018-01-15 18:59:16 +01:00
Willy Tarreau
4576424174 CONTRIB: hpack: add an hpack decoder
This decoder takes a series of hex codes on stdin using one line
per HEADERS frame and shows the decoded headers.
2017-12-30 17:43:28 +01:00
Ryan O'Hara
8cb9993469 CONTRIB: halog: Fix compiler warnings in halog.c
There were several unused variables in halog.c that each caused a
compiler warning [-Wunused-but-set-variable]. This patch simply
removes the declaration of said vairables and any instance where the
unused variable was assigned a value.
2017-12-20 09:36:58 +01:00
Ryan O'Hara
957d12028e CONTRIB: iprange: Fix compiler warning in iprange.c
The declaration of main() in iprange.c did not specify a type, causing
a compiler warning [-Wimplicit-int]. This patch simply declares main()
to be type 'int' and calls exit(0) at the end of the function.
2017-12-20 09:36:58 +01:00
Vincent Bernat
190bbb8f53 MINOR: systemd: remove comment about HAPROXY_STATS_SOCKET
This variable was used by the wrapper which was removed in
a6cfa9098e. The correct way to do seamless reload is now to enable
"expose-fd listeners" on the stat socket.
2017-12-10 07:15:50 +01:00
Aleksandar Lazic
f2b5d75ae2 CONTRIB: halog: Add help text for -s switch in halog program
It was not documented. May be backported to older releases.
2017-12-07 19:27:47 +01:00
Christian Ruppert
57dc283014 BUILD: Fix LDFLAGS vs. LIBS re linking order in various makefiles
Libraries should always be listed last. Should be backported to 1.8.

Signed-off-by: Christian Ruppert <idl0r@qasl.de>
2017-12-02 14:36:15 +01:00
Eric Salama
5438183276 CONTRIB: spoa_example: remove SPOE enums that are useless for clients 2017-11-21 21:33:27 +01:00
Willy Tarreau
75f42466c0 CONTRIB: spoa_example: remove last dependencies on type "sample"
Being an external agent, it's confusing that it uses haproxy's internal
types and it seems to have encouraged other implementations to do so.
Let's completely remove any reference to struct sample and use the
native DATA types instead of converting to and from haproxy's sample
types.
2017-11-21 21:32:52 +01:00
Willy Tarreau
9f95ff0647 CONTRIB: spoa_example: remove bref, wordlist, cond_wordlist
These ones are not needed, let's further reduce the include file.
2017-11-21 21:32:52 +01:00
Eric Salama
8a9c6c2154 CONTRIB: spoa_example: allow to compile outside HAProxy.
Don't include haproxy's includes anymore and use a local copy instead.
2017-11-21 21:32:52 +01:00
Tim Duesterhus
d6942c8297 MEDIUM: mworker: Add systemd Type=notify support
This patch adds support for `Type=notify` to the systemd unit.

Supporting `Type=notify` improves both starting as well as reloading
of the unit, because systemd will be let known when the action completed.

See this quote from `systemd.service(5)`:
> Note however that reloading a daemon by sending a signal (as with the
> example line above) is usually not a good choice, because this is an
> asynchronous operation and hence not suitable to order reloads of
> multiple services against each other. It is strongly recommended to
> set ExecReload= to a command that not only triggers a configuration
> reload of the daemon, but also synchronously waits for it to complete.

By making systemd aware of a reload in progress it is able to wait until
the reload actually succeeded.

This patch introduces both a new `USE_SYSTEMD` build option which controls
including the sd-daemon library as well as a `-Ws` runtime option which
runs haproxy in master-worker mode with systemd support.

When haproxy is running in master-worker mode with systemd support it will
send status messages to systemd using `sd_notify(3)` in the following cases:

- The master process forked off the worker processes (READY=1)
- The master process entered the `mworker_reload()` function (RELOADING=1)
- The master process received the SIGUSR1 or SIGTERM signal (STOPPING=1)

Change the unit file to specify `Type=notify` and replace master-worker
mode (`-W`) with master-worker mode with systemd support (`-Ws`).

Future evolutions of this feature could include making use of the `STATUS`
feature of `sd_notify()` to send information about the number of active
connections to systemd. This would require bidirectional communication
between the master and the workers and thus is left for future work.
2017-11-20 18:39:41 +01:00
Frdric Lcaille
6d889500e2 CONTRIB: Wireshark dissector for HAProxy Peer Protocol. 2017-11-15 19:49:41 +01:00
Christopher Faulet
f4cfcf9a26 MINOR: debug/flags: Add missing flags 2017-11-13 11:38:14 +01:00
Willy Tarreau
8071eae6b9 CONTRIB: hpack: implement a reverse huffman table generator for hpack
This one was built by studying the HPACK Huffman table (RFC7541
appendix B). It creates 5 small tables (4*512 bytes, 1*64 bytes) to
map one byte at a time from the input stream based on the following
observations :

 * rht_bit31_24[256]   is indexed on bits 31..24 when < 0xfe
 * rht_bit24_17[256]   is indexed on bits 24..17 when 31..24 >= 0xfe
 * rht_bit15_11_fe[32] is indexed on bits 15..11 when 24..17 == 0xfe
 * rht_bit15_8[256]    is indexed on bits 15..8 when 24..17 == 0xff
 * rht_bit11_4[256]    is indexed on bits 11..4 when 15..8 == 0xff
 * when 11..4 == 0xff, 3..2 provide the following mapping :
 *   00 => 0x0a, 01 => 0x0d, 10 => 0x16, 11 => EOS
2017-10-31 18:03:24 +01:00
Christopher Faulet
94bb4c6a48 BUG/MINOR: spoa: Update pointer on the end of the frame when a reply is encoded
The same buffer is used for a request and its response. So we need to be sure
to correctly reset info when the response is encoded. And here there was a
bug. The pointer on the end of the frame was not updated.  So it was not
possible to encode a response bigger than the corresponding request.
2017-10-31 11:36:12 +01:00
Willy Tarreau
f5e0d1f9fd CONTRIB: trace: report the base name only for file names
The output is hard to read when the full path to the sources files is
emitted, let's strip it.
2017-10-24 19:54:25 +02:00
Willy Tarreau
e8f0f1265f CONTRIB: trace: try to display the function's return value on exit
On x86_64, when gcc instruments functions and compiles at -O0, it saves
the function's return value in register rbx before calling the trace
callback. It provides a nice opportunity to display certain useful
values (flags, booleans etc) during trace sessions. It's absolutely
not guaranteed that it will always work but it provides a considerable
help when it does so it's worth activating it. When building on a
different architecture, the value 0 is always reported as the return
value. On x86_64 with optimizations (-O), the RBX register will not
necessarily match and random values will be reported, but since it's
not the primary target it's not a problem.
2017-10-24 19:54:25 +02:00
Willy Tarreau
1296382d0b CONTRIB: trace: add the possibility to place trace calls in the code
Now any call to trace() in the code will automatically appear interleaved
with the call sequence and timestamped in the trace file. They appear with
a '#' on the 3rd argument (caller's pointer) in order to make them easy to
spot. If the trace functionality is not used, a dmumy weak function is used
instead so that it doesn't require to recompile every time traces are
enabled/disabled.

The trace decoder knows how to deal with these messages, detects them and
indents them similarly to the currently traced function. This can be used
to print function arguments for example.

Note that we systematically flush the log when calling trace() to ensure we
never miss important events, so this may impact performance.

The trace() function uses the same format as printf() so it should be easy
to setup during debugging sessions.
2017-10-24 19:54:25 +02:00
Olivier Houchard
1a0545f3d7 REORG: connection: rename CO_FL_DATA_* -> CO_FL_XPRT_*
These flags are not exactly for the data layer, they instead indicate
what is expected from the transport layer. Since we're going to split
the connection between the transport and the data layers to insert a
mux layer, it's important to have a clear idea of what each layer does.

All function conn_data_* used to manipulate these flags were renamed to
conn_xprt_*.
2017-10-22 09:54:15 +02:00
Ilya Shipitsin
4473a2e9aa BUG/MINOR: contrib/halog: fixing small memory leak
Issue was identified by cppcheck
2017-10-03 13:52:45 +02:00
Dragan Dosen
2f1cacb1aa BUG/MINOR: contrib/modsecurity: close the va_list ap before return
Make sure the va_list ap is closed before return in case the function
vsnprintf() returned an error.
2017-09-18 11:18:34 +02:00
Dragan Dosen
ccf6100e11 BUG/MINOR: contrib/mod_defender: close the va_list argp before return
Fix the case when the va_list argp could be left open if the function
vsnprintf() returned a negative value.
2017-09-18 11:18:09 +02:00
Willy Tarreau
bbae3f0170 MEDIUM: connection: remove useless flag CO_FL_DATA_WR_SH
After careful inspection, this flag is set at exactly two places :
  - once in the health-check receive callback after receipt of a
    response
  - once in the stream interface's shutw() code where CF_SHUTW is
    always set on chn->flags

The flag was checked in the checks before deciding to send data, but
when it is set, the wake() callback immediately closes the connection
so the CO_FL_SOCK_WR_SH flag is also set.

The flag was also checked in si_conn_send(), but checking the channel's
flag instead is enough and even reveals that one check involving it
could never match.

So it's time to remove this flag and replace its check with a check of
CF_SHUTW in the stream interface. This way each layer is responsible
for its shutdown, this will ease insertion of the mux layer.
2017-08-30 10:05:49 +02:00
Willy Tarreau
54e917cfa1 MEDIUM: connection: remove useless flag CO_FL_DATA_RD_SH
This flag is both confusing and wrong. It is supposed to report the
fact that the data layer has received a shutdown, but in fact this is
reported by CO_FL_SOCK_RD_SH which is set by the transport layer after
this condition is detected. The only case where the flag above is set
is in the stream interface where CF_SHUTR is also set on the receiving
channel.

In addition, it was checked in the health checks code (while never set)
and was always test jointly with CO_FL_SOCK_RD_SH everywhere, except in
conn_data_read0_pending() which incorrectly doesn't match the second
time it's called and is fortunately protected by an extra check on
(ic->flags & CF_SHUTR).

This patch gets rid of the flag completely. Now conn_data_read0_pending()
accurately reports the fact that the transport layer has detected the end
of the stream, regardless of the fact that this state was already consumed,
and the stream interface watches ic->flags&CF_SHUTR to know if the channel
was already closed by the upper layer (which it already used to do).

The now unused conn_data_read0() function was removed.
2017-08-30 08:18:50 +02:00
Willy Tarreau
8e3c6ce75a MEDIUM: connection: get rid of data->init() which was not for data
The ->init() callback of the connection's data layer was only used to
complete the session's initialisation since sessions and streams were
split apart in 1.6. The problem is that it creates a big confusion in
the layers' roles as the session has to register a dummy data layer
when waiting for a handshake to complete, then hand it off to the
stream which will replace it.

The real need is to notify that the transport has finished initializing.
This should enable a better splitting between these layers.

This patch thus introduces a connection-specific callback called
xprt_done_cb() which informs about handshake successes or failures. With
this, data->init() can disappear, CO_FL_INIT_DATA as well, and we don't
need to register a dummy data->wake() callback to be notified of errors.
2017-08-30 07:04:04 +02:00
David Carlier
80ebd30c96 BUG/MINOR: contrib/mod_defender: build fix
In similar manner than modsecurity, making the build possible under *BSD flavors, the -lm for ceilf function.
2017-07-19 14:35:24 +02:00
David Carlier
0f4df640d2 BUG/MINOR: contrib/modsecurity: BSD build fix
previous version introduced in the last commit was not the correct one.
2017-07-19 14:34:31 +02:00
Frdric Lcaille
a4d0361969 CONTRIB: plug qdiscs: Plug queuing disciplines mini HOWTO.
Add plug_qdisc.c source file which may help in how to programatically
use plug queueing disciplines with its README file.
Such code may be useful to reproduce painful network application bugs.
2017-06-14 16:22:34 +02:00
Willy Tarreau
2686dcad1e CLEANUP: connection: remove unused CO_FL_WAIT_DATA
Very early in the connection rework process leading to v1.5-dev12, commit
56a77e5 ("MEDIUM: connection: complete the polling cleanups") marked the
end of use for this flag which since was never set anymore, but it continues
to be tested. Let's kill it now.
2017-06-02 15:50:27 +02:00
Dragan Dosen
59bb97a192 MINOR: Add Mod Defender integration as contrib
This is a service that talks SPOE protocol and uses the Mod Defender (a
NAXSI clone) functionality to detect HTTP attacks. It returns a HTTP
status code to indicate whether the request is suspicious or not, based on
NAXSI rules. The value of the returned code can be used in HAProxy rules
to determine if the HTTP request should be blocked/rejected.
2017-06-02 12:14:55 +02:00
William Lallemand
a6cfa9098e MAJOR: systemd-wrapper: get rid of the wrapper
The master worker mode obsoletes the systemd-wrapper, to ensure that
nobody uses it anymore, the code has been removed.
2017-06-02 10:56:32 +02:00
William Lallemand
6db884d961 MEDIUM: systemd: Type=forking in unit file
Adding Type=forking in the unit file ensure better monitoring from
systemd. During a systemctl start the tool is able to return an error if
it didn't work with this option.
2017-06-02 10:56:32 +02:00