Commit Graph

158 Commits

Author SHA1 Message Date
Willy Tarreau
05b9b64afb MINOR: stream-int: replace SI_FL_WANT_PUT with !SI_FL_RX_WAIT_EP
The SI_FL_WANT_PUT flag is used in an awkward way, sometimes it's
set by the stream-interface to mean "I have something to deliver",
sometimes it's cleared by the channel to say "I don't want you to
send what you have", and it has to be set back once CF_DONT_READ
is cleared. This will have to be split between SI_FL_RX_WAIT_EP
and SI_FL_RXBLK_CHAN. This patch only replaces all uses of the
flag with its natural (but negated) replacement SI_FL_RX_WAIT_EP.
The code is expected to be strictly equivalent. The now unused flag
was completely removed.
2018-11-18 21:41:46 +01:00
Willy Tarreau
94f7907d65 MINOR: stream-int: introduce new SI_FL_RXBLK flags
The plan is to have the following flags to describe why a stream interface
doesn't produce data :

    - SI_FL_RXBLK_CHAN : the channel doesn't want it to receive
    - SI_FL_RXBLK_BUFF : waiting for a buffer allocation to complete
    - SI_FL_RXBLK_ROOM : more room is required in the channel to receive
    - SI_FL_RXBLK_SHUT : input now closed, nothing new will come
    - SI_FL_RX_WAIT_EP : waiting for the endpoint to produce more data

Applets like the CLI which consume complete commands at once and produce
large chunks of responses will for example be able to stop being woken up
by clearing SI_FL_WANT_GET and setting SI_FL_RXBLK_ROOM when the rx buffer
is full. Once called they will unblock WANT_GET. The flags were moved
together in readable form with the Rx bits using 2 hex digits and still
have some room to do a similar operation on the Tx path later, with the
WAIT_EP flag being represented alone on a digit.
2018-11-18 21:41:45 +01:00
Willy Tarreau
d0f5bbcd64 MINOR: stream-int: rename SI_FL_WAIT_ROOM to SI_FL_RXBLK_ROOM
This flag is not enough to describe all blocking situations, as can be
seen in each case we remove it. The muxes has taught us that using multiple
blocking flags in parallel will be much easier, so let's start to do this
now. This patch only renames this flags in order to make next changes more
readable.
2018-11-18 21:41:45 +01:00
Willy Tarreau
d5016469bf CONTRIB: debug: fix build related to conn_stream flags change
Commit 53216e7db ("MEDIUM: connections: Don't directly mess with the
polling from the upper layers.") removed the CS_FL_DATA_RD_ENA and
CS_FL_DATA_WR_ENA flags without updating flags.c, thus breaking the
build. This patch also adds flag CL_FL_NOT_FIRST which was brought
by commit 08088e77c.
2018-11-16 10:39:50 +01:00
Joseph Herlant
bd0f83f80b CLEANUP: fix typos in comments for contrib/wireshark-dissectors
This fixes a typo in the README of the peers section of this subsystem
and 2 typos in code comments. Groupped together as cleanup to avoid too
many 1 char patches.
2018-11-12 08:53:16 +01:00
Joseph Herlant
ebe14bbbef CLEANUP: fix typos in comments for contrib/spoa_example
Fixes 3 common typos in the comments of the contrib/spoa_example
subsystem.
2018-11-12 08:52:54 +01:00
Joseph Herlant
9fe83fa639 CLEANUP: fix typos in comments for the contrib/modsecurity subsystem
3 typos detected in code comments in the contrib/modsecurity subsystem.
2018-11-12 08:52:36 +01:00
Joseph Herlant
42172bdc97 CLEANUP: fix a typo in a comment for the contrib/halog subsystem
Typo in comment, not visible by end-users.
2018-11-12 08:52:16 +01:00
Willy Tarreau
ede3d884fc MEDIUM: channel: merge back flags CF_WRITE_PARTIAL and CF_WRITE_EVENT
The behaviour of the flag CF_WRITE_PARTIAL was modified by commit
95fad5ba4 ("BUG/MAJOR: stream-int: don't re-arm recv if send fails") due
to a situation where it could trigger an immediate wake up of the other
side, both acting in loops via the FD cache. This loss has caused the
need to introduce CF_WRITE_EVENT as commit c5a9d5bf, to replace it, but
both flags express more or less the same thing and this distinction
creates a lot of confusion and complexity in the code.

Since the FD cache now acts via tasklets, the issue worked around in the
first patch no longer exists, so it's more than time to kill this hack
and to restore CF_WRITE_PARTIAL's semantics (i.e.: there has been some
write activity since we last left process_stream).

This patch mostly reverts the two commits above. Only the part making
use of CF_WROTE_DATA instead of CF_WRITE_PARTIAL to detect the loss of
data upon connection setup was kept because it's more accurate and
better suited.
2018-10-26 08:32:57 +02:00
Willy Tarreau
5cc8a0a7ac CLEANUP/CONTRIB: hpack: remove some h1 build warnings
These are inherited by recent reorganization to the H1 code.
2018-09-20 09:41:24 +02:00
Christopher Faulet
5f8ef13d5d MINOR: debug: Add checks for conn_stream flags
This may be carefully backported to 1.8 (a few flags don't exist there).
2018-07-20 13:39:30 +02:00
Christopher Faulet
aff9328739 MINOR: debug: Add check for CO_FL_WILL_UPDATE
This could be backported to 1.8.
2018-07-20 13:39:30 +02:00
Willy Tarreau
83061a820e MAJOR: chunks: replace struct chunk with struct buffer
Now all the code used to manipulate chunks uses a struct buffer instead.
The functions are still called "chunk*", and some of them will progressively
move to the generic buffer handling code as they are cleaned up.
2018-07-19 16:23:43 +02:00
Willy Tarreau
843b7cbe9d MEDIUM: chunks: make the chunk struct's fields match the buffer struct
Chunks are only a subset of a buffer (a non-wrapping version with no head
offset). Despite this we still carry a lot of duplicated code between
buffers and chunks. Replacing chunks with buffers would significantly
reduce the maintenance efforts. This first patch renames the chunk's
fields to match the name and types used by struct buffers, with the goal
of isolating the code changes from the declaration changes.

Most of the changes were made with spatch using this coccinelle script :

  @rule_d1@
  typedef chunk;
  struct chunk chunk;
  @@
  - chunk.str
  + chunk.area

  @rule_d2@
  typedef chunk;
  struct chunk chunk;
  @@
  - chunk.len
  + chunk.data

  @rule_i1@
  typedef chunk;
  struct chunk *chunk;
  @@
  - chunk->str
  + chunk->area

  @rule_i2@
  typedef chunk;
  struct chunk *chunk;
  @@
  - chunk->len
  + chunk->data

Some minor updates to 3 http functions had to be performed to take size_t
ints instead of ints in order to match the unsigned length here.
2018-07-19 16:23:43 +02:00
Vincent Bernat
3b479bd5f5 MINOR: systemd: consider exit status 143 as successful
The master process will exit with the status of the last worker. When
the worker is killed with SIGTERM, it is expected to get 143 as an
exit status. Therefore, we consider this exit status as normal from a
systemd point of view. If it happens when not stopping, the systemd
unit is configured to always restart, so it has no adverse effect.

This has mostly a cosmetic effect. Without the patch, stopping HAProxy
leads to the following status:

    â— haproxy.service - HAProxy Load Balancer
       Loaded: loaded (/lib/systemd/system/haproxy.service; disabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Fri 2018-06-22 20:35:42 CEST; 8min ago
         Docs: man:haproxy(1)
               file:/usr/share/doc/haproxy/configuration.txt.gz
      Process: 32715 ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS (code=exited, status=143)
      Process: 32714 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
     Main PID: 32715 (code=exited, status=143)

After the patch:

    â— haproxy.service - HAProxy Load Balancer
       Loaded: loaded (/lib/systemd/system/haproxy.service; disabled; vendor preset: enabled)
       Active: inactive (dead)
         Docs: man:haproxy(1)
               file:/usr/share/doc/haproxy/configuration.txt.gz
2018-07-12 17:44:51 +02:00
Dragan Dosen
ee07dbeca5 BUG/MINOR: contrib/modsecurity: update pointer on the end of the frame
Similar to commit 94bb4c6 ("BUG/MINOR: spoa: Update pointer on the end of
the frame when a reply is encoded").

This patch should be backported to 1.8.
2018-06-04 17:41:25 +02:00
Dragan Dosen
f21185f236 BUG/MINOR: contrib/mod_defender: update pointer on the end of the frame
Similar to commit 94bb4c6 ("BUG/MINOR: spoa: Update pointer on the end of
the frame when a reply is encoded").

This patch should be backported to 1.8.
2018-06-04 17:41:25 +02:00
Christopher Faulet
58b8e492a3 BUG/MINOR: contrib/modsecurity: Don't reset the status code during disconnect
When the connection is closed by HAProxy, the status code provided in the
DISCONNECT frame is lost. By retransmitting it in the agent's reply, we are sure
to have it in the SPOE logs.

This patch may be backported in 1.8.
2018-06-04 17:34:51 +02:00
Christopher Faulet
72de8eb7aa BUG/MINOR: contrib/mod_defender: Don't reset the status code during disconnect
When the connection is closed by HAProxy, the status code provided in the
DISCONNECT frame is lost. By retransmitting it in the agent's reply, we are sure
to have it in the SPOE logs.

This patch may be backported in 1.8.
2018-06-04 17:34:50 +02:00
Christopher Faulet
b47e438593 BUG/MINOR: contrib/spoa_example: Don't reset the status code during disconnect
When the connection is closed by HAProxy, the status code provided in the
DISCONNECT frame is lost. By retransmitting it in the agent's reply, we are sure
to have it in the SPOE logs.

This patch may be backported in 1.8.
2018-06-04 17:34:50 +02:00
Christopher Faulet
6381650516 MAJOR: spoe: upgrade the SPOP version to 2.0 and remove the support for 1.0
The commit c4dcaff3 ("BUG/MEDIUM: spoe: Flags are not encoded in network order")
introduced an incompatibility with older agents. So the major version of the
SPOP is increased to make the situation unambiguous. And because before the fix,
the protocol is buggy, the support of the version 1.0 is removed to be sure to
not continue to support buggy agents.

The agents in the contrib folder (spoa_example, modsecurity and mod_defender)
are also updated to announce the SPOP version 2.0.

So, to be clear, from the patch, connections to agents announcing the SPOP
version 1.0 will be rejected.

This patch must be backported in 1.8.
2018-06-04 17:33:48 +02:00
David Carlier
53d31594bf DOC: contrib/modsecurity: few typo fixes
Few typo fixes.
2018-05-31 20:47:39 +02:00
Christopher Faulet
633f3bffed BUG/MEDIUM: contrib/modsecurity: Use network order to encode/decode flags
A recent fix on the SPOE revealed a mismatch between the SPOE specification and
the modsecurity implementation on the way flags are encoded or decoded. They
must be exchanged using the network bytes order and not the host one.

Be careful though, this patch breaks the compatiblity with HAProxy SPOE before
commit c4dcaff3 ("BUG/MEDIUM: spoe: Flags are not encoded in network order").
2018-05-18 15:06:31 +02:00
Christopher Faulet
48d02d0d21 BUG/MEDIUM: contrib/mod_defender: Use network order to encode/decode flags
A recent fix on the SPOE revealed a mismatch between the SPOE specification and
the mod_defender implementation on the way flags are encoded or decoded. They
must be exchanged using the network bytes order and not the host one.

Be careful though, this patch breaks the compatiblity with HAProxy SPOE before
commit c4dcaff3 ("BUG/MEDIUM: spoe: Flags are not encoded in network order").
2018-05-18 15:06:18 +02:00
Thierry FOURNIER
c4dcaff3f0 BUG/MEDIUM: spoe: Flags are not encoded in network order
The flags are direct copy of the "unsigned int" in the network stream,
so the stream contains a 32 bits field encoded with the host endian.
 - This is not reliable for stream betwen different architecture host
 - For x86, the bits doesn't correspond to the documentation.

This patch add some precision in the documentation and put the bitfield
in the stream usig network butes order.

Warning: this patch can break compatibility with existing agents.

This patch should be backported in all version supporing SPOE

Original network capture:

   12:28:16.181343 IP 127.0.0.1.46782 > 127.0.0.1.12345: Flags [P.], seq 134:168, ack 59, win 342, options [nop,nop,TS val 2855241281 ecr 2855241281], length 34
           0x0000:  4500 0056 6b94 4000 4006 d10b 7f00 0001  E..Vk.@.@.......
           0x0010:  7f00 0001 b6be 3039 a3d1 ee54 7d61 d6f7  ......09...T}a..
           0x0020:  8018 0156 fe4a 0000 0101 080a aa2f 8641  ...V.J......./.A
           0x0030:  aa2f 8641 0000 001e 0301 0000 0000 010f  ./.A............
                                          ^^^^^^^^^^
           0x0040:  6368 6563 6b2d 636c 6965 6e74 2d69 7001  check-client-ip.
           0x0050:  0006 7f00 0001                           ......

Fixed network capture:

   12:24:26.948165 IP 127.0.0.1.46706 > 127.0.0.1.12345: Flags [P.], seq 4066280627:4066280661, ack 3148908096, win 342, options [nop,nop,TS val 2855183972 ecr 2855177690], length 34
           0x0000:  4500 0056 0538 4000 4006 3768 7f00 0001  E..V.8@.@.7h....
           0x0010:  7f00 0001 b672 3039 f25e 84b3 bbb0 8640  .....r09.^.....@
           0x0020:  8018 0156 fe4a 0000 0101 080a aa2e a664  ...V.J.........d
           0x0030:  aa2e 8dda 0000 001e 0300 0000 0114 010f  ................
                                          ^^^^^^^^^^
           0x0040:  6368 6563 6b2d 636c 6965 6e74 2d69 7001  check-client-ip.
           0x0050:  0006 7f00 0001                           ......
2018-05-18 13:50:53 +02:00
Thierry FOURNIER
29a05c13d1 BUG/MINOR: spoa-example: unexpected behavior for more than 127 args
Buf is unsigned, so nbargs will be negative for more then 127 args.

Note that I cant test this bug because I cant put sufficient args
on the configuration line. It is just detected reading code.

[wt: this can be backported to 1.8 & 1.7]
2018-03-19 12:59:10 +01:00
Tim Duesterhus
2788a39c07 MINOR: systemd: Add SystemD's SystemCallFilter option to the unit file
This option takes away system calls that are unneeded for haproxy's
operation and thus is a good defense in depth measure.
2018-03-01 15:57:15 +01:00
Tim Duesterhus
8a9659212e MINOR: systemd: Add SystemD's Protect*= options to the unit file
While the haproxy workers usually are running chrooted the master
process is not. This patch is a pretty safe defense in depth measure
to ensure haproxy cannot touch sensitive parts of the file system.

ProtectSystem takes non-boolean arguments in newer SystemD versions,
but setting those would leave older systems such as Ubuntu Xenial
unprotected. Distro maintainers and system administrators could
adapt the ProtectSystem value to the SystemD version they ship.
2018-03-01 15:57:15 +01:00
Tim Duesterhus
1ce8de2d93 MINOR: systemd: Add section for SystemD sandboxing to unit file
This commit adds a warning for settings that possibly provide better
sandboxing and explains their tradeoffs.
2018-03-01 15:57:15 +01:00
Christopher Faulet
0b89f72e88 MINOR: spoa_example: Count the number of frames processed by each worker
This is done for debug purpose. This way, it is easy to know if the load is
equally distributed between workers.
2018-02-02 16:00:32 +01:00
Willy Tarreau
260bf5c106 CONTRIB: debug: fix a few flags definitions
Commit f4cfcf9 ("MINOR: debug/flags: Add missing flags") added a number
of missing flags but a few of them were incorrect, hiding real values.
This can be backported to 1.8.
2018-01-15 18:59:16 +01:00
Willy Tarreau
4576424174 CONTRIB: hpack: add an hpack decoder
This decoder takes a series of hex codes on stdin using one line
per HEADERS frame and shows the decoded headers.
2017-12-30 17:43:28 +01:00
Ryan O'Hara
8cb9993469 CONTRIB: halog: Fix compiler warnings in halog.c
There were several unused variables in halog.c that each caused a
compiler warning [-Wunused-but-set-variable]. This patch simply
removes the declaration of said vairables and any instance where the
unused variable was assigned a value.
2017-12-20 09:36:58 +01:00
Ryan O'Hara
957d12028e CONTRIB: iprange: Fix compiler warning in iprange.c
The declaration of main() in iprange.c did not specify a type, causing
a compiler warning [-Wimplicit-int]. This patch simply declares main()
to be type 'int' and calls exit(0) at the end of the function.
2017-12-20 09:36:58 +01:00
Vincent Bernat
190bbb8f53 MINOR: systemd: remove comment about HAPROXY_STATS_SOCKET
This variable was used by the wrapper which was removed in
a6cfa9098e. The correct way to do seamless reload is now to enable
"expose-fd listeners" on the stat socket.
2017-12-10 07:15:50 +01:00
Aleksandar Lazic
f2b5d75ae2 CONTRIB: halog: Add help text for -s switch in halog program
It was not documented. May be backported to older releases.
2017-12-07 19:27:47 +01:00
Christian Ruppert
57dc283014 BUILD: Fix LDFLAGS vs. LIBS re linking order in various makefiles
Libraries should always be listed last. Should be backported to 1.8.

Signed-off-by: Christian Ruppert <idl0r@qasl.de>
2017-12-02 14:36:15 +01:00
Eric Salama
5438183276 CONTRIB: spoa_example: remove SPOE enums that are useless for clients 2017-11-21 21:33:27 +01:00
Willy Tarreau
75f42466c0 CONTRIB: spoa_example: remove last dependencies on type "sample"
Being an external agent, it's confusing that it uses haproxy's internal
types and it seems to have encouraged other implementations to do so.
Let's completely remove any reference to struct sample and use the
native DATA types instead of converting to and from haproxy's sample
types.
2017-11-21 21:32:52 +01:00
Willy Tarreau
9f95ff0647 CONTRIB: spoa_example: remove bref, wordlist, cond_wordlist
These ones are not needed, let's further reduce the include file.
2017-11-21 21:32:52 +01:00
Eric Salama
8a9c6c2154 CONTRIB: spoa_example: allow to compile outside HAProxy.
Don't include haproxy's includes anymore and use a local copy instead.
2017-11-21 21:32:52 +01:00
Tim Duesterhus
d6942c8297 MEDIUM: mworker: Add systemd Type=notify support
This patch adds support for `Type=notify` to the systemd unit.

Supporting `Type=notify` improves both starting as well as reloading
of the unit, because systemd will be let known when the action completed.

See this quote from `systemd.service(5)`:
> Note however that reloading a daemon by sending a signal (as with the
> example line above) is usually not a good choice, because this is an
> asynchronous operation and hence not suitable to order reloads of
> multiple services against each other. It is strongly recommended to
> set ExecReload= to a command that not only triggers a configuration
> reload of the daemon, but also synchronously waits for it to complete.

By making systemd aware of a reload in progress it is able to wait until
the reload actually succeeded.

This patch introduces both a new `USE_SYSTEMD` build option which controls
including the sd-daemon library as well as a `-Ws` runtime option which
runs haproxy in master-worker mode with systemd support.

When haproxy is running in master-worker mode with systemd support it will
send status messages to systemd using `sd_notify(3)` in the following cases:

- The master process forked off the worker processes (READY=1)
- The master process entered the `mworker_reload()` function (RELOADING=1)
- The master process received the SIGUSR1 or SIGTERM signal (STOPPING=1)

Change the unit file to specify `Type=notify` and replace master-worker
mode (`-W`) with master-worker mode with systemd support (`-Ws`).

Future evolutions of this feature could include making use of the `STATUS`
feature of `sd_notify()` to send information about the number of active
connections to systemd. This would require bidirectional communication
between the master and the workers and thus is left for future work.
2017-11-20 18:39:41 +01:00
Frédéric Lécaille
6d889500e2 CONTRIB: Wireshark dissector for HAProxy Peer Protocol. 2017-11-15 19:49:41 +01:00
Christopher Faulet
f4cfcf9a26 MINOR: debug/flags: Add missing flags 2017-11-13 11:38:14 +01:00
Willy Tarreau
8071eae6b9 CONTRIB: hpack: implement a reverse huffman table generator for hpack
This one was built by studying the HPACK Huffman table (RFC7541
appendix B). It creates 5 small tables (4*512 bytes, 1*64 bytes) to
map one byte at a time from the input stream based on the following
observations :

 * rht_bit31_24[256]   is indexed on bits 31..24 when < 0xfe
 * rht_bit24_17[256]   is indexed on bits 24..17 when 31..24 >= 0xfe
 * rht_bit15_11_fe[32] is indexed on bits 15..11 when 24..17 == 0xfe
 * rht_bit15_8[256]    is indexed on bits 15..8 when 24..17 == 0xff
 * rht_bit11_4[256]    is indexed on bits 11..4 when 15..8 == 0xff
 * when 11..4 == 0xff, 3..2 provide the following mapping :
 *   00 => 0x0a, 01 => 0x0d, 10 => 0x16, 11 => EOS
2017-10-31 18:03:24 +01:00
Christopher Faulet
94bb4c6a48 BUG/MINOR: spoa: Update pointer on the end of the frame when a reply is encoded
The same buffer is used for a request and its response. So we need to be sure
to correctly reset info when the response is encoded. And here there was a
bug. The pointer on the end of the frame was not updated.  So it was not
possible to encode a response bigger than the corresponding request.
2017-10-31 11:36:12 +01:00
Willy Tarreau
f5e0d1f9fd CONTRIB: trace: report the base name only for file names
The output is hard to read when the full path to the sources files is
emitted, let's strip it.
2017-10-24 19:54:25 +02:00
Willy Tarreau
e8f0f1265f CONTRIB: trace: try to display the function's return value on exit
On x86_64, when gcc instruments functions and compiles at -O0, it saves
the function's return value in register rbx before calling the trace
callback. It provides a nice opportunity to display certain useful
values (flags, booleans etc) during trace sessions. It's absolutely
not guaranteed that it will always work but it provides a considerable
help when it does so it's worth activating it. When building on a
different architecture, the value 0 is always reported as the return
value. On x86_64 with optimizations (-O), the RBX register will not
necessarily match and random values will be reported, but since it's
not the primary target it's not a problem.
2017-10-24 19:54:25 +02:00
Willy Tarreau
1296382d0b CONTRIB: trace: add the possibility to place trace calls in the code
Now any call to trace() in the code will automatically appear interleaved
with the call sequence and timestamped in the trace file. They appear with
a '#' on the 3rd argument (caller's pointer) in order to make them easy to
spot. If the trace functionality is not used, a dmumy weak function is used
instead so that it doesn't require to recompile every time traces are
enabled/disabled.

The trace decoder knows how to deal with these messages, detects them and
indents them similarly to the currently traced function. This can be used
to print function arguments for example.

Note that we systematically flush the log when calling trace() to ensure we
never miss important events, so this may impact performance.

The trace() function uses the same format as printf() so it should be easy
to setup during debugging sessions.
2017-10-24 19:54:25 +02:00
Olivier Houchard
1a0545f3d7 REORG: connection: rename CO_FL_DATA_* -> CO_FL_XPRT_*
These flags are not exactly for the data layer, they instead indicate
what is expected from the transport layer. Since we're going to split
the connection between the transport and the data layers to insert a
mux layer, it's important to have a clear idea of what each layer does.

All function conn_data_* used to manipulate these flags were renamed to
conn_xprt_*.
2017-10-22 09:54:15 +02:00