Commit Graph

640 Commits

Author SHA1 Message Date
Tim Duesterhus
22586524e3 BUG/MINOR: hlua: Fix memory leak in hlua_alloc
During a configuration check valgrind reports:

    ==14425== 0 bytes in 106 blocks are definitely lost in loss record 1 of 107
    ==14425==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==14425==    by 0x4C2FDEF: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==14425==    by 0x443CFC: hlua_alloc (hlua.c:8662)
    ==14425==    by 0x5F72B11: luaM_realloc_ (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
    ==14425==    by 0x5F78089: luaH_free (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
    ==14425==    by 0x5F707D3: sweeplist (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
    ==14425==    by 0x5F710D0: luaC_freeallobjects (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
    ==14425==    by 0x5F7715D: close_state (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
    ==14425==    by 0x443D4C: hlua_deinit (hlua.c:9302)
    ==14425==    by 0x543F88: deinit (haproxy.c:2742)
    ==14425==    by 0x5448E7: deinit_and_exit (haproxy.c:2830)
    ==14425==    by 0x5455D9: init (haproxy.c:2044)

This is due to Lua calling `hlua_alloc()` with `ptr = NULL` and `nsize = 0`.
While `realloc` is supposed to be equivalent `free()` if the size is `0` this
is only required for a non-NULL pointer. Apparently my allocator (or valgrind)
actually allocates a zero size area if the pointer is NULL, possibly taking up
some memory for management structures.

Fix this leak by specifically handling the case where both the pointer and the
size are `0`.

This bug appears to have been introduced with the introduction of the
multi-threaded Lua, thus this fix is specific for 2.4. No backport needed.
2021-01-08 14:46:43 +01:00
Ilya Shipitsin
b8888ab557 CLEANUP: assorted typo fixes in the code and comments
This is 15th iteration of typo fixes
2021-01-06 17:32:03 +01:00
Tim Duesterhus
e5ff14100a CLEANUP: Compare the return value of XXXcmp() functions with zero
According to coding-style.txt it is recommended to use:

`strcmp(a, b) == 0` instead of `!strcmp(a, b)`

So let's do this.

The change was performed by running the following (very long) coccinelle patch
on src/:

    @@
    statement S;
    expression E;
    expression F;
    @@

      if (
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
      )
    (
      S
    |
      { ... }
    )

    @@
    statement S;
    expression E;
    expression F;
    @@

      if (
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
      )
    (
      S
    |
      { ... }
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    G &&
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    G ||
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
    && G
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) != 0
    || G
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    G &&
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    G ||
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    && G
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    || G
    )

    @@
    expression E;
    expression F;
    expression G;
    @@

    (
    - !
    (
    dns_hostname_cmp
    |
    eb_memcmp
    |
    memcmp
    |
    strcasecmp
    |
    strcmp
    |
    strncasecmp
    |
    strncmp
    )
    -  (E, F)
    +  (E, F) == 0
    )
2021-01-04 10:09:02 +01:00
Tim Duesterhus
f89d43a381 MINOR: lua: Use consistent error message 'memory allocation failed'
Other locations in the configuration parser use 'memory allocation failed', so
use this one as well.
2021-01-03 20:37:16 +01:00
Tim Duesterhus
621e74afd1 BUG/MINOR: lua: Fix memory leak error cases in hlua_config_prepend_path
In case of an error `p` is not properly freed.

Minor leak during configuration parsing in out of memory situations, no
backport needed.
2021-01-03 20:37:16 +01:00
Christopher Faulet
26256f86e1 MINOR: stream: Pass an optional input buffer when a stream is created
It is now possible to set the buffer used by the channel request buffer when
a stream is created. It may be useful if input data are already received,
instead of waiting the first call to the mux rcv_buf() callback. This change
is mandatory to support H1 connection with no stream attached.

For now, the multiplexers don't pass any buffer. BUF_NULL is thus used to
call stream_create_from_cs().
2020-12-04 14:41:48 +01:00
Willy Tarreau
186f37674c BUG/MINOR: lua-thread: close all states on deinit
It seems to me that lua_close() must be called on all states at deinit
time, not just the first two ones. This is likely a remnant of commit
59f11be43 ("MEDIUM: lua-thread: Add the lua-load-per-thread directive").
There should likely be some memory leak reports when using Lua without
this fix, though none were observed for now.

No backport is needed as this was merged into 2.4-dev.
2020-12-04 12:00:11 +01:00
Thierry Fournier
aafc777854 BUG/MEDIUM: lua-thread: some parts must be initialized once
Lua dedicated TCP, HTTP and SSL socket and proxies must be initialized
once. Right now, they are initialized from the Lua init state, but since
commit 59f11be43 ("MEDIUM: lua-thread: Add the lua-load-per-thread
directive") this function is called one time per lua context. This
caused some fields to be cleared and overwritten, and pre-allocated
object to be lost. This is why the address sanitizer detected memory
leaks from the socket_ssl server initialization.

Let's move all the state-independent part of the function to the
hlua_init() function to avoid this.

No backport is needed, this is only 2.4-dev.
2020-12-04 11:55:05 +01:00
Thierry Fournier
46278ff828 MINOR: lua-thread: Add verbosity in errors
Because lua-load-per-thread could not load the same code for each thread,
this patch displays the state-id associated with the error.
2020-12-02 21:53:16 +01:00
Thierry Fournier
59f11be436 MEDIUM: lua-thread: Add the lua-load-per-thread directive
The goal is to allow execution of one main lua state per thread.

This patch contains the main job. The lua init is done using these
steps:
 - "lua-load-per-thread" loads the lua code in the first thread
 - it creates the structs
 - it stores loaded files
 - the 1st step load is completed (execution of hlua_post_init)
   and now, we known the number of threads
 - we initilize lua states for all remaining threads
 - for each one, we load the lua file
 - for each one, we execute post-init

Once all is loaded, we control consistency of functions references.
The rules are:
 - a function reference cannot be in the shared lua state and in
   a per-thread lua state at the same time.
 - if a function reference is declared in a per-thread lua state, it
   must be declared in all per-thread lua states
2020-12-02 21:53:16 +01:00
Thierry Fournier
c749259dff MINOR: lua-thread: Store each function reference and init reference in array
The goal is to allow execution of one main lua state per thread.

The array introduces storage of one reference per thread, because each
lua state can have different reference id for a same function. A function
returns the preferred state id according to configuration and current
thread id.
2020-12-02 21:53:16 +01:00
Thierry Fournier
021d986ecc MINOR: lua-thread: Replace state_from by state_id
The goal is to allow execution of one main lua state per thread.

"state_from" is a pointer to the parent lua state. "state_id"
is the index of the parent state id in the reference lua states
array. "state_id" is better because the lock is a "== 0" test
which is quick than pointer comparison. In other way, the state_id
index could index other things the the Lua state concerned. I
think to the function references.
2020-12-02 21:53:16 +01:00
Thierry Fournier
62a22aa23f MINOR: lua-thread: Replace "struct hlua_function" allocation by dedicated function
The goal is to allow execution of one main lua state per thread.

This function will initialize the struct with other things than 0.
With this function helper, the initialization is centralized and
it prevents mistakes. This patch also keeps a reference to each
declared function in a list. It will be useful in next patches to
control consistency of declared references.
2020-12-02 21:53:16 +01:00
Thierry Fournier
afc63e2cb1 MINOR: lua-thread: Replace global gL var with an array of states
The goal is to allow execution of one main lua state per thread.

The array of states is initialized at the max number of thread +1.
We define the index 0 is the common state shared by all threads
and should be locked. Other index index are dedicated to each
one thread. The old gL now becomes hlua_states[0].
2020-12-02 21:53:16 +01:00
Thierry Fournier
7cbe5046e8 MEDIUM: lua-thread: Apply lock only if the parent state is the main thread
The goal is to allow execution of one main lua state per thread.

This patch opens the way to addition of a per-thread dedicated lua state.
By passing the hlua we can figure the original state that's been used
and decide to lock or not.
2020-12-02 21:53:16 +01:00
Thierry Fournier
3c539327f4 MEDIUM: lua-thread: No longer use locked context in initialization parts
The goal is to allow execution of one main lua state per thread.

Stop using locks in init part, we will use only in parts where
the parent lua state is known, so we could take decision about lock
according with the lua parent state.
2020-12-02 21:53:16 +01:00
Thierry Fournier
ecb83c24c4 MINOR: lua-thread: Add the "thread" core variable
The goal is to allow execution of one main lua state per thread.

This commit introduces this variable in the core. Lua state initialized
by thread will have access to this variable, which reports the executing
thread. 0 indicates the shared thread. Programs which must be executed
only once can check for core.thread <= 1.
2020-12-02 21:53:16 +01:00
Thierry Fournier
b8cef175bd MINOR: lua-thread: Split hlua_post_init() function in two parts
The goal is to allow execution of one main lua state per thread.

This function will be called for each initialized lua state, so
one per thread. The split transforms the lua state variable from
global to local.
2020-12-02 21:53:16 +01:00
Thierry Fournier
c93c15cf8c MINOR: lua-thread: Split hlua_load function in two parts
The goal is to allow execution of one main lua state per thread.

This function will be called once per thread, using different Lua
states. This patch prepares the work.
2020-12-02 21:53:16 +01:00
Thierry Fournier
75fc02956b MINOR: lua-thread: make hlua_ctx_init() get L from its caller
The goal is to allow execution of one main lua state per thread.

The function hlua_ctx_init() now gets the original lua state from
its caller. This allows the initialisation of lua_thread (coroutines)
from any master lua state.

The parent lua state is stored in the hlua struct.

This patch is a temporary transition, it will be modified later.
2020-12-02 21:53:16 +01:00
Thierry Fournier
1eac28f5fc MINOR: lua-thread: Split hlua_init() function in two parts
The goal is to allow execution of one main lua state per thread.

This is a preparative work in order to init more than one stack
in the lua-thread objective.
2020-12-02 21:53:16 +01:00
Thierry Fournier
ad5345fed7 MINOR: lua-thread: Replace embedded struct hlua_function by a pointer
The goal is to allow execution of one main lua state per thread.

Because this struct will be filled after the configuration parser, we
cannot copy the content. The actual state of the Haproxy code doesn't
justify this change, it is an update preparing next steps.
2020-12-02 21:53:16 +01:00
Thierry Fournier
92689e651e MINOR: lua-thread: Stop usage of struct hlua for the global lua state
The goal is to no longer use "struct hlua" with global main lua_state.

The usage of the "struct hlua" is no longer required. This patch replaces
this struct by another one.

Now, the usage of runtime Lua phase is separated from the start lua phase.
2020-12-02 21:53:16 +01:00
Thierry Fournier
4234dbd03b MINOR: lua-thread: Use NULL context for main lua state
The goal is to no longer use "struct hlua" with global main lua_state.

This patch returns NULL value when some code tries go get the hlua struct
associated with a task through hlua_gethlua(). This functions is useful
only during runtime because the struct hlua contains only runtime states.

Some Lua functions allowed to yield are called from init environment.
I'm not sure this is a good practice. Maybe it will be clever to
disallow calling this kind of functions.
2020-12-02 21:53:16 +01:00
Thierry Fournier
9eb3230b7c MINOR: lua-thread: hlua_ctx_renew() is never called with main gL lua state
The goal is no longer using "struct hlua" with global main lua_state.

if somewhere in the code, hlua_ctx_renew() is called with a global Lua
context, we have a serious bug. A crash is better than working with
this bug, so this patch remove a useless control.

In other way, this control were used during hlua_post_init() function.
The function hlua_post_init() used a call to the runtime hlua_ctx_resume()
function. This call no longer exists.
2020-12-02 21:53:16 +01:00
Thierry Fournier
670db24329 MEDIUM: lua-thread: make hlua_post_init() no longer use the runtime execution function
The goal is to no longer use "struct hlua" with global main lua_state.

The hlua_post_init() is executed during start phase, it does not require
yielding nor any advanced runtime error processing. Let's simplify this
by re-implementing the code using lower-level functions which directly
take a state and not an hlua anymore.
2020-12-02 21:53:16 +01:00
Thierry Fournier
3fb9e5133a MINOR: lua-thread: remove struct hlua from function hlua_prepend_path()
The goal is to no longer use "struct hlua" with global main lua_state
and directly take the state instead.

This patch removes the implicit dependency to this struct with
the function hlua_prepend_path()
2020-12-02 21:53:16 +01:00
Willy Tarreau
cdb53465f4 MEDIUM: lua-thread: use atomics for memory accounting
Let's switch memory accounting to atomics so that the allocator function
may safely be used from concurrent Lua states.

Given that this function is extremely hot on the call path, we try to
optimize it for the most common case, which is:
  - no limit
  - there's enough memory

The accounting is what is particuarly expensive in threads since all
CPUs compete for a cache line, so when the limit is not used, we don't
want to use accounting. However we need to preserve it during the boot
phase until we may parse a "tune.lua.maxmem" value. For this, we turn
the unlimited "0" value to ~0 at the end of the boot phase to mark the
definite end of accounting. The function then detects this value and
directly jumps to realloc() in this case.

When the limit is enforced however, we use a CAS to check and reserve
our share of memory, and we roll back on failure. The CAS is used both
for increments and decrements so that a single operation is enough to
update the counters.
2020-12-02 21:53:16 +01:00
Willy Tarreau
d36c7fa5ec MINOR: lua: simplify hlua_alloc() to only rely on realloc()
The function really has the semantics of a realloc() except that it
also passes the old size to help with accounting. No need to special
case the free or malloc, realloc does everything we need.
2020-12-02 21:53:16 +01:00
Thierry Fournier
f67442efdb BUG/MINOR: lua: warn when registering action, conv, sf, cli or applet multiple times
Lua allows registering multiple sample-fetches, converters, action, cli,
applet/services with the same name. This is absolutely useless since only
the first registration will be used. This patch sends a warning if the case
is encountered.

This pach could be backported until 1.8, with the 3 associated patches:
 - MINOR: actions: Export actions lookup functions
 - MINOR: actions: add a function returning a service pointer from its name
 - MINOR: cli: add a function to look up a CLI service description
2020-12-02 09:45:18 +01:00
Thierry Fournier
2f05cc6f86 BUG/MINOR: lua: Some lua init operation are processed unsafe
Operation luaL_openlibs() and lua_prepend path are processed whithout
the safe context, so in case of failure Haproxy aborts or stops without
error message.

This patch could be backported until 1.8
2020-12-02 09:45:18 +01:00
Thierry Fournier
13d08b73eb BUG/MINOR: lua: Post init register function are not executed beyond the first one
Just because if the first init is a success we return success in place
of continuing the loop.

This patch could be backported until 1.8
2020-12-02 09:45:18 +01:00
Thierry Fournier
77a88943d6 BUG/MINOR: lua: lua-load doesn't check its parameters
"lua-load" doesn't check if the expected parameter is present. It tries to
open() directly the argument at second position. So if the filename is
omitted, it tries to load an empty filename.

This patch could be backported until 1.8
2020-12-02 09:42:43 +01:00
Thierry Fournier
de6145f747 BUG/MINOR: lua: missing "\n" in error message
Just replace ".n" by "\n"

This could be backported until 1.9, but it is not so important.
2020-12-02 09:31:33 +01:00
Thierry Fournier
91dc0c0d8f BUG/MINOR: lua: set buffer size during map lookups
This size is used by some pattern matching to determine if there
is sufficient room in the buffer to add final \0 if necessary.
If the size is not set, the conditions use uninitialized value.

Note: it seems this bug can't cause a crash.

Should be backported until 2.2 (at least)
2020-11-11 10:43:21 +01:00
Amaury Denoyelle
bc0af6a199 BUG/MINOR: lua: initialize sample before using it
Memset the sample before using it through hlua_lua2smp. This function is
ORing the smp.flags, so this field need to be cleared before its use.
This was reported by a coverity warning.

Fixes the github issue #929.
This bug can be backported up to 1.8.
2020-10-29 18:52:44 +01:00
Willy Tarreau
9b7587a6af MINOR: connection: make sockaddr_alloc() take the address to be copied
Roughly half of the calls to sockadr_alloc() are made to copy an already
known address. Let's optionally pass it in argument so that the function
can handle the copy at the same time, this slightly simplifies its usage.
2020-10-15 21:47:56 +02:00
Christopher Faulet
f98d821b94 MINOR: hlua: Display debug messages on stderr only in debug mode
Debug Messages emitted in lua using core.Debug() or core.log() are now only
displayed on stderr if HAProxy is started in debug mode (-d parameter on the
command line). There is no change for other message levels.

This patch should fix the issue #879. It may be backported to all stable
versions.
2020-10-05 11:11:36 +02:00
Willy Tarreau
5fc9328aa2 MINOR: tools: make str2sa_range() directly return the protocol
We'll need this so that it can return pointers to stacked protocol in
the future (for QUIC). In addition this removes a lot of tests for
protocol validity in the callers.

Some of them were checked further apart, or after a call to
str2listener() and they were simplified as well.

There's still a trick, we can fail to return a protocol in case the caller
accepts an fqdn for use later. This is what servers do and in this case it
is valid to return no protocol. A typical example is:

   server foo localhost:1111
2020-09-16 22:08:08 +02:00
Willy Tarreau
a93e5c7fae MINOR: tools: make str2sa_range() optionally return the fd
If a file descriptor was passed, we can optionally return it. This will
be useful for listening sockets which are both a pre-bound FD and a ready
socket.
2020-09-16 22:08:08 +02:00
Willy Tarreau
328199348b MINOR: tools: add several PA_O_* flags in str2sa_range() callers
These flags indicate whether the call is made to fill a bind or a server
line, or even just send/recv calls (like logs or dns). Some special cases
are made for outgoing FDs (e.g. pipes for logs) or socket FDs (e.g external
listeners), and there's a distinction between stream or dgram usage that's
expected to significantly help str2sa_range() proceed appropriately with
the input information. For now they are not used yet.
2020-09-16 22:08:08 +02:00
Willy Tarreau
8b0fa8f0ab MEDIUM: config: remove all checks for missing/invalid ports/ranges
Now that str2sa_range() checks for appropriate port specification, we
don't need to implement adhoc test cases in every call place, if the
result is valid, the conditions are met otherwise the error message is
appropriately filled.
2020-09-16 22:08:08 +02:00
Willy Tarreau
809587635e MINOR: tools: add several PA_O_PORT_* flags in str2sa_range() callers
These flags indicate what is expected regarding port specifications. Some
callers accept none, some need fixed ports, some have it mandatory, some
support ranges, and some take an offset. Each possibilty is reflected by
an option. For now they are not exploited, but the goal is to instrument
str2sa_range() to properly parse that.
2020-09-16 22:08:07 +02:00
Tim Duesterhus
e52b6e5456 CLEANUP: Do not use a fixed type for 'sizeof' in 'calloc'
Changes performed using the following coccinelle patch:

    @@
    type T;
    expression E;
    expression t;
    @@

    (
      t = calloc(E, sizeof(*t))
    |
    - t = calloc(E, sizeof(T))
    + t = calloc(E, sizeof(*t))
    )

Looking through the commit history, grepping for coccinelle shows that the same
replacement with a different patch was already performed in the past in commit
02779b6263.
2020-09-12 20:31:25 +02:00
Thierry Fournier
77016daabe MINOR: hlua: Add error message relative to the Channel manipulation and HTTP mode
When the developper try to manipulate HAProxy channels in HTTP mode,
an error throws without explanation. This patch adds an explanation.
2020-08-17 12:50:43 +02:00
Christopher Faulet
6ad7df423b MINOR: arg: Use chunk_destroy() to release string arguments
This way, all fields of the buffer structure are reset when a string argument
(ARGT_STR) is released.  It is also a good way to explicitly specify this kind
of argument is a chunk. So .data and .size fields must be set.

This patch may be backported to ease backports.
2020-08-07 14:27:54 +02:00
Christopher Faulet
fd2e906084 MINOR: lua: Add support for regex as fetches and converters arguments
It means now regsub() converter is now exported to the lua. Map converters based
on regex are not available because the map arguments are not supported.
2020-08-07 14:27:54 +02:00
Christopher Faulet
d25d926806 MINOR: lua: Add support for userlist as fetches and converters arguments
It means now http_auth() and http_auth_group() sample fetches are now exported
to the lua.
2020-08-07 14:27:54 +02:00
Christopher Faulet
05e2d77441 MEDIUM: lua: Don't filter exported fetches and converters
Thanks to previous commits, it is now safe to use from lua the sample fetches
and sample converters that convert arguments, especially the strings
(ARGT_STR). So now, there are all exported to the lua. They was filtered on the
validation functions. Only fetches without validation functions or with val_hdr
or val_payload_lv functions were exported, and converters without validation
functions.

This patch depends on following commits :

  * aec27ef44 "BUG/MINOR: lua: Duplicate lua strings in sample fetches/converters arg array"
  * fdea1b631 "MINOR: hlua: Don't needlessly copy lua strings in trash during args validation"

It must be backported as far as 2.1 because the date() and http_date()
converters are no longer exported because of the filter on the validation
function, since the commit ae6f125c7 ("MINOR: sample: add us/ms support to
date/http_date)".
2020-08-07 14:27:37 +02:00
Christopher Faulet
aec27ef443 BUG/MINOR: lua: Duplicate lua strings in sample fetches/converters arg array
Strings in the argument array used by sample fetches and converters must be
duplicated. This is mandatory because, during the arguments validations, these
strings may be converted and released. It works this way during the
configuration parsing and there is no reason to adapt this behavior during the
runtime when a sample fetch or a sample converter is called from the lua. In
fact, there is a reason to not change the behavior. It must reamain simple for
everyone to add new fetches or converters.

Thus, lua strings are duplicated. It is only performed at the end of the
hlua_lua2arg_check() function, if the argument is still a ARGT_STR. Of course,
it requires a cleanup loop after the call or when an error is triggered.

This patch depends on following commits:

  * 959171376 "BUG/MINOR: arg: Fix leaks during arguments validation for fetches/converters"
  * fdea1b631 "MINOR: hlua: Don't needlessly copy lua strings in trash during args validation"

It may be backported to all supported versions, most probably as far as 2.1
only.
2020-08-07 14:26:35 +02:00
Christopher Faulet
fdea1b6319 MINOR: hlua: Don't needlessly copy lua strings in trash during args validation
Lua strings are NULL terminated. So in the hlua_lua2arg_check() function, used
to check arguments against the sample fetches specification, there is no reason
to copy these strings in a trash to add a terminating null byte.

In addition, when the array of arguments is built from lua values, we must take
care to count this terminating null bytes in the size of the buffer where a
string is stored. The same must be done when a sample is built from a lua value.

This patch may be backported to easy backports.
2020-08-07 14:25:31 +02:00
Christopher Faulet
e663a6e326 BUG/MINOR: lua: Check argument type to convert it to IP mask in arg validation
In hlua_lua2arg_check() function, before converting an argument to an IPv4 or
IPv6 mask, we must be sure to have an integer or a string argument (ARGT_SINT or
ARGT_STR).

This patch must be backported to all supported versions.
2020-08-07 14:25:31 +02:00
Christopher Faulet
8e09ac8592 BUG/MINOR: lua: Check argument type to convert it to IPv4/IPv6 arg validation
In hlua_lua2arg_check() function, before converting a string to an IP address,
we must be to sure to have a string argument (ARGT_STR).

This patch must be backported to all supported versions.
2020-08-07 14:25:31 +02:00
Christopher Faulet
73292e9e66 BUG/MINOR: lua: Duplicate map name to load it when a new Map object is created
When a new map is created, the sample_load_map() function is called. To do so,
an argument array is created with the name as first argument. Because it is a
lua string, owned by the lua, it must be duplicated. The sample_load_map()
function will convert this argument to a map. In theory, after the conversion,
it must release the original string. It is not performed for now and it is a bug
that will be fixed in the next commit.

This patch may be backported to all supported versions, most probably as far as
2.1 only. But it must be backported with the next commit "BUG/MINOR: arg: Fix
leaks during arguments validation for fetches/converters".
2020-08-07 14:24:30 +02:00
Christopher Faulet
2361fd9487 BUG/MINOR: lua: Fix a possible null pointer deref on lua ctx
This bug was introduced by the commit 8f587ea3 ("MEDIUM: lua: Set the analyse
expiration date with smaller wake_time only"). At the end of hlua_action(), the
lua context may be null if the alloc failed.

No backport needed, this is 2.3-dev.
2020-07-30 10:40:59 +02:00
Christopher Faulet
8f587ea347 MEDIUM: lua: Set the analyse expiration date with smaller wake_time only
If a lua action yields for any reason and if the wake timeout is set, it only
override the analyse expiration date if it is smaller. This way, a lower
inspect-delay will be respected, if any.
2020-07-30 09:31:09 +02:00
Christopher Faulet
498c483009 BUG/MINOR: lua: Abort execution of actions that yield on a final evaluation
A Lua action may yield. It may happen because the action returns explicitly
act.YIELD or because the script itself yield. In the first case, we must abort
the script execution if it is the final rule evaluation, i.e if the
ACT_OPT_FINAL flag is set. The second case is already covered.

This patch must be backported to 2.2.
2020-07-30 09:31:09 +02:00
Christopher Faulet
08ed98fd79 MEDIUM: lua: Add support for the Lua 5.4
On Lua 5.4, some API changes make HAProxy compilation to fail. Among other
things, the lua_resume() function has changed and now takes an extra argument in
Lua 5.4 and the error LUA_ERRGCMM was removed. Thus the LUA_VERSION_NUM macro is
now tested to know the lua version is used and adapt the code accordingly.

Here are listed the incompatibilities with the previous Lua versions :

   http://www.lua.org/manual/5.4/manual.html#8

This patch comes from the HAproxy's fedora RPM, committed by Tom Callaway :

  https://src.fedoraproject.org/rpms/haproxy/blob/db970613/f/haproxy-2.2.0-lua-5.4.patch

This patch should fix the issue #730. It must be backported to 2.2 and probably
as far as 2.0.
2020-07-30 09:31:09 +02:00
Willy Tarreau
f1ea47d896 BUG/MINOR: htx: add two missing HTX_FL_EOI and remove an unexpected one
A workaround for some difficulties encountered to anticipate end of
messages was addressed by commit 810df0614 ("MEDIUM: htx: Add a flag on
a HTX message when no more data are expected"), but there were 3 issues
in it (with minor impact):
  - the flag was mistakenly set before an EOH in Lua, which would only
    cause incomplete packets to be emitted for now but could cause
    truncated responses in the future. It's not needed to add it on
    the next EOM block as http_forward_proxy_resp() already does it.

  - one was still missing in hlua_applet_http_fct(), possibly causing
    delays on Lua services

  - one was missing in the Prometheus exporter.

All this simply shows that this mechanism is still quite fragile and
not trivial to use, especially in order to deal with the impossibility
to append the EOM, so we'll need to improve the solution in the future
and future backports should not be completely ruled out.

This fix must be backported where the patch above is backported,
typically 2.1 and later as it was required for a set of fixes.
2020-07-23 06:53:27 +02:00
Christopher Faulet
810df06145 MEDIUM: htx: Add a flag on a HTX message when no more data are expected
The HTX_FL_EOI flag must now be set on a HTX message when no more data are
expected. Most of time, it must be set before adding the EOM block. Thus, if
there is no space for the EOM, there is still an information to know all data
were received and pushed in the HTX message. There is only an exception for the
HTTP replies (deny, return...). For these messages, the flag is set after all
blocks are pushed in the message, including the EOM block, because, on error,
we remove all inserted data.
2020-07-22 16:43:32 +02:00
Tim Duesterhus
d0c0ca2720 CLEANUP: Add static void hlua_deinit()
Compiling HAProxy with USE_LUA=1 and running a configuration check within
valgrind with a very simple configuration such as:

    listen foo
    	bind *:8080

Will report quite a few possible leaks afterwards:

    ==24048== LEAK SUMMARY:
    ==24048==    definitely lost: 0 bytes in 0 blocks
    ==24048==    indirectly lost: 0 bytes in 0 blocks
    ==24048==      possibly lost: 95,513 bytes in 1,209 blocks
    ==24048==    still reachable: 329,960 bytes in 71 blocks
    ==24048==         suppressed: 0 bytes in 0 blocks

Printing these possible leaks shows that all of them are caused by Lua.
Luckily Lua makes it *very* easy to free all used memory, so let's do
this on shutdown.

Afterwards this patch is applied the output looks much better:

    ==24199== LEAK SUMMARY:
    ==24199==    definitely lost: 0 bytes in 0 blocks
    ==24199==    indirectly lost: 0 bytes in 0 blocks
    ==24199==      possibly lost: 0 bytes in 0 blocks
    ==24199==    still reachable: 329,960 bytes in 71 blocks
    ==24199==         suppressed: 0 bytes in 0 blocks
2020-07-07 16:52:35 +02:00
Ilya Shipitsin
46a030cdda CLEANUP: assorted typo fixes in the code and comments
This is 11th iteration of typo fixes
2020-07-06 14:34:32 +02:00
Tim Duesterhus
588b3148d9 BUILD: Re-enable -Wimplicit-fallthrough
Getting rid of this warning is cleaner solved using a 'fall through' comment,
because it clarifies intent to a human reader.

This patch adjust a few places that cause -Wimplicit-fallthrough to trigger:

- Fix typos in the comment.
- Remove redundant 'no break' that trips up gcc from comment.
- Move the comment out of the block when the 'case' is completely surrounded
  by braces.
- Add comments where I could determine that the fall through was intentional.

Changes tested on

    gcc (Debian 9.3.0-13) 9.3.0
    Copyright (C) 2019 Free Software Foundation, Inc.
    This is free software; see the source for copying conditions.  There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

using

    make -j4 all TARGET=linux-glibc USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1
2020-06-11 16:49:37 +02:00
Willy Tarreau
b2551057af CLEANUP: include: tree-wide alphabetical sort of include files
This patch fixes all the leftovers from the include cleanup campaign. There
were not that many (~400 entries in ~150 files) but it was definitely worth
doing it as it revealed a few duplicates.
2020-06-11 10:18:59 +02:00
Willy Tarreau
36979d9ad5 REORG: include: move the error reporting functions to from log.h to errors.h
Most of the files dealing with error reports have to include log.h in order
to access ha_alert(), ha_warning() etc. But while these functions don't
depend on anything, log.h depends on a lot of stuff because it deals with
log-formats and samples. As a result it's impossible not to embark long
dependencies when using ha_warning() or qfprintf().

This patch moves these low-level functions to errors.h, which already
defines the error codes used at the same places. About half of the users
of log.h could be adjusted, sometimes revealing other issues such as
missing tools.h. Interestingly the total preprocessed size shrunk by
4%.
2020-06-11 10:18:59 +02:00
Willy Tarreau
6be7849f39 REORG: include: move cfgparse.h to haproxy/cfgparse.h
There's no point splitting the file in two since only cfgparse uses the
types defined there. A few call places were updated and cleaned up. All
of them were in C files which register keywords.

There is nothing left in common/ now so this directory must not be used
anymore.
2020-06-11 10:18:58 +02:00
Willy Tarreau
dfd3de8826 REORG: include: move stream.h to haproxy/stream{,-t}.h
This one was not easy because it was embarking many includes with it,
which other files would automatically find. At least global.h, arg.h
and tools.h were identified. 93 total locations were identified, 8
additional includes had to be added.

In the rare files where it was possible to finalize the sorting of
includes by adjusting only one or two extra lines, it was done. But
all files would need to be rechecked and cleaned up now.

It was the last set of files in types/ and proto/ and these directories
must not be reused anymore.
2020-06-11 10:18:58 +02:00
Willy Tarreau
1e56f92693 REORG: include: move server.h to haproxy/server{,-t}.h
extern struct dict server_name_dict was moved from the type file to the
main file. A handful of inlined functions were moved at the bottom of
the file. Call places were updated to use server-t.h when relevant, or
to simply drop the entry when not needed.
2020-06-11 10:18:58 +02:00
Willy Tarreau
a55c45470f REORG: include: move queue.h to haproxy/queue{,-t}.h
Nothing outstanding here. A number of call places were not justified and
removed.
2020-06-11 10:18:58 +02:00
Willy Tarreau
a264d960f6 REORG: include: move proxy.h to haproxy/proxy{,-t}.h
This one is particularly difficult to split because it provides all the
functions used to manipulate a proxy state and to retrieve names or IDs
for error reporting, and as such, it was included in 73 files (down to
68 after cleanup). It would deserve a small cleanup though the cut points
are not obvious at the moment given the number of structs involved in
the struct proxy itself.
2020-06-11 10:18:58 +02:00
Willy Tarreau
c2b1ff04e5 REORG: include: move http_ana.h to haproxy/http_ana{,-t}.h
It was moved without any change, however many callers didn't need it at
all. This was a consequence of the split of proto_http.c into several
parts that resulted in many locations to still reference it.
2020-06-11 10:18:58 +02:00
Willy Tarreau
f1d32c475c REORG: include: move channel.h to haproxy/channel{,-t}.h
The files were moved with no change. The callers were cleaned up a bit
and a few of them had channel.h removed since not needed.
2020-06-11 10:18:58 +02:00
Willy Tarreau
5e539c9b8d REORG: include: move stream_interface.h to haproxy/stream_interface{,-t}.h
Almost no changes, removed stdlib and added buf-t and connection-t to
the types to avoid a warning.
2020-06-11 10:18:58 +02:00
Willy Tarreau
83487a833c REORG: include: move cli.h to haproxy/cli{,-t}.h
Almost no change except moving the cli_kw struct definition after the
defines. Almost all users had both types&proto included, which is not
surprizing since this code is old and it used to be the norm a decade
ago. These places were cleaned.
2020-06-11 10:18:58 +02:00
Willy Tarreau
2eec9b5f95 REORG: include: move stats.h to haproxy/stats{,-t}.h
Just some minor reordering, and the usual cleanup of call places for
those which didn't need it. We don't include the whole tools.h into
stats-t anymore but just tools-t.h.
2020-06-11 10:18:58 +02:00
Willy Tarreau
3f0f82e7a9 REORG: move applet.h to haproxy/applet{,-t}.h
The type file was slightly tidied. The cli-specific APPCTX_CLI_ST1_* flag
definitions were moved to cli.h. The type file was adjusted to include
buf-t.h and not the huge buf.h. A few call places were fixed because they
did not need this include.
2020-06-11 10:18:58 +02:00
Willy Tarreau
48d25b3bc9 REORG: include: move session.h to haproxy/session{,-t}.h
Almost no change was needed beyond a little bit of reordering of the
types file and adjustments to use session-t instead of session at a
few places.
2020-06-11 10:18:58 +02:00
Willy Tarreau
126ba3a1e1 REORG: include: move http_fetch.h to haproxy/http_fetch.h
There's no type file for this trivial one. The unneeded dependency on
htx.h was dropped.
2020-06-11 10:18:58 +02:00
Willy Tarreau
7ea393d95e REORG: include: move connection.h to haproxy/connection{,-t}.h
The type file is becoming a mess, half of it is for the proxy protocol,
another good part describes conn_streams and mux ops, it would deserve
being split again. At least it was reordered so that elements are easier
to find, with the PP-stuff left at the end. The MAX_SEND_FD macro was moved
to compat.h as it's said to be the value for Linux.
2020-06-11 10:18:58 +02:00
Willy Tarreau
8b550afe1e REORG: include: move tcp_rules.h to haproxy/tcp_rules.h
There's no type file on this one which is pretty simple.
2020-06-11 10:18:58 +02:00
Willy Tarreau
cea0e1bb19 REORG: include: move task.h to haproxy/task{,-t}.h
The TASK_IS_TASKLET() macro was moved to the proto file instead of the
type one. The proto part was a bit reordered to remove a number of ugly
forward declaration of static inline functions. About a tens of C and H
files had their dependency dropped since they were not using anything
from task.h.
2020-06-11 10:18:58 +02:00
Willy Tarreau
a171892501 REORG: include: move vars.h to haproxy/vars{,-t}.h
A few includes (sessions.h, stream.h, api-t.h) were added for arguments
that were first declared in function prototypes.
2020-06-11 10:18:58 +02:00
Willy Tarreau
e6ce10be85 REORG: include: move sample.h to haproxy/sample{,-t}.h
This one is particularly tricky to move because everyone uses it
and it depends on a lot of other types. For example it cannot include
arg-t.h and must absolutely only rely on forward declarations to avoid
dependency loops between vars -> sample_data -> arg. In order to address
this one, it would be nice to split the sample_data part out of sample.h.
2020-06-11 10:18:58 +02:00
Willy Tarreau
469509b39e REORG: include: move payload.h to haproxy/payload.h
There's no type file, it only contains fetch_rdp_cookie_name() and
val_payload_lv() which probably ought to move somewhere else instead
of staying there.
2020-06-11 10:18:58 +02:00
Willy Tarreau
2cd5809f94 REORG: include: move map to haproxy/map{,-t}.h
Only small cleanups, and removal of a few includes from files that
didn't need them.
2020-06-11 10:18:58 +02:00
Willy Tarreau
225a90aaec REORG: include: move pattern.h to haproxy/pattern{,-t}.h
It was moved as-is, except for extern declaration of pattern_reference.
A few C files used to include it but didn't need it anymore after having
been split apart so this was cleaned.
2020-06-11 10:18:58 +02:00
Willy Tarreau
c761f843da REORG: include: move http_rules.h to haproxy/http_rules.h
There was no include file. This one still includes types/proxy.h.
2020-06-11 10:18:57 +02:00
Willy Tarreau
8efbdfb77b REORG: include: move obj_type.h to haproxy/obj_type{,-t}.h
No change was necessary. It still includes lots of types/* files.
2020-06-11 10:18:57 +02:00
Willy Tarreau
aa74c4e1b3 REORG: include: move arg.h to haproxy/arg{,-t}.h
Almost no change was needed; chunk.h was replaced with buf-t.h.
It dpeends on types/vars.h and types/protocol_buffers.h.
2020-06-11 10:18:57 +02:00
Willy Tarreau
8c794000c4 REORG: include: move hlua_fcn.h to haproxy/hlua_fcn.h
Added lua.h which was missing from the includes.
2020-06-11 10:18:57 +02:00
Willy Tarreau
8641605ff6 REORG: include: move hlua.h to haproxy/hlua{,-t}.h
This one required a few more includes as it uses list and ebpt_node.
It still references lots of types/ files for now.
2020-06-11 10:18:57 +02:00
Willy Tarreau
87735330d1 REORG: include: move http_htx.h to haproxy/http_htx{,-t}.h
A few includes had to be added, namely list-t.h in the type file and
types/proxy.h in the proto file. actions.h was including http-htx.h
but didn't need it so it was dropped.
2020-06-11 10:18:57 +02:00
Willy Tarreau
48fbcae07c REORG: tools: split common/standard.h into haproxy/tools{,-t}.h
And also rename standard.c to tools.c. The original split between
tools.h and standard.h dates from version 1.3-dev and was mostly an
accident. This patch moves the files back to what they were expected
to be, and takes care of not changing anything else. However this
time tools.h was split between functions and types, because it contains
a small number of commonly used macros and structures (e.g. name_desc)
which in turn cause the massive list of includes of tools.h to conflict
with the callers.

They remain the ugliest files of the whole project and definitely need
to be cleaned and split apart. A few types are defined there only for
functions provided there, and some parts are even OS-specific and should
move somewhere else, such as the symbol resolution code.
2020-06-11 10:18:57 +02:00
Willy Tarreau
5413a87ad3 REORG: include: move common/h1.h to haproxy/h1.h
The file was moved as-is. There was a wrong dependency on dynbuf.h
instead of buf.h which was addressed. There was no benefit to
splitting this between types and functions.
2020-06-11 10:18:57 +02:00
Willy Tarreau
374b442cbc REORG: include: split common/xref.h into haproxy/xref{,-t}.h
The type is the only element needed by applet.h and hlua.h, while hlua.c
needs the various functions. XREF_BUSY was placed into the types as well
since it's better to have the special values there.
2020-06-11 10:18:57 +02:00
Willy Tarreau
7cd8b6e3a4 REORG: include: split common/regex.h into haproxy/regex{,-t}.h
Regex are essentially included for myregex_t but it turns out that
several of the C files didn't include it directly, relying on the
one included by their own .h. This has been cleanly addressed so
that only the type is included by H files which need it, and adding
the missing includes for the other ones.
2020-06-11 10:18:57 +02:00
Willy Tarreau
889faf467b CLEANUP: include: remove excessive includes of common/standard.h
Some of them were simply removed as unused (possibly some leftovers
from an older cleanup session), some were turned to haproxy/bitops.h
and a few had to be added (hlua.c and stick-table.h need standard.h
for parse_time_err; htx.h requires chunk.h but used to get it through
standard.h).
2020-06-11 10:18:56 +02:00
Willy Tarreau
3f567e4949 REORG: include: split hathreads into haproxy/thread.h and haproxy/thread-t.h
This splits the hathreads.h file into types+macros and functions. Given
that most users of this file used to include it only to get the definition
of THREAD_LOCAL and MAXTHREADS, the bare minimum was placed into thread-t.h
(i.e. types and macros).

All the thread management was left to haproxy/thread.h. It's worth noting
the drop of the trailing "s" in the name, to remove the permanent confusion
that arises between this one and the system implementation (no "s") and the
makefile's option (no "s").

For consistency, src/hathreads.c was also renamed thread.c.

A number of files were updated to only include thread-t which is the one
they really needed.

Some future improvements are possible like replacing empty inlined
functions with macros for the thread-less case, as building at -O0 disables
inlining and causes these ones to be emitted. But this really is cosmetic.
2020-06-11 10:18:56 +02:00
Willy Tarreau
4c7e4b7738 REORG: include: update all files to use haproxy/api.h or api-t.h if needed
All files that were including one of the following include files have
been updated to only include haproxy/api.h or haproxy/api-t.h once instead:

  - common/config.h
  - common/compat.h
  - common/compiler.h
  - common/defaults.h
  - common/initcall.h
  - common/tools.h

The choice is simple: if the file only requires type definitions, it includes
api-t.h, otherwise it includes the full api.h.

In addition, in these files, explicit includes for inttypes.h and limits.h
were dropped since these are now covered by api.h and api-t.h.

No other change was performed, given that this patch is large and
affects 201 files. At least one (tools.h) was already freestanding and
didn't get the new one added.
2020-06-11 10:18:42 +02:00
Willy Tarreau
8d2b777fe3 REORG: ebtree: move the include files from ebtree to include/import/
This is where other imported components are located. All files which
used to directly include ebtree were touched to update their include
path so that "import/" is now prefixed before the ebtree-related files.

The ebtree.h file was slightly adjusted to read compiler.h from the
common/ subdirectory (this is the only change).

A build issue was encountered when eb32sctree.h is loaded before
eb32tree.h because only the former checks for the latter before
defining type u32. This was addressed by adding the reverse ifdef
in eb32tree.h.

No further cleanup was done yet in order to keep changes minimal.
2020-06-11 09:31:11 +02:00