Commit Graph

329 Commits

Author SHA1 Message Date
Gilchrist Dadaglo
bdd034947c BUG/MEDIUM: spoa/python: Fixing references to None
As per https://docs.python.org/3/c-api/none.html, None has to be treated
exactly like other objects for reference counting.
So, when we use it, we need to INCREF and when we are done, DECREF

This patch must be backported as far as 2.0.
2020-12-09 08:39:31 +01:00
Gilchrist Dadaglo
042f697a0f BUG/MEDIUM: spoa/python: Fixing PyObject_Call positional arguments
As per https://docs.python.org/3/c-api/object.html#c.PyObject_Call,
positional arguments should be an empty tuple when not used.
Previously the code had a dictionary instead of tuple. This commit is to
fix it and use tuple to avoid unexpected consequences

This patch must be backported as far as 2.0.
2020-12-09 08:39:31 +01:00
Gilchrist Dadaglo
d1c0cf80bc BUG/MINOR: spoa/python: Cleanup ipaddress objects if initialization fails
This change is to ensure objects from the ipaddress module are cleaned
up when spoa module initialization fails.
In general the interpreter would just crash, but in a code where import
is conditional (try/except), then we would keep those objects around

This patch must be backported as far as 2.0.
2020-12-09 08:39:31 +01:00
Gilchrist Dadaglo
132d8f61e9 BUG/MINOR: spoa/python: Cleanup references for failed Module Addobject operations
As per https://docs.python.org/3/c-api/module.html#c.PyModule_AddObject,
references are stolen by the function only for success. We must do
cleanup manually if there is a failure

This patch must be backported as far as 2.0.
2020-12-09 08:39:31 +01:00
Gilchrist Dadaglo
c7485ac101 DOC: spoa/python: Fixing typos in comments
Fixing a missing letter in a comment

This patch must be backported as far as 2.0.
2020-12-09 08:39:31 +01:00
Gilchrist Dadaglo
85b25829bc DOC: spoa/python: Rephrasing memory related error messages
The old message "No more space left available" was redundant with "left
available". This commit is to rephrase that sentence and make it more
explicit we are talking about memory

This patch must be backported as far as 2.0.
2020-12-09 08:39:31 +01:00
Gilchrist Dadaglo
6884742c65 DOC: spoa/python: Fixing typo in IP related error messages
This commit fixes typos in the ps_python_set_var_ip* byte manipulation error
messages

This patch must be backported as far as 2.0.
2020-12-09 08:39:31 +01:00
Gilchrist Dadaglo
d5c428e761 BUG/MAJOR: spoa/python: Fixing return None
As per https://docs.python.org/3/c-api/none.html, None requires to be
incremented before being returned to prevent deallocating none

This patch must be backported as far as 2.0.
2020-12-09 08:39:31 +01:00
Christopher Faulet
6ad06066cd CLEANUP: connection: Remove CS_FL_READ_PARTIAL flag
Since the recent refactoring of the H1 multiplexer, this flag is no more
used. Thus it is removed.
2020-12-04 14:41:49 +01:00
Christopher Faulet
da831fa068 CLEANUP: http-ana: Remove TX_WAIT_NEXT_RQ unsued flag
This flags is now unused. It was used in REQ_WAIT_HTTP analyser, when a
stream was waiting for a request, to set the keep-alive timeout or to avoid
to send HTTP errors to client.
2020-12-04 14:41:49 +01:00
Daniel Corbett
e2370307be CONTRIB: release-estimator: Add release estimating tool
This tool monitors the HAProxy stable branches and calculates a proposed
release date for the next minor release based on the bug fixes that are in
the queue.

Print only:
    ./release-estimator.py --print

Send email:
    ./release-estimator.py --send-mail --from-email from@domain.local --to-email to@domain.local

See contrib/release-estimator/README.md for details.
2020-10-24 12:27:17 +02:00
Willy Tarreau
9018ca9655 CONTRIB: tcploop: remove unused local variables in tcp_pause()
Building with -Wall shows that "pollfd" and "ret" are not used. Silly
copy-paste...
2020-10-22 05:17:08 +02:00
Willy Tarreau
2e065cbbf6 CONTRIB: tcploop: implement a disconnect operation 'D'
This performs a connect(AF_UNSPEC) over an existing connection. This is
mainly for compatibility testing. At this step it only seems to work on
linux for TCP sockets (both listening and established), while SO_LINGER
successfully resets established connections on freebsd and aix.
2020-10-14 08:46:23 +02:00
Willy Tarreau
c3914d4fff MEDIUM: proxy: replace proxy->state with proxy->disabled
The remaining proxy states were only used to distinguish an enabled
proxy from a disabled one. Due to the initialization order, both
PR_STNEW and PR_STREADY were equivalent after startup, and they
would only differ from PR_STSTOPPED when the proxy is disabled or
shutdown (which is effectively another way to disable it).

Now we just have a "disabled" field which allows to distinguish them.
It's becoming obvious that start_proxies() is only used to print a
greeting message now, that we'd rather get rid of. Probably that
zombify_proxy() and stop_proxy() should be merged once their
differences move to the right place.
2020-10-09 11:27:30 +02:00
Willy Tarreau
6b3bf733dd MEDIUM: proxy: remove the unused PR_STFULL state
Since v1.4 or so, it's almost not possible anymore to set this state. The
only exception is by using the CLI to change a frontend's maxconn setting
below its current usage. This case makes no sense, and for other cases it
doesn't make sense either because "full" is a vague concept when only
certain listeners are full and not all. Let's just remove this unused
state and make it clear that it's not reported. The "ready" or "open"
states will continue to be reported without being misleading as they
will be opposed to "stop".
2020-10-09 11:27:30 +02:00
Amaury Denoyelle
da5b6d1cd9 MINOR: stats: hide px/sv/li fields in applet struct
Use an opaque pointer to store proxy instance. Regroup server/listener
as a single opaque pointer. This has the benefit to render the structure
more evolutive to support statistics on other types of objects in the
future.

This patch is needed to extend stat support for components other than
proxies objects.

The prometheus module has been adapted for these changes.
2020-10-05 10:48:58 +02:00
Gilchrist Dadaglo
d00ce06d9d BUG/MEDIUM: contrib/spoa-server: Fix ipv4_address used instead of ipv6_address
Typo leads to checking the wrong object for memory issues

This patch must be backported as far as 2.0.
2020-09-01 18:28:40 +02:00
Gilchrist Dadaglo
2417ebcc01 BUG/MINOR: contrib/spoa-server: Updating references to free in case of failure
When we encounter a failure, all previously borrowed references should
be freed. Especially if the program is not failing immediately

This patch must be backported as far as 2.0.
2020-09-01 18:28:40 +02:00
Gilchrist Dadaglo
c7d303a0a8 BUG/MINOR: contrib/spoa-server: Do not free reference to NULL
As per https://docs.python.org/3/c-api/refcounting.html, Py_DECREF
should not be called on NULL objects.

This patch must be backported as far as 2.0.
2020-09-01 18:28:40 +02:00
Gilchrist Dadaglo
9f0c984cf5 BUG/MINOR: contrib/spoa-server: Ensure ip address references are freed
IP addresses references passed in argument for ps_python are not freed after
they have been used. Leading to a small chance of mem leak if a lot of ip
addresses are passed around

This patch must be backported as far as 2.0.
2020-09-01 18:28:21 +02:00
Gilchrist Dadaglo
222f060be3 BUG/MAJOR: contrib/spoa-server: Fix unhandled python call leading to memory leak
The result from spoa evaluation of the user provided python code is
never passed back to the main spoa process nor freed.
Same for the keyword list passed.
This results into the elements never freed by Python as reference count
never goes down.
https://docs.python.org/3/extending/extending.html#reference-counting-in-python

This patch must be backported as far as 2.0.
2020-09-01 18:28:16 +02:00
Bertrand Jacquin
ae2132cffb MINOR: contrib/spoa-server: allow MAX_FRAME_SIZE override
MAX_FRAME_SIZE is forced to the default value of tune.bufsize, however
they don't necessarily have to be tight together.
2020-09-01 18:25:04 +02:00
William Dauchy
4896d279aa DOC: spoa-server: fix false friends actually
it seems like `actually` was wrongly used instead of `currently`

Signed-off-by: William Dauchy <w.dauchy@criteo.com>
2020-08-05 22:12:54 +02:00
William Dauchy
bbe18ac5df BUG/MINOR: spoa-server: fix size_t format printing
From https://www.python.org/dev/peps/pep-0353/
"A new type Py_ssize_t is introduced, which has the same size as the
compiler's size_t type, but is signed. It will be a typedef for ssize_t
where available."

For integer types, causes printf to expect a size_t-sized integer
argument.

This should fix github issue #702

This should be backported to >= v2.2

Signed-off-by: William Dauchy <w.dauchy@criteo.com>
2020-08-05 22:12:47 +02:00
Willy Tarreau
f1ea47d896 BUG/MINOR: htx: add two missing HTX_FL_EOI and remove an unexpected one
A workaround for some difficulties encountered to anticipate end of
messages was addressed by commit 810df0614 ("MEDIUM: htx: Add a flag on
a HTX message when no more data are expected"), but there were 3 issues
in it (with minor impact):
  - the flag was mistakenly set before an EOH in Lua, which would only
    cause incomplete packets to be emitted for now but could cause
    truncated responses in the future. It's not needed to add it on
    the next EOM block as http_forward_proxy_resp() already does it.

  - one was still missing in hlua_applet_http_fct(), possibly causing
    delays on Lua services

  - one was missing in the Prometheus exporter.

All this simply shows that this mechanism is still quite fragile and
not trivial to use, especially in order to deal with the impossibility
to append the EOM, so we'll need to improve the solution in the future
and future backports should not be completely ruled out.

This fix must be backported where the patch above is backported,
typically 2.1 and later as it was required for a set of fixes.
2020-07-23 06:53:27 +02:00
Christopher Faulet
c55a626217 MINOR: contrib/prometheus-exporter: Add missing global and per-server metrics
Following metrics are now exported by the prometheus exporter to reflect recent
changes on HAProxy :

  * haproxy_process_failed_resolutions
  * haproxy_process_bytes_out_total
  * haproxy_process_spliced_bytes_out_total
  * haproxy_process_bytes_out_rate

and

 * haproxy_server_unsafe_idle_connections_current
 * haproxy_server_safe_idle_connections_current
 * haproxy_server_used_connections_current
 * haproxy_server_need_connections_current
2020-07-15 14:08:14 +02:00
Willy Tarreau
62fd12149f CONTRIB: da: fix memory leak in dummy function da_atlas_open()
The dummy function takes care of doing a bit of work using a malloc()
to avoid returning a constant but it doesn't free the tested pointer,
which coverity noticed in issue #741. Let's free it before testing it
for the return value.

This may be backported but is not important since this code is only
present to allow to build the device detection code and not to actually
run it.
2020-07-12 09:12:07 +02:00
Pierre Cheynier
1e36976352 CLEANUP: contrib/prometheus-exporter: typo fixes for ssl reuse metric
A typo I identified while having a look to our metric inventory.
(s/frontent/frontend)
2020-07-07 23:15:03 +02:00
Willy Tarreau
fc402cf3bc CONTRIB: debug: add missing flags SI_FL_L7_RETRY & SI_FL_D_L7_RETRY
These ones were missing from debug/flags.
2020-06-28 16:05:39 +02:00
Ryan O'Hara
f49a6049b8 BUG/MINOR: systemd: Wait for network to be online
Change systemd service file to wait for network to be completely
online. This solves two problems:

If haproxy is configured to bind to IP address(es) that are not yet
assigned, haproxy would previously fail. The workaround is to use
"option transparent".

If haproxy us configured to use a resolver to resolve servers via DNS,
haproxy would previously fail due to the fact that the network is not
fully online yet. This is the most compelling reason for this patch.

Signed-off-by: Ryan O'Hara <rohara@redhat.com>
Acked-by: Lukas Tribus <lukas@ltri.eu>
2020-06-19 21:31:10 +02:00
Willy Tarreau
54067e9d38 CONTRIB: debug: add the missing flags CO_FL_SAFE_LIST and CO_FL_IDLE_LIST
As often when flags are added they're not updated here. These ones were
missing. They're 2.2 only so no backport is needed.
2020-06-19 14:21:46 +02:00
Willy Tarreau
0a4b0ab177 BUILD: include: add sys/types before netinet/tcp.h
Apparently Cygwin requires sys/types.h before netinet/tcp.h but doesn't
include it by itself, as shown here:
  https://github.com/haproxy/haproxy/actions/runs/131943890

This patch makes sure it's always present, which is in server.c and
the SPOA example.
2020-06-11 11:22:44 +02:00
Willy Tarreau
b2551057af CLEANUP: include: tree-wide alphabetical sort of include files
This patch fixes all the leftovers from the include cleanup campaign. There
were not that many (~400 entries in ~150 files) but it was definitely worth
doing it as it revealed a few duplicates.
2020-06-11 10:18:59 +02:00
Willy Tarreau
36979d9ad5 REORG: include: move the error reporting functions to from log.h to errors.h
Most of the files dealing with error reports have to include log.h in order
to access ha_alert(), ha_warning() etc. But while these functions don't
depend on anything, log.h depends on a lot of stuff because it deals with
log-formats and samples. As a result it's impossible not to embark long
dependencies when using ha_warning() or qfprintf().

This patch moves these low-level functions to errors.h, which already
defines the error codes used at the same places. About half of the users
of log.h could be adjusted, sometimes revealing other issues such as
missing tools.h. Interestingly the total preprocessed size shrunk by
4%.
2020-06-11 10:18:59 +02:00
Willy Tarreau
6be7849f39 REORG: include: move cfgparse.h to haproxy/cfgparse.h
There's no point splitting the file in two since only cfgparse uses the
types defined there. A few call places were updated and cleaned up. All
of them were in C files which register keywords.

There is nothing left in common/ now so this directory must not be used
anymore.
2020-06-11 10:18:58 +02:00
Willy Tarreau
dfd3de8826 REORG: include: move stream.h to haproxy/stream{,-t}.h
This one was not easy because it was embarking many includes with it,
which other files would automatically find. At least global.h, arg.h
and tools.h were identified. 93 total locations were identified, 8
additional includes had to be added.

In the rare files where it was possible to finalize the sorting of
includes by adjusting only one or two extra lines, it was done. But
all files would need to be rechecked and cleaned up now.

It was the last set of files in types/ and proto/ and these directories
must not be reused anymore.
2020-06-11 10:18:58 +02:00
Willy Tarreau
1e56f92693 REORG: include: move server.h to haproxy/server{,-t}.h
extern struct dict server_name_dict was moved from the type file to the
main file. A handful of inlined functions were moved at the bottom of
the file. Call places were updated to use server-t.h when relevant, or
to simply drop the entry when not needed.
2020-06-11 10:18:58 +02:00
Willy Tarreau
4980160ecc REORG: include: move backend.h to haproxy/backend{,-t}.h
The files remained mostly unchanged since they were OK. However, half of
the users didn't need to include them, and about as many actually needed
to have it and used to find functions like srv_currently_usable() through
a long chain that broke when moving the file.
2020-06-11 10:18:58 +02:00
Willy Tarreau
6c58ab0304 REORG: include: move spoe.h to haproxy/spoe{,-t}.h
Only minor change was to make sure all defines were before the structs
in spoe-t.h, everything else went smoothly.
2020-06-11 10:18:58 +02:00
Willy Tarreau
a264d960f6 REORG: include: move proxy.h to haproxy/proxy{,-t}.h
This one is particularly difficult to split because it provides all the
functions used to manipulate a proxy state and to retrieve names or IDs
for error reporting, and as such, it was included in 73 files (down to
68 after cleanup). It would deserve a small cleanup though the cut points
are not obvious at the moment given the number of structs involved in
the struct proxy itself.
2020-06-11 10:18:58 +02:00
Willy Tarreau
aeed4a85d6 REORG: include: move log.h to haproxy/log{,-t}.h
The current state of the logging is a real mess. The main problem is
that almost all files include log.h just in order to have access to
the alert/warning functions like ha_alert() etc, and don't care about
logs. But log.h also deals with real logging as well as log-format and
depends on stream.h and various other things. As such it forces a few
heavy files like stream.h to be loaded early and to hide missing
dependencies depending where it's loaded. Among the missing ones is
syslog.h which was often automatically included resulting in no less
than 3 users missing it.

Among 76 users, only 5 could be removed, and probably 70 don't need the
full set of dependencies.

A good approach would consist in splitting that file in 3 parts:
  - one for error output ("errors" ?).
  - one for log_format processing
  - and one for actual logging.
2020-06-11 10:18:58 +02:00
Willy Tarreau
c2b1ff04e5 REORG: include: move http_ana.h to haproxy/http_ana{,-t}.h
It was moved without any change, however many callers didn't need it at
all. This was a consequence of the split of proto_http.c into several
parts that resulted in many locations to still reference it.
2020-06-11 10:18:58 +02:00
Willy Tarreau
f1d32c475c REORG: include: move channel.h to haproxy/channel{,-t}.h
The files were moved with no change. The callers were cleaned up a bit
and a few of them had channel.h removed since not needed.
2020-06-11 10:18:58 +02:00
Willy Tarreau
5e539c9b8d REORG: include: move stream_interface.h to haproxy/stream_interface{,-t}.h
Almost no changes, removed stdlib and added buf-t and connection-t to
the types to avoid a warning.
2020-06-11 10:18:58 +02:00
Willy Tarreau
209108dbbd REORG: include: move ssl_sock.h to haproxy/ssl_sock{,-t}.h
Almost nothing changed, just moved a static inline at the end and moved
an export from the types to the main file.
2020-06-11 10:18:58 +02:00
Willy Tarreau
2eec9b5f95 REORG: include: move stats.h to haproxy/stats{,-t}.h
Just some minor reordering, and the usual cleanup of call places for
those which didn't need it. We don't include the whole tools.h into
stats-t anymore but just tools-t.h.
2020-06-11 10:18:58 +02:00
Willy Tarreau
3f0f82e7a9 REORG: move applet.h to haproxy/applet{,-t}.h
The type file was slightly tidied. The cli-specific APPCTX_CLI_ST1_* flag
definitions were moved to cli.h. The type file was adjusted to include
buf-t.h and not the huge buf.h. A few call places were fixed because they
did not need this include.
2020-06-11 10:18:58 +02:00
Willy Tarreau
7ea393d95e REORG: include: move connection.h to haproxy/connection{,-t}.h
The type file is becoming a mess, half of it is for the proxy protocol,
another good part describes conn_streams and mux ops, it would deserve
being split again. At least it was reordered so that elements are easier
to find, with the PP-stuff left at the end. The MAX_SEND_FD macro was moved
to compat.h as it's said to be the value for Linux.
2020-06-11 10:18:58 +02:00
Willy Tarreau
cea0e1bb19 REORG: include: move task.h to haproxy/task{,-t}.h
The TASK_IS_TASKLET() macro was moved to the proto file instead of the
type one. The proto part was a bit reordered to remove a number of ugly
forward declaration of static inline functions. About a tens of C and H
files had their dependency dropped since they were not using anything
from task.h.
2020-06-11 10:18:58 +02:00
Willy Tarreau
f268ee8795 REORG: include: split global.h into haproxy/global{,-t}.h
global.h was one of the messiest files, it has accumulated tons of
implicit dependencies and declares many globals that make almost all
other file include it. It managed to silence a dependency loop between
server.h and proxy.h by being well placed to pre-define the required
structs, forcing struct proxy and struct server to be forward-declared
in a significant number of files.

It was split in to, one which is the global struct definition and the
few macros and flags, and the rest containing the functions prototypes.

The UNIX_MAX_PATH definition was moved to compat.h.
2020-06-11 10:18:58 +02:00