Commit Graph

138 Commits

Author SHA1 Message Date
Cedric Paillet
e06e31ea3b MINOR: promex: introduce haproxy_backend_agg_check_status
This patch introduces haproxy_backend_agg_check_status metric
as we wanted in 42d7c402d but with the right data source.

This patch could be backported as far as 2.4.
2022-12-09 10:54:48 +01:00
Cedric Paillet
7d6644e689 BUG/MINOR: promex: create haproxy_backend_agg_server_status
haproxy_backend_agg_server_check_status currently aggregates
haproxy_server_status instead of haproxy_server_check_status.
We deprecate this and create a new one,
haproxy_backend_agg_server_status to clarify what it really
does.

This patch could be backported as far as 2.4.
2022-12-09 10:54:27 +01:00
scientiamobile
6d6787ba7c LICENSE: wurfl: clarify the dummy library license.
This clarifies that LGPL is also permitted for the wurfl.h dummy file.
Should be backported where relevant.

Signed-off-by: Luca Passani <luca.passani@scientiamobile.com>
2022-12-02 05:32:46 +01:00
Dragan Dosen
a9800a0f58 MEDIUM: 51d: add support for 51Degrees V4 with Hash algorithm
This patch also adds a set of new global options:

- 51degrees-use-performance-graph { on | off }
- 51degrees-use-predictive-graph { on | off }
- 51degrees-drift <number>
- 51degrees-difference <number>
- 51degrees-allow-unmatched { on | off }

To build using the latest 51Degrees V4 engine with Hash algorithm, set
USE_51DEGREES_V4=1.

Other supported build options are 51DEGREES_INC, 51DEGREES_LIB and
51DEGREES_SRC which needs to be set to the directory that contains
headers and C files. For example:

make TARGET=<target> USE_51DEGREES_V4=1 51DEGREES_SRC='51D_REPO_PATH'/src
2022-12-01 16:12:21 +01:00
Willy Tarreau
913bea55fc BUILD: prometheus: use __fallthrough in promex_dump_metrics() and IO handler()
This avoids 11 build warnings when preprocessing happens before compiling
with gcc >= 7.
2022-11-14 11:14:02 +01:00
Amaury Denoyelle
56f50a03b7 CLEANUP: cli: rename dynamic error printing state
Rename CLI_ST_PRINT_FREE to CLI_ST_PRINT_DYNERR.

Most notably, this highlights that this is reserved to error printing.

This is done to ensure consistency between CLI_ST_PRINT/CLI_ST_PRINT_DYN
and CLI_ST_PRINT_ERR/CLI_ST_PRINT_DYNERR. The name is also consistent
with the function cli_dynerr() which activates it.
2022-11-10 16:42:47 +01:00
Ilya Shipitsin
3b64a28e15 CLEANUP: assorted typo fixes in the code and comments
This is 31st iteration of typo fixes
2022-08-06 17:12:51 +02:00
Willy Tarreau
c12b321661 CLEANUP: applet: rename appctx_cs() to appctx_sc()
It returns a stream connector, not a conn_stream anymore, so let's
fix its name.
2022-05-27 19:33:35 +02:00
Willy Tarreau
4c218fb25a CLEANUP: promex: rename all occurrences of stconn "cs" to "sc"
Function arguments and local variables called "cs" were renamed to "sc"
to avoid future confusion.
2022-05-27 19:33:35 +02:00
Willy Tarreau
cb086c6de1 REORG: stconn: rename conn_stream.{c,h} to stconn.{c,h}
There's no more reason for keepin the code and definitions in conn_stream,
let's move all that to stconn. The alphabetical ordering of include files
was adjusted.
2022-05-27 19:33:35 +02:00
Willy Tarreau
5edca2f0e1 REORG: rename cs_utils.h to sc_strm.h
This file contains all the stream-connector functions that are specific
to application layers of type stream. So let's name it accordingly so
that it's easier to figure what's located there.

The alphabetical ordering of include files was preserved.
2022-05-27 19:33:35 +02:00
Willy Tarreau
f61dd19284 CLEANUP: stconn: rename cs_{shut,chk}* to sc_*
This applies the following renaming:

cs_shutr() -> sc_shutr()
cs_shutw() -> sc_shutw()
cs_chk_rcv() -> sc_chk_rcv()
cs_chk_snd() -> sc_chk_snd()
cs_must_kill_conn() -> sc_must_kill_conn()
2022-05-27 19:33:35 +02:00
Willy Tarreau
99615ed85d CLEANUP: stconn: rename cs_rx_room_{blk,rdy} to sc_{need,have}_room()
The new name mor eclearly indicates that a stream connector cannot make
any more progress because it needs room in the channel buffer, or that
it may be unblocked because the buffer now has more room available. The
testing function is sc_waiting_room(). This is mostly used by applets.
Note that the flags will change soon.
2022-05-27 19:33:35 +02:00
Willy Tarreau
ea27f48c5a CLEANUP: stconn: rename cs_{check,strm,strm_task} to sc_strm_*
These functions return the app-layer associated with an stconn, which
is a check, a stream or a stream's task. They're used a lot to access
channels, flags and for waking up tasks. Let's just name them
appropriately for the stream connector.
2022-05-27 19:33:34 +02:00
Willy Tarreau
40a9c32e3a CLEANUP: stconn: rename cs_{i,o}{b,c} to sc_{i,o}{b,c}
We're starting to propagate the stream connector's new name through the
API. Most call places of these functions that retrieve the channel or its
buffer are in applets. The local variable names are not changed in order
to keep the changes small and reviewable. There were ~92 uses of cs_ic(),
~96 of cs_oc() (due to co_get*() being less factorizable than ci_put*),
and ~5 accesses to the buffer itself.
2022-05-27 19:33:34 +02:00
Willy Tarreau
026e8fb290 CLEANUP: stconn: tree-wide rename stconn states CS_ST/SB_* to SC_ST/SB_*
This also follows the natural naming. There are roughly 238 changes, all
totally trivial. conn_stream-t.h has become completely void of any
"conn_stream" related stuff now (except its name).
2022-05-27 19:33:34 +02:00
Willy Tarreau
4596fe20d9 CLEANUP: conn_stream: tree-wide rename to stconn (stream connector)
This renames the "struct conn_stream" to "struct stconn" and updates
the descriptions in all comments (and the rare help descriptions) to
"stream connector" or "connector". This touches a lot of files but
the change is minimal. The local variables were not even renamed, so
there's still a lot of "cs" everywhere.
2022-05-27 19:33:34 +02:00
Willy Tarreau
d869e13ed8 CLEANUP: applet: rename the sedesc pointer from "endp" to "sedesc"
Now at least it makes it obvious that it's the stream endpoint descriptor
and not an endpoint. There were few changes thanks to the previous refactor
of the flags.
2022-05-27 19:33:34 +02:00
Willy Tarreau
b605c4213f CLEANUP: conn_stream: rename the stream endpoint flags CS_EP_* to SE_FL_*
Let's now use the new flag names for the stream endpoint.
2022-05-27 19:33:34 +02:00
Willy Tarreau
d56377c5eb CLEANUP: conn_stream: apply endp_flags.cocci tree-wide
This changes all main uses of endp->flags to the se_fl_*() equivalent
by applying coccinelle script endp_flags.cocci. The se_fl_*() functions
themselves were manually excluded from the change, of course.

Note: 144 locations were touched, manually reviewed and found to be OK.

The script was applied with all includes:

  spatch --in-place --recursive-includes -I include --sp-file $script $files
2022-05-27 19:33:34 +02:00
Christopher Faulet
c9929380a4 MINOR: applet: Change return value for .init callback function
0 is now returned on success and -1 on error.
2022-05-17 16:13:21 +02:00
Willy Tarreau
0698c80a58 CLEANUP: applet: remove the unneeded appctx->owner
This one is the pointer to the conn_stream which is always in the
endpoint that is always present in the appctx, thus it's not needed.
This patch removes it and replaces it with appctx_cs() instead. A
few occurences that were using __cs_strm(appctx->owner) were moved
directly to appctx_strm() which does the equivalent.
2022-05-13 14:28:48 +02:00
Willy Tarreau
66435e5f63 CLEANUP: applet: use the appctx's endp instead of cs->endp
The few applets that set CS_EP_EOI or CS_EP_ERROR used to set it on the
endpoint retrieved from the conn_stream while it's already available on
the appctx itself. Better use the appctx one to limit the unneeded
interactions between the two sides.
2022-05-13 14:28:46 +02:00
Willy Tarreau
1c0715b12a CLEANUP: cli: move the status print context into its own context
Now that the CLI's print context is alone in the appctx, it's possible
to refine the appctx's ctx layout so that the cli part matches exactly
a regular svcctx, and as such move the CLI context into an svcctx like
other applets. External code will still build and work because the
struct cli perfectly maps onto the struct cli_print_ctx that's located
into svc.storage. This is of course only to make a smooth transition
during 2.6 and will disappear immediately after.

A tiny change had to be applied to the opentracing addon which performs
direct accesses to the CLI's err pointer in its own print function. The
rest uses the standard cli_print_* which were the only ones that needed
a small change.

The whole "ctx.cli" struct could be tagged as deprecated so that any
possibly existing external code that relies on it will get build
warnings, and the comments in the struct are pretty clear about the
way to fix it, and the lack of future of this old API.
2022-05-06 18:33:22 +02:00
Willy Tarreau
de58d245d6 CLEANUP: promex: stop using appctx->st2
Now that we have out own context, there's no need to use the cryptic st2
struct member. It's solely used to carry a field number here, so let's
add this into the context.
2022-05-06 18:13:35 +02:00
Willy Tarreau
ae1747db85 CLEANUP: promex: make the applet use its own context
The prometheus applet used to map to the stats context since it was not
convenient to have one's own context, and to reuse the fields with its
own values and enums. The obj1 pointer was used both for proxies and
stick-tables; obj2 was used both for servers and listeners.

This change makes use of the generic command context allocation so that
the there's now a properly typed context for prometheus, defined in the
code itself and independent on the stats or appctx ones. For clarity,
the types are correctly set and there's one proxy, one table, one server
and one listener. Some could be compacted using unions but that's not
necessary since the context is reasonably compact. The stats' st_code
field was used as the object state so the new field name is obj_state.

An attempt was made to change the types to const for what us only visited
but some calls pass through the stats code to retrieve info and that code
uses non-const variables due to internal API limitations (read_freq_ctr()
being used and requiring variable). That could change in the future,
though.
2022-05-06 18:13:35 +02:00
Christopher Faulet
6b0a0fb2f9 CLEANUP: tree-wide: Remove any ref to stream-interfaces
Stream-interfaces are gone. Corresponding files can be safely be removed. In
addition, comments are updated accordingly.
2022-04-13 15:10:16 +02:00
Christopher Faulet
a0bdec350f MEDIUM: stream-int/conn-stream: Move blocking flags from SI to CS
Remaining flags and associated functions are move in the conn-stream
scope. These flags are added on the endpoint and not the conn-stream
itself. This way it will be possible to get them from the mux or the
applet. The functions to get or set these flags are renamed accordingly with
the "cs_" prefix and updated to manipualte a conn-stream instead of a
stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
da098e6c17 MINOR: stream-int/conn-stream: Move si_shut* and si_chk* in conn-stream scope
si_shutr(), si_shutw(), si_chk_rcv() and si_chk_snd() are moved in the
conn-stream scope and renamed, respectively, cs_shutr(), cs_shutw(),
cs_chk_rcv(), cs_chk_snd() and manipulate a conn-stream instead of a
stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
62e757470a MEDIUM: stream-int/conn-stream: Move stream-interface state in the conn-stream
The stream-interface state (SI_ST_*) is now in the conn-stream. It is a
mechanical replacement for now. Nothing special. SI_ST_* and SI_SB_* were
renamed accordingly. Utils functions to manipulate these infos were moved
under the conn-stream scope.

But it could be good to keep in mind that this part should be
reworked. Indeed, at the CS level, we only need to know if it is ready to
receive or to send. The state of conn-stream from INI to EST is only used on
the server side. The client CS is immediately set to EST. Thus current
SI_ST_* states should probably be moved to the stream to reflect the server
connection state during the establishment stage.
2022-04-13 15:10:15 +02:00
Christopher Faulet
908628c4c0 MEDIUM: tree-wide: Use CS util functions instead of SI ones
At many places, we now use the new CS functions to get a stream or a channel
from a conn-stream instead of using the stream-interface API. It is the
first step to reduce the scope of the stream-interfaces. The main change
here is about the applet I/O callback functions. Before the refactoring, the
stream-interface was the appctx owner. Thus, it was heavily used. Now, as
far as possible,the conn-stream is used. Of course, it remains many calls to
the stream-interface API.
2022-04-13 15:10:14 +02:00
Christopher Faulet
b041b23ae4 MEDIUM: conn-stream: Move remaning flags from CS to endpoint
All old flags CS_FL_* are now moved in the endpoint scope and renamed
CS_EP_* accordingly. It is a systematic replacement. There is no true change
except for the health-check and the endpoint reset. Here it is a bit special
because the same conn-stream is reused. Thus, we must handle endpoint
allocation errors. To do so, cs_reset_endp() has been adapted.

Thanks to this last change, it will now be possible to simplify the
multiplexer and probably the applets too. A review must also be performed to
remove some flags in the channel or the stream-interface. The HTX will
probably be simplified too. Finally, there is now some place in the
conn-stream to move info from the stream-interface.
2022-04-13 15:10:14 +02:00
Christopher Faulet
4aa1d2838c MINOR: applet: Make .init callback more generic
For now there is no much change. Only the appctx is passed as argument when
the .init callback function is called. And it is not possible to yield at
this stage. It is not a problem because the feature is not used. Only the
lua defines this callback function for the lua TCP/HTTP services. The idea
is to be able to use it for all applets to initialize the appctx context.
2022-04-13 15:10:13 +02:00
Miroslav Zagorac
4e4b813bde Revert "BUILD: opentracing: display warning in case of using OT_USE_VARS at compile time"
This reverts commit 6c9f7faa62.

This warning is no longer needed because source code can be compiled with
enabled support for OpenTracing context transfer via HAProxy variables.

This patch must be backported in 2.5.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
52c2ec3202 MAJOR: opentracing: reenable usage of vars to transmit opentracing context
Since commit 3a4bedccc ("MEDIUM: vars: replace the global name index with
a hash") the names of HAProxy variables are no longer saved, ie their
64-bit hashes are saved instead.

This is very convenient for the HAProxy itself, but for the OpenTracing
module it is a problem because the names of the variables are important
when transferring the OpenTracing context.  Namely, this context consists
of an unknown amount of data stored in a key-value format.  The number
of these data (and the name of the variable used for this purpose) is
determined with the configuration of the OpenTracing filter, as well as
with the tracer used.  The two previous sentences seem to be in conflict,
but that is only so at first glance.  The function in the OpenTracing
filter used to read the context does not really know this, neither their
number nor its name.  The only thing that function actually knows is the
prefix of the variable names used for context transfer, and by that it
could find all the necessary data.  Of course, until the application of
the above-mentioned commit.

The problem is solved in a very simple way: in a common variable that
the filter always knows its name, the names of all variables that are the
product of the OpenTracing context are saved.  The names of these context
variables can only be added to that common variable.  When that variable
is no longer needed (when we no longer need context), it is deleted.

The format for saving data to this common variable is as follows:
  +-----+---------------+-- .. --+-----+---------------+
  | len | variable name |        | len | variable name |
  +-----+---------------+-- .. --+-----+---------------+

The amount of memory space used to store the length of the name is 1 byte,
with a sign (the minus sign is provided for inactive records, but this is
not currently used).  This means that the maximum length of the variable
name to be saved is 127 characters, which is quite enough for use in the
filter.  The buffer size for such data storage is global.tune.bufsize.

This patch must be backported in 2.5.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
6a1dfdde67 DEBUG: opentracing: display the contents of the err variable after setting
A display of the contents of the err variable has been added to the
FLT_OT_ERR() macro, once it has been set.

This patch must be backported as far as 2.4.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
854384f400 CLEANUP: opentracing: added FLT_OT_PARSE_INVALID_enum enum
It's always good to replace "hard-coded" values with something that looks
less "hard-coded" (even though that doesn't change anything in the code).
This is done here using FLT_OT_PARSE_INVALID_enum enum.

This patch must be backported as far as 2.4.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
ca09e01a13 DEBUG: opentracing: show return values of all functions in the debug output
If the OpenTracing filter is compiled using the 'OT_DEBUG=1' option, then
log messages are printed to stderr when the filter is running.  In the log
one can then find (among other things) the order in which the function is
called and the value that the function returns (if it is not a void type).

Prior to applying this patch, no value returned by a function was logged.

Log output example:
  [ 1]    0.038807 [OT]: flt_ot_init_per_thread(0x56365bd45ec0, 0x56365bd48210) {
  [ 1]    0.038807 [OT]:    ot_start(0x56365bd58920, 0x56365bd4e3a0, 0x7f561acba168:(nil)) {
  [ 1]    0.038807 [OT]:    } = 0
  [ 1]    0.038807 [OT]: } = 0

This patch must be backported as far as 2.4.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
b536cec2dc MINOR: opentracing: improved normalization of context variable names
The flag_cpy parameter has been added to the flt_ot_normalize_name()
function, through which we can determine whether the function converts
special characters (which are part of that name) when copying a variable
name.

This patch must be backported in 2.5.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
aec19a7d0d CLEANUP: opentracing: added variable to store variable length
The same variable should not be used to store multiple different results,
because it can be confusing.  Therefore, the var_name_len variable has
been added in several functions, in order to avoid the use of the retval
variable for several different purposes.

This patch must be backported as far as 2.4.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
854efefd6c CLEANUP: opentracing: added flt_ot_smp_init() function
The flt_ot_smp_init() function has been added to make initializing the
sample structure easier.  The contents of the structure in question are
set in several places in the source of the OpenTracing filter, so it is
better to do this in one place.

This patch must be backported as far as 2.4.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
6163e0357e MINOR: opentracing: only takes the variables lock on shared entries
Regarding commit #61ecf2838:
  There's no point taking the variables locks for sess/txn/req/res
  contexts since these ones always run inside the same thread anyway.

This patch must be backported in 2.5.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
220a1ad33b Revert "MINOR: opentracing: change the scope of the variable 'ot.uuid' from 'sess' to 'txn'"
This reverts commit 560c7b874a.

The ot.uuid variable should have the 'sess' scope because it is created
when an OpenTracing filter is attached to a stream.  After that, the
stream processing is started and on that occasion the contexts for the
variables that have the range 'txn' and 'req' are initialized.  This
means that we cannot use variables with the specified scopes before that
point.

This patch must be backported in 2.5.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
e6f76f0663 CLEANUP: opentracing: removed unused function flt_ot_var_get()
The flt_ot_var_get() function is not used anywhere and is unnecessary
in the existing implementation of the OpenTracing filter.

This patch must be backported as far as 2.4.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
e40a34cea1 CLEANUP: opentracing: removed unused function flt_ot_var_unset()
The flt_ot_var_unset() function is not used anywhere and is unnecessary
in the existing implementation of the OpenTracing filter.

This patch must be backported as far as 2.4.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
b51e94b0b3 DOC: opentracing: corrected comments in function descriptions
Several comments in the function descriptions have been corrected.

This patch must be backported as far as 2.4.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
9154e00f70 EXAMPLES: opentracing: refined shell scripts for testing filter performance
When calling the 'basename' command, the argument ${0} is enclosed in
quotation marks.  This is necessary if the path of the executable script
(contained in that argument) has "non-standard" elements, such as space
and the like.

Also, in the script 'test-speed.sh' the function sh_exit() has been added
for easier printing of messages at the end of execution.

This patch must be backported as far as 2.4.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
9964ad8436 BUG/BUILD: opentracing: fixed OT_DEFINE variable setting
In case of using parameter 'OT_USE_VARS=1', the value of the OT_DEFINE
variable is set incorrectly (i.e. the previous value was deleted and a
new one set instead of adding new content).

This patch must be backported in 2.5.
2022-04-08 16:31:33 +02:00
Miroslav Zagorac
728627f932 BUG/MINOR: opentracing: setting the return value in function flt_ot_var_set()
Function flt_ot_var_set() did not check whether the variable was
successfully set or not.  In case of failure, the value -1 is returned.

This patch must be backported as far as 2.4.
2022-04-08 16:31:33 +02:00
Christopher Faulet
be69cbdafe BUG/MEDIUM: promex: Be sure to never set EOM flag on an empty HTX message
It is the same bug than "BUG/MEDIUM: stats: Be sure to never set EOM flag on
an empty HTX message". We must be sure to set the EOM flag on a non empyt
message to be sure the mux on the client will properly handle
shutdowns. Otherwise, it may miss the end of the message and considers any
shutdown as an abort.

This patch must be backported as far as 2.4. On previous version there is
still the EOM HTX block.
2022-04-07 11:04:07 +02:00
Christopher Faulet
bef64b23b7 BUG/MINOR: promex: Set conn-stream/channel EOI flags at the end of request
This bug is the same than for the HTTP client. See "BUG/MINOR: httpclient:
Set conn-stream/channel EOI flags at the end of request" for details.

This patch must be backported as far as 2.0. But only CF_EOI must be set
because applets are not attached to a conn-stream on older versions.
2022-03-08 18:24:16 +01:00
Christopher Faulet
86e1c3381b MEDIUM: applet: Set the conn-stream as appctx owner instead of the stream-int
Because appctx is now an endpoint of the conn-stream, there is no reason to
still have the stream-interface as appctx owner. Thus, the conn-stream is
now the appctx owner.
2022-02-24 11:00:02 +01:00
David Carlier
629cbdf5cc MEDIUM: da: update module to handle schedule mode.
The DeviceAtlas addon can optionally interacts with the new service
 without change of configuration in the HAProxy part.
Note that however it requires the DeviceAtlas Identification C API
2.4.0 minimum from this point.

Signed-off-by: David Carlier <dcarlier@deviceatlas.com>
2022-01-28 07:29:01 +01:00
David Carlier
b81483cf2d MEDIUM: da: update doc and build for new scheduler mode service.
Mentions of the new database update runtime mode and update of
the legit module and the dummy part too.
Note the DeviceAtlas C API version 2.4.0 minimum required
alongside with libCURL, libzip and libgz.
2022-01-28 07:28:53 +01:00
David Carlier
e9cff619c1 MEDIUM: da: new optional data file download scheduler service.
New specialized service to daily handle the update of download file without
interruption of service and to be preemptively started before HAProxy.
It consists on a standalone utility which shared a memory block with
the DeviceAtlas module which handles the JSON data file update on
a daily basis.

Signed-off-by: David Carlier <dcarlier@deviceatlas.com>
2022-01-28 07:23:07 +01:00
Miroslav Zagorac
6c9f7faa62 BUILD: opentracing: display warning in case of using OT_USE_VARS at compile time
Please do not set the OT_USE_VARS configuration variable, as the source
will probably not be able to compile!  For now, this variable can only
be used for experimental purposes, and is not intended for wider use.

For further clarification, please see commit 4cb2c83f4.

Must be backported to 2.5.
2021-12-28 14:51:40 +01:00
William Dauchy
42d7c402d5 MINOR: promex: backend aggregated server check status
- add new metric: `haproxy_backend_agg_server_check_status`
  it counts the number of servers matching a specific check status
  this permits to exclude per server check status as the usage is often
  to rely on the total. Indeed in large setup having thousands of
  servers per backend the memory impact is not neglible to store the per
  server metric.
- realign promex_str_metrics array

quite simple implementation - we could improve it later by adding an
internal state to the prometheus exporter, thus to avoid counting at
every dump.

this patch is an attempt to close github issue #1312. It may bebackported
to 2.4 if requested.

Signed-off-by: William Dauchy <wdauchy@gmail.com>
2021-11-09 10:51:08 +01:00
Christopher Faulet
dfd10ab5ee MINOR: proxy: Introduce proxy flags to replace disabled bitfield
This change is required to support TCP/HTTP rules in defaults sections. The
'disabled' bitfield in the proxy structure, used to know if a proxy is
disabled or stopped, is replaced a generic bitfield named 'flags'.

PR_DISABLED and PR_STOPPED flags are renamed to PR_FL_DISABLED and
PR_FL_STOPPED respectively. In addition, everywhere there is a test to know
if a proxy is disabled or stopped, there is now a bitwise AND operation on
PR_FL_DISABLED and/or PR_FL_STOPPED flags.
2021-10-15 14:12:19 +02:00
Willy Tarreau
5554264f31 REORG: time: move time-keeping code and variables to clock.c
There is currently a problem related to time keeping. We're mixing
the functions to perform calculations with the os-dependent code
needed to retrieve and adjust the local time.

This patch extracts from time.{c,h} the parts that are solely dedicated
to time keeping. These are the "now" or "before_poll" variables for
example, as well as the various now_*() functions that make use of
gettimeofday() and clock_gettime() to retrieve the current time.

The "tv_*" functions moved there were also more appropriately renamed
to "clock_*".

Other parts used to compute stolen time are in other files, they will
have to be picked next.
2021-10-08 17:22:26 +02:00
Willy Tarreau
b7fc4c4e9f BUILD: tree-wide: add missing http_ana.h from many places
At least 6 files make use of s->txn without including http_ana which
defines it. They used to get it from other includes.
2021-10-07 01:36:51 +02:00
Tim Duesterhus
b113b5ca24 CLEANUP: Apply ist.cocci
This cleans up ist handling.
2021-09-17 17:22:05 +02:00
Miroslav Zagorac
e956c9cdac CLEANUP: opentracing: simplify the condition on the empty header
The second patch from the last series of patches has been redesigned
here, the ist() function is used to set an empty string instead of
working directly with the string pointer.

I thank Tim Düsterhus for his advice.
2021-09-12 08:17:02 +02:00
Miroslav Zagorac
560c7b874a MINOR: opentracing: change the scope of the variable 'ot.uuid' from 'sess' to 'txn'
At the suggestion of Willy Tarreau, the scope of the 'ot.uuid' variable was
changed from 'sess' to 'txn', so it is now limited to the transaction only.
2021-09-12 07:08:14 +02:00
Miroslav Zagorac
a75f3404c8 CLEANUP: opentracing: use the haproxy function to generate uuid
To avoid duplicate source code, the original haproxy function is used to
generate the OpenTracing runtime context UUID.

Also, the structure flt_ot_runtime_context is simplified because the
detailed definition of UUID is removed from it (struct flt_ot_uuid),
ie the UUID is left only in the form of a string.
2021-09-12 07:08:14 +02:00
Miroslav Zagorac
71cb1ab6b9 BUG/MINOR: opentracing: enable the use of http headers without a set value
In case we transfer some data that does not have a set value via the HTTP
header, adding that header in the text map was done incorrectly.

This simple patch allows the use of HTTP headers without a set value.
2021-09-12 07:08:14 +02:00
Miroslav Zagorac
4cb2c83f46 BUILD: opentracing: exclude the use of haproxy variables for the OpenTracing context
Due to a recent change in the handling of haproxy variables, their use for
OpenTracing context transfer has been excluded from the compilation process.

The use of variables can be re-enabled if the newly defined variable
OT_USE_VARS is set to 1 when calling the 'make' utility.  However,
this should not be used for now as the compilation will end in error.

This change prevents the use of haproxy variables to convey the OpenTracing
context.  This means that the 'use-vars' parameter cannot be used in the
OpenTracing filter configuration for 'inject' and 'extract' operations.

An example configuration that uses this feature is in the test/ctx
directory, while the script to run that test is test/run-ctx.sh.

Then, the 'sess.ot.uuid' variable is no longer set when initializing the
OpenTracing session.  This means that this variable can still be used in
the OpenTracing configuration, but its contents will be empty.
2021-09-12 07:07:13 +02:00
Tim Duesterhus
d5fc8fcb86 CLEANUP: Add haproxy/xxhash.h to avoid modifying import/xxhash.h
This solves setting XXH_INLINE_ALL in a cleaner way, because the imported
header is not modified, easing future updates.

see 6f7cc11e6d
2021-09-11 19:58:45 +02:00
Willy Tarreau
0c36d0938c BUILD: ot: add argument for default value to vars_get_by_name()
The API was extended by commit e352b9dac ("MINOR: vars: make vars_get_by_*
support an optional default value") but I didn't notice that opentracing
was using it, so it broke the build. No backport needed.
2021-09-03 14:21:40 +02:00
Miroslav Zagorac
3571111de3 BUILD: opentracing: fixed build when using pkg-config utility
In case the OpenTracing C Wrapper library was installed as part of the
system (ie in a directory that pkg-config considers part of the system),
HAProxy building was not possible because in that case pkg-config did
not set the value of the OT_CFLAGS variable in the addon Makefile.

This resolves GitHub issue #1323.
2021-08-01 18:18:29 +02:00
Miroslav Zagorac
9425ed488f BUG/MEDIUM: opentracing: initialization before establishing daemon and/or chroot mode
This patch solves the problem reported in github issue #1204, where the
OpenTracing filter cannot communicate with the selected tracer if HAProxy
is run in daemon mode.

This commit also solves github issue #1274, where the problem manifests
itself when using the 'chroot' keyword in the HAProxy configuration.

This is solved so that the initialization of the OpenTracing plugin is
split into two operations, first the plugin (dynamic library) is loaded
before switching the HAProxy to daemon mode (or chroot) and then the
tracer thread is started.

This means that nothing is retrieved from the file system in runtime.

After applying this commit, opentracing C wrapper version 1.1.0 should be
used because the earlier version does not have separated initialization
functions.

This resolves GitHub issues #1204 and #1274.
2021-06-10 06:45:39 +02:00
Miroslav Zagorac
1642474054 Revert "BUG/MINOR: opentracing: initialization after establishing daemon mode"
This reverts commit f2263435d7.

This commit is unnecessary because although it solves the problem of using
the OpenTracing filter in daemon mode, it does not solve the same problem
if chroot is used.

The following commit related to the OpenTracing filter solves both problems
efficiently.
2021-06-10 06:45:39 +02:00
Miroslav Zagorac
65b86685e1 BUILD/MINOR: opentracing: fixed build when using clang
The arguments of the snprintf() function are now consistent with the
format used.

This should fix the github issue #1242.
2021-05-20 05:25:02 +02:00
Miroslav Zagorac
d2acd0b3a7 BUILD/MINOR: opentracing: fixed compilation with filter enabled
The inclusion of header files proxy.h and tools.h was added to the
addons/ot/include/include.h file.  Without this HAProxy cannot be
compiled if the OpenTracing filter is to be used.
2021-05-12 09:45:16 +02:00
Willy Tarreau
0fd04fd620 BUILD: promex: service-prometheus.c needs tools.h
It calls url_decode() and memprintf() but used to inherit them through
others.
2021-05-08 12:58:12 +02:00
Willy Tarreau
2bd86284e6 BUILD: wurfl: wurfl.c needs tools.h
It calls memprintf() which is defined there but used to inherit it
through others.
2021-05-08 12:56:20 +02:00
Willy Tarreau
cfc4f24d80 REORG: vars: move the "proc" scope variables out of the global struct
The presence of this field causes a long dependency chain because almost
everyone includes global-t.h, and vars include sample_data which include
some system includes as well as HTTP parts.

There is absolutely no reason for having the process-wide variables in
the global struct, let's just move them into vars.c and vars.h. This
reduces from ~190k to ~170k the preprocessed output of version.c.
2021-05-08 12:11:29 +02:00
Willy Tarreau
0b26b3866c MINOR: stats: pass the appctx flags to stats_fill_info()
Currently the stats filling function knows nothing about the caller's
needs, so let's pass the STAT_* flags so that it can adapt to the
requester's constraints.
2021-05-08 10:52:12 +02:00
Willy Tarreau
2b71810cb3 CLEANUP: lists/tree-wide: rename some list operations to avoid some confusion
The current "ADD" vs "ADDQ" is confusing because when thinking in terms
of appending at the end of a list, "ADD" naturally comes to mind, but
here it does the opposite, it inserts. Several times already it's been
incorrectly used where ADDQ was expected, the latest of which was a
fortunate accident explained in 6fa922562 ("CLEANUP: stream: explain
why we queue the stream at the head of the server list").

Let's use more explicit (but slightly longer) names now:

   LIST_ADD        ->       LIST_INSERT
   LIST_ADDQ       ->       LIST_APPEND
   LIST_ADDED      ->       LIST_INLIST
   LIST_DEL        ->       LIST_DELETE

The same is true for MT_LISTs, including their "TRY" variant.
LIST_DEL_INIT keeps its short name to encourage to use it instead of the
lazier LIST_DELETE which is often less safe.

The change is large (~674 non-comment entries) but is mechanical enough
to remain safe. No permutation was performed, so any out-of-tree code
can easily map older names to new ones.

The list doc was updated.
2021-04-21 09:20:17 +02:00
Miroslav Zagorac
a8bdf2b655 MINOR: opentracing: transfer of context names without prefix
In order to enable the assignment of a context name, and yet exclude the
use of that name (prefix in this case) when extracting the context from
the HTTP header, a special character '-' has been added, which can be
specified at the beginning of the prefix.

So let's say if we look at examples of the fe-be configuration, we can
transfer the context via an HTTP header without a prefix like this:

  fe/ot.cfg:
        ..
        span "HAProxy session"
            inject "" use-headers
        event on-backend-http-request

Such a context can be read in another process using a name that has a
special '-' sign at the beginning:

  be/ot.cfg:
    ot-scope frontend_http_request
        extract "-ot-ctx" use-headers
        span "HAProxy session" child-of "-ot-ctx" root
        ..

This means that the context name will be '-ot-ctx' but it will not be
used when extracting data from HTTP headers.

Of course, if the context does not have a prefix set, all HTTP headers
will be inserted into the OpenTracing library as context.  All of the
above will only work correctly if that library can figure out what is
relevant to the context and what is not.
2021-04-15 08:40:08 +02:00
Miroslav Zagorac
4b3eb0a940 MINOR: opentracing: correct calculation of the number of arguments in the args[]
It is possible that some arguments within the configuration line are not
specified; that is, they are set to a blank string.

For example:
  keyword '' arg_2

In that case the content of the args field will be like this:
  args[0]:                  'keyword'
  args[1]:                  NULL pointer
  args[2]:                  'arg_2'
  args[3 .. MAX_LINE_ARGS): NULL pointers

The previous way of calculating the number of arguments (as soon as a
null pointer is encountered) could not place an argument on an empty
string.

All of the above is essential for passing the OpenTracing context via
the HTTP headers (keyword 'inject'), where one of the arguments is the
context name prefix.  This way we can set an empty prefix, which is very
useful if we get context from some other process that can't add a prefix
to that data; or we want to pass the context to some process that cannot
handle the prefix of that data.
2021-04-15 08:40:05 +02:00
Willy Tarreau
ff88270ef9 MINOR: pool: move pool declarations to read_mostly
All pool heads are accessed via a pointer and should not be shared with
highly written variables. Move them to the read_mostly section.
2021-04-10 19:27:41 +02:00
Miroslav Zagorac
98272253d8 MINOR: opentracing: register config file and line number on log servers
In commit 9533a7038 new parameters have been added to the declaration
of function parse_logsrv().

This patch should be backported to all branches where the OpenTracing
filter is located.
2021-04-08 11:10:27 +02:00
Miroslav Zagorac
f2263435d7 BUG/MINOR: opentracing: initialization after establishing daemon mode
This patch solves the problem reported in github issue #1204, where the
OpenTracing filter cannot communicate with the selected tracer if HAProxy
is run in daemon mode.  The author of the reported issue uses Zipkin
tracer, while in this example Jaeger tracer is used (see gdb output below).

The problem is that the OpenTracing library is initialized before HAProxy
initialize the daemon mode.  Establishing this mode kills the OpenTracing
thread, after which the correct operation of the OpenTracing filter is no
longer possible.  Also, HAProxy crashes on deinitialization of the
OpenTracing library.

The initialization of the OpenTracing library has been moved from the
flt_ot_init() function (which is started before switching the HAProxy to
daemon mode) to the flt_ot_init_per_thread() function (which is run after
switching the HAProxy to daemon mode).

Gdb output of crashed HAProxy process:

  [Thread debugging using libthread_db enabled]
  Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
  Core was generated by `../../../haproxy -f sa/haproxy.cfg'.
  Program terminated with signal SIGSEGV, Segmentation fault.
  #0  0x00007f8131fd5629 in pthread_join (threadid=140192831239936, thread_return=0x0) at pthread_join.c:45
  45      pthread_join.c: No such file or directory.
  (gdb) where
  #0  0x00007f8131fd5629 in pthread_join (threadid=140192831239936, thread_return=0x0) at pthread_join.c:45
  #1  0x00007f812f15abc7 in std:🧵:join() ()
     from /tmp/haproxy-os-master/contrib/opentracing/test/libjaeger_opentracing_plugin-0.5.0.so
  #2  0x00007f812f0fb6f7 in jaegertracing::reporters::RemoteReporter::close() ()
        from /tmp/haproxy-os-master/contrib/opentracing/test/libjaeger_opentracing_plugin-0.5.0.so
  #3  0x00007f812f0b7055 in jaegertracing::reporters::CompositeReporter::close() ()
           from /tmp/haproxy-os-master/contrib/opentracing/test/libjaeger_opentracing_plugin-0.5.0.so
  #4  0x00007f812f0b9136 in jaegertracing::Tracer::Close() ()
              from /tmp/haproxy-os-master/contrib/opentracing/test/libjaeger_opentracing_plugin-0.5.0.so
  #5  0x00007f81309def32 in ot_tracer_close (tracer=0x55fb48057390) at ../../src/tracer.cpp:91
  #6  0x000055fb41785705 in ot_close (tracer=0x55fb48061168) at contrib/opentracing/src/opentracing.c:208
  #7  0x000055fb4177fc64 in flt_ot_deinit (p=<optimized out>, fconf=<optimized out>) at contrib/opentracing/src/filter.c:215
  #8  0x000055fb418bc038 in flt_deinit (proxy=proxy@entry=0x55fb4805ce50) at src/filters.c:360
  #9  0x000055fb41893ed1 in free_proxy (p=0x55fb4805ce50) at src/proxy.c:315
  #10 0x000055fb41888809 in deinit () at src/haproxy.c:2217
  #11 0x000055fb41889078 in deinit_and_exit (status=0) at src/haproxy.c:2343
  #12 0x000055fb4173d809 in main (argc=<optimized out>, argv=<optimized out>) at src/haproxy.c:3230

This patch should be backported to all branches where the OpenTracing
filter is located.
2021-04-02 18:21:19 +02:00
Willy Tarreau
a63d1a0863 CONTRIB: move contrib/opentracing to addons/ot
This one is the last optional module to build with haproxy, so let's move
it to addons/. It was renamed to "ot" as it was the only one whose USE_*
option did not match the directory name, now this is consistent.

Few changes were required, only the Makefile, and doc were adjusted, as
the directory was already self-contained and relocatable.
2021-04-02 17:48:42 +02:00
Willy Tarreau
57610c694e CONTRIB: move src/wurfl.c and contrib/wurfl to addons/wurfl
Both the source file and the dummy library are now at the same place.
Maybe the build howto could be moved there as well to make things even
cleaner.

The Makefile, MAINTAINERS, doc, and vtest matrix were updated.
2021-04-02 17:48:42 +02:00
Willy Tarreau
f8d9ec57f0 CONTRIB: move src/da.c and contrib/deviceatlas to addons/deviceatlas
Both the source file and the dummy library are now at the same place.
Maybe the build howto could be moved there as well to make things even
cleaner.

The Makefile, MAINTAINERS, doc, github build matrix, coverity checks
and travis CI's build were updated.
2021-04-02 17:48:42 +02:00
Willy Tarreau
977209d1d8 CONTRIB: move 51Degrees to addons/51degrees
Now it's much cleaner, both 51d.c and the dummy library live together and
are easier to spot and maintain. The build howto probably ought to be moved
there as well. Makefile, docs and MAINTAINERS were updated, as well as
the github CI's build matrix, travis CI's, and coverity checks.
2021-04-02 17:48:42 +02:00
Willy Tarreau
3dfadc7ae7 CONTRIB: move prometheus-exporter to addons/promex
Let's start to better organize the addons by moving promex there (and
with an easier directory name). The makefile and maintainers files were
updated, as well as the CI's build matrix.
2021-04-02 17:48:42 +02:00