There's no more reason for keepin the code and definitions in conn_stream,
let's move all that to stconn. The alphabetical ordering of include files
was adjusted.
This file contains all the stream-connector functions that are specific
to application layers of type stream. So let's name it accordingly so
that it's easier to figure what's located there.
The alphabetical ordering of include files was preserved.
The new name mor eclearly indicates that a stream connector cannot make
any more progress because it needs room in the channel buffer, or that
it may be unblocked because the buffer now has more room available. The
testing function is sc_waiting_room(). This is mostly used by applets.
Note that the flags will change soon.
These functions return the app-layer associated with an stconn, which
is a check, a stream or a stream's task. They're used a lot to access
channels, flags and for waking up tasks. Let's just name them
appropriately for the stream connector.
We're starting to propagate the stream connector's new name through the
API. Most call places of these functions that retrieve the channel or its
buffer are in applets. The local variable names are not changed in order
to keep the changes small and reviewable. There were ~92 uses of cs_ic(),
~96 of cs_oc() (due to co_get*() being less factorizable than ci_put*),
and ~5 accesses to the buffer itself.
This also follows the natural naming. There are roughly 238 changes, all
totally trivial. conn_stream-t.h has become completely void of any
"conn_stream" related stuff now (except its name).
This renames the "struct conn_stream" to "struct stconn" and updates
the descriptions in all comments (and the rare help descriptions) to
"stream connector" or "connector". This touches a lot of files but
the change is minimal. The local variables were not even renamed, so
there's still a lot of "cs" everywhere.
Now at least it makes it obvious that it's the stream endpoint descriptor
and not an endpoint. There were few changes thanks to the previous refactor
of the flags.
This changes all main uses of endp->flags to the se_fl_*() equivalent
by applying coccinelle script endp_flags.cocci. The se_fl_*() functions
themselves were manually excluded from the change, of course.
Note: 144 locations were touched, manually reviewed and found to be OK.
The script was applied with all includes:
spatch --in-place --recursive-includes -I include --sp-file $script $files
This one is the pointer to the conn_stream which is always in the
endpoint that is always present in the appctx, thus it's not needed.
This patch removes it and replaces it with appctx_cs() instead. A
few occurences that were using __cs_strm(appctx->owner) were moved
directly to appctx_strm() which does the equivalent.
The few applets that set CS_EP_EOI or CS_EP_ERROR used to set it on the
endpoint retrieved from the conn_stream while it's already available on
the appctx itself. Better use the appctx one to limit the unneeded
interactions between the two sides.
Now that the CLI's print context is alone in the appctx, it's possible
to refine the appctx's ctx layout so that the cli part matches exactly
a regular svcctx, and as such move the CLI context into an svcctx like
other applets. External code will still build and work because the
struct cli perfectly maps onto the struct cli_print_ctx that's located
into svc.storage. This is of course only to make a smooth transition
during 2.6 and will disappear immediately after.
A tiny change had to be applied to the opentracing addon which performs
direct accesses to the CLI's err pointer in its own print function. The
rest uses the standard cli_print_* which were the only ones that needed
a small change.
The whole "ctx.cli" struct could be tagged as deprecated so that any
possibly existing external code that relies on it will get build
warnings, and the comments in the struct are pretty clear about the
way to fix it, and the lack of future of this old API.
Now that we have out own context, there's no need to use the cryptic st2
struct member. It's solely used to carry a field number here, so let's
add this into the context.
The prometheus applet used to map to the stats context since it was not
convenient to have one's own context, and to reuse the fields with its
own values and enums. The obj1 pointer was used both for proxies and
stick-tables; obj2 was used both for servers and listeners.
This change makes use of the generic command context allocation so that
the there's now a properly typed context for prometheus, defined in the
code itself and independent on the stats or appctx ones. For clarity,
the types are correctly set and there's one proxy, one table, one server
and one listener. Some could be compacted using unions but that's not
necessary since the context is reasonably compact. The stats' st_code
field was used as the object state so the new field name is obj_state.
An attempt was made to change the types to const for what us only visited
but some calls pass through the stats code to retrieve info and that code
uses non-const variables due to internal API limitations (read_freq_ctr()
being used and requiring variable). That could change in the future,
though.
Remaining flags and associated functions are move in the conn-stream
scope. These flags are added on the endpoint and not the conn-stream
itself. This way it will be possible to get them from the mux or the
applet. The functions to get or set these flags are renamed accordingly with
the "cs_" prefix and updated to manipualte a conn-stream instead of a
stream-interface.
si_shutr(), si_shutw(), si_chk_rcv() and si_chk_snd() are moved in the
conn-stream scope and renamed, respectively, cs_shutr(), cs_shutw(),
cs_chk_rcv(), cs_chk_snd() and manipulate a conn-stream instead of a
stream-interface.
The stream-interface state (SI_ST_*) is now in the conn-stream. It is a
mechanical replacement for now. Nothing special. SI_ST_* and SI_SB_* were
renamed accordingly. Utils functions to manipulate these infos were moved
under the conn-stream scope.
But it could be good to keep in mind that this part should be
reworked. Indeed, at the CS level, we only need to know if it is ready to
receive or to send. The state of conn-stream from INI to EST is only used on
the server side. The client CS is immediately set to EST. Thus current
SI_ST_* states should probably be moved to the stream to reflect the server
connection state during the establishment stage.
At many places, we now use the new CS functions to get a stream or a channel
from a conn-stream instead of using the stream-interface API. It is the
first step to reduce the scope of the stream-interfaces. The main change
here is about the applet I/O callback functions. Before the refactoring, the
stream-interface was the appctx owner. Thus, it was heavily used. Now, as
far as possible,the conn-stream is used. Of course, it remains many calls to
the stream-interface API.
All old flags CS_FL_* are now moved in the endpoint scope and renamed
CS_EP_* accordingly. It is a systematic replacement. There is no true change
except for the health-check and the endpoint reset. Here it is a bit special
because the same conn-stream is reused. Thus, we must handle endpoint
allocation errors. To do so, cs_reset_endp() has been adapted.
Thanks to this last change, it will now be possible to simplify the
multiplexer and probably the applets too. A review must also be performed to
remove some flags in the channel or the stream-interface. The HTX will
probably be simplified too. Finally, there is now some place in the
conn-stream to move info from the stream-interface.
For now there is no much change. Only the appctx is passed as argument when
the .init callback function is called. And it is not possible to yield at
this stage. It is not a problem because the feature is not used. Only the
lua defines this callback function for the lua TCP/HTTP services. The idea
is to be able to use it for all applets to initialize the appctx context.
This reverts commit 6c9f7faa62.
This warning is no longer needed because source code can be compiled with
enabled support for OpenTracing context transfer via HAProxy variables.
This patch must be backported in 2.5.
Since commit 3a4bedccc ("MEDIUM: vars: replace the global name index with
a hash") the names of HAProxy variables are no longer saved, ie their
64-bit hashes are saved instead.
This is very convenient for the HAProxy itself, but for the OpenTracing
module it is a problem because the names of the variables are important
when transferring the OpenTracing context. Namely, this context consists
of an unknown amount of data stored in a key-value format. The number
of these data (and the name of the variable used for this purpose) is
determined with the configuration of the OpenTracing filter, as well as
with the tracer used. The two previous sentences seem to be in conflict,
but that is only so at first glance. The function in the OpenTracing
filter used to read the context does not really know this, neither their
number nor its name. The only thing that function actually knows is the
prefix of the variable names used for context transfer, and by that it
could find all the necessary data. Of course, until the application of
the above-mentioned commit.
The problem is solved in a very simple way: in a common variable that
the filter always knows its name, the names of all variables that are the
product of the OpenTracing context are saved. The names of these context
variables can only be added to that common variable. When that variable
is no longer needed (when we no longer need context), it is deleted.
The format for saving data to this common variable is as follows:
+-----+---------------+-- .. --+-----+---------------+
| len | variable name | | len | variable name |
+-----+---------------+-- .. --+-----+---------------+
The amount of memory space used to store the length of the name is 1 byte,
with a sign (the minus sign is provided for inactive records, but this is
not currently used). This means that the maximum length of the variable
name to be saved is 127 characters, which is quite enough for use in the
filter. The buffer size for such data storage is global.tune.bufsize.
This patch must be backported in 2.5.
A display of the contents of the err variable has been added to the
FLT_OT_ERR() macro, once it has been set.
This patch must be backported as far as 2.4.
It's always good to replace "hard-coded" values with something that looks
less "hard-coded" (even though that doesn't change anything in the code).
This is done here using FLT_OT_PARSE_INVALID_enum enum.
This patch must be backported as far as 2.4.
If the OpenTracing filter is compiled using the 'OT_DEBUG=1' option, then
log messages are printed to stderr when the filter is running. In the log
one can then find (among other things) the order in which the function is
called and the value that the function returns (if it is not a void type).
Prior to applying this patch, no value returned by a function was logged.
Log output example:
[ 1] 0.038807 [OT]: flt_ot_init_per_thread(0x56365bd45ec0, 0x56365bd48210) {
[ 1] 0.038807 [OT]: ot_start(0x56365bd58920, 0x56365bd4e3a0, 0x7f561acba168:(nil)) {
[ 1] 0.038807 [OT]: } = 0
[ 1] 0.038807 [OT]: } = 0
This patch must be backported as far as 2.4.
The flag_cpy parameter has been added to the flt_ot_normalize_name()
function, through which we can determine whether the function converts
special characters (which are part of that name) when copying a variable
name.
This patch must be backported in 2.5.
The same variable should not be used to store multiple different results,
because it can be confusing. Therefore, the var_name_len variable has
been added in several functions, in order to avoid the use of the retval
variable for several different purposes.
This patch must be backported as far as 2.4.
The flt_ot_smp_init() function has been added to make initializing the
sample structure easier. The contents of the structure in question are
set in several places in the source of the OpenTracing filter, so it is
better to do this in one place.
This patch must be backported as far as 2.4.
Regarding commit #61ecf2838:
There's no point taking the variables locks for sess/txn/req/res
contexts since these ones always run inside the same thread anyway.
This patch must be backported in 2.5.
This reverts commit 560c7b874a.
The ot.uuid variable should have the 'sess' scope because it is created
when an OpenTracing filter is attached to a stream. After that, the
stream processing is started and on that occasion the contexts for the
variables that have the range 'txn' and 'req' are initialized. This
means that we cannot use variables with the specified scopes before that
point.
This patch must be backported in 2.5.
The flt_ot_var_get() function is not used anywhere and is unnecessary
in the existing implementation of the OpenTracing filter.
This patch must be backported as far as 2.4.
The flt_ot_var_unset() function is not used anywhere and is unnecessary
in the existing implementation of the OpenTracing filter.
This patch must be backported as far as 2.4.
When calling the 'basename' command, the argument ${0} is enclosed in
quotation marks. This is necessary if the path of the executable script
(contained in that argument) has "non-standard" elements, such as space
and the like.
Also, in the script 'test-speed.sh' the function sh_exit() has been added
for easier printing of messages at the end of execution.
This patch must be backported as far as 2.4.
In case of using parameter 'OT_USE_VARS=1', the value of the OT_DEFINE
variable is set incorrectly (i.e. the previous value was deleted and a
new one set instead of adding new content).
This patch must be backported in 2.5.
Function flt_ot_var_set() did not check whether the variable was
successfully set or not. In case of failure, the value -1 is returned.
This patch must be backported as far as 2.4.
It is the same bug than "BUG/MEDIUM: stats: Be sure to never set EOM flag on
an empty HTX message". We must be sure to set the EOM flag on a non empyt
message to be sure the mux on the client will properly handle
shutdowns. Otherwise, it may miss the end of the message and considers any
shutdown as an abort.
This patch must be backported as far as 2.4. On previous version there is
still the EOM HTX block.
This bug is the same than for the HTTP client. See "BUG/MINOR: httpclient:
Set conn-stream/channel EOI flags at the end of request" for details.
This patch must be backported as far as 2.0. But only CF_EOI must be set
because applets are not attached to a conn-stream on older versions.
Because appctx is now an endpoint of the conn-stream, there is no reason to
still have the stream-interface as appctx owner. Thus, the conn-stream is
now the appctx owner.
The DeviceAtlas addon can optionally interacts with the new service
without change of configuration in the HAProxy part.
Note that however it requires the DeviceAtlas Identification C API
2.4.0 minimum from this point.
Signed-off-by: David Carlier <dcarlier@deviceatlas.com>
Mentions of the new database update runtime mode and update of
the legit module and the dummy part too.
Note the DeviceAtlas C API version 2.4.0 minimum required
alongside with libCURL, libzip and libgz.
New specialized service to daily handle the update of download file without
interruption of service and to be preemptively started before HAProxy.
It consists on a standalone utility which shared a memory block with
the DeviceAtlas module which handles the JSON data file update on
a daily basis.
Signed-off-by: David Carlier <dcarlier@deviceatlas.com>
Please do not set the OT_USE_VARS configuration variable, as the source
will probably not be able to compile! For now, this variable can only
be used for experimental purposes, and is not intended for wider use.
For further clarification, please see commit 4cb2c83f4.
Must be backported to 2.5.
- add new metric: `haproxy_backend_agg_server_check_status`
it counts the number of servers matching a specific check status
this permits to exclude per server check status as the usage is often
to rely on the total. Indeed in large setup having thousands of
servers per backend the memory impact is not neglible to store the per
server metric.
- realign promex_str_metrics array
quite simple implementation - we could improve it later by adding an
internal state to the prometheus exporter, thus to avoid counting at
every dump.
this patch is an attempt to close github issue #1312. It may bebackported
to 2.4 if requested.
Signed-off-by: William Dauchy <wdauchy@gmail.com>
