mirror of
http://git.haproxy.org/git/haproxy.git/
synced 2024-12-18 01:14:38 +00:00
70230c6484
This commit adds the OpenTracing filter (hereinafter we will use the abbreviated name 'the OT filter') to the contrib tree. The OT filter adds native support for using distributed tracing in HAProxy. This is enabled by sending an OpenTracing compliant request to one of the supported tracers; such as Datadog, Jaeger, Lightstep and Zipkin tracers. Please note: tracers are not listed by any preference, but alphabetically. The OT filter is a standard HAProxy filter, so what applies to others also applies to this one (of course, by that I mean what is described in the documentation, more precisely in the doc/internals/filters.txt file). The OT filter activation is done explicitly by specifying it in the HAProxy configuration. If this is not done, the OT filter in no way participates in the work of HAProxy. As for the impact on HAProxy speed, this is documented with several tests located in the test directory, and the result is found in the README-speed-* files. In short, the speed of operation depends on the way it is used and the complexity of the configuration, from an almost immeasurable impact to a significant deceleration (5x and more). I think that in some normal use the speed of HAProxy with the filter on will be quite satisfactory with a slowdown of less than 4%. The OT filter allows intensive use of ACLs, which can be defined anywhere in the configuration. Thus, it is possible to use the filter only for those connections that are of interest to us. More detailed documentation related to the operation, configuration and use of the filter can be found in the contrib/opentracing directory. To make the OpenTracing filter easier to configure and compile, several entries have been added to the Makefile. When running the make utility, it is possible to use several new arguments: USE_OT=1 : enable the OpenTracing filter OT_DEBUG=1 : compile the OpenTracing filter in debug mode OT_INC=path : force the include path to libopentracing-c-wrapper OT_LIB=path : force the lib path to libopentracing-c-wrapper OT_RUNPATH=1 : add libopentracing-c-wrapper RUNPATH to haproxy executable If USE_OT is set, then an additional Makefile from the contrib/opentracing directory is included in the compilation process.
795 lines
31 KiB
Plaintext
795 lines
31 KiB
Plaintext
-----------------------------------------
|
|
The HAProxy OpenTracing filter (OT)
|
|
Version 1.0
|
|
( Last update: 2020-12-10 )
|
|
-----------------------------------------
|
|
Author : Miroslav Zagorac
|
|
Contact : mzagorac at haproxy dot com
|
|
|
|
|
|
SUMMARY
|
|
--------
|
|
|
|
0. Terms
|
|
1. Introduction
|
|
2. Build instructions
|
|
3. Basic concepts in OpenTracing
|
|
4. OT configuration
|
|
4.1. OT scope
|
|
4.2. "ot-tracer" section
|
|
4.3. "ot-scope" section
|
|
4.4. "ot-group" section
|
|
5. Examples
|
|
5.1 Benchmarking results
|
|
6. OT CLI
|
|
7. Known bugs and limitations
|
|
|
|
|
|
0. Terms
|
|
---------
|
|
|
|
* OT: The HAProxy OpenTracing filter
|
|
|
|
OT is the HAProxy filter that allows you to send data to distributed
|
|
tracing systems via the OpenTracing API.
|
|
|
|
|
|
1. Introduction
|
|
----------------
|
|
|
|
Nowadays there is a growing need to divide a process into microservices and
|
|
there is a problem of monitoring the work of the same process. One way to
|
|
solve this problem is to use distributed tracing service in a central location,
|
|
the so-called tracer.
|
|
|
|
OT is a feature introduced in HAProxy 2.4. This filter enables communication
|
|
via the OpenTracing API with OpenTracing compatible servers (tracers).
|
|
Currently, tracers that support this API include Datadog, Jaeger, LightStep
|
|
and Zipkin.
|
|
|
|
The OT filter was primarily tested with the Jaeger tracer, while configurations
|
|
for both Datadog and Zipkin tracers were also set in the test directory.
|
|
|
|
The OT filter is a standard HAProxy filter, so what applies to others also
|
|
applies to this one (of course, by that I mean what is described in the
|
|
documentation, more precisely in the doc/internals/filters.txt file).
|
|
|
|
The OT filter activation is done explicitly by specifying it in the HAProxy
|
|
configuration. If this is not done, the OT filter in no way participates
|
|
in the work of HAProxy.
|
|
|
|
As for the impact on HAProxy speed, this is documented with several tests
|
|
located in the test directory, and the result is found in the README-speed-*
|
|
files. In short, the speed of operation depends on the way it is used and
|
|
the complexity of the configuration, from an almost immeasurable impact to
|
|
a significant deceleration (5x and more). I think that in some normal use
|
|
the speed of HAProxy with the filter on will be quite satisfactory with a
|
|
slowdown of less than 4% (provided that no more than 10% of requests are
|
|
sent to the tracer, which is determined by the keyword 'rate-limit').
|
|
|
|
The OT filter allows intensive use of ACLs, which can be defined anywhere in
|
|
the configuration. Thus, it is possible to use the filter only for those
|
|
connections that are of interest to us.
|
|
|
|
|
|
2. Build instructions
|
|
----------------------
|
|
|
|
OT is the HAProxy filter and as such is compiled together with HAProxy.
|
|
|
|
To communicate with some OpenTracing compatible tracer, the OT filter uses the
|
|
OpenTracing C Wrapper library (which again uses the OpenTracing CPP library).
|
|
This means that we must have both libraries installed on the system on which
|
|
we want to compile or use HAProxy.
|
|
|
|
Instructions for compiling and installing both required libraries can be
|
|
found at https://github.com/haproxytech/opentracing-c-wrapper .
|
|
|
|
Also, to use the OT filter when running HAProxy we need to have an OpenTracing
|
|
plugin for the tracer we want to use. We will return to this later, in
|
|
section 5.
|
|
|
|
The OT filter can be more easily compiled using the pkg-config tool, if we
|
|
have the OpenTracing C Wrapper library installed so that it contains pkg-config
|
|
files (which have the .pc extension). If the pkg-config tool cannot be used,
|
|
then the path to the directory where the include files and libraries are
|
|
located can be explicitly specified.
|
|
|
|
Below are examples of the two ways to compile HAProxy with the OT filter, the
|
|
first using the pkg-congfig tool and the second explicitly specifying the path
|
|
to the OpenTracing C Wrapper include and library.
|
|
|
|
Note: prompt '%' indicates that the command is executed under a unprivileged
|
|
user, while prompt '#' indicates that the command is executed under the
|
|
root user.
|
|
|
|
Example of compiling HAProxy using the pkg-congfig tool (assuming the
|
|
OpenTracing C Wrapper library is installed in the /opt directory):
|
|
|
|
% PKG_CONFIG_PATH=/opt/lib/pkgconfig make USE_OT=1 TARGET=linux-glibc
|
|
|
|
The OT filter can also be compiled in debug mode as follows:
|
|
|
|
% PKG_CONFIG_PATH=/opt/lib/pkgconfig make USE_OT=1 OT_DEBUG=1 TARGET=linux-glibc
|
|
|
|
HAProxy compilation example explicitly specifying path to the OpenTracing C
|
|
Wrapper include and library:
|
|
|
|
% make USE_OT=1 OT_INC=/opt/include OT_LIB=/opt/lib TARGET=linux-glibc
|
|
|
|
In case we want to use debug mode, then it looks like this:
|
|
|
|
% make USE_OT=1 OT_DEBUG=1 OT_INC=/opt/include OT_LIB=/opt/lib TARGET=linux-glibc
|
|
|
|
If the library we want to use is not installed on a unix system, then a locally
|
|
installed library can be used (say, which is compiled and installed in the user
|
|
home directory). In this case instead of /opt/include and /opt/lib the
|
|
equivalent paths to the local installation should be specified. Of course,
|
|
in that case the pkg-config tool can also be used if we have a complete
|
|
installation (with .pc files).
|
|
|
|
last but not least, if the pkg-config tool is not used when compiling, then
|
|
HAProxy executable may not be able to find the OpenTracing C Wrapper library
|
|
at startup. This can be solved in several ways, for example using the
|
|
LD_LIBRARY_PATH environment variable which should be set to the path where the
|
|
library is located before starting the HAProxy.
|
|
|
|
% LD_LIBRARY_PATH=/opt/lib /path-to/haproxy ...
|
|
|
|
Another way is to add RUNPATH to HAProxy executable that contains the path to
|
|
the library in question.
|
|
|
|
% make USE_OT=1 OT_RUNPATH=1 OT_INC=/opt/include OT_LIB=/opt/lib TARGET=linux-glibc
|
|
|
|
After HAProxy is compiled, we can check if the OT filter is enabled:
|
|
|
|
% ./haproxy -vv | grep opentracing
|
|
--- command output ----------
|
|
[ OT] opentracing
|
|
--- command output ----------
|
|
|
|
|
|
3. Basic concepts in OpenTracing
|
|
---------------------------------
|
|
|
|
Basic concepts of OpenTracing can be read on the OpenTracing documentation
|
|
website https://opentracing.io/docs/overview/.
|
|
|
|
Here we will list only the most important elements of distributed tracing and
|
|
these are 'trace', 'span' and 'span context'. Trace is a description of the
|
|
complete transaction we want to record in the tracing system. A span is an
|
|
operation that represents a unit of work that is recorded in a tracing system.
|
|
Span context is a group of information related to a particular span that is
|
|
passed on to the system (from service to service). Using this context, we can
|
|
add new spans to already open trace (or supplement data in already open spans).
|
|
|
|
An individual span may contain one or more tags, logs and baggage items.
|
|
The tag is a key-value element that is valid for the entire span. Log is a
|
|
key-value element that allows you to write some data at a certain time, it
|
|
can be used for debugging. A baggage item is a key-value data pair that can
|
|
be used for the duration of an entire trace, from the moment it is added to
|
|
the span.
|
|
|
|
|
|
4. OT configuration
|
|
--------------------
|
|
|
|
In order for the OT filter to be used, it must be included in the HAProxy
|
|
configuration, in the proxy section (frontend / listen / backend):
|
|
|
|
frontend ot-test
|
|
...
|
|
filter opentracing [id <id>] config <file>
|
|
...
|
|
|
|
If no filter id is specified, 'ot-filter' is used as default. The 'config'
|
|
parameter must be specified and it contains the path of the file used to
|
|
configure the OT filter.
|
|
|
|
|
|
4.1 OT scope
|
|
-------------
|
|
|
|
If the filter id is defined for the OT filter, then the OT scope with
|
|
the same name should be defined in the configuration file. In the same
|
|
configuration file we can have several defined OT scopes.
|
|
|
|
Each OT scope must have a defined (only one) "ot-tracer" section that is
|
|
used to configure the operation of the OT filter and define the used groups
|
|
and scopes.
|
|
|
|
OT scope starts with the id of the filter specified in square brackets and
|
|
ends with the end of the file or when a new OT scope is defined.
|
|
|
|
For example, this defines two OT scopes in the same configuration file:
|
|
[my-first-ot-filter]
|
|
ot-tracer tracer1
|
|
...
|
|
ot-group group1
|
|
...
|
|
ot-scope scope1
|
|
...
|
|
|
|
[my-second-ot-filter]
|
|
...
|
|
|
|
|
|
4.2. "ot-tracer" section
|
|
-------------------------
|
|
|
|
Only one "ot-tracer" section must be defined for each OT scope.
|
|
|
|
There are several keywords that must be defined for the OT filter to work.
|
|
These are 'config' which defines the configuration file for the OpenTracing
|
|
API, and 'plugin' which defines the OpenTracing plugin used.
|
|
|
|
Through optional keywords can be defined ACLs, logging, rate limit, and groups
|
|
and scopes that define the tracing model.
|
|
|
|
|
|
ot-tracer <name>
|
|
A new OT with the name <name> is created.
|
|
|
|
Arguments :
|
|
name - the name of the tracer section
|
|
|
|
|
|
The following keywords are supported in this section:
|
|
- mandatory keywords:
|
|
- config
|
|
- plugin
|
|
|
|
- optional keywords:
|
|
- acl
|
|
- debug-level
|
|
- groups
|
|
- [no] log
|
|
- [no] option disabled
|
|
- [no] option dontlog-normal
|
|
- [no] option hard-errors
|
|
- rate-limit
|
|
- scopes
|
|
|
|
|
|
acl <aclname> <criterion> [flags] [operator] <value> ...
|
|
Declare or complete an access list.
|
|
|
|
To configure and use the ACL, see section 7 of the HAProxy Configuration
|
|
Manual.
|
|
|
|
|
|
config <file>
|
|
'config' is one of the two mandatory keywords associated with the OT tracer
|
|
configuration. This keyword sets the path of the configuration file for the
|
|
OpenTracing tracer plugin. To set the contents of this configuration file,
|
|
it is best to look at the documentation related to the OpenTracing tracer we
|
|
want to use.
|
|
|
|
Arguments :
|
|
file - the path of the configuration file
|
|
|
|
|
|
debug-level <value>
|
|
This keyword sets the value of the debug level related to the display of
|
|
debug messages in the OT filter. The 'debug-level' value is binary, ie
|
|
a single value bit enables or disables the display of the corresponding
|
|
debug message that uses that bit. The default value is set via the
|
|
FLT_OT_DEBUG_LEVEL macro in the include/config.h file. Debug level value
|
|
is used only if the OT filter is compiled with the debug mode enabled,
|
|
otherwise it is ignored.
|
|
|
|
Arguments :
|
|
value - binary value ranging from 0 to 255 (8 bits)
|
|
|
|
|
|
groups <name> ...
|
|
A list of "ot-group" groups used for the currently defined tracer is declared.
|
|
Several groups can be specified in one line.
|
|
|
|
Arguments :
|
|
name - the name of the OT group
|
|
|
|
|
|
log global
|
|
log <addr> [len <len>] [format <fmt>] <facility> [<level> [<minlevel>]]
|
|
no log
|
|
Enable per-instance logging of events and traffic.
|
|
|
|
To configure and use the logging system, see section 4.2 of the HAProxy
|
|
Configuration Manual.
|
|
|
|
|
|
option disabled
|
|
no option disabled
|
|
Keyword which turns the operation of the OT filter on or off. By default
|
|
the filter is on.
|
|
|
|
|
|
option dontlog-normal
|
|
no option dontlog-normal
|
|
Enable or disable logging of normal, successful processing. By default,
|
|
this option is disabled. For this option to be considered, logging must
|
|
be turned on.
|
|
|
|
See also: 'log' keyword description.
|
|
|
|
|
|
option hard-errors
|
|
no option hard-errors
|
|
During the operation of the filter, some errors may occur, caused by
|
|
incorrect configuration of the tracer or some error related to the operation
|
|
of HAProxy. By default, such an error will not interrupt the filter
|
|
operation for the stream in which the error occurred. If the 'hard-error'
|
|
option is enabled, the operation error prohibits all further processing of
|
|
events and groups in the stream in which the error occurred.
|
|
|
|
|
|
plugin <file>
|
|
'plugin' is one of the two mandatory keywords associated with the OT tracer
|
|
configuration. This keyword sets the path of the OpenTracing tracer plugin.
|
|
|
|
Arguments :
|
|
file - the name of the plugin used
|
|
|
|
|
|
rate-limit <value>
|
|
This option allows limiting the use of the OT filter, ie it can be influenced
|
|
whether the OT filter is activated for a stream or not. Determining whether
|
|
or not a filter is activated depends on the value of this option that is
|
|
compared to a randomly selected value when attaching the filter to the stream.
|
|
By default, the value of this option is set to 100.0, ie the OT filter is
|
|
activated for each stream.
|
|
|
|
Arguments :
|
|
value - floating point value ranging from 0.0 to 100.0
|
|
|
|
|
|
scopes <name> ...
|
|
This keyword declares a list of "ot-scope" definitions used for the currently
|
|
defined tracer. Multiple scopes can be specified in the same line.
|
|
|
|
Arguments :
|
|
name - the name of the OT scope
|
|
|
|
|
|
4.3. "ot-scope" section
|
|
------------------------
|
|
|
|
Stream processing begins with filter attachment, then continues with the
|
|
processing of a number of defined events and groups, and ends with filter
|
|
detachment. The "ot-scope" section is used to define actions related to
|
|
individual events. However, this section may be part of a group, so the
|
|
event does not have to be part of the definition.
|
|
|
|
|
|
ot-scope <name>
|
|
Creates a new OT scope definition named <name>.
|
|
|
|
Arguments :
|
|
name - the name of the OT scope
|
|
|
|
|
|
The following keywords are supported in this section:
|
|
- acl
|
|
- baggage
|
|
- event
|
|
- extract
|
|
- finish
|
|
- inject
|
|
- log
|
|
- span
|
|
- tag
|
|
|
|
|
|
acl <aclname> <criterion> [flags] [operator] <value> ...
|
|
Declare or complete an access list.
|
|
|
|
To configure and use the ACL, see section 7 of the HAProxy Configuration
|
|
Manual.
|
|
|
|
|
|
baggage <name> <sample> ...
|
|
Baggage items allow the propagation of data between spans, ie allow the
|
|
assignment of metadata that is propagated to future children spans.
|
|
This data is formatted in the style of key-value pairs and is part of
|
|
the context that can be transferred between processes that are part of
|
|
a server architecture.
|
|
|
|
This kewyord allows setting the baggage for the currently active span. The
|
|
data type is always a string, ie any sample type is converted to a string.
|
|
The exception is a binary value that is not supported by the OT filter.
|
|
|
|
See the 'tag' keyword description for the data type conversion table.
|
|
|
|
Arguments :
|
|
name - key part of a data pair
|
|
sample - sample expression (value part of a data pair), at least one
|
|
sample must be present
|
|
|
|
|
|
event <name> [{ if | unless } <condition>]
|
|
Set the event that triggers the 'ot-scope' to which it is assigned.
|
|
Optionally, it can be followed by an ACL-based condition, in which case it
|
|
will only be evaluated if the condition is true.
|
|
|
|
ACL-based conditions are executed in the context of a stream that processes
|
|
the client and server connections. To configure and use the ACL, see
|
|
section 7 of the HAProxy Configuration Manual.
|
|
|
|
Arguments :
|
|
name - the event name
|
|
condition - a standard ACL-based condition
|
|
|
|
Supported events are (the table gives the names of the events in the OT
|
|
filter and the corresponding equivalent in the SPOE filter):
|
|
|
|
-------------------------------------|------------------------------
|
|
the OT filter | the SPOE filter
|
|
-------------------------------------|------------------------------
|
|
on-client-session-start | on-client-session
|
|
on-frontend-tcp-request | on-frontend-tcp-request
|
|
on-http-wait-request | -
|
|
on-http-body-request | -
|
|
on-frontend-http-request | on-frontend-http-request
|
|
on-switching-rules-request | -
|
|
on-backend-tcp-request | on-backend-tcp-request
|
|
on-backend-http-request | on-backend-http-request
|
|
on-process-server-rules-request | -
|
|
on-http-process-request | -
|
|
on-tcp-rdp-cookie-request | -
|
|
on-process-sticking-rules-request | -
|
|
on-client-session-end | -
|
|
on-server-unavailable | -
|
|
-------------------------------------|------------------------------
|
|
on-server-session-start | on-server-session
|
|
on-tcp-response | on-tcp-response
|
|
on-http-wait-response | -
|
|
on-process-store-rules-response | -
|
|
on-http-response | on-http-response
|
|
on-server-session-end | -
|
|
-------------------------------------|------------------------------
|
|
|
|
|
|
extract <name-prefix> [use-vars | use-headers]
|
|
For a more detailed description of the propagation process of the span
|
|
context, see the description of the keyword 'inject'. Only the process
|
|
of extracting data from the carrier is described here.
|
|
|
|
Arguments :
|
|
name-prefix - data name prefix (ie key element prefix)
|
|
use-vars - data is extracted from HAProxy variables
|
|
use-headers - data is extracted from the HTTP header
|
|
|
|
|
|
Below is an example of using HAProxy variables to transfer span context data:
|
|
|
|
--- test/ctx/ot.cfg --------------------------------------------------------
|
|
...
|
|
ot-scope client_session_start_2
|
|
extract "ot_ctx_1" use-vars
|
|
span "Client session" child-of "ot_ctx_1"
|
|
...
|
|
----------------------------------------------------------------------------
|
|
|
|
|
|
finish <name> ...
|
|
Closing a particular span or span context. Instead of the name of the span,
|
|
there are several specially predefined names with which we can finish certain
|
|
groups of spans. So it can be used as the name '*req*' for all open spans
|
|
related to the request channel, '*res*' for all open spans related to the
|
|
response channel and '*' for all open spans regardless of which channel they
|
|
are related to. Several spans and/or span contexts can be specified in one
|
|
line.
|
|
|
|
Arguments :
|
|
name - the name of the span or context context
|
|
|
|
|
|
inject <name-prefix> [use-vars] [use-headers]
|
|
In OpenTracing, the transfer of data related to the tracing process between
|
|
microservices that are part of a larger service is done through the
|
|
propagation of the span context. The basic operations that allow us to
|
|
access and transfer this data are 'inject' and 'extract'.
|
|
|
|
'inject' allows us to extract span context so that the obtained data can
|
|
be forwarded to another process (microservice) via the selected carrier.
|
|
'inject' in the name actually means inject data into carrier. Carrier is
|
|
an interface here (ie a data structure) that allows us to transfer tracing
|
|
state from one process to another.
|
|
|
|
Data transfer can take place via one of two selected storage methods, the
|
|
first is by adding data to the HTTP header and the second is by using HAProxy
|
|
variables. Only data transfer via HTTP header can be used to transfer data
|
|
to another process (ie microservice). All data is organized in the form of
|
|
key-value data pairs.
|
|
|
|
No matter which data transfer method you use, we need to specify a prefix
|
|
for the key element. All alphanumerics (lowercase only) and underline
|
|
character can be used to construct the data name prefix. Uppercase letters
|
|
can actually be used, but they will be converted to lowercase when creating
|
|
the prefix.
|
|
|
|
Arguments :
|
|
name-prefix - data name prefix (ie key element prefix)
|
|
use-vars - HAProxy variables are used to store and transfer data
|
|
use-headers - HTTP headers are used to store and transfer data
|
|
|
|
|
|
Below is an example of using HTTP headers and variables, and how this is
|
|
reflected in the internal data of the HAProxy process.
|
|
|
|
--- test/ctx/ot.cfg --------------------------------------------------------
|
|
...
|
|
ot-scope client_session_start_1
|
|
span "HAProxy session" root
|
|
inject "ot_ctx_1" use-headers use-vars
|
|
...
|
|
----------------------------------------------------------------------------
|
|
|
|
- generated HAProxy variable (key -> value):
|
|
txn.ot_ctx_1.uberDtraceDid -> 8f1a05a3518d2283:8f1a05a3518d2283:0:1
|
|
|
|
- generated HTTP header (key: value):
|
|
ot_ctx_1-uber-trace-id: 8f1a05a3518d2283:8f1a05a3518d2283:0:1
|
|
|
|
Because HAProxy does not allow the '-' character in the variable name (which
|
|
is automatically generated by the OpenTracing API and on which we have no
|
|
influence), it is converted to the letter 'D'. We can see that there is no
|
|
such conversion in the name of the HTTP header because the '-' sign is allowed
|
|
there. Due to this conversion, initially all uppercase letters are converted
|
|
to lowercase because otherwise we would not be able to distinguish whether
|
|
the disputed sign '-' is used or not.
|
|
|
|
Thus created HTTP headers and variables are deleted when executing the
|
|
'finish' keyword or when detaching the stream from the filter.
|
|
|
|
|
|
log <name> <sample> ...
|
|
This kewyord allows setting the log for the currently active span. The
|
|
data type is always a string, ie any sample type is converted to a string.
|
|
The exception is a binary value that is not supported by the OT filter.
|
|
|
|
See the 'tag' keyword description for the data type conversion table.
|
|
|
|
Arguments :
|
|
name - key part of a data pair
|
|
sample - sample expression (value part of a data pair), at least one
|
|
sample must be present
|
|
|
|
|
|
span <name> [<reference>]
|
|
Creating a new span (or referencing an already opened one). If a new span
|
|
is created, it can be a child of the referenced span, follow from the
|
|
referenced span, or be root 'span'. In case we did not specify a reference
|
|
to the previously created span, the new span will become the root span.
|
|
We need to pay attention to the fact that in one trace there can be only
|
|
one root span. In case we have specified a non-existent span as a reference,
|
|
a new span will not be created.
|
|
|
|
Arguments :
|
|
name - the name of the span being created or referenced (operation
|
|
name)
|
|
reference - span or span context to which the created span is referenced
|
|
|
|
|
|
tag <name> <sample> ...
|
|
This kewyord allows setting a tag for the currently active span. The first
|
|
argument is the name of the tag (tag ID) and the second its value. A value
|
|
can consist of one or more data. If the value is only one data, then the
|
|
type of that data depends on the type of the HAProxy sample. If the value
|
|
contains more data, then the data type is string. The data conversion table
|
|
is below:
|
|
|
|
HAProxy sample data type | the OpenTracing data type
|
|
--------------------------+---------------------------
|
|
NULL | NULL
|
|
BOOL | BOOL
|
|
INT32 | INT64
|
|
UINT32 | UINT64
|
|
INT64 | INT64
|
|
UINT64 | UINT64
|
|
IPV4 | STRING
|
|
IPV6 | STRING
|
|
STRING | STRING
|
|
BINARY | UNSUPPORTED
|
|
--------------------------+---------------------------
|
|
|
|
Arguments :
|
|
name - key part of a data pair
|
|
sample - sample expression (value part of a data pair), at least one
|
|
sample must be present
|
|
|
|
|
|
4.4. "ot-group" section
|
|
------------------------
|
|
|
|
This section allows us to define a group of OT scopes, that is not activated
|
|
via an event but is triggered from TCP or HTTP rules. More precisely, these
|
|
are the following rules: 'tcp-request', 'tcp-response', 'http-request',
|
|
'http-response' and 'http-after-response'. These rules can be defined in the
|
|
HAProxy configuration file.
|
|
|
|
|
|
ot-group <name>
|
|
Creates a new OT group definition named <name>.
|
|
|
|
Arguments :
|
|
name - the name of the OT group
|
|
|
|
|
|
The following keywords are supported in this section:
|
|
- scopes
|
|
|
|
|
|
scopes <name> ...
|
|
'ot-scope' sections that are part of the specified group are defined. If
|
|
the mentioned 'ot-scope' sections are used only in some OT group, they do
|
|
not have to have defined events. Several 'ot-scope' sections can be
|
|
specified in one line.
|
|
|
|
Arguments :
|
|
name - the name of the 'ot-scope' section
|
|
|
|
|
|
5. Examples
|
|
------------
|
|
|
|
Several examples of the OT filter configuration can be found in the test
|
|
directory. A brief description of the prepared configurations follows:
|
|
|
|
cmp - the configuration very similar to that of the spoa-opentracing project.
|
|
It was made to compare the speed of the OT filter with the
|
|
implementation of distributed tracing via spoa-opentracing application.
|
|
|
|
sa - the configuration in which all possible events are used.
|
|
|
|
ctx - the configuration is very similar to the previous one, with the only
|
|
difference that the spans are opened using the span context as a span
|
|
reference.
|
|
|
|
fe be - a slightly more complicated example of the OT filter configuration
|
|
that uses two cascaded HAProxy services. The span context between
|
|
HAProxy processes is transmitted via the HTTP header.
|
|
|
|
empty - the empty configuration in which the OT filter is initialized but
|
|
no event is triggered. It is not very usable, except to check the
|
|
behavior of the OT filter in the case of a similar configuration.
|
|
|
|
|
|
In order to be able to collect data (and view results via the web interface)
|
|
we need to install some of the supported tracers. We will use the Jaeger
|
|
tracer as an example. Installation instructions can be found on the website
|
|
https://www.jaegertracing.io/download/. For the impatient, here we will list
|
|
how the image to test the operation of the tracer system can be installed
|
|
without much reading of the documentation.
|
|
|
|
# docker pull jaegertracing/all-in-one:latest
|
|
# docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
|
|
-p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 \
|
|
-p 16686:16686 -p 14268:14268 -p 9411:9411 jaegertracing/all-in-one:latest
|
|
|
|
The last command will also initialize and run the Jaeger container. If we
|
|
want to use that container later, it can be started and stopped in the classic
|
|
way, using the 'docker container start/stop' commands.
|
|
|
|
|
|
In order to be able to use any of the configurations from the test directory,
|
|
we must also have a tracer plugin in that directory (all examples use the
|
|
Jaeger tracer plugin). The simplest way is to download the tracer plugin
|
|
using the already prepared shell script get-opentracing-plugins.sh.
|
|
The script accepts one argument, the directory in which the download is made.
|
|
If run without an argument, the script downloads all plugins to the current
|
|
directory.
|
|
|
|
% ./get-opentracing-plugins.sh
|
|
|
|
After that, we can run one of the pre-configured configurations using the
|
|
provided script run-xxx.sh (where xxx is the name of the configuration being
|
|
tested). For example:
|
|
|
|
% ./run-sa.sh
|
|
|
|
The script will create a new log file each time it is run (because part of the
|
|
log file name is the start time of the script).
|
|
|
|
Eh, someone will surely notice that all test configurations use the Jaeger
|
|
tracing plugin that cannot be downloaded using the get-opentracing-plugins.sh
|
|
script. Unfortunately, the latest precompiled version that can be downloaded
|
|
is 0.4.2, for newer ones only the source code can be found. Version 0.4.2 has
|
|
a bug that can cause the operation of the OT filter to get stuck, so it is
|
|
better not to use this version. Here is the procedure by which we can compile
|
|
a newer version of the plugin (in our example it is 0.5.0).
|
|
|
|
Important note: the GCC version must be at least 4.9 or later.
|
|
|
|
% wget https://github.com/jaegertracing/jaeger-client-cpp/archive/v0.5.0.tar.gz
|
|
% tar xf v0.5.0.tar.gz
|
|
% cd jaeger-client-cpp-0.5.0
|
|
% mkdir build
|
|
% cd build
|
|
% cmake -DCMAKE_INSTALL_PREFIX=/opt -DJAEGERTRACING_PLUGIN=ON -DHUNTER_CONFIGURATION_TYPES=Release -DHUNTER_BUILD_SHARED_LIBS=OFF ..
|
|
% make
|
|
|
|
After the plugin is compiled, it will be in the current directory. The name
|
|
of the plugin is libjaegertracing_plugin.so.
|
|
|
|
|
|
5.1. Benchmarking results
|
|
--------------------------
|
|
|
|
To check the operation of the OT filter, several different test configurations
|
|
have been made which are located in the test directory. The test results of
|
|
the same configurations (with the names README-speed-xxx, where xxx is the name
|
|
of the configuration being tested) are also in the directory of the same name.
|
|
|
|
All tests were performed on the same debian 9.13 system, CPU i7-4770, 32 GB RAM.
|
|
For the purpose of testing, the thttpd web server on port 8000 was used.
|
|
Testing was done with the wrk utility running via run-xxx.sh scripts; that is,
|
|
via the test-speed.sh script that is run as follows:
|
|
|
|
% ./test-speed.sh all
|
|
|
|
The above mentioned thttpd web server is run from that script and it should be
|
|
noted that we need to have the same installed on the system (or change the path
|
|
to the thttpd server in that script if it is installed elsewhere).
|
|
|
|
Each test is performed several times over a period of 5 minutes per individual
|
|
test. The only difference when running the tests for the same configuration
|
|
was in changing the 'rate-limit' parameter (and the 'option disabled' option),
|
|
which is set to the following values: 100.0, 50.0, 10.0, 2.5 and 0.0 percent.
|
|
Then a test is performed with the OT filter active but disabled for request
|
|
processing ('option disabled' is included in the ot.cfg configuration). In
|
|
the last test, the OT filter is not used at all, ie it is not active and does
|
|
not affect the operation of HAProxy in any way.
|
|
|
|
|
|
6. OT CLI
|
|
----------
|
|
|
|
Via the HAProxy CLI interface we can find out the current status of the OT
|
|
filter and change several of its settings.
|
|
|
|
All supported CLI commands can be found in the following way, using the
|
|
socat utility with the assumption that the HAProxy CLI socket path is set
|
|
to /tmp/haproxy.sock (of course, instead of socat, nc or other utility can
|
|
be used with a change in arguments when running the same):
|
|
|
|
% echo "help" | socat - UNIX-CONNECT:/tmp/haproxy.sock | grep flt-ot
|
|
--- command output ----------
|
|
flt-ot debug [level] : set the OT filter debug level (default: get current debug level)
|
|
flt-ot disable : disable the OT filter
|
|
flt-ot enable : enable the OT filter
|
|
flt-ot soft-errors : turning off hard-errors mode
|
|
flt-ot hard-errors : enabling hard-errors mode
|
|
flt-ot logging [state] : set logging state (default: get current logging state)
|
|
flt-ot rate [value] : set the rate limit (default: get current rate value)
|
|
flt-ot status : show the OT filter status
|
|
--- command output ----------
|
|
|
|
'flt-ot debug' can only be used in case the OT filter is compiled with the
|
|
debug mode enabled.
|
|
|
|
|
|
7. Known bugs and limitations
|
|
------------------------------
|
|
|
|
The name of the span context definition can contain only letters, numbers and
|
|
characters '_' and '-'. Also, all uppercase letters in the name are converted
|
|
to lowercase. The character '-' is converted internally to the 'D' character,
|
|
and since a HAProxy variable is generated from that name, this should be taken
|
|
into account if we want to use it somewhere in the HAProxy configuration.
|
|
The above mentioned span context is used in the 'inject' and 'extract' keywords.
|
|
|
|
Let's look a little at the example test/fe-be (configurations are in the
|
|
test/fe and test/be directories, 'fe' is here the abbreviation for frontend
|
|
and 'be' for backend). In case we have the 'rate-limit' set to a value less
|
|
than 100.0, then distributed tracing will not be started with each new HTTP
|
|
request. It also means that the span context will not be delivered (via the
|
|
HTTP header) to the backend HAProxy process. The 'rate-limit' on the backend
|
|
HAProxy must be set to 100.0, but because the frontend HAProxy does not send
|
|
a span context every time, all such cases will cause an error to be reported
|
|
on the backend server. Therefore, the 'hard-errors' option must be set on the
|
|
backend server, so that processing on that stream is stopped as soon as the
|
|
first error occurs. Such cases will slow down the backend server's response
|
|
a bit (in the example in question it is about 3%).
|