Certain types and enums are very specific to the HTTP/1 parser, and we'll
need to share them with the HTTP/2 to HTTP/1 translation code. Let's move
them to h1.c/h1.h. Those with very few occurrences or only used locally
were renamed to explicitly mention the relevant HTTP version :
enum ht_state -> h1_state.
http_msg_state_str -> h1_msg_state_str
HTTP_FLG_* -> H1_FLG_*
http_char_classes -> h1_char_classes
Others like HTTP_IS_*, HTTP_MSG_* are left to be done later.
Some compiler versions don't emit an error when facing an unknown
no-warning unless another error is reported, resulting in all -Wno-*
options being enabled by default and being reported as wrong with
build errors. Let's create a new "cc-nowarn" function to disable
warnings only after checking that the positive one is supported.
The recent gcc and clang are utterly broken and apparently written by
people who don't use them anymore, because they emit warnings that are
impossible to disable in the code, which is the opposite of what a
warning should do. It is however possible to disable these warnings on
the command line.
This patch adds when supported :
-Wno-format-truncation: bogus warning which is triggered on each
snprintf() call based on the input type instead of the variables
ranges, resulting in the impossibility to use "%02d" and similar.
-Wno-address-of-packed-member: emitted for each and every line in
ebtree.h by recent clang. Probably that the warning's author has
never understood the use cases of packed structs and should be
taught the use cases of the language he writes the compiler for.
-Wno-null-dereference: emitted by clang on *(int *)0 = 0. The code
will be updated to use a volatile instead but this recent change
of behaviour will certainly cause quite some bugs in decades of
existing code.
Feel free to report new such stupid warnings and to propose patches
to complete this list.
The recent gcc and clang are utterly broken and apparently written by
people who don't use them anymore, because they emit warnings that are
impossible to disable in the code, which is the opposite of what a
warning should do. It is however possible to disable these warnings on
the command line, but not in a backwards-compatible way.
Thus here we create a new function which detect if the compiler supports
certain options, and which adds them if supported.
Haproxy relies on signed integer wraparound on overflow, however this is
really an undefined behavior, so the C compiler is allowed to do whatever
it wants, and clang does exactly that, and that causes problems when the
timer goes from <= INT_MAX to > INT_MAX, and explains the various hangs
reported on FreeBSD every 49.7 days. To make sure we get the intended
behavior, use -fwrapv for now. A proper fix is to switch everything to
unsigned, and it will happen later, but this is simpler, and more likely to
be backported to the stable branches.
Many thanks to David King, Mark S, Dave Cottlehuber, Slawa Olhovchenkov,
Piotr Pawel Stefaniak, and any other I may have forgotten for reporting that
and investigating.
this adds a support of the newest pcre2 library,
more secure than its older sibling in a cost of a
more complex API.
It works pretty similarly to pcre's part to keep
the overall change smooth, except :
- we define the string class supported at compile time.
- after matching the ovec data is properly sized, althought
we do not take advantage of it here.
- the lack of jit support is treated less 'dramatically'
as pcre2_jit_compile in this case is 'no-op'.
When doing a parallel build on multiple CPUs it's common that at the end
a few CPUs only are busy compiling very large files while the other ones
have finished. By placing the largest files first, we can ensure that in
the worst case they are present from the beginning to the end, and that
other processes are free to take smaller files. This ordering was made
based on a measurement consisting in counting the number of times a given
file appears in the build. The top ten looks like this :
145 src/cfgparse.c
131 src/proto_http.c
83 src/ssl_sock.c
74 src/stats.c
73 src/stream.c
55 src/flt_spoe.c
48 src/server.c
46 src/pattern.c
43 src/checks.c
42 src/flt_http_comp.c
Only a few files were moved, ssl_sock would need to be moved as well but
that would not be a convenient thing to do in the makefile. This new
order allows to save about 10-15% of build time on 4 CPUs, which is nice.
There's no more reason to keep tcp rules processing inside proto_tcp.c
given that there is nothing in common there except these 3 letters : tcp.
The tcp rules are in fact connection, session and content processing rules.
Let's move them to "tcp-rules" and let them live their life there.
SPOE makes possible the communication with external components to retrieve some
info using an in-house binary protocol, the Stream Processing Offload Protocol
(SPOP). In the long term, its aim is to allow any kind of offloading on the
streams. This first version, besides being experimental, won't do lot of
things. The most important today is to validate the protocol design and lay the
foundations of what will, one day, be a full offload engine for the stream
processing.
So, for now, the SPOE can offload the stream processing before "tcp-request
content", "tcp-response content", "http-request" and "http-response" rules. And
it only supports variables creation/suppression. But, in spite of these limited
features, we can easily imagine to implement a SSO solution, an ip reputation
service or an ip geolocation service.
Internally, the SPOE is implemented as a filter. So, to use it, you must use
following line in a proxy proxy section:
frontend my-front
...
filter spoe [engine <name>] config <file>
...
It uses its own configuration file to keep the HAProxy configuration clean. It
is also a easy way to disable it by commenting out the filter line.
See "doc/SPOE.txt" for all details about the SPOE configuration.
In the last release a lot of the structures have become opaque for an
end user. This means the code using these needs to be changed to use the
proper functions to interact with these structures instead of trying to
manipulate them directly.
This does not fix any deprecations yet that are part of 1.1.0, it only
ensures that it can be compiled against that version and is still
compatible with older ones.
[wt: openssl-0.9.8 doesn't build with it, there are conflicts on certain
function prototypes which we declare as inline here and which are
defined differently there. But openssl-0.9.8 is not supported anymore
so probably it's OK to go without it for now and we'll see later if
some users still need it. Emeric has reviewed this change and didn't
spot anything obvious which requires special care. Let's try it for
real now]
WURFL is a high-performance and low-memory footprint mobile device
detection software component that can quickly and accurately detect
over 500 capabilities of visiting devices. It can differentiate between
portable mobile devices, desktop devices, SmartTVs and any other types
of devices on which a web browser can be installed.
In order to add WURFL device detection support, you would need to
download Scientiamobile InFuze C API and install it on your system.
Refer to www.scientiamobile.com to obtain a valid InFuze license.
Any useful information on how to configure HAProxy working with WURFL
may be found in:
doc/WURFL-device-detection.txt
doc/configuration.txt
examples/wurfl-example.cfg
Please find more information about WURFL device detection API detection
at https://docs.scientiamobile.com/documentation/infuze/infuze-c-api-user-guide
SOL_IPV6 is not defined on OSX, breaking the compile. Also libcrypt is
not available for installation neither in Macports nor as a Brew recipe,
so we're disabling implicit dependancy.
Signed-off-by: Dinko Korunic <dinko.korunic@gmail.com>
Kay Fuchs reported that the recent changes to automatically rebuild files
on config option changes caused "make install" to rebuild the whole code
with the wrong options. That's caused by the fact that the "install-bin"
target depends on the "haproxy" target, which detects the lack of options
and causes a rebuild with different ones.
This patch makes a simple change, it removes this automatic dependency
which was already wrong since it could cause some files to be built with
different options prior to these changes, and instead emits an error
message indicating that "make" should be run prior to "make install".
The patches were backported into 1.6 so this fix must go there as well.
We now instrument the makefile to keep a copy of previous build options.
The goal is to ensure that we'll rebuild everything when build options
change. The options that are watched are TARGET, VERBOSE_CFLAGS, and
BUILD_OPTIONS. These ones are copied into a file ".build_opts" and
compared to the new ones upon each build. This file is referenced in
the DEP variable which all .o files depend on, and it depends on the
code which updates it only upon changes. This ensures that a new file
is regenerated and detected upon change and that everything is rebuilt.
Some users tend to get caught by incorrect builds when they try patches
that modify some include file after they forget to run "make clean".
While we can't blame users who are not developers, forcing developers
to rely on a painful autodepend is not nice either and will cause them
to test their changes less often. Here we propose a reasonable tradeoff.
This patch introduces a new "INCLUDES" variable which enumerates all
the ".h" files and sets them as a build dependency for all ".o" files.
This list is then copied into a "DEP" variable which can safely be
overridden if desired. This way by default all .c files are rebuilt if
any include file changes. This is the safe method for all users. And
developers can simply add "DEP=" to their quick build scripts to keep
the old fast and efficient behaviour.
Libraries requires the export of embedded Lua symbols. If a library
is loaded by HAProxy or by an Lua program, an error like the following
error raises:
[ALERT] 085/135722 (7224) : parsing [test.cfg:8] : lua runtime error: error loading module 'test' from file './test.so':
./test.so: undefined symbol: lua_createtable
This patch modify the Makefile, and allow exports of the Lua symbols.
This patch must be backported in version 1.6
When Lua executes functions from its API, these can throws an error.
These function must be executed in a special environment which catch
these error, otherwise a critical error (like segfault) can raise.
This patch add a c file called "hlua_fcn.c" which collect all the
Lua/c function needing safe environment for its execution.
The "trace" filter has been added. It defines all available callbacks and for
each one it prints a trace message. To enable it:
listener test
...
filter trace
...
HTTP compression will be moved in a true filter. To prepare the ground, some
functions have been moved in a dedicated file. Idea is to keep everything about
compression algos in compression.c and everything related to the filtering in
flt_http_comp.c.
For now, a header has been added to help during the transition. It will be
removed later.
Unused empty ACL keyword list was removed. The "compression" keyword
parser was moved from cfgparse.c to flt_http_comp.c.
This patch adds the support of filters in HAProxy. The main idea is to have a
way to "easely" extend HAProxy by adding some "modules", called filters, that
will be able to change HAProxy behavior in a programmatic way.
To do so, many entry points has been added in code to let filters to hook up to
different steps of the processing. A filter must define a flt_ops sutrctures
(see include/types/filters.h for details). This structure contains all available
callbacks that a filter can define:
struct flt_ops {
/*
* Callbacks to manage the filter lifecycle
*/
int (*init) (struct proxy *p);
void (*deinit)(struct proxy *p);
int (*check) (struct proxy *p);
/*
* Stream callbacks
*/
void (*stream_start) (struct stream *s);
void (*stream_accept) (struct stream *s);
void (*session_establish)(struct stream *s);
void (*stream_stop) (struct stream *s);
/*
* HTTP callbacks
*/
int (*http_start) (struct stream *s, struct http_msg *msg);
int (*http_start_body) (struct stream *s, struct http_msg *msg);
int (*http_start_chunk) (struct stream *s, struct http_msg *msg);
int (*http_data) (struct stream *s, struct http_msg *msg);
int (*http_last_chunk) (struct stream *s, struct http_msg *msg);
int (*http_end_chunk) (struct stream *s, struct http_msg *msg);
int (*http_chunk_trailers)(struct stream *s, struct http_msg *msg);
int (*http_end_body) (struct stream *s, struct http_msg *msg);
void (*http_end) (struct stream *s, struct http_msg *msg);
void (*http_reset) (struct stream *s, struct http_msg *msg);
int (*http_pre_process) (struct stream *s, struct http_msg *msg);
int (*http_post_process) (struct stream *s, struct http_msg *msg);
void (*http_reply) (struct stream *s, short status,
const struct chunk *msg);
};
To declare and use a filter, in the configuration, the "filter" keyword must be
used in a listener/frontend section:
frontend test
...
filter <FILTER-NAME> [OPTIONS...]
The filter referenced by the <FILTER-NAME> must declare a configuration parser
on its own name to fill flt_ops and filter_conf field in the proxy's
structure. An exemple will be provided later to make it perfectly clear.
For now, filters cannot be used in backend section. But this is only a matter of
time. Documentation will also be added later. This is the first commit of a long
list about filters.
It is possible to have several filters on the same listener/frontend. These
filters are stored in an array of at most MAX_FILTERS elements (define in
include/types/filters.h). Again, this will be replaced later by a list of
filters.
The filter API has been highly refactored. Main changes are:
* Now, HA supports an infinite number of filters per proxy. To do so, filters
are stored in list.
* Because filters are stored in list, filters state has been moved from the
channel structure to the filter structure. This is cleaner because there is no
more info about filters in channel structure.
* It is possible to defined filters on backends only. For such filters,
stream_start/stream_stop callbacks are not called. Of course, it is possible
to mix frontend and backend filters.
* Now, TCP streams are also filtered. All callbacks without the 'http_' prefix
are called for all kind of streams. In addition, 2 new callbacks were added to
filter data exchanged through a TCP stream:
- tcp_data: it is called when new data are available or when old unprocessed
data are still waiting.
- tcp_forward_data: it is called when some data can be consumed.
* New callbacks attached to channel were added:
- channel_start_analyze: it is called when a filter is ready to process data
exchanged through a channel. 2 new analyzers (a frontend and a backend)
are attached to channels to call this callback. For a frontend filter, it
is called before any other analyzer. For a backend filter, it is called
when a backend is attached to a stream. So some processing cannot be
filtered in that case.
- channel_analyze: it is called before each analyzer attached to a channel,
expects analyzers responsible for data sending.
- channel_end_analyze: it is called when all other analyzers have finished
their processing. A new analyzers is attached to channels to call this
callback. For a TCP stream, this is always the last one called. For a HTTP
one, the callback is called when a request/response ends, so it is called
one time for each request/response.
* 'session_established' callback has been removed. Everything that is done in
this callback can be handled by 'channel_start_analyze' on the response
channel.
* 'http_pre_process' and 'http_post_process' callbacks have been replaced by
'channel_analyze'.
* 'http_start' callback has been replaced by 'http_headers'. This new one is
called just before headers sending and parsing of the body.
* 'http_end' callback has been replaced by 'channel_end_analyze'.
* It is possible to set a forwarder for TCP channels. It was already possible to
do it for HTTP ones.
* Forwarders can partially consumed forwardable data. For this reason a new
HTTP message state was added before HTTP_MSG_DONE : HTTP_MSG_ENDING.
Now all filters can define corresponding callbacks (http_forward_data
and tcp_forward_data). Each filter owns 2 offsets relative to buf->p, next and
forward, to track, respectively, input data already parsed but not forwarded yet
by the filter and parsed data considered as forwarded by the filter. A any time,
we have the warranty that a filter cannot parse or forward more input than
previous ones. And, of course, it cannot forward more input than it has
parsed. 2 macros has been added to retrieve these offets: FLT_NXT and FLT_FWD.
In addition, 2 functions has been added to change the 'next size' and the
'forward size' of a filter. When a filter parses input data, it can alter these
data, so the size of these data can vary. This action has an effet on all
previous filters that must be handled. To do so, the function
'filter_change_next_size' must be called, passing the size variation. In the
same spirit, if a filter alter forwarded data, it must call the function
'filter_change_forward_size'. 'filter_change_next_size' can be called in
'http_data' and 'tcp_data' callbacks and only these ones. And
'filter_change_forward_size' can be called in 'http_forward_data' and
'tcp_forward_data' callbacks and only these ones. The data changes are the
filter responsability, but with some limitation. It must not change already
parsed/forwarded data or data that previous filters have not parsed/forwarded
yet.
Because filters can be used on backends, when we the backend is set for a
stream, we add filters defined for this backend in the filter list of the
stream. But we must only do that when the backend and the frontend of the stream
are not the same. Else same filters are added a second time leading to undefined
behavior.
The HTTP compression code had to be moved.
So it simplifies http_response_forward_body function. To do so, the way the data
are forwarded has changed. Now, a filter (and only one) can forward data. In a
commit to come, this limitation will be removed to let all filters take part to
data forwarding. There are 2 new functions that filters should use to deal with
this feature:
* flt_set_http_data_forwarder: This function sets the filter (using its id)
that will forward data for the specified HTTP message. It is possible if it
was not already set by another filter _AND_ if no data was yet forwarded
(msg->msg_state <= HTTP_MSG_BODY). It returns -1 if an error occurs.
* flt_http_data_forwarder: This function returns the filter id that will
forward data for the specified HTTP message. If there is no forwarder set, it
returns -1.
When an HTTP data forwarder is set for the response, the HTTP compression is
disabled. Of course, this is not definitive.
Makefile deviceatlas throwing an error if the necessary pcre flag
is not passed avoiding surprising bunch of 'undefined reference'
for the user. Plus a tiny typo in OPENSSL area.
[wt: backport to 1.6]
doc/haproxy-{en,fr}.txt have been removed recently but they were still
referenced in the Makefile. Many other documents have also been
added. Instead of hard-coding a list of documents to install, install
all those in doc/ with some exceptions:
- coding-style.txt is more for developers
- gpl.txt and lgpl.txt are usually present at other places (and I would
have to remove them in the Debian packaging, less work for me)
The documentation in the subdirectories is not installed as it is more
targeted to developers.
Added support for city hash method, turned off multi threading support
and included maths library. Removed reference to compression library
which was never needed.
This was the first transparent proxy technology supported by haproxy
circa 2005 but it was obsoleted in 2007 by Tproxy 4.0 which removed a
lot of the earlier versions' shortcomings and was finally merged into
the kernel. Since nobody has been using cttproxy for many years now
and nobody has even just tried to compile the files, it's time to
remove it. The doc was updated as well.
On platforms where the dl*() functions are not part of the libc, a
program linking Lua also needs to link to libdl.
Moreover, on platforms using a gold linker with the --as-needed flag,
the libdl library needs to be linked after linking Lua, otherwise, it
won't be marked as needed and will be discarded and its symbols won't be
present at the end of the linking phase.
Ubuntu enables the --as-needed flag by default. Other distributions may
advertise its use, like Gentoo.
Moved 51Degrees code from src/haproxy.c, src/sample.c and src/cfgparse.c
into a separate files src/51d.c and include/import/51d.h.
Added two new functions init_51degrees() and deinit_51degrees(), updated
Makefile and other code reorganizations related to 51Degrees.
This patch adds support of variables during the processing of each stream. The
variables scope can be set as 'session', 'transaction', 'request' or 'response'.
The variable type is the type returned by the assignment expression. The type
can change while the processing.
The allocated memory can be controlled for each scope and each request, and for
the global process.
Implementation of a DNS client in HAProxy to perform name resolution to
IP addresses.
It relies on the freshly created UDP client to perform the DNS
resolution. For now, all UDP socket calls are performed in the
DNS layer, but this might change later when the protocols are
extended to be more suited to datagram mode.
A new section called 'resolvers' is introduced thanks to this patch. It
is used to describe DNS servers IP address and also many parameters.
Basic introduction of a UDP layer in HAProxy. It can be used as a
client only and manages UDP exchanges with servers.
It can't be used to load-balance UDP protocols, but only used by
internal features such as DNS resolution.
Since both DEVICEATLAS_INC and DEVICEATLAS_LIB are set to the same path
when building from sources, simply allow DEVICEATLAS_SRC to be set alone
to simplify the build procedure.
Till now 3 paths were needed, 51DEGREES_SRC, 51DEGREES_INC, and
51DEGREES_LIB. Let's make the last two default to 51DEGREES_SRC since
it's the same location, and fix the doc to reflect this (all three were
documented but inconsistently).
To build with 51Degrees set USE_51DEGREES=1. 51DEGREES_INC, 51DEGREES_LIB,
and 51DEGREES_SRC will need to be set to the 51Degrees pattern header and
C file.
This diff updates the Makefile to compile conditionally via
some new sets of flags, USE_DEVICEATLAS to enable the module
and the couple DEVICEATLAS_INC/DEVICEATLAS_LIB which needs to
point to the API root folder in order to compile the API and
the module.
The xxhash library provides a very fast and excellent hash algorithm
suitable for many purposes. It excels at hashing large blocks but is
also extremely fast on small ones. It's distributed under a 2-clause
BSD license (GPL-compatible) so it can be included here. Updates are
distributed here :
https://github.com/Cyan4973/xxHash
This will be usable to implement some maps/acl caches for heavy datasets
loaded from files (mostly regex-based but in general anything that cannot
be indexed in a tree).
There is now a pointer to the session in the stream, which is NULL
for now. The session pool is created as well. Some parts will move
from the stream to the session now.
With HTTP/2, we'll have to support multiplexed streams. A stream is in
fact the largest part of what we currently call a session, it has buffers,
logs, etc.
In order to catch any error, this commit removes any reference to the
struct session and tries to rename most "session" occurrences in function
names to "stream" and "sess" to "strm" when that's related to a session.
The files stream.{c,h} were added and session.{c,h} removed.
The session will be reintroduced later and a few parts of the stream
will progressively be moved overthere. It will more or less contain
only what we need in an embryonic session.
Sample fetch functions and converters will have to change a bit so
that they'll use an L5 (session) instead of what's currently called
"L4" which is in fact L6 for now.
Once all changes are completed, we should see approximately this :
L7 - http_txn
L6 - stream
L5 - session
L4 - connection | applet
There will be at most one http_txn per stream, and a same session will
possibly be referenced by multiple streams. A connection will point to
a session and to a stream. The session will hold all the information
we need to keep even when we don't yet have a stream.
Some more cleanup is needed because some code was already far from
being clean. The server queue management still refers to sessions at
many places while comments talk about connections. This will have to
be cleaned up once we have a server-side connection pool manager.
Stream flags "SN_*" still need to be renamed, it doesn't seem like
any of them will need to move to the session.
Willy Tarreau
Nan Liu
David Carlier
William Lallemand
Dmitry Sivachenko
Olivier Houchard
Christopher Faulet
Dirkjan Bussink
scientiamobile
Bertrand Jacquin
Daniel Jakots
Dinko Korunic
Thierry Fournier
Christopher Faulet
David CARLIER
Jerome Duval
Vincent Bernat
James Rosewell
Cyril Bonté
Dragan Dosen
Baptiste Assmann
Thomas Holmes