Commit Graph

52 Commits

Author SHA1 Message Date
Emeric Brun
bd163817ed MEDIUM: sink: build header in sink_write for log formats
This patch extends the sink_write prototype and code to
handle the rfc5424 and rfc3164 header.

It uses header building tools from log.c. Doing this some
functions/vars have been externalized.

facility and minlevel have been removed from the struct sink
and passed to args at sink_write because they depends of the log
and not of the sink (they remained unused by rest of the code
until now).
2020-05-19 11:04:11 +02:00
Ilya Shipitsin
77e3b4a2c4 CLEANUP: assorted typo fixes in the code and comments
These are mostly comments in the code. A few error messages were fixed
and are of low enough importance not to deserve a backport. Some regtests
were also fixed.
2020-03-14 09:42:07 +01:00
Christopher Faulet
5c6fefc8eb MINOR: log: Provide a function to emit a log for an application
Application is a generic term here. It is a modules which handle its own log
server list, with no dependency on a proxy. Such applications can now call the
function app_log() to log messages, passing a log server list and a tag as
parameters. Internally, the function __send_log() has been adapted accordingly.
2019-09-17 10:18:54 +02:00
Frédéric Lécaille
d803e475e5 MINOR: log: Enable the log sampling and load-balancing feature.
This patch implements the sampling and load-balancing of log servers configured
with "sample" new keyword implemented by this commit:
    'MINOR: log: Add "sample" new keyword to "log" lines'.
As the list of ranges used to sample the log to balance is ordered, we only
have to maintain ->curr_idx member of smp_info struct which is the index of
the sample and check if it belongs or not to the current range to decide if we
must send it to the log server or not.
2019-04-30 09:25:09 +02:00
Willy Tarreau
b6b3df3ed3 MEDIUM: initcall: use initcalls for a few initialization functions
signal_init(), init_log(), init_stream(), and init_task() all used to
only preset some values and lists. This needs to be done very early to
provide a reliable interface to all other users. The calls used to be
explicit in haproxy.c:init(). Now they're placed in initcalls at the
STG_PREPARE stage. The functions are not exported anymore.
2018-11-26 19:50:32 +01:00
William Lallemand
9c56a22b20 MINOR: log: introduce ha_notice()
It's like ha_warning() or ha_alert() but with a NOTICE prefix.
2018-11-21 19:02:23 +01:00
Joseph Herlant
85b4059b82 CLEANUP: Fix typos in the log subsystem
Fix some misspells in the code comments of the log subsystem.
2018-11-18 22:26:42 +01:00
Willy Tarreau
13ef773722 MINOR: log: report the number of dropped logs in the stats
It's easy to detect when logs on some paths are lost as sendmsg() will
return EAGAIN. This is particularly true when sending to /dev/log, which
often doesn't support a big logging capacity. Let's keep track of these
and report the total number of dropped messages in "show info".
2018-11-12 18:37:55 +01:00
Olivier Houchard
54620523e2 MINOR: log: One const should be enough.
"const const" doesn't bring much more constness, so only use one.
2018-09-06 18:52:15 +02:00
Willy Tarreau
5383935856 MINOR: log: provide a function to emit a log for a session
The new function sess_log() only needs a session to emit a log. It will
ignore the parts that depend on the stream. It is usable to emit a log
to report early errors in muxes. These ones will typically mention
"<BADREQ>" for the request and 0 for the HTTP status code.
2018-09-06 09:43:41 +02:00
Willy Tarreau
26ffa8544d CLEANUP: log: make the low_level lf_{ip,port,text,text_len} functions take consts
These ones were abusively relying on variables making it hard to integrate
with const arguments.
2018-09-05 20:01:23 +02:00
Willy Tarreau
43c538eab6 MINOR: log: move the log code to sess_build_logline() to add extra arguments
The current build_logline() can only be used with valid streams, which
means it is not suitable for use from muxes. We start by moving it into
another more generic function which takes the session as an argument,
to avoid complexifying all the internal API for jsut a few use cases.
This new function is not supposed to be called directly from outside so
we'll be able to instrument it to support several calling conventions.

For now the behaviour and conditions remain unchanged.
2018-09-05 20:01:23 +02:00
Christopher Faulet
4b0b79dd56 MINOR: log: move 'log' keyword parsing in dedicated function
Now, the function parse_logsrv should be used to parse a "log" line. This
function will update the list of loggers passed in argument. It can release all
log servers when "no log" line was parsed (by the caller) or it can parse "log
global" or "log <address> ... " lines. It takes care of checking the caller
context (global or not) to prohibit "log global" usage in the global section.
2018-04-05 15:13:54 +02:00
Willy Tarreau
bafbe01028 CLEANUP: pools: rename all pool functions and pointers to remove this "2"
During the migration to the second version of the pools, the new
functions and pool pointers were all called "pool_something2()" and
"pool2_something". Now there's no more pool v1 code and it's a real
pain to still have to deal with this. Let's clean this up now by
removing the "2" everywhere, and by renaming the pool heads
"pool_head_something".
2017-11-24 17:49:53 +01:00
Christopher Faulet
767a84bcc0 CLEANUP: log: Rename Alert/Warning in ha_alert/ha_warning 2017-11-24 17:19:12 +01:00
Christopher Faulet
f8188c69fa MEDIUM: threads/logs: Make logs thread-safe
log buffers and static variables used in log functions are now thread-local. So
there is no need to lock anything to log messages. Moreover, per-thread
init/deinit functions are now used to initialize these buffers.
2017-10-31 13:58:30 +01:00
Christopher Faulet
0132d06f68 MINOR: logs: Use dedicated function to init/deinit log buffers
Now, we use init_log_buffers and deinit_log_buffers to, respectively, initialize
and deinitialize log buffers used for syslog messages.

These functions have been introduced to be used by threads, to deal with
thread-local log buffers.
2017-09-05 10:29:31 +02:00
Thierry FOURNIER / OZON.IO
8a4e4420fb MEDIUM: log-format: Use standard HAProxy log system to report errors
The function log format emit its own error message using Alert(). This
patch replaces this behavior and uses the standard HAProxy error system
(with memprintf).

The benefits are:
 - cleaning the log system

 - the logformat can ignore the caller (actually the caller must set
   a flag designing the caller function).

 - Make the usage of the logformat function easy for future components.
2016-11-25 07:32:58 +01:00
Thierry FOURNIER / OZON.IO
a2c38d7904 MEDIUM: log-format: strict parsing and enable fail
Until now, the function parse_logformat_string() never fails. It
send warnings when it parses bad format, and returns expression in
best effort.

This patch replaces warnings by alert and returns a fail code.

Maybe the warning mode is designed for a compatibility with old
configuration versions. If it is the case, now this compatibility
is broken.

[wt: no, the reason is that an alert must cause a startup failure,
 but this will be OK with next patch]
2016-11-24 18:54:26 +01:00
Thierry FOURNIER / OZON.IO
6fe0e1b977 CLEANUP: log-format: remove unused arguments
The log-format function parse_logformat_string() takes file and line
for building parsing logs. These two parameters are embedded in the
struct proxy curproxy, which is the current parsing context.

This patch removes these two unused arguments.
2016-11-24 18:54:26 +01:00
Thierry FOURNIER / OZON.IO
eb51e16306 CLEANUP/MINOR: log-format: unexport functions parse_logformat_var_args() and parse_logformat_var()
Remove export of the fucntion parse_logformat_var_args() and
parse_logformat_var(). These functions are a part of the
logformat parser, and this export is useless.
2016-11-24 18:54:25 +01:00
Dragan Dosen
835b9212f6 MEDIUM: log: add a new log format flag "E"
The +E mode escapes characters '"', '\' and ']' with '\' as prefix. It
mostly makes sense to use it in the RFC5424 structured-data log formats.

Example:

log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]]
2016-02-12 13:36:47 +01:00
Dragan Dosen
43885c728e BUG/MEDIUM: logs: segfault writing to log from Lua
Michael Ezzell reported a bug causing haproxy to segfault during startup
when trying to send syslog message from Lua. The function __send_log() can
be called with *p that is NULL and/or when the configuration is not fully
parsed, as is the case with Lua.

This patch fixes this problem by using individual vectors instead of the
pre-generated strings log_htp and log_htp_rfc5424.

Also, this patch fixes a problem causing haproxy to write the wrong pid in
the logs -- the log_htp(_rfc5424) strings were generated at the haproxy
start, but "pid" value would be changed after haproxy is started in
daemon/systemd mode.
2015-10-02 00:57:45 +02:00
Dragan Dosen
c8cfa7b4f3 MEDIUM: logs: have global.log_send_hostname not contain the trailing space
This patch unifies global.log_send_hostname addition in the log header
processing.
2015-09-28 18:27:45 +02:00
Dragan Dosen
0b85ecee53 MEDIUM: logs: add a new RFC5424 log-format for the structured-data
This patch adds a new RFC5424-specific log-format for the structured-data
that is automatically send by __send_log() when the sender is in RFC5424
mode.

A new statement "log-format-sd" should be used in order to set log-format
for the structured-data part in RFC5424 formatted syslog messages.
Example:

    log-format-sd [exampleSDID@1234\ bytes=\"%B\"\ status=\"%ST\"]
2015-09-28 14:01:27 +02:00
Dragan Dosen
1322d09a6f MEDIUM: logs: add support for RFC5424 header format per logger
The function __send_log() iterates over senders and passes the header as
the first vector to sendmsg(), thus it can send a logger-specific header
in each message.

A new logger arguments "format rfc5424" should be used in order to enable
RFC5424 header format. For example:

    log 10.2.3.4:1234 len 2048 format rfc5424 local2 info
2015-09-28 14:01:27 +02:00
Dragan Dosen
68d2e3a742 MEDIUM: logs: remove the hostname, tag and pid part from the logheader
At the moment we have to call snprintf() for every log line just to
rebuild a constant. Thanks to sendmsg(), we send the message in 3 parts:
time-based header, proxy-specific hostname+log-tag+pid, session-specific
message.
2015-09-28 14:01:27 +02:00
Dragan Dosen
59cee973cd MEDIUM: log: use a separate buffer for the header and for the message
Make sendmsg() use two vectors, one for the message header that is updated
by update_log_hdr() and one for the message buffer.
2015-09-28 14:01:27 +02:00
Willy Tarreau
87b09668be REORG/MAJOR: session: rename the "session" entity to "stream"
With HTTP/2, we'll have to support multiplexed streams. A stream is in
fact the largest part of what we currently call a session, it has buffers,
logs, etc.

In order to catch any error, this commit removes any reference to the
struct session and tries to rename most "session" occurrences in function
names to "stream" and "sess" to "strm" when that's related to a session.

The files stream.{c,h} were added and session.{c,h} removed.

The session will be reintroduced later and a few parts of the stream
will progressively be moved overthere. It will more or less contain
only what we need in an embryonic session.

Sample fetch functions and converters will have to change a bit so
that they'll use an L5 (session) instead of what's currently called
"L4" which is in fact L6 for now.

Once all changes are completed, we should see approximately this :

   L7 - http_txn
   L6 - stream
   L5 - session
   L4 - connection | applet

There will be at most one http_txn per stream, and a same session will
possibly be referenced by multiple streams. A connection will point to
a session and to a stream. The session will hold all the information
we need to keep even when we don't yet have a stream.

Some more cleanup is needed because some code was already far from
being clean. The server queue management still refers to sessions at
many places while comments talk about connections. This will have to
be cleaned up once we have a server-side connection pool manager.
Stream flags "SN_*" still need to be renamed, it doesn't seem like
any of them will need to move to the session.
2015-04-06 11:23:56 +02:00
Willy Tarreau
18324f574f MEDIUM: log: support a user-configurable max log line length
With all the goodies supported by logformat, people find that the limit
of 1024 chars for log lines is too short. Some servers do not support
larger lines and can simply drop them, so changing the default value is
not always the best choice.

This patch takes a different approach. Log line length is specified per
log server on the "log" line, with a value between 80 and 65535. That
way it's possibly to satisfy all needs, even with some fat local servers
and small remote ones.
2014-06-27 18:13:53 +02:00
Thierry FOURNIER
eeaa951726 MINOR: configuration: File and line propagation
This patch permits to communicate file and line of the
configuration file at the configuration parser.
2014-03-17 18:06:08 +01:00
Willy Tarreau
434c57c95c MINOR: log: indicate it when some unreliable sample fetches are logged
If a log-format involves some sample fetches that may not be present at
the logging instant, we can now report a warning.

Note that this is done both for log-format and for add-header and carefully
respects the original fetch keyword's capabilities.
2013-04-03 02:12:56 +02:00
Willy Tarreau
6cbbdbf3f3 BUG/MEDIUM: log: emit '-' for empty fields again
Commit 2b0108ad accidently got rid of the ability to emit a "-" for
empty log fields. This can happen for captured request and response
cookies, as well as for fetches. Since we don't want to have this done
for headers however, we set the default log method when parsing the
format. It is still possible to force the desired mode using +M/-M.
2013-02-05 18:55:09 +01:00
Willy Tarreau
b83bc1e1c1 MINOR: log: make parse_logformat_string() take a const char *
Sometimes we can't pass a char *, and there is no need for this since we strdup() it.
2012-12-24 12:36:33 +01:00
Willy Tarreau
2b0108adf6 MINOR: log: add lf_text_len
This function allows to log a text of a specific length.
2012-12-21 19:24:48 +01:00
Willy Tarreau
8a3f52fc2e MEDIUM: log-format: make the format parser more robust and more extensible
The log-format parser reached a limit making it hard to add new features.
It also suffers from a weak handling of certain incorrect corner cases,
for example "%{foo}" is emitted as a litteral while syntactically it's an
argument to no variable. Also the argument parser had to redo some of the
job with some cases causing minor memory leaks (eg: ignored args).

This work aims at improving the situation so that slightly better reporting
is possible and that it becomes possible to extend the log format. The code
has a few more states but looks significantly simpler. The parser is now
capable of reporting ignored arguments and truncated lines.
2012-12-20 23:34:20 +01:00
Willy Tarreau
4f65356a22 MINOR: log: make lf_text use a const char *
lf_text() should use a const char * otherwise it makes it more complex
to use data coming from const strings.
2012-10-12 20:30:51 +02:00
William Lallemand
a73203e3dc MEDIUM: log: Unique ID
The Unique ID, is an ID generated with several informations. You can use
a log-format string to customize it, with the "unique-id-format" keyword,
and insert it in the request header, with the "unique-id-header" keyword.
2012-04-07 16:25:26 +02:00
William Lallemand
5f2324019d MEDIUM: log: New format-log flags: %Fi %Fp %Si %Sp %Ts %rt %H %pid
%Fi: Frontend IP
%Fp: Frontend Port
%Si: Server IP
%Sp: Server Port
%Ts: Timestamp
%rt: HTTP request counter
%H: hostname
%pid: PID

+X: Hexadecimal represenation

The +X mode in logformat displays hexadecimal for the following flags
%Ci %Cp %Fi %Fp %Bi %Bp %Si %Sp %Ts %ct %pid

rename logformat_write_string() to lf_text()

Optimize size computation
2012-04-07 16:05:39 +02:00
William Lallemand
1d7055675e MEDIUM: log: split of log_format generation
* logformat functions now take a format linked list as argument
* build_logline() build a logline using a format linked list
* rename LOG_* by LOG_FMT_* in enum
* improve error management in build_logline()
2012-04-07 16:05:02 +02:00
Willy Tarreau
b1a2faf7c9 BUG/CRITICAL: log: fix risk of crash in development snapshot
Commit a1cc38 introduced a regression which was easy to trigger till ad4cd58
(snapshots 20120222 to 20120311 included). The bug was still present after
that but harder to trigger.

The bug is caused by the use of two distinct log buffers due to intermediary
changes. The issue happens when an HTTP request is logged just after a TCP
request during the same second and the HTTP request is too large for the buffer.
In this case, it happens that the HTTP request is logged into the TCP buffer
instead and that length controls can't detect anything.

Starting with bddd4f, the issue is still possible when logging too large an
HTTP request just after a send_log() call (typically a server status change).

We owe a big thanks to Sander Klein for testing several snapshots and more
specifically for taking significant risks in production by letting the buggy
version crash several times in order to provide an exploitable core ! The bug
could not have been found without this precious help. Thank you Sander !

This fix does not need to be backported, it did not affect any released version.
2012-03-19 17:09:30 +01:00
William Lallemand
81f5117a24 BUG/MINOR: log-format: fix %o flag
The %o flag was not working at all.
2012-03-12 15:50:53 +01:00
William Lallemand
bddd4fd93b MEDIUM: log: use log_format for mode tcplog
Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to
log.c

A new field in logformat_type define if you can use a logformat
variable in TCP or HTTP mode.

doc: log-format in tcp mode

Note that due to the way log buffer allocation currently works, trying to
log an HTTP request without "option httplog" is still not possible. This
will change in the near future.
2012-03-12 15:47:13 +01:00
William Lallemand
a1cc381151 MEDIUM: log: make http_sess_log use log_format
http_sess_log now use the logformat linked list to make the log
string, snprintf is not used for speed issue.

CLF mode also uses logformat.

NOTE: as of now, empty fields in CLF now are "" not "-" anymore.
2012-02-09 17:03:28 +01:00
William Lallemand
723b73ad75 MINOR: config: Parse the string of the log-format config keyword
parse_logformat_string: parse the string, detect the type: text,
        separator or variable

parse_logformat_var: dectect variable name

parse_logformat_var_args: parse arguments and flags

add_to_logformat_list: add to the logformat linked list
2012-02-09 17:03:24 +01:00
William Lallemand
2a4a44f0f9 REORG: log: split send_log function
send_log function is now splited in 3 functions
* hdr_log: generate the syslog header
* send_log: send a syslog message with a printf format string
* __send_log: send a syslog message
2012-02-09 15:54:43 +01:00
Willy Tarreau
40d2516371 [BUILD] add format(printf) to printf-like functions
Doing this helps catching warnings about wrong output formats.
2009-04-03 12:01:47 +02:00
Willy Tarreau
ec6c5df018 [CLEANUP] remove many #include <types/xxx> from C files
It should be stated as a rule that a C file should never
include types/xxx.h when proto/xxx.h exists, as it gives
less exposure to declaration conflicts (one of which was
caught and fixed here) and it complicates the file headers
for nothing.

Only types/global.h, types/capture.h and types/polling.h
have been found to be valid includes from C files.
2008-07-16 10:30:42 +02:00
Willy Tarreau
332f8bfc5b [MAJOR] ported requri to use mempools v2 2007-05-13 21:36:56 +02:00
Willy Tarreau
422505801f [MEDIUM] splitted logs into two versions : TCP and HTTP
logs are handled better with dedicated functions. The HTTP implementation
moved to proto_http.c. It has been cleaned up a bit. Now a frontend with
option httplog and no log will not call the function anymore.
2007-04-01 01:30:43 +02:00