1
0
mirror of http://git.haproxy.org/git/haproxy.git/ synced 2024-12-13 15:04:42 +00:00
Commit Graph

77 Commits

Author SHA1 Message Date
Willy Tarreau
0034cd9e0a BUILD: contrib: fix ip6range build on Centos 7
Jarno Huuskonen reported that ip6range doesn't build anymore on
Centos 7 (and possibly other distros) due to "in6_u" not being known.
Using s6_addr32 instead of in6_u.u6_addr32 apparently works fine, and
it's also what the Lua code uses so it should be OK.

This patch may be backported to 1.6.
2016-11-22 11:50:51 +01:00
Christopher Faulet
03a3449e1a MINOR: spoe: Remove useless 'timeout ack' option
To limit the time to process an event, you should set 'timeout processing'
option. So 'timeout ack' option is redundant and useless.
2016-11-21 15:29:59 +01:00
Willy Tarreau
29cc11ca41 CONTRIB: tcploop: add basic loops via a jump instruction
This one jumps back to the oldest post-fork and post-accept action,
so it allows to recv(), pause() and send() in loops after a fork()
and an accept() for example. This is handy for bugs that reproduce
once in a while or to keep idle connections working.
2016-11-12 19:16:29 +01:00
Willy Tarreau
1973e81c06 CONTRIB: tcploop: don't report failed send() or recv()
Many clients close with an RST on the last response or when they get
their response, so let's not report an error.
2016-11-12 19:16:19 +01:00
Willy Tarreau
59623e0aa8 CONTRIB: tcploop: support sending plain strings
By passing "S:<string>" instead of S<size> it's possible to send
a pre-defined string, which is convenient to write HTTP requests or
responses.

Example : produce two responses, one in keep-alive, one not for ab :

  ./tcploop 8001 L W N2 A R S:"HTTP/1.0 200 OK\r\nConnection: keep-alive\r\nContent-length: 50\r\n\r\n0123456789.123456789.123456789.123456789.123456789" R S:"HTTP/1.0 200 OK\r\nContent-length: 50\r\n\r\n0123456789.123456789.123456789.123456789.123456789"

With 20 such keep-alive responses and 10 parallel processes, ab achieves
350kreq/s, so it should be possible to get precise timings.
2016-11-12 18:39:32 +01:00
Willy Tarreau
9557bacfff CONTRIB: tcploop: update the usage output
Otherwise we have no other help.
2016-11-12 18:28:29 +01:00
Willy Tarreau
869c759153 CONTRIB: tcploop: implement logging when called with -v
This is helpful to show what state we're dealing with. The pid is
written, optionally followed by the time in 3 different formats
(relative/absolute) depending on the command line option (-t, -tt, -ttt).
2016-11-12 18:28:29 +01:00
Willy Tarreau
95a6b786fc CONTRIB: tcploop: implement fork()
Fork is a very convenient way to deal with independant yet properly
timed connections. It's particularly useful here for accept(), and
ensures that any accepted FD will automatically be released. The
principle is that when we hit a fork command, the parent restarts
evaluating the actions from the beginning and the child continues
to evaluate the next actions. Listen and connect are skipped if the
connection is already established. Fork() is amazingly cheap on
Linux, 21k forked connections per second are handled on a single
core, and 38k on two cores.

For now it's not possible to have two different code paths so in order
to have both a listener and a connector, two distinct commands are
still needed.
2016-11-12 18:26:43 +01:00
Willy Tarreau
84393aa863 CONTRIB: tcploop: scriptable TCP I/O for debugging purposes
netcat, nc6 and socat are only partially convenient as reproducers for
state machine bugs, but when it comes to adding delays, forcing resets,
waiting for data to be acked, they become useless.

The purpose of this utility is to be able to easily script some TCP
operations such as connect, accept, send, receive, shutdown and of
course pauses.
2016-11-12 18:04:05 +01:00
Willy Tarreau
a7da4d24f5 CONTRIB: debug/flags: add check for SF_ERR_CHK_PORT
This flag was added by commit 95db2bc ("MAJOR: check: find out which
port to use for health check at run time"), let's check for it.
2016-11-11 08:05:34 +01:00
Willy Tarreau
b01b3ada6b BUILD: debug/flags: remove test for SF_COMP_READY
It doesn't exist anymore.
2016-11-11 08:04:44 +01:00
Christopher Faulet
ba7bc164f7 MINOR: spoe/checks: Add support for SPOP health checks
A new "option spop-check" statement has been added to enable server health
checks based on SPOP HELLO handshake. SPOP is the protocol used by SPOE filters
to talk to servers.
2016-11-09 22:57:02 +01:00
Christopher Faulet
010fdedc37 MINOR: spoe: add random ip-reputation service as SPOA example
This is a very simple service that implement a "random" ip reputation
service. It will return random scores for all checked IP addresses. It only
shows you how to implement a ip reputation service or such kind of services
using the SPOE.
2016-11-09 22:57:02 +01:00
Pavlos Parissis
68535bddf3 MINOR: systemd: Perform sanity check on config before reload 2016-06-20 11:03:19 +02:00
Pavlos Parissis
cc23a79f34 MINOR: systemd: Use variable for config and pidfile paths
Users can set the location of haproxy.cfg and pidfile files by providing
a systemd overwrite file /etc/systemd/system/haproxy.service.d/overwrite.conf
with the following content:

    [Service]
    Environment=CONFIG=/etc/foobar/haproxy.cfg
2016-06-20 11:03:13 +02:00
Willy Tarreau
074d65bf25 CONTRIB: initiate a debugging suite to make debugging easier
The goal is to have a collection of quick-n-dirty utilities that make
debugging easier and that can easily be modified when needed. The first
utility in this series is called "flags". For a given numeric argument,
it reports the various known combinations of flags for channels, streams
and so on. This way it's easy to copy-paste values from the CLI or from
gdb and immediately know what state a stream-interface or connection is
in.
2015-12-20 23:21:57 +01:00
Kristoffer Gr�nlund
91a9f1a2e1 MINOR: systemd: Check configuration before start
Adds a configuration check before starting the haproxy service.
2014-10-15 20:01:09 +02:00
Apollon Oikonomopoulos
74f016985a BUG/MEDIUM: systemd: set KillMode to 'mixed'
By default systemd will send SIGTERM to all processes in the service's
control group. In our case, this includes the wrapper, the master
process and all worker processes.

Since commit c54bdd2a the wrapper actually catches SIGTERM and survives
to see the master process getting killed by systemd and regard this as
an error, placing the unit in a failed state during "systemctl stop".

Since the wrapper now handles SIGTERM by itself, we switch the kill mode
to 'mixed', which means that systemd will deliver the initial SIGTERM to
the wrapper only, and if the actual haproxy processes don't exit after a
given amount of time (default: 90s), a SIGKILL is sent to all remaining
processes in the control group. See systemd.kill(5) for more
information.

This should also be backported to 1.5.
2014-10-09 11:44:42 +02:00
Willy Tarreau
c874653bb4 BUILD: don't use type "uint" which is not portable
Dmitry Sivachenko reported that "uint" doesn't build on FreeBSD 10.
On Linux it's defined in sys/types.h and indicated as "old". Just
get rid of the very few occurrences.
2014-05-28 23:05:07 +02:00
Willy Tarreau
9f66aa9cc4 CONTRIB: halog: avoid calling time/localtime/mktime for each line
The last commit provides time-based filtering. Unfortunately, it wastes
90% of the time calling the expensive time()/localtime()/mktime()
functions.

This patch does 3 things :
  - call time()/localtime() only once to initialize the correct
    struct timeinfo ;

  - call mktime() only when the time has changed regardless of
    the current second.

  - manually add the current second to the cached result.

Doing just this is enough to multiply the parsing speed by 8.
2014-05-23 16:40:25 +02:00
Olivier Burgard
e97b904801 CONTRIB: halog: Filter input lines by date and time through timestamp
I wanted to make a graph with average answer time in nagios that takes only
the last 5 mn of the log. Filtering the log before using halog was too
slow, so I added that filter to halog.

The patch attached to this mail is a proposal to add a new option : -time
[min][:max]

The values are min timestamp and/or max timestamp of the lines to be used
for stats. The date and time of the log lines between '[' and ']' are
converted to timestamp and compared to these values.

Here is an exemple of usage :
cat /var/log/haproxy.log | ./halog -srv -H -q -time $(date --date '-5 min' +%s)
2014-05-23 16:18:48 +02:00
Thierry FOURNIER
1a0fb5dd35 CONTRIB: ip6range: add a network IPv6 range to mask converter
It can be used to convert some Maxmind geolocation IPv6 lists using the
CVS format to IPv6 networks format.
2014-03-17 18:06:06 +01:00
Willy Tarreau
7cf479cc09 MEDIUM: halog: add support for counting per source address (-ic)
This is the same as -uc except that instead of counting URLs, it
counts source addresses. The reported times are request times and
not response times.

The code becomes heavily ugly, the url struct is being abused to
store an address, and there are no more bit fields available. The
code needs a major revamp.
2013-02-16 23:49:04 +01:00
Marc-Antoine Perennou
f825580da8 MEDIUM: add systemd service
Signed-off-by: Marc-Antoine Perennou <Marc-Antoine@Perennou.com>
2013-02-13 10:47:59 +01:00
Willy Tarreau
04281bd1ad MINOR: contrib: make the iprange tool grep for addresses
The iprange tool is handy for transforming network range formats, but
it's common to need a tool for running quick checks on the output.
The tool now supports a list of addresses on the command line, and it
will only output those which match. It's absolutely inefficient but is
handy for debugging.
2012-12-13 00:11:02 +01:00
Willy Tarreau
a1629a59d1 BUG: halog: fix broken output limitation
Commit 667c905f introduced parameter -m to halog which limits the size
of the output. Unfortunately it is completely broken in that it doesn't
check that the limit was previously set or not, and also prevents a
simple counting operation from returning anything if a limit is not set.

Note that the -gt and -pct outputs behave differently in face of this
limit, since they count the valid output lines BEFORE actually producing
the data, so the limit really applies to valid input lines.
2012-11-13 20:48:15 +01:00
Willy Tarreau
667c905fe5 MINOR: halog: add a parameter to limit output line count
Sometimes it's useful to limit the output to a number of lines, for
example when output is already sorted (eg: 10 slowest URLs, ...). Now
we can use -m for this.
2012-10-10 16:49:28 +02:00
Willy Tarreau
4201df77df BUG/MINOR: halog: fix help message for -ut/-uto
Erroneous copy-paste suggesting wrong option.
2012-10-10 14:57:35 +02:00
Willy Tarreau
0a70688016 BUG/MINOR: halog: -ad/-ac report the correct number of output lines
There was a lines_out++ left from earlier code, causing each input
line to be counted as an output line.

This fix also affects 1.4 and should be backported.
2012-10-10 13:43:17 +02:00
Willy Tarreau
8a09b663a8 MINOR: halog: sort output by cookie code
It's sometimes useful to have the output sorted by cookie code to see
the ratios of NI vs VN for example. This is now possible with -cc.
2012-10-10 10:27:18 +02:00
Baptiste
61aaad06e8 CONTRIB: halog: sort URLs by avg bytes_read or total bytes_read
The patch attached to this mail brings ability to sort URLs by
averaged bytes read and total bytes read in HALog tool.
In most cases, bytes read is also the object size.
The purpose of this patch is to know which URL consume the most
bandwith, in average or in total.
It may be interesting as well to know the standard deviation (ecart
type in french) for some counters (like bytes_read).

The results:
- Sorting by average bytes read per URL:
./halog -uba <~/tmp/haproxy.log | column -t | head
2246 lines in, 302 lines out, 194 parsing errors
18    0    5101     283    5101   283    126573  2278327  /lib/exe/js.php
1     0    1        1      1      1      106734  106734   /wp-admin/images/screenshots/theme-customizer.png
2     0    2        1      2      1      106511  213022   /wp-admin/css/wp-admin.css
1     0    1        1      1      1      96698   96698    /wp-admin/images/screenshots/captions-1.png
1     0    1        1      1      1      73165   73165    /wp-admin/images/screenshots/flex-header-1.png
4     0    0        0      0      0      64832   259328   /cuisine/wp-content/plugins/stats/open-flash-chart.swf
1     0    0        0      0      0      48647   48647    /wp-admin/images/screenshots/flex-header-3.png
1     0    0        0      0      0      44046   44046    /wp-admin/images/screenshots/captions-2.png
1     0    1        1      1      1      38830   38830    /wp-admin/images/screenshots/flex-header-2.png

- Sorting by total bytes read per URL:
./halog -ubt <~/tmp/haproxy.log | column -t | head
2246 lines in, 302 lines out, 194 parsing errors
18    0    5101     283    5101   283    126573  2278327  /lib/exe/js.php
60    0    14387    239    14387  239    10081   604865   /lib/exe/css.php
64    2    8820     137    8819   142    7742    495524   /doku.php
14    0    250      17     250    17     24045   336632   /wp-admin/load-scripts.php
71    0    6422     90     6422   90     4048    287419   /wp-admin/
4     0    0        0      0      0      64832   259328   /cuisine/wp-content/plugins/stats/open-flash-chart.swf
2     0    2        1      2      1      106511  213022   /wp-admin/css/wp-admin.css
31    3    5423     174    5040   180    6804    210931   /index
10    0    429      42     429    42     18009   180093   /cuisine/files/2011/10/tarte_figue_amande-e1318281546905-225x300.jpg
2012-09-09 08:44:01 +02:00
Willy Tarreau
f8c95d2a25 OPTIM: halog: improve cold-cache behaviour when loading a file
Using posix_fadvise() it is possible to tell the system that we're
going to read a whole file at once. The kernel then doubles the
read-ahead size for this file. On Linux with an SSD, this has improved
cold-cache performance by around 20%. Hot-cache is not affected at all.
2012-06-12 09:16:56 +02:00
Willy Tarreau
419a598eae OPTIM: halog: make use of memchr() on platforms which provide a fast one
glibc-2.11 on x86_64 provides a machine-specific memchr() which is faster
than the generic C implementation by around 40%, so let's make it possible
to use it instead of the hand-coded version.
2012-06-12 08:52:22 +02:00
Willy Tarreau
8ad4193100 CLEANUP: halog: make clean should also remove .o files 2012-06-12 07:59:16 +02:00
Willy Tarreau
de5dc0509c MINOR: halog: use the more recent dual-mode fgets2 implementation
This version implements both 32 and 64 bit versions at once, it
avoids the need to have two separate output files. It also improves
efficiency on i386 platforms by adding a little bit of assembly where
gcc isn't efficient.
2012-06-09 11:22:27 +02:00
Willy Tarreau
7de211c88b MINOR: add a new function call tracer for debugging purposes
This feature relies on GCC's ability to call helpers at function entry/exit
points. We define these helpers to quickly dump the minimum info into a trace
file that can be converted to a human readable format using a script in the
contrib/trace directory. This has only been implemented in the GNU makefile
for now on as it is unsure whether it's supported on all OSes.

The feature is enabled by building with "TRACE=1". The performance impact is
huge, so this feature should only be used when debugging. To limit the loss
of performance, fprintf() has been disabled and the output is hand-crafted
and emitted using fwrite(), resulting in doubling the performance. Using the
TSC instead of gettimeofday() also doubles the performance. Around 1200 conns/s
may be achieved on a Pentium-M 1.7 GHz which leads to around 50 MB/s of traces.

The entry and exits of all functions will be dumped into a file designated
by the HAPROXY_TRACE environment variable, or by default "trace.out". If the
trace file name is empty or "/dev/null", then traces are disabled. If
opening the trace file fails, then stderr is used. If HAPROXY_TRACE_FAST is
used, then the time is taken from the global <now> variable. Last, if
HAPROXY_TRACE_TSC is used, then the machine's TSC is used instead of the
real time (almost twice as fast).

The output format is :

  <sec.usec> <level> <caller_ptr> <dir> <callee_ptr>
or :
  <tsc> <level> <caller_ptr> <dir> <callee_ptr>

where <dir> is '>' when entering a function and '<' when leaving.

The awk script in contrib/trace provides a nicer indented output :

6f74989e6f8 ->->->   run_poll_loop > signal_process_queue [src/haproxy.c:1097:0x804bd69] > [include/proto/signal.h:32:0x8049cd0]
6f74989eb00          run_poll_loop < signal_process_queue [src/haproxy.c:1097:0x804bd69] < [include/proto/signal.h:32:0x8049cd0]
6f74989ef44 ->->->   run_poll_loop > wake_expired_tasks [src/haproxy.c:1100:0x804bd72] > [src/task.c:123:0x8055060]
6f74989f3a6 ->->->->   wake_expired_tasks > eb32_lookup_ge [src/task.c:128:0x8055091] > [ebtree/eb32tree.c:138:0x80a8c70]
6f74989f7e9            wake_expired_tasks < eb32_lookup_ge [src/task.c:128:0x8055091] < [ebtree/eb32tree.c:138:0x80a8c70]
6f74989fc0d ->->->->   wake_expired_tasks > eb32_first [src/task.c:134:0x80550d5] > [ebtree/eb32tree.h:55:0x8054ad0]
6f7498a003d ->->->->->   eb32_first > eb_first [ebtree/eb32tree.h:56:0x8054af1] > [ebtree/ebtree.h:520:0x8054a10]
6f7498a0436 ->->->->->->   eb_first > eb_walk_down [ebtree/ebtree.h:521:0x8054a33] > [ebtree/ebtree.h:442:0x80549a0]
6f7498a0843 ->->->->->->->   eb_walk_down > eb_gettag [ebtree/ebtree.h:445:0x80549d6] > [ebtree/ebtree.h:418:0x80548e0]
6f7498a0c2b                  eb_walk_down < eb_gettag [ebtree/ebtree.h:445:0x80549d6] < [ebtree/ebtree.h:418:0x80548e0]
6f7498a1042 ->->->->->->->   eb_walk_down > eb_untag [ebtree/ebtree.h:447:0x80549e2] > [ebtree/ebtree.h:412:0x80548a0]
6f7498a1498                  eb_walk_down < eb_untag [ebtree/ebtree.h:447:0x80549e2] < [ebtree/ebtree.h:412:0x80548a0]
6f7498a18c6 ->->->->->->->   eb_walk_down > eb_root_to_node [ebtree/ebtree.h:448:0x80549e7] > [ebtree/ebtree.h:432:0x8054960]
6f7498a1cd4                  eb_walk_down < eb_root_to_node [ebtree/ebtree.h:448:0x80549e7] < [ebtree/ebtree.h:432:0x8054960]
6f7498a20c4                eb_first < eb_walk_down [ebtree/ebtree.h:521:0x8054a33] < [ebtree/ebtree.h:442:0x80549a0]
6f7498a24b4              eb32_first < eb_first [ebtree/eb32tree.h:56:0x8054af1] < [ebtree/ebtree.h:520:0x8054a10]
6f7498a289c            wake_expired_tasks < eb32_first [src/task.c:134:0x80550d5] < [ebtree/eb32tree.h:55:0x8054ad0]
6f7498a2c8c          run_poll_loop < wake_expired_tasks [src/haproxy.c:1100:0x804bd72] < [src/task.c:123:0x8055060]
6f7498a3095 ->->->   run_poll_loop > process_runnable_tasks [src/haproxy.c:1103:0x804bd7a] > [src/task.c:190:0x8055150]

A nice improvement would possibly consist in trying to get the function's
arguments in the stack and to dump a few more infor for some well-known
functions (eg: the session's status for process_session).
2012-05-26 00:12:37 +02:00
Willy Tarreau
9bb0e2042e MINOR: contrib/iprange: add a network IP range to mask converter
This tool has remained uncommitted in my development tree for almost a year.
Just minor polish and commit.

It can be used to convert some geolocation IP lists to ACLs.
2012-04-02 21:44:05 +02:00
Willy Tarreau
615674cdec MINOR: halog: add some help on the command line 2012-01-23 08:17:59 +01:00
Willy Tarreau
e1a908c369 OPTIM: halog: keep a fast path for the lines-count only
Using "halog -c" is still something quite common to perform on logs,
but unfortunately since the recent added controls, it was sensibly
slowed down due to the parsing of the accept date field.

Now we use a specific loop for the case where nothing is needed from
the input, and this sped up the line counting by 2.5x. A 2.4 GHz Xeon
now counts lines at a rate of 2 GB of logs per second.
2012-01-03 09:28:05 +01:00
Willy Tarreau
08911ff896 MINOR: halog: add support for matching queued requests
-Q outputs all requests which went through at least one queue.
-QS outputs all requests which went through a server queue.
2011-10-13 13:28:36 +02:00
Willy Tarreau
6ee71754e2 BUILD: halog: make halog build on solaris
Solaris' "rm" command does not support -v. Also, specify CC=gcc
because "cc" generally is not gcc there.
2011-09-16 15:03:37 +02:00
Willy Tarreau
f9042060c9 [OPTIM] halog: add assembly version of the field lookup code
Gcc tries to be a bit too smart in these small loops and the result is
that on i386 we waste a lot of time there. By recoding these loops in
assembly, we save up to 23% total processing time on i386! The savings
on x86_64 are much lower, probably because there are more registers and
gcc has to do less tricks. However, those savings vary a lot between gcc
versions and even cause harm on some of them (eg: 4.4) because gcc does
not know how to optimize the code once inlined.

However, by recoding field_start() in C to try to match the assembly
code as much as possible, we can significantly reduce its execution
time without risking the negative impacts. Thus, the assembly version
is less interesting there but still worth being used on some compilers.
2011-09-10 12:39:30 +02:00
Willy Tarreau
31a02e9c5b [OPTIM] halog: make fgets parse more bytes by blocks
By adding a "landing area" at the end of the buffer, it becomes safe to
parse more bytes at once. On 32-bit this makes fgets run about 4% faster
but it does not save anything on 64-bit.
2011-09-10 10:46:39 +02:00
Willy Tarreau
96c148b0d2 [MINOR] halog: do not consider byte 0x8A as end of line
A bug in the algorithm used to find an LF in multiple bytes at once
made byte 0x80 trigger detection of byte 0x00, thus 0x8A matches byte
0x0A. In practice, this issue never happens since byte 0x8A won't be
displayed in logs (or it will be encoded). This could still possibly
happen in mixed logs.
2011-09-09 08:21:55 +02:00
Willy Tarreau
61a40c7402 [MINOR] halog: support backslash-escaped quotes
Some syslog servers escape quotes, which make the resulting logs unusable
for URL processing since the parser looks for the first field beginning
with a quote. It now supports also fields starting with backslash and
quote in order to address this. No performance impact was measured.
2011-09-06 08:11:27 +02:00
Willy Tarreau
d3007ffa6f [MINOR] halog: add -hs/-HS to filter by HTTP status code range
The code was merged with the error code checking which is very similar and
which shares the same information. The new test adds about 1% slowdown to
error checking but makes it more reliable when facing wrongly formated
status codes.
2011-09-05 02:09:24 +02:00
Herv� COMMOWICK
927cdddf9c [MINOR] halog: add support for termination code matching (-tcn/-TCN)
It is now possible to filter by termination code with -tcn <termcode>, to be
able to track one kind of errors, for example after counting it with -tc.
Use -TCN <termcode> gives you the opposite.
2011-08-10 18:04:50 +02:00
Willy Tarreau
14389e7036 [OPTIM] halog: remove support for tab delimiters in input data
Haproxy does not use tabs when sending logs, and checking for them
wastes no less than 4% of CPU cycles. Better get rid of these tests.
2011-07-11 06:48:04 +02:00
Willy Tarreau
a2b39fb5c5 [OPTIM] halog: remove many 'if' by using a function pointer for the filters
There were too many filters, we were losing time in all the "if" statements.
By moving all the filters to independant functions, we made the code cleaner
and slightly faster (3%).

One minor bug was found, the -tc and -st options did not report the number
of output lines, but always zero.
2011-07-11 06:48:04 +02:00
Willy Tarreau
26deaf51d9 [OPTIM] halog: check once for correct line format and reuse the pointer
Almost all filters first check the line format, which takes a lot of code
and requires parsing back and forth. By centralizing this test, we can
save about 15-20 more percent of performance for all filters.

Also, the test was wrong, it was checking that the source IP address was
starting with a digit, which is not always true with local IPv6 addresses.
Instead, we now check that the next field (accept field) starts with an
opening bracket and is followed by a digit between 0 and 3 (day of the
month). Doing this has contributed a 2% speedup because all other field
calculations were relative to a closer field.
2011-07-11 06:48:04 +02:00