Commit Graph

2931 Commits

Author SHA1 Message Date
Willy Tarreau
0ede5a3318 BUG/MEDIUM: session: fix FD leak when transport layer logging is enabled
Commit 2b199c9a attempted to fix all places where the transport layer
is improperly closed, but it missed one place in session_free(). If
SSL ciphers are logged, the close() is delayed post-log and performed
in session_free(). However, conn_xprt_close() only closes the transport
layer but not the file descriptor, resulting in a slow FD leak which is
hardly noticeable until the process cannot accept any new connection.

A workaround consisted in disabling %sslv/%sslc in log-format.

So use conn_full_close() instead of conn_xprt_close() to fix this there
too.

A similar pending issue existed in the close during outgoing connection
failure, though on this side, the transport layer is never tracked at the
moment.
2012-12-08 08:48:04 +01:00
Willy Tarreau
26d7cfce32 BUG/MAJOR: polling: do not set speculative events on ERR nor HUP
Errors and Hangups are sticky events, which means that once they're
detected, we never clear them, allowing them to be handled later if
needed.

Till now when an error was reported, it used to register a speculative
I/O event for both recv and send. Since the connection had not requested
such events, it was not able to detect a change and did not clear them,
so the events were called in loops until a timeout caused their owner
task to die.

So this patch does two things :
  - stop registering spec events when no I/O activity was requested,
    so that we don't end up with non-disablable polling state ;

  - keep the sticky polling flags (ERR and HUP) when leaving the
    connection handler so that an error notification doesn't
    magically become a normal recv() or send() report once the
    event is converted to a spec event.

It is normally not needed to make the connection handler emit an
error when it detects POLL_ERR because either a registered data
handler will have done it, or the event will be disabled by the
wake() callback.
2012-12-07 00:09:43 +01:00
Willy Tarreau
debdc4b657 BUG/MAJOR: raw_sock: must check error code on hangup
In raw_sock, we already check for FD_POLL_HUP after a short recv()
to avoid a useless syscall and detect the end of stream. However,
we fail to check for FD_POLL_ERR here, which causes major issues
as some errors might be delivered and ignored if they are delivered
at the same time as a HUP, and there is no data to send to detect
them on the other direction.

Since the connections flags do not have the CO_FL_ERROR flag, the
polling is not disabled on the socket and the pollers immediately
call the conn_fd_handler() again, resulting in CPU spikes for as
long as the timeouts allow them.

Note that this patch alone fixes the issue but a few patches will
follow to strengthen this fragile area.

Big thanks to Bryan Berry who reported the issue with significant
amounts of detailed traces that helped rule out many other initially
suspected causes and to finally reproduce the issue in the lab.
2012-12-07 00:01:33 +01:00
Willy Tarreau
ee2663b1cd BUILD: ssl: NAME_MAX is not portable, use MAXPATHLEN instead
At least Solaris doesn't know about NAME_MAX, so let's use the more portable
MAXPATHLEN instead. This issue was reported by Benjamin Polidore.
2012-12-06 11:36:59 +01:00
Tait Clarridge
7896d5293d MINOR: acl: add fetch for server session rate
Considering there is no option yet for maxconnrate for servers, I wrote
an ACL to check a backend server session rate which we use to send to an
"overflow" backend to prevent latency responses to our clients (very
sensitive latency requirements).
2012-12-06 07:52:09 +01:00
Willy Tarreau
4445502351 BUILD: stdbool is not portable
Benjamin Polidore reported a build issue on Solaris with gcc 4.2.4 where
stdbool is not usable without c99. It only appeared at one location in
dumpstats and is totally useless, let's use the more common and portable
int as everywhere else.
2012-12-05 23:01:12 +01:00
Emeric Brun
af9619da3e MEDIUM: ssl: manage shared cache by blocks for huge sessions.
Sessions using client certs are huge (more than 1 kB) and do not fit
in session cache, or require a huge cache.

In this new implementation sshcachesize set a number of available blocks
instead a number of available sessions.

Each block is large enough (128 bytes) to store a simple session (without
client certs).

Huge sessions will take multiple blocks depending on client certificate size.

Note: some unused code for session sync with remote peers was temporarily
      removed.
2012-12-04 10:56:56 +01:00
Willy Tarreau
dc979f2492 BUG/MINOR: http: don't log a 503 on client errors while waiting for requests
If a client aborts a request with an error (typically a TCP reset), we must
log a 400. Till now we did not set the status nor close the stream interface,
causing the request to attempt to be forwarded and logging a 503.

Should be backported to 1.4 which is affected as well.
2012-12-04 10:52:22 +01:00
Emeric Brun
1eb20efe70 BUG/MEDIUM: ssl: first outgoing connection would fail with {ca,crt}-ignore-err
When using ca_ignore_err/crt_ignore_err, a connection to an untrusted
server raises an error which is ignored. But the next SSL_read() that
encounters EAGAIN raises the error again, breaking the connection.

Subsequent connections don't have this problem because the session has
been stored and is correctly reused without performing a verify again.

The solution consists in correctly flushing the SSL error stack when
ignoring the crt/ca error.
2012-12-03 19:39:40 +01:00
Emeric Brun
78617e51fd BUG/MINOR: ssl: One free session in cache remains unused. 2012-12-03 19:39:40 +01:00
Willy Tarreau
5f51e1ad81 DOC: document the connection error format in logs
This is for failed connection handshakes that are now logged.
2012-12-03 18:40:10 +01:00
Willy Tarreau
20879a0233 MEDIUM: connection: add error reporting for the SSL
Get a bit more info in the logs when client-side SSL handshakes fail.
2012-12-03 17:21:52 +01:00
Willy Tarreau
8e3bf699db MEDIUM: connection: add error reporting for the PROXY protocol header
When the PROXY protocol header is expected and fails, leading to an
abort of the incoming connection, we now emit a log message. If option
dontlognull is set and it was just a port probe, then nothing is logged.
2012-12-03 17:21:51 +01:00
Willy Tarreau
0af2912fd1 MEDIUM: connection: add minimal error reporting in logs for incomplete connections
Since the introduction of SSL, it became quite annoying not to get any useful
info in logs about handshake failures. Let's improve reporting for embryonic
sessions by checking a per-connection error code and reporting it into the logs
if an error happens before the session is completely instanciated.

The "dontlognull" option is supported in that if a connection does not talk
before being aborted, nothing will be emitted.

At the moment, only timeouts are considered for SSL and the PROXY protocol,
but next patches will handle more errors.
2012-12-03 15:38:23 +01:00
Willy Tarreau
14cba4b0b1 MEDIUM: connection: add an error code in connections
This will be needed to improve error reporting, especially for SSL.
2012-12-03 14:22:13 +01:00
Willy Tarreau
d1b3f0498d MINOR: connection: don't remove failed handshake flags
It's annoying that handshake handlers remove themselves from the
connection flags when they fail because there is no way to tell
which one fails. So now we only remove them when they succeed.
2012-12-03 14:22:12 +01:00
Willy Tarreau
5a94037644 BUG/MEDIUM: comp: DEFAULT_MAXZLIBMEM was expressed in bytes and not megabytes
The value is stored in bytes but was not multiplied. It would only affect
packagers.
2012-12-03 14:22:12 +01:00
Dmitry Sivachenko
c9f3b45d7a DOC: add a few precisions on compression
Please consider the attached patch, I hope it will clarify haproxy's behavior a
bit.
2012-12-03 10:50:17 +01:00
Willy Tarreau
8139b9959f MINOR: compression: make the stats a bit more robust
To ensure that we only count when a response was compressed, we also
check for the SN_COMP_READY flag which indicates that the compression
was effectively initialized. Comp_algo alone is meaningless.
2012-11-27 09:34:00 +01:00
Willy Tarreau
9101535038 BUG/MINOR: http: disable compression when message has no body
Compression was not disabled on 1xx, 204, 304 nor HEAD requests. This
is not really a problem, but it reports more compressed responses than
really done.
2012-11-27 09:34:00 +01:00
Willy Tarreau
7d588eed78 BUILD: ssl: OpenSSL 0.9.6 has no renegociation
It did not build anymore on 0.9.6. Not very important but better fix it.
2012-11-26 18:47:31 +01:00
Emeric Brun
786991e8b7 BUG/MEDIUM: ssl: Fix handshake failure on session resumption with client cert.
Openssl session_id_context was not set on cached sessions so handshake returns an error.
2012-11-26 18:43:21 +01:00
Willy Tarreau
78bbeb4a99 BUG/MAJOR: stats: correctly check for a possible divide error when showing compression ratios
Commit 5730c68b changed to display compression ratios based on 2xx
responses, but we should then check that there are such responses
instead of checking for requests. The risk is a divide error if there
are some requests but no 2xx yet (eg: redirect).
2012-11-26 16:44:48 +01:00
Willy Tarreau
0a80a8dbb2 MINOR: http: factor out the content-type checks
Let's only look up the content-type header once. This involves
inverting the condition which is not dramatic.

Also, we now always check the value length before comparing it, and we
always reset the ctx.idx before looking a header up. Otherwise that
could make header lookups depend on their on-wire order. It would be
a minor issue however since at worst it would cause some responses not
to be compressed.
2012-11-26 16:36:00 +01:00
Willy Tarreau
5730c68b46 MINOR: stats: compute the ratio of compressed response based on 2xx responses
Since only responses with status 200 can be compressed, let's only count the
ratio of compressed responses on the basis of the 2xx responses and not all
of them. Note that responses 206 are still included in this count but it gives
a better figure, especially for places where authentication is used and 401 is
common.
2012-11-26 16:19:46 +01:00
William Lallemand
d300261bab MINOR: compression: disable on multipart or status != 200
The compression is disabled when the HTTP status code is not 200, indeed
compression on some HTTP code can create issues (ex: 206, 416).

Multipart message should not be compressed eitherway.
2012-11-26 16:02:58 +01:00
William Lallemand
859550e068 BUG/MINOR: compression: Content-Type is case insensitive
The Content-Type parameter must be case insensitive.
2012-11-26 16:02:58 +01:00
Willy Tarreau
f003d375ec BUG/MINOR: http: don't report client aborts as server errors
If a client aborts with an abortonclose flag, the close is forwarded
to the server and when server response is processed, the analyser thinks
it's the server who has closed first, and logs flags "SD" or "SH" and
counts a server error. In order to avoid this, we now first detect that
the client has closed and log a client abort instead.

This likely is the reason why many people have been observing a small rate
of SD/SH flags without being able to find what the error was.

This fix should probably be backported to 1.4.
2012-11-26 13:50:02 +01:00
Willy Tarreau
fee48ce452 [RELEASE] Released version 1.5-dev14
Released version 1.5-dev14 with the following main changes :
    - DOC: fix minor typos
    - BUG/MEDIUM: compression: does not forward trailers
    - MINOR: buffer_dump with ASCII
    - BUG/MEDIUM: checks: mark the check as stopped after a connect error
    - BUG/MEDIUM: checks: ensure we completely disable polling upon success
    - BUG/MINOR: checks: don't mark the FD as closed before transport close
    - MEDIUM: checks: avoid accumulating TIME_WAITs during checks
    - MINOR: cli: report the msg state in full text in "show sess $PTR"
    - CLEANUP: checks: rename some server check flags
    - MAJOR: checks: rework completely bogus state machine
    - BUG/MINOR: checks: slightly clean the state machine up
    - MEDIUM: checks: avoid waking the application up for pure TCP checks
    - MEDIUM: checks: close the socket as soon as we have a response
    - BUG/MAJOR: checks: close FD on all timeouts
    - MINOR: checks: fix recv polling after connect()
    - MEDIUM: connection: provide a common conn_full_close() function
    - BUG/MEDIUM: checks: prevent TIME_WAITs from appearing also on timeouts
    - BUG/MAJOR: peers: the listener's maxaccept was not set and caused loops
    - MINOR: listeners: make the accept loop more robust when maxaccept==0
    - BUG/MEDIUM: acl: correctly resolve all args, not just the first one
    - BUG/MEDIUM: acl: make prue_acl_expr() correctly free ACL expressions upon exit
    - BUG/MINOR: stats: fix inversion of the report of a check in progress
    - MEDIUM: tcp: add explicit support for delayed ACK in connect()
    - BUG/MEDIUM: connection: always disable polling upon error
    - MINOR: connection: abort earlier when errors are detected
    - BUG/MEDIUM: checks: report handshake failures
    - BUG/MEDIUM: connection: local_send_proxy must wait for connection to establish
    - MINOR: tcp: add support for the "v6only" bind option
    - MINOR: stats: also report the computed compression savings in html stats
    - MINOR: stats: report the total number of compressed responses per front/back
    - MINOR: tcp: add support for the "v4v6" bind option
    - DOC: stats: document the comp_rsp stats column
    - BUILD: buffer: fix another isprint() warning on solaris
    - MINOR: cli: add support for the "show sess all" command
    - BUG/MAJOR: cli: show sess <id> may randomly corrupt the back-ref list
    - MINOR: cli: improve output format for show sess $ptr
2012-11-26 03:11:05 +01:00
Willy Tarreau
909d517e3f MINOR: cli: improve output format for show sess $ptr
This change removes pointers for known types (stream_interface, ...),
adds buffer pointers and sizes, and moves buffer information to their
own line. The output is cleaner with shorter lines and slightly more
lines.
2012-11-26 03:04:41 +01:00
Willy Tarreau
5f9a8779b3 BUG/MAJOR: cli: show sess <id> may randomly corrupt the back-ref list
show sess <id> puts a backref into the session it's dumping. If the output
is interrupted, the backref cannot always be removed because it's only done
in the I/O handler. This can randomly corrupt the backref list when the
session closes, because it passes the pointer to the next session which
itself might be watched.

The case is hard to reproduce (hundreds of attempts) but monitoring systems
might encounter it frequently.

Thus we have to add a release handler which does the cleanup even when the
I/O handler is not called.

This issue should also be present in 1.4 so the patch should be backported.
2012-11-26 02:22:40 +01:00
Willy Tarreau
7615366c70 MINOR: cli: add support for the "show sess all" command
Sometimes when debugging haproxy, it is important to take a full
snapshot of all sessions and their respective states. Till now it
was complicated to do because we had to use scripts and sessions
would vanish between two runs.

Now with this command we have the same output as "show sess $id"
but for all sessions in the table. This is a debugging command only,
it should only be used by developers as it is never guaranteed to
perfectly work !
2012-11-26 01:18:33 +01:00
Willy Tarreau
95898ac211 BUILD: buffer: fix another isprint() warning on solaris
This one came with commit recent be0efd8. Solaris wants ints, not chars.
2012-11-26 00:57:40 +01:00
Willy Tarreau
11d4ec85d1 DOC: stats: document the comp_rsp stats column
This was forgotten in recent patch 5e16cbc3.
2012-11-26 00:49:03 +01:00
Willy Tarreau
77e3af9e6f MINOR: tcp: add support for the "v4v6" bind option
Commit 9b6700f added "v6only". As suggested by Vincent Bernat, it is
sometimes useful to have the opposite option to force binding to the
two protocols when the system is configured to bind to v6 only by
default. This option does exactly this. v6only still has precedence.
2012-11-24 15:07:23 +01:00
Willy Tarreau
5e16cbc3bd MINOR: stats: report the total number of compressed responses per front/back
Depending on the content-types and accept-encoding fields, some responses
might or might not be compressed. Let's have a counter of the number of
compressed responses and report it in the stats to help improve compression
usage.

Some cosmetic issues were fixed in the CSV output too (missing commas at the
end).
2012-11-24 14:54:13 +01:00
Willy Tarreau
f149d8f21e MINOR: stats: also report the computed compression savings in html stats
It's interesting to know the average compression ratio obtained on
frontends and backends without having to compute it by hand, so let's
report it in the HTML stats.
2012-11-24 14:06:49 +01:00
Willy Tarreau
9b6700f673 MINOR: tcp: add support for the "v6only" bind option
This option forces a socket to bind to IPv6 only when it uses the
default address (eg: ":::80").
2012-11-24 12:20:28 +01:00
Willy Tarreau
e3635edc88 BUG/MEDIUM: connection: local_send_proxy must wait for connection to establish
The conn_local_send_proxy() function has to retrieve the local and remote
addresses, but the getpeername() and getsockname() functions may fail until
the connection is established. So now we catch this error and poll for write
when this happens.
2012-11-24 11:23:04 +01:00
Willy Tarreau
6c560da279 BUG/MEDIUM: checks: report handshake failures
Up to now, only data layer failures were reported to the task, but
if a handshake failed from the beginning, the error was not reported
as a failure.
2012-11-24 11:14:45 +01:00
Willy Tarreau
9a92cd5985 MINOR: connection: abort earlier when errors are detected
If an uncaught CO_FL_ERROR flag on a connection is detected, we
immediately go to the wakeup function. This ensures that even if
an error is asynchronously delivered, we don't risk re-enabling
polling or doing unexpected things in the handshake handlers.
2012-11-24 11:12:13 +01:00
Willy Tarreau
36fb02c526 BUG/MEDIUM: connection: always disable polling upon error
Commit 0ffde2cc in 1.5-dev13 tried to always disable polling on file
descriptors when errors were encountered. Unfortunately it did not
always succeed in doing so because it relied on detecting polling
changes to disable it. Let's use a dedicated conn_stop_polling()
function that is inconditionally called upon error instead.

This managed to stop a busy loop observed when a health check makes
use of the send-proxy protocol and fails before the connection can
be established.
2012-11-24 11:09:07 +01:00
Willy Tarreau
f0837b259b MEDIUM: tcp: add explicit support for delayed ACK in connect()
Commit 24db47e0 tried to improve support for delayed ACK upon connect
but it was incomplete, because checks with the proxy protocol would
always enable polling for data receive and there was no way of
distinguishing data polling and delayed ack.

So we add a distinct delack flag to the connect() function so that
the caller decides whether or not to use a delayed ack regardless
of pending data (eg: when send-proxy is in use). Doing so covers all
combinations of { (check with data), (sendproxy), (smart-connect) }.
2012-11-24 10:24:27 +01:00
Willy Tarreau
0eb2bed561 BUG/MINOR: stats: fix inversion of the report of a check in progress
Recent fix for health checks 5a78f36d inverted the condition to display
a "*" in front of the check status on the stats page.
2012-11-24 00:20:24 +01:00
Willy Tarreau
4a6e5c6d69 BUG/MEDIUM: acl: make prue_acl_expr() correctly free ACL expressions upon exit
When leaving, during the deinit() process, prune_acl_expr() is called to
delete all ACL expressions. A bug was introduced with commit 34db1084 that
caused every other expression argument to be skipped, and more annoyingly,
it introduced the risk of scanning past the arg list and crashing or
freezing the old process during a reload.

Credits for finding this issue go to Dmitry Sivachenko who first reported
it, and second did a lot of research to narrow it down to a minimal
configuration.
2012-11-24 00:02:14 +01:00
Willy Tarreau
7d1df41171 BUG/MEDIUM: acl: correctly resolve all args, not just the first one
Since 1.5-dev9, ACLs support multiple args. The changes performed in
acl_find_targets() were bogus as they were not always applied to the
current argument being processed, but sometimes to the first one only.

Fortunately till now, all ACLs which support resolvable arguments have
it in the first place only, so there was no impact.
2012-11-23 23:47:36 +01:00
Dmitry Sivachenko
87c208b2c5 DOC: fix minor typos
I am attaching a small patch which fixes 2 typos.
2012-11-23 20:35:13 +01:00
Willy Tarreau
50de90a228 MINOR: listeners: make the accept loop more robust when maxaccept==0
If some listeners are mistakenly configured with 0 as the maxaccept value,
then we now consider them as limited to one accept() at a time. This will
avoid some issues as fixed by the past commit.
2012-11-23 20:22:10 +01:00
Willy Tarreau
ca57de3e7b BUG/MAJOR: peers: the listener's maxaccept was not set and caused loops
Recent commit 16a214 to move the maxaccept parameter to listeners didn't
set it on the peers' listeners, resulting in the value zero being used
there. This caused a busy loop for each peers section, because no incoming
connection could be accepted.

Thanks to Hervé Commowick for reporting this issue.
2012-11-23 20:21:37 +01:00
Willy Tarreau
cfd97c6f04 BUG/MEDIUM: checks: prevent TIME_WAITs from appearing also on timeouts
We need to disable lingering before closing on timeout too, otherwise
we accumulate TIME_WAITs.
2012-11-23 17:35:59 +01:00