160 lines
8.2 KiB
Plaintext
160 lines
8.2 KiB
Plaintext
The PROXY protocol - 2010/10/29 - Willy TARREAU
|
|
-----------------------------------------------
|
|
|
|
Relaying TCP connections through proxies generally involves a loss of the
|
|
original TCP connection parameters such as source and destination addresses,
|
|
ports, and so on. Some protocols make it a little bit easier to transfer such
|
|
information. For SMTP, Postfix authors have proposed the XCLIENT protocol which
|
|
received broad adoption and is particularly suited to mail exchanges. In HTTP,
|
|
we have the non-standard but omnipresent X-Forwarded-For header which relays
|
|
information about the original source address, and the less common
|
|
X-Original-To which relays information about the destination address.
|
|
|
|
However, both mechanisms require a knowledge of the underlying protocol to be
|
|
implemented in intermediaries.
|
|
|
|
Then comes a new class of products which we'll call "dumb proxies", not because
|
|
they don't do anything, but because they're processing protocol-agnostic data.
|
|
Stunnel is an example of such a "dumb proxy". It talks raw TCP on one side, and
|
|
raw SSL on the other one, and does that reliably.
|
|
|
|
The problem with such a proxy when it is combined with another one such as
|
|
haproxy is to adapt it to talk the higher level protocol. A patch is available
|
|
for Stunnel to make it capable to insert an X-Forwarded-For header in the first
|
|
HTTP request of each incoming connection. Haproxy is able not to add another
|
|
one when the connection comes from Stunnel, so that it's possible to hide it
|
|
from the servers.
|
|
|
|
The typical architecture becomes the following one :
|
|
|
|
|
|
+--------+ HTTP :80 +----------+
|
|
| client | --------------------------------> | |
|
|
| | | haproxy, |
|
|
+--------+ +---------+ | 1 or 2 |
|
|
/ / HTTPS | stunnel | HTTP :81 | listening|
|
|
<________/ ---------> | (server | ---------> | ports |
|
|
| mode) | | |
|
|
+---------+ +----------+
|
|
|
|
|
|
The problem appears when haproxy runs with keep-alive on the side towards the
|
|
client. The Stunnel patch will only add the X-Forwarded-For header to the first
|
|
request of each connection and all subsequent requests will not have it. One
|
|
solution could be to improve the patch to make it support keep-alive and parse
|
|
all forwarded data, whether they're announced with a Content-Length or with a
|
|
Transfer-Encoding, taking care of special methods such as HEAD which announce
|
|
data without transfering them, etc... In fact, it would require implementing a
|
|
full HTTP stack in Stunnel. It would then become a lot more complex, a lot less
|
|
reliable and would not anymore be the "dumb proxy" that fits every purposes.
|
|
|
|
In practice, we don't need to add a header for each request because we'll emit
|
|
the exact same information every time : the information related to the client
|
|
side connection. We could then cache that information in haproxy and use it for
|
|
every other request. But that becomes dangerous and is still limited to HTTP
|
|
only.
|
|
|
|
Another approach would be to prepend each connection with a line reporting the
|
|
characteristics of the other side's connection. This method is a lot simpler to
|
|
implement, does not require any protocol-specific knowledge on either side, and
|
|
completely fits the purpose. That's finally what we did with a small patch to
|
|
Stunnel and another one to haproxy. We have called this protocol the PROXY
|
|
protocol.
|
|
|
|
The PROXY protocol's goal is to fill the receiver's internal structures with
|
|
the information it could have found itself if it performed the accept from the
|
|
client. Thus right now we're supporting the following :
|
|
- INET protocol and family (TCP over IPv4 or IPv6)
|
|
- layer 3 source and destination addresses
|
|
- layer 4 source and destination ports if any
|
|
|
|
Unlike the XCLIENT protocol, the PROXY protocol was designed with limited
|
|
extensibility in order to help the receiver parse it very fast, while keeping
|
|
it human-readable for better debugging possibilities. So it consists in exactly
|
|
the following block prepended before any data flowing from the dumb proxy to
|
|
the next hop :
|
|
|
|
- a string identifying the protocol : "PROXY" ( \x50 \x52 \x4F \x58 \x59 )
|
|
|
|
- exactly one space : " " ( \x20 )
|
|
|
|
- a string indicating the proxied INET protocol and family. At the moment,
|
|
only "TCP4" ( \x54 \x43 \x50 \x34 ) for TCP over IPv4, and "TCP6"
|
|
( \x54 \x43 \x50 \x36 ) for TCP over IPv6 are allowed. Unsupported or
|
|
unknown protocols must be reported with the name "UNKNOWN" ( \x55 \x4E \x4B
|
|
\x4E \x4F \x57 \x4E). The remaining fields of the line are then optional
|
|
and may be ignored, until the CRLF is found.
|
|
|
|
- exactly one space : " " ( \x20 )
|
|
|
|
- the layer 3 source address in its canonical format. IPv4 addresses must be
|
|
indicated as a series of exactly 4 integers in the range [0..255] inclusive
|
|
written in decimal representation separated by exactly one dot between each
|
|
other. Heading zeroes are not permitted in front of numbers in order to
|
|
avoid any possible confusion with octal numbers. IPv6 addresses must be
|
|
indicated as series of 4 hexadecimal digits (upper or lower case) delimited
|
|
by colons between each other, with the acceptance of one double colon
|
|
sequence to replace the largest acceptable range of consecutive zeroes. The
|
|
total number of decoded bits must exactly be 128. The advertised protocol
|
|
family dictates what format to use.
|
|
|
|
- exactly one space : " " ( \x20 )
|
|
|
|
- the layer 3 destination address in its canonical format. It is the same
|
|
format as the layer 3 source address and matches the same family.
|
|
|
|
- exactly one space : " " ( \x20 )
|
|
|
|
- the TCP source port represented as a decimal integer in the range
|
|
[0..65535] inclusive. Heading zeroes are not permitted in front of numbers
|
|
in order to avoid any possible confusion with octal numbers.
|
|
|
|
- exactly one space : " " ( \x20 )
|
|
|
|
- the TCP destination port represented as a decimal integer in the range
|
|
[0..65535] inclusive. Heading zeroes are not permitted in front of numbers
|
|
in order to avoid any possible confusion with octal numbers.
|
|
|
|
- the CRLF sequence ( \x0D \x0A )
|
|
|
|
The receiver MUST be configured to only receive this protocol and MUST not try
|
|
to guess whether the line is prepended or not. That means that the protocol
|
|
explicitly prevents port sharing between public and private access. Otherwise
|
|
it would become a big security issue. The receiver should ensure proper access
|
|
filtering so that only trusted proxies are allowed to use this protocol. The
|
|
receiver must wait for the CRLF sequence to decode the addresses in order to
|
|
ensure they are complete. Any sequence which does not exactly match the
|
|
protocol must be discarded and cause a connection abort. It is recommended
|
|
to abort the connection as soon as possible to that the emitter notices the
|
|
anomaly.
|
|
|
|
If the announced transport protocol is "UNKNOWN", then the receiver knows that
|
|
the emitter talks the correct protocol, any may or may not decide to accept the
|
|
connection and use the real connection's parameters as if there was no such
|
|
protocol on the wire.
|
|
|
|
An example of such a line before an HTTP request would look like this (CR
|
|
marked as "\r" and LF marked as "\n") :
|
|
|
|
PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n
|
|
GET / HTTP/1.1\r\n
|
|
Host: 192.168.0.11\r\n
|
|
\r\n
|
|
|
|
For the emitter, the line is easy to put into the output buffers once the
|
|
connection is established. For the receiver, once the line is parsed, it's
|
|
easy to skip it from the input buffers.
|
|
|
|
We have a patch available for recent versions of Stunnel that brings it the
|
|
ability to be an emitter. The feature is called "sendproxy" there. The code
|
|
for the receiving side has been merged into haproxy and is enabled using the
|
|
"accept-proxy" keyword on a "bind" statement. Haproxy will use the transport
|
|
information from the PROXY protocol for logging, ACLs, etc... everywhere an
|
|
information about the original connection is required.
|
|
|
|
It is possible that the protocol may slightly evolve to present other
|
|
information such as the incoming network interface, or the origin addresses in
|
|
case of network address translation happening before the first proxy, but this
|
|
is not identified as a requirement right now.
|
|
--
|