haproxy/doc/proxy-protocol.txt

                               The PROXY protocol
                                  Willy Tarreau
                                    2011/03/20

Abstract

   The PROXY protocol provides a convenient way to safely transport connection
   information such as a client's address across multiple layers of NAT or TCP
   proxies. It is designed to require little changes to existing components and
   to limit the performance impact caused by the processing of the transported
   information.


Revision history

   2010/10/29 - first version
   2011/03/20 - update: implementation and security considerations


1. Background

Relaying TCP connections through proxies generally involves a loss of the
original TCP connection parameters such as source and destination addresses,
ports, and so on. Some protocols make it a little bit easier to transfer such
information. For SMTP, Postfix authors have proposed the XCLIENT protocol which
received broad adoption and is particularly suited to mail exchanges. In HTTP,
we have the non-standard but omnipresent X-Forwarded-For header which relays
information about the original source address, and the less common
X-Original-To which relays information about the destination address.

However, both mechanisms require a knowledge of the underlying protocol to be
implemented in intermediaries.

Then comes a new class of products which we'll call "dumb proxies", not because
they don't do anything, but because they're processing protocol-agnostic data.
Stunnel is an example of such a "dumb proxy". It talks raw TCP on one side, and
raw SSL on the other one, and does that reliably.

The problem with such a proxy when it is combined with another one such as
haproxy is to adapt it to talk the higher level protocol. A patch is available
for Stunnel to make it capable to insert an X-Forwarded-For header in the first
HTTP request of each incoming connection. Haproxy is able not to add another
one when the connection comes from Stunnel, so that it's possible to hide it
from the servers.

The typical architecture becomes the following one :


      +--------+      HTTP                      :80 +----------+
      | client |  --------------------------------> |          |
      |        |                                    | haproxy, |
      +--------+             +---------+            |  1 or 2  |
     /        /     HTTPS    | stunnel |  HTTP  :81 | listening|
    <________/    ---------> | (server | ---------> |  ports   |
                             |  mode)  |            |          |
                             +---------+            +----------+


The problem appears when haproxy runs with keep-alive on the side towards the
client. The Stunnel patch will only add the X-Forwarded-For header to the first
request of each connection and all subsequent requests will not have it. One
solution could be to improve the patch to make it support keep-alive and parse
all forwarded data, whether they're announced with a Content-Length or with a
Transfer-Encoding, taking care of special methods such as HEAD which announce
data without transfering them, etc... In fact, it would require implementing a
full HTTP stack in Stunnel. It would then become a lot more complex, a lot less
reliable and would not anymore be the "dumb proxy" that fits every purposes.

In practice, we don't need to add a header for each request because we'll emit
the exact same information every time : the information related to the client
side connection. We could then cache that information in haproxy and use it for
every other request. But that becomes dangerous and is still limited to HTTP
only.

Another approach would be to prepend each connection with a line reporting the
characteristics of the other side's connection. This method is a lot simpler to
implement, does not require any protocol-specific knowledge on either side, and
completely fits the purpose. That's finally what we did with a small patch to
Stunnel and another one to haproxy. We have called this protocol the PROXY
protocol.


2. The PROXY protocol

The PROXY protocol's goal is to fill the receiver's internal structures with
the information it could have found itself if it performed the accept from the
client. Thus right now we're supporting the following :
  - INET protocol and family (TCP over IPv4 or IPv6)
  - layer 3 source and destination addresses
  - layer 4 source and destination ports if any

Unlike the XCLIENT protocol, the PROXY protocol was designed with limited
extensibility in order to help the receiver parse it very fast, while keeping
it human-readable for better debugging possibilities. So it consists in exactly
the following block prepended before any data flowing from the dumb proxy to
the next hop :

  - a string identifying the protocol : "PROXY" ( \x50 \x52 \x4F \x58 \x59 )

  - exactly one space : " " ( \x20 )

  - a string indicating the proxied INET protocol and family. At the moment,
    only "TCP4" ( \x54 \x43 \x50 \x34 ) for TCP over IPv4, and "TCP6"
    ( \x54 \x43 \x50 \x36 ) for TCP over IPv6 are allowed. Unsupported or
    unknown protocols must be reported with the name "UNKNOWN" ( \x55 \x4E \x4B
    \x4E \x4F \x57 \x4E). The remaining fields of the line are then optional
    and may be ignored, until the CRLF is found.

  - exactly one space : " " ( \x20 )

  - the layer 3 source address in its canonical format. IPv4 addresses must be
    indicated as a series of exactly 4 integers in the range [0..255] inclusive
    written in decimal representation separated by exactly one dot between each
    other. Heading zeroes are not permitted in front of numbers in order to
    avoid any possible confusion with octal numbers. IPv6 addresses must be
    indicated as series of 4 hexadecimal digits (upper or lower case) delimited
    by colons between each other, with the acceptance of one double colon
    sequence to replace the largest acceptable range of consecutive zeroes. The
    total number of decoded bits must exactly be 128. The advertised protocol
    family dictates what format to use.

  - exactly one space : " " ( \x20 )

  - the layer 3 destination address in its canonical format. It is the same
    format as the layer 3 source address and matches the same family.

  - exactly one space : " " ( \x20 )

  - the TCP source port represented as a decimal integer in the range
    [0..65535] inclusive. Heading zeroes are not permitted in front of numbers
    in order to avoid any possible confusion with octal numbers.

  - exactly one space : " " ( \x20 )

  - the TCP destination port represented as a decimal integer in the range
    [0..65535] inclusive. Heading zeroes are not permitted in front of numbers
    in order to avoid any possible confusion with octal numbers.

  - the CRLF sequence ( \x0D \x0A )

The receiver MUST be configured to only receive this protocol and MUST not try
to guess whether the line is prepended or not. That means that the protocol
explicitly prevents port sharing between public and private access. Otherwise
it would become a big security issue. The receiver should ensure proper access
filtering so that only trusted proxies are allowed to use this protocol. The
receiver must wait for the CRLF sequence to decode the addresses in order to
ensure they are complete. Any sequence which does not exactly match the
protocol must be discarded and cause a connection abort. It is recommended
to abort the connection as soon as possible to that the emitter notices the
anomaly.

If the announced transport protocol is "UNKNOWN", then the receiver knows that
the emitter talks the correct protocol, and may or may not decide to accept the
connection and use the real connection's parameters as if there was no such
protocol on the wire.

An example of such a line before an HTTP request would look like this (CR
marked as "\r" and LF marked as "\n") :

    PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n
    GET / HTTP/1.1\r\n
    Host: 192.168.0.11\r\n
    \r\n

For the emitter, the line is easy to put into the output buffers once the
connection is established. For the receiver, once the line is parsed, it's
easy to skip it from the input buffers.


3. Implementations

Haproxy 1.5 implements the PROXY protocol on both sides :
  - the listening sockets accept the protocol when the "accept-proxy" setting
    is passed to the "bind" keyword. Connections accepted on such listeners
    will behave just as if the source really was the one advertised in the
    protocol. This is true for logging, ACLs, content filtering, transparent
    proxying, etc...

  - the protocol may be used to connect to servers if the "send-proxy" setting
    is present on the "server" line. It is enabled on a per-server basis, so it
    is possible to have it enabled for remote servers only and still have local
    ones behave differently. If the incoming connection was accepted with the
    "accept-proxy", then the relayed information is the one advertised in this
    connection's PROXY line.

We have a patch available for recent versions of Stunnel that brings it the
ability to be an emitter. The feature is called "sendproxy" there.

The protocol is so simple that it is expected that other implementations will
appear, especially in environments such as SMTP, IMAP, FTP, RDP where the
client's address is an important piece of information for the server and some
intermediaries.

Proxy developers are encouraged to implement this protocol, because it will
make their products much more transparent in complex infrastructures, and will
get rid of a number of issues related to logging and access control.


4. Security considerations

The protocol was designed so as to be distinguishable from HTTP. It will not
parse as a valid HTTP request and an HTTP request will not parse as a valid
proxy request. That makes it easier to enfore its use certain connections.
Implementers should be very careful about not trying to automatically detect
whether they have to decode the line or not, but rather to only rely on a
configuration parameter. Indeed, if the opportunity is left to a normal client
to use the protocol, he will be able to hide his activities or make them appear
as coming from someone else. However, accepting the line only from a number of
known sources should be safe.


5. Future developments

It is possible that the protocol may slightly evolve to present other
information such as the incoming network interface, or the origin addresses in
case of network address translation happening before the first proxy, but this
is not identified as a requirement right now. Suggestions on improvements are
welcome.


6. Contacts

Please use w@1wt.eu to send any comments to the author.