The PROXY protocol - 2010/10/29 - Willy TARREAU ----------------------------------------------- Relaying TCP connections through proxies generally involves a loss of the original TCP connection parameters such as source and destination addresses, ports, and so on. Some protocols make it a little bit easier to transfer such information. For SMTP, Postfix authors have proposed the XCLIENT protocol which received broad adoption and is particularly suited to mail exchanges. In HTTP, we have the non-standard but omnipresent X-Forwarded-For header which relays information about the original source address, and the less common X-Original-To which relays information about the destination address. However, both mechanisms require a knowledge of the underlying protocol to be implemented in intermediaries. Then comes a new class of products which we'll call "dumb proxies", not because they don't do anything, but because they're processing protocol-agnostic data. Stunnel is an example of such a "dumb proxy". It talks raw TCP on one side, and raw SSL on the other one, and does that reliably. The problem with such a proxy when it is combined with another one such as haproxy is to adapt it to talk the higher level protocol. A patch is available for Stunnel to make it capable to insert an X-Forwarded-For header in the first HTTP request of each incoming connection. Haproxy is able not to add another one when the connection comes from Stunnel, so that it's possible to hide it from the servers. The typical architecture becomes the following one : +--------+ HTTP :80 +----------+ | client | --------------------------------> | | | | | haproxy, | +--------+ +---------+ | 1 or 2 | / / HTTPS | stunnel | HTTP :81 | listening| <________/ ---------> | (server | ---------> | ports | | mode) | | | +---------+ +----------+ The problem appears when haproxy runs with keep-alive on the side towards the client. The Stunnel patch will only add the X-Forwarded-For header to the first request of each connection and all subsequent requests will not have it. One solution could be to improve the patch to make it support keep-alive and parse all forwarded data, whether they're announced with a Content-Length or with a Transfer-Encoding, taking care of special methods such as HEAD which announce data without transfering them, etc... In fact, it would require implementing a full HTTP stack in Stunnel. It would then become a lot more complex, a lot less reliable and would not anymore be the "dumb proxy" that fits every purposes. In practice, we don't need to add a header for each request because we'll emit the exact same information every time : the information related to the client side connection. We could then cache that information in haproxy and use it for every other request. But that becomes dangerous and is still limited to HTTP only. Another approach would be to prepend each connection with a line reporting the characteristics of the other side's connection. This method is a lot simpler to implement, does not require any protocol-specific knowledge on either side, and completely fits the purpose. That's finally what we did with a small patch to Stunnel and another one to haproxy. We have called this protocol the PROXY protocol. The PROXY protocol's goal is to fill the receiver's internal structures with the information it could have found itself if it performed the accept from the client. Thus right now we're supporting the following : - INET protocol and family (TCP over IPv4 or IPv6) - layer 3 source and destination addresses - layer 4 source and destination ports if any Unlike the XCLIENT protocol, the PROXY protocol was designed with limited extensibility in order to help the receiver parse it very fast, while keeping it human-readable for better debugging possibilities. So it consists in exactly the following block prepended before any data flowing from the dumb proxy to the next hop : - a string identifying the protocol : "PROXY" ( \x50 \x52 \x4F \x58 \x59 ) - exactly one space : " " ( \x20 ) - a string indicating the proxied INET protocol and family. At the moment, only "TCP4" ( \x54 \x43 \x50 \x34 ) for TCP over IPv4, and "TCP6" ( \x54 \x43 \x50 \x36 ) for TCP over IPv6 are allowed. Unsupported or unknown protocols must be reported with the name "UNKNOWN" ( \x55 \x4E \x4B \x4E \x4F \x57 \x4E). The remaining fields of the line are then optional and may be ignored, until the CRLF is found. - exactly one space : " " ( \x20 ) - the layer 3 source address in its canonical format. IPv4 addresses must be indicated as a series of exactly 4 integers in the range [0..255] inclusive written in decimal representation separated by exactly one dot between each other. Heading zeroes are not permitted in front of numbers in order to avoid any possible confusion with octal numbers. IPv6 addresses must be indicated as series of 4 hexadecimal digits (upper or lower case) delimited by colons between each other, with the acceptance of one double colon sequence to replace the largest acceptable range of consecutive zeroes. The total number of decoded bits must exactly be 128. The advertised protocol family dictates what format to use. - exactly one space : " " ( \x20 ) - the layer 3 destination address in its canonical format. It is the same format as the layer 3 source address and matches the same family. - exactly one space : " " ( \x20 ) - the TCP source port represented as a decimal integer in the range [0..65535] inclusive. Heading zeroes are not permitted in front of numbers in order to avoid any possible confusion with octal numbers. - exactly one space : " " ( \x20 ) - the TCP destination port represented as a decimal integer in the range [0..65535] inclusive. Heading zeroes are not permitted in front of numbers in order to avoid any possible confusion with octal numbers. - the CRLF sequence ( \x0D \x0A ) The receiver MUST be configured to only receive this protocol and MUST not try to guess whether the line is prepended or not. That means that the protocol explicitly prevents port sharing between public and private access. Otherwise it would become a big security issue. The receiver should ensure proper access filtering so that only trusted proxies are allowed to use this protocol. The receiver must wait for the CRLF sequence to decode the addresses in order to ensure they are complete. Any sequence which does not exactly match the protocol must be discarded and cause a connection abort. It is recommended to abort the connection as soon as possible to that the emitter notices the anomaly. If the announced transport protocol is "UNKNOWN", then the receiver knows that the emitter talks the correct protocol, any may or may not decide to accept the connection and use the real connection's parameters as if there was no such protocol on the wire. An example of such a line before an HTTP request would look like this (CR marked as "\r" and LF marked as "\n") : PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n GET / HTTP/1.1\r\n Host: 192.168.0.11\r\n \r\n For the emitter, the line is easy to put into the output buffers once the connection is established. For the receiver, once the line is parsed, it's easy to skip it from the input buffers. We have a patch available for recent versions of Stunnel that brings it the ability to be an emitter. The feature is called "send-proxy" there. The code for the receiving side has been merged into haproxy and is enabled using the "accept-proxy" keyword on a "bind" statement. Haproxy will use the transport information from the PROXY protocol for logging, ACLs, etc... everywhere an information about the original connection is required. It is possible that the protocol may slightly evolve to present other information such as the incoming network interface, or the origin addresses in case of network address translation happening before the first proxy, but this is not identified as a requirement right now. --