DOC: add more design feedback on the new layering model

Introduce the distinction between structured messages and raw data,
and how to make them coexist in a buffer. This is still a design draft.

doc/internals/notes-layers.txt

@@ -102,3 +102,172 @@ Both operations should return a composite status :
   - number of bytes transfered
   - status flags (shutr, shutw, reset, empty, full, ...)
 
+
+2018-07-23 - Update after merging rxbuf
+---------------------------------------
+
+It becomes visible that the mux will not always be welcome to decode incoming
+data because it will sometimes imply extra memory copies and/or usage for no
+benefit.
+
+Ideally, when when a stream is instanciated based on incoming data, these
+incoming data should be passed and the upper layers called, but it should then
+be up these upper layers to peek more data in certain circumstances. Typically
+if the pending connection data are larger than what is expected to be passed
+above, it means some data may cause head-of-line blocking (HOL) to other
+streams, and needs to be pushed up through the layers to let other streams
+continue to work. Similarly very large H2 data frames after header frames
+should probably not be passed as they may require copies that could be avoided
+if passed later. However if the decoded frame fits into the conn_stream's
+buffer, there is an opportunity to use a single buffer for the conn_stream
+and the channel. The H2 demux could set a blocking flag indicating it's waiting
+for the upper stream to take over demuxing. This flag would be purged once the
+upper stream would start reading, or when extra data come and change the
+conditions.
+
+Forcing structured headers and raw data to coexist within a single buffer is
+quite challenging for many code parts. For example it's perfectly possible to
+see a fragmented buffer containing series of headers, then a small data chunk
+that was received at the same time, then a few other headers added by request
+processing, then another data block received afterwards, then possibly yet
+another header added by option http-send-name-header, and yet another data
+block. This causes some pain for compression which still needs to know where
+compressed and uncompressed data start/stop. It also makes it very difficult
+to account the exact bytes to pass through the various layers.
+
+One solution consists in thinking about buffers using 3 representations :
+
+  - a structured message, which is used for the internal HTTP representation.
+    This message may only be atomically processed. It has no clear byte count,
+    it's a message.
+
+  - a raw stream, consisting in sequences of bytes. That's typically what
+    happens in data sequences or in tunnel.
+
+  - a pipe, which contains data to be forwarded, and that haproxy cannot have
+    access to.
+
+The processing efficiency decreases with the higher complexity above, but the
+capabilities increase. The structured message can contain anything including
+serialized data blocks to be processed or forwarded. The raw stream contains
+data blocks to be processed or forwarded. The pipe only contains data blocks
+to be forwarded. The the latter ones are only an optimization of the former
+ones.
+
+Thus ideally a channel should have access to all such 3 storage areas at once,
+depending on the use case :
+  (1) a structured message,
+  (2) a raw stream,
+  (3) a pipe
+
+Right now a channel only has (2) and (3) but after the native HTTP rework, it
+will only have (1) and (3). Placing a raw stream exclusively in (1) comes with
+some performance drawbacks which are not easily recovered, and with some quite
+difficult management still involving the reserve to ensure that a data block
+doesn't prevent headers from being appended. But during header processing, the
+payload may be necessary so we cannot decide to drop this option.
+
+A long-term approach would consist in ensuring that a single channel may have
+access to all 3 representations at once, and to enumerate priority rules to
+define how they interact together. That's exactly what is currently being done
+with the pipe and the raw buffer right now. Doing so would also save the need
+for storing payload in the structured message and void the requirement for the
+reserve. But it would cost more memory to process POST data and server
+responses. Thus an intermediary step consists in keeping this model in mind but
+not implementing everything yet.
+
+Short term proposal : a channel has access to a buffer and a pipe. A non-empty
+buffer is either in structured message format OR raw stream format. Only the
+channel knows. However a structured buffer MAY contain raw data in a properly
+formated way (using the envelope defined by the structured message format).
+
+By default, when a demux writes to a CS rxbuf, it will try to use the lowest
+possible level for what is being done (i.e. splice if possible, otherwise raw
+stream, otherwise structured message). If the buffer already contains a
+structured message, then this format is exclusive. From this point the MUX has
+two options : either encode the incoming data to match the structured message
+format, or refrain from receiving into the CS's rxbuf and wait until the upper
+layer request those data.
+
+This opens a simplified option which could be suited even for the long term :
+  - cs_recv() will take one or two flags to indicate if a buffer already
+    contains a structured message or not ; the upper layer knows it.
+
+  - cs_recv() will take two flags to indicate what the upper layer is willing
+    to take :
+      - structured message only
+      - raw stream only
+      - any of them
+
+    From this point the mux can decide to either pass anything or refrain from
+    doing so.
+
+  - the demux stores the knowledge it has from the contents into some CS flags
+    to indicate whether or not some structured message are still available, and
+    whether or not some raw data are still available. Thus the caller knows
+    whether or not extra data are available.
+
+  - when the demux works on its own, it refrains from passing structured data
+    to a non-empty buffer, unless these data are causing trouble to other
+    streams (HOL).
+
+  - when a demux has to encapsulate raw data into a structured message, it will
+    always have to respect a configured reserve so that extra header processing
+    can be done on the structured message inside the buffer, regardless of the
+    supposed available room. In addition, the upper layer may indicate using an
+    extra recv() flag whether it wants the demux to defragment serialized data
+    (for example by moving trailing headers apart) or if it's not necessary.
+    This flag will be set by the stream interface if compression is required or
+    if the http-buffer-request option is set for example. Probably that using
+    to_forward==0 is a stronger indication that the reserve must be respected.
+
+  - cs_recv() and cs_send() when fed with a message, should not return byte
+    counts but message counts (i.e. 0 or 1). This implies that a single call to
+    either of these functions cannot mix raw data and structured messages at
+    the same time.
+
+At this point it looks like the conn_stream will have some encapsulation work
+to do for the payload if it needs to be encapsulated into a message. This
+further magnifies the importance of *not* decoding DATA frames into the CS's
+rxbuf until really needed.
+
+The CS will probably need to hold indication of what is available at the mux
+level, not only in the CS. Eg: we know that payload is still available.
+
+Using these elements, it should be possible to ensure that full header frames
+may be received without enforcing any reserve, that too large frames that do
+not fit will be detected because they return 0 message and indicate that such
+a message is still pending, and that data availability is correctly detected
+(later we may expect that the stream-interface allocates a larger or second
+buffer to place the payload).
+
+Regarding the ability for the channel to forward data, it looks like having a
+new function "cs_xfer(src_cs, dst_cs, count)" could be very productive in
+optimizing the forwarding to make use of splicing when available. It is not yet
+totally clear whether it will split into "cs_xfer_in(src_cs, pipe, count)"
+followed by "cs_xfer_out(dst_cs, pipe, count)" or anything different, and it
+still needs to be studied. The general idea seems to be that the receiver might
+have to call the sender directly once they agree on how to transfer data (pipe
+or buffer). If the transfer is incomplete, the cs_xfer() return value and/or
+flags will indicate the current situation (src empty, dst full, etc) so that
+the caller may register for notifications on the appropriate event and wait to
+be called again to continue.
+
+Short term implementation :
+  1) add new CS flags to qualify what the buffer contains and what we expect
+     to read into it;
+
+  2) set these flags to pretend we have a structured message when receiving
+     headers (after all, H1 is an atomic header as well) and see what it
+     implies for the code; for H1 it's unclear whether it makes sense to try
+     to set it without the H1 mux.
+
+  3) use these flags to refrain from sending DATA frames after HEADERS frames
+     in H2.
+
+  4) flush the flags at the stream interface layer when performing a cs_send().
+
+  5) use the flags to enforce receipt of data only when necessary
+
+We should be able to end up with sequencial receipt in H2 modelling what is
+needed for other protocols without interfering with the native H1 devs.