haproxy/doc/internals/notes-polling.txt

2019-09-03

u8 fd.state;
u8 fd.ev;


ev = one of :
	#define FD_POLL_IN	0x01
	#define FD_POLL_PRI	0x02
	#define FD_POLL_OUT	0x04
	#define FD_POLL_ERR	0x08
	#define FD_POLL_HUP	0x10

Could we instead have :

  FD_WAIT_IN  0x01
  FD_WAIT_OUT 0x02
  FD_WAIT_PRI 0x04
  FD_SEEN_HUP 0x08
  FD_SEEN_HUP 0x10
  FD_WAIT_CON 0x20  <<= shouldn't this be in the connection itself in fact ?

=> not needed, covered by the state instead.

What is missing though is :
  - FD_DATA_PENDING  -- overlaps with READY_R, OK if passed by pollers only
  - FD_EOI_PENDING
  - FD_ERR_PENDING
  - FD_EOI
  - FD_SHW
  - FD_ERR

fd_update_events() could do that :

    if ((fd_data_pending|fd_eoi_pending|fd_err_pending) && !(fd_err|fd_eoi))
        may_recv()

    if (fd_send_ok && !(fd_err|fd_shw))
        may_send()

    if (fd_err)
        wake()

the poller could do that :
  HUP+OUT => always indicates a failed connect(), it should not lack ERR. Is this err_pending ?

   ERR  HUP  OUT   IN
    0    0    0    0  => nothing
    0    0    0    1  => FD_DATA_PENDING
    0    0    1    0  => FD_SEND_OK
    0    0    1    1  => FD_DATA_PENDING|FD_SEND_OK
    0    1    0    0  => FD_EOI (|FD_SHW)
    0    1    0    1  => FD_DATA_PENDING|FD_EOI_PENDING (|FD_SHW)
    0    1    1    0  => FD_EOI |FD_ERR (|FD_SHW)
    0    1    1    1  => FD_EOI_PENDING (|FD_ERR_PENDING) |FD_DATA_PENDING (|FD_SHW)
    1    X    0    0  => FD_ERR | FD_EOI (|FD_SHW)
    1    X    X    1  => FD_ERR_PENDING | FD_EOI_PENDING | FD_DATA_PENDING (|FD_SHW)
    1    X    1    0  => FD_ERR | FD_EOI (|FD_SHW)

   OUT+HUP,OUT+HUP+ERR => FD_ERR

This reorders to:

    IN  ERR  HUP  OUT 
    0    0    0    0   => nothing
    0    0    0    1   => FD_SEND_OK
    0    0    1    0   => FD_EOI (|FD_SHW)

    0    X    1    1   => FD_ERR | FD_EOI (|FD_SHW)
    0    1    X    0   => FD_ERR | FD_EOI (|FD_SHW)
    0    1    X    1   => FD_ERR | FD_EOI (|FD_SHW)

    1    0    0    0   => FD_DATA_PENDING
    1    0    0    1   => FD_DATA_PENDING|FD_SEND_OK
    1    0    1    0   => FD_DATA_PENDING|FD_EOI_PENDING (|FD_SHW)
    1    0    1    1   => FD_EOI_PENDING (|FD_ERR_PENDING) |FD_DATA_PENDING (|FD_SHW)
    1    1    X    X   => FD_ERR_PENDING | FD_EOI_PENDING | FD_DATA_PENDING (|FD_SHW)

Regarding "|SHW", it's normally useless since it will already have been done,
except on connect() error where this indicates there's no need for SHW.

FD_EOI and FD_SHW could be part of the state (FD_EV_SHUT_R, FD_EV_SHUT_W).
Then all states having these bit and another one would be transient and need
to resync. We could then have "fd_shut_recv" and "fd_shut_send" to turn these
states.

The FD's ev then only needs to update EOI_PENDING, ERR_PENDING, ERR, DATA_PENDING.
With this said, these are not exactly polling states either, as err/eoi/shw are
orthogonal to the other states and are required to update them so that the polling
state really is DISABLED in the end. So we need more of an operational status for
the FD containing EOI_PENDING, EOI, ERR_PENDING, ERR, SHW, CLO?. These could be
classified in 3 categories: read:(OPEN, EOI_PENDING, EOI); write:(OPEN,SHW),
ctrl:(OPEN,ERR_PENDING,ERR,CLO). That would be 2 bits for R, 1 for W, 2 for ctrl
or total 5 vs 6 for individual ones, but would be harder to manipulate.

Proposal:
  - rename fdtab[].state to "polling_state"
  - rename fdtab[].ev    to "status"

Note: POLLHUP is also reported is a listen() socket has gone in shutdown()
TEMPORARILY! Thus we may not always consider this as a final error.


Work hypothesis:

SHUT RDY ACT
  0   0   0   => disabled
  0   0   1   => active
  0   1   0   => stopped
  0   1   1   => ready
  1   0   0   => final shut
  1   0   1   => shut pending without data
  1   1   0   => shut pending, stopped
  1   1   1   => shut pending

PB: we can land into final shut if one thread disables the FD while another
    one that was waiting on it reports it as shut. Theorically it should be
    implicitly ready though, since reported. But if no data is reported, it
    will be reportedly shut only. And no event will be reported then. This
    might still make sense since it's not active, thus we don't want events.
    But it will not be enabled later either in this case so the shut really
    risks not to be properly reported. The issue is that there's no difference
    between a shut coming from the bottom and a shut coming from the top, and
    we need an event to report activity here. Or we may consider that a poller
    never leaves a final shut by itself (100) and always reports it as
    shut+stop (thus ready) if it was not active. Alternately, if active is
    disabled, shut should possibly be ignored, then a poller cannot report
    shut. But shut+stopped seems the most suitable as it corresponds to
    disabled->stopped transition.

Now let's add ERR. ERR necessarily implies SHUT as there doesn't seem to be a
valid case of ERR pending without shut pending.

ERR SHUT RDY ACT
 0    0   0   0   => disabled
 0    0   0   1   => active
 0    0   1   0   => stopped
 0    0   1   1   => ready

 0    1   0   0   => final shut, no error
 0    1   0   1   => shut pending without data
 0    1   1   0   => shut pending, stopped
 0    1   1   1   => shut pending

 1    0   X   X   => invalid

 1    1   0   0   => final shut, error encountered
 1    1   0   1   => error pending without data
 1    1   1   0   => error pending after data, stopped
 1    1   1   1   => error pending

So the algorithm for the poller is:
  - if (shutdown_pending or error) reported and ACT==0,
    report SHUT|RDY or SHUT|ERR|RDY

For read handlers :
  - if (!(flags & (RDY|ACT)))
     return
  - if (ready)
     try_to_read
  - if (err)
     report error
  - if (shut)
     read0

For write handlers:
  - if (!(flags & (RDY|ACT)))
     return
  - if (err||shut)
     report error
  - if (ready)
     try_to_write

For listeners:
  - if (!(flags & (RDY|ACT)))
     return
  - if (err||shut)
     pause
  - if (ready)
     try_to_accept

Kqueue reports events differently, it says EV_EOF() on READ or WRITE, that
we currently map to FD_POLL_HUP and FD_POLL_ERR. Thus kqueue reports only
POLLRDHUP and not POLLHUP, so for now a direct mapping of POLLHUP to
FD_POLL_HUP does NOT imply write closed with kqueue while it does for others.

Other approach, use the {RD,WR}_{ERR,SHUT,RDY} flags to build a composite
status in each poller and pass this to fd_update_events(). We normally
have enough to be precise, and this latter will rework the events.

FIXME: Normally on KQUEUE we're supposed to look at kev[].fflags to get the error
on EV_EOF() on read or write.