BUG/MAJOR: connection: always recompute polling status upon I/O

Bryan Berry and Baptiste Assmann both reported some occasional CPU
spinning loops where haproxy was still processing I/O but burning
CPU for apparently uncaught events.

What happens is the following sequence :
  - proxy is in TCP mode
  - a connection from a client initiates a connection to a server
  - the connection to the server does not immediately happen and is
    polled for
  - in the mean time, the client speaks and the stream interface
    calls ->chk_snd() on the peer connection to send the new data
  - chk_snd() calls send_loop() to send the data. This last one
    makes the connection succeed and empties the buffer, so it
    disables polling on the connection and on the FD by creating
    an update entry.
  - before the update is processed, poll() succeeds and reports
    a write event for this fd. The poller does fd_ev_set() on the
    FD to switch it to speculative mode
  - the IO handler is called with a connection which has no write
    flag but an FD which is enabled in speculative mode.
  - the connection does nothing useful.
  - conn_update_polling() at the end of conn_fd_handler() cannot
    disable the FD because there were no changes on this FD.
  - the handler is left with speculative polling still enabled on
    the FD, and will be called over and over until a poll event is
    needed to transfer data.

There is no perfectly elegant solution to this. At least we should
update the flags indicating the current polling status to reflect
what is being done at the FD level. This will allow to detect that
the FD needs to be disabled upon exit.

chk_snd() also needs minor changes to correctly switch to speculative
polling before calling send_loop(), and to reflect this in the connection
flags. This is needed so that no event remains stuck there without any
polling. In fact, chk_snd() and chk_rcv() should perform the same number
of preparations and cleanups as conn_fd_handler().

src/connection.c

@@ -52,6 +52,13 @@ int conn_fd_handler(int fd)
 	flags = conn->flags &= ~(CO_FL_WAIT_DATA|CO_FL_WAIT_ROOM|CO_FL_WAIT_RD|CO_FL_WAIT_WR);
 	flags &= ~CO_FL_ERROR; /* ensure to call the wake handler upon error */
 
+	/* adjust current polling status if it has been updated below us */
+	if (fd_ev_is_set(conn->t.sock.fd, DIR_RD))
+		conn->flags |= CO_FL_CURR_RD_ENA;
+
+	if (fd_ev_is_set(conn->t.sock.fd, DIR_WR))
+		conn->flags |= CO_FL_CURR_WR_ENA;
+
 	if (unlikely(conn->flags & CO_FL_ERROR))
 		goto leave;
 

src/stream_interface.c

@@ -799,15 +799,23 @@ static void stream_int_chk_snd_conn(struct stream_interface *si)
 	     (fdtab[si->conn->t.sock.fd].ev & FD_POLL_OUT)))   /* we'll be called anyway */
 		return;
 
-	if (!(si->conn->flags & CO_FL_HANDSHAKE) && si_conn_send_loop(si->conn) < 0) {
-		/* Write error on the file descriptor. We mark the FD as STERROR so
-		 * that we don't use it anymore and we notify the task.
+	if (!(si->conn->flags & (CO_FL_HANDSHAKE|CO_FL_WAIT_L4_CONN|CO_FL_WAIT_L6_CONN))) {
+		/* Before calling the data-level operations, we have to prepare
+		 * the polling flags to ensure we properly detect changes.
 		 */
-		fdtab[si->conn->t.sock.fd].ev &= ~FD_POLL_STICKY;
-		__conn_data_stop_both(si->conn);
-		si->flags |= SI_FL_ERR;
-		si->conn->flags |= CO_FL_ERROR;
-		goto out_wakeup;
+		if (si->conn->ctrl)
+			fd_want_send(si->conn->t.sock.fd);
+		si->conn->flags &= ~(CO_FL_WAIT_DATA|CO_FL_WAIT_ROOM|CO_FL_WAIT_RD|CO_FL_WAIT_WR);
+		si->conn->flags |= CO_FL_CURR_WR_ENA;
+
+		if (si_conn_send_loop(si->conn) < 0) {
+			/* Write error on the file descriptor */
+			fd_stop_both(si->conn->t.sock.fd);
+			__conn_data_stop_both(si->conn);
+			si->flags |= SI_FL_ERR;
+			si->conn->flags |= CO_FL_ERROR;
+			goto out_wakeup;
+		}
 	}
 
 	/* OK, so now we know that some data might have been sent, and that we may
@@ -819,6 +827,7 @@ static void stream_int_chk_snd_conn(struct stream_interface *si)
 		 * ->o limit was reached. Maybe we just wrote the last
 		 * chunk and need to close.
 		 */
+		__conn_data_stop_send(si->conn);
 		if (((ob->flags & (CF_SHUTW|CF_AUTO_CLOSE|CF_SHUTW_NOW)) ==
 		     (CF_AUTO_CLOSE|CF_SHUTW_NOW)) &&
 		    (si->state == SI_ST_EST)) {