RepoMirrors/haproxy
1
0
Fork 0
mirror of http://git.haproxy.org/git/haproxy.git/ synced 2025-02-17 02:56:51 +00:00
Code Issues Packages Projects Releases Wiki Activity

BUG/MEDIUM: peers: limit reconnect attempts of the old process on reload

Browse Source 
When peers are configured and HAProxy is reloaded or restarted, a
synchronization is performed between the old process and the new one. To do
so, the old process connects on the new one. If the synchronization fails,
it retries. However, there is no delay and reconnect attempts are not
bounded. Thus, it may loop for a while, consuming all the CPU. Of course, it
is unexpected, but it is possible. For instance, if the local peer is
misconfigured, an infinite loop can be observed if the connection succeeds
but not the synchronization. This prevents the old process to exit, except
if "hard-stop-after" option is set.

To fix the bug, the reconnect is delayed. The local peer already has a
expiration date to delay the reconnects. But it was not used on stopping
mode. So we use it not. Thanks to the previous fix, the reconnect timeout is
shorter in this case (500ms against 5s on running mode). In addition, we
also use the peers resync expiration date to not infinitely retries. It is
accurate because the new process, on its side, use this timeout to switch
from a local resync to a remote resync.

This patch depends on "MINOR: peers: Use a dedicated reconnect timeout when
stopping the local peer". It fixes the issue #1799. It should be backported
as far as 2.0.
This commit is contained in:
Christopher Faulet 2022-07-26 19:19:18 +02:00
parent ab4b094055
commit 160fff665e
1 changed files with 23 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

34
src/peers.c
View File

@ -3421,6 +3421,10 @@ struct task *process_peer_sync(struct task * task, void *context, unsigned int s
peer_session_forceshutdown(ps);
}
}
/* Set resync timeout for the local peer and request a immediate reconnect */
peers->resync_timeout = tick_add(now_ms, MS_TO_TICKS(PEER_RESYNC_TIMEOUT));
peers->local->reconnect = now_ms;
}
}
@ -3436,18 +3440,26 @@ struct task *process_peer_sync(struct task * task, void *context, unsigned int s
}
else if (!ps->appctx) {
/* If there's no active peer connection */
if (ps->statuscode == 0 ||
ps->statuscode == PEER_SESS_SC_SUCCESSCODE ||
ps->statuscode == PEER_SESS_SC_CONNECTEDCODE ||
ps->statuscode == PEER_SESS_SC_TRYAGAIN) {
/* connection never tried
* or previous peer connection was successfully established
* or previous tcp connect succeeded but init state incomplete
* or during previous connect, peer replies a try again statuscode */
if (!tick_is_expired(peers->resync_timeout, now_ms) &&
(ps->statuscode == 0 ||
ps->statuscode == PEER_SESS_SC_SUCCESSCODE ||
ps->statuscode == PEER_SESS_SC_CONNECTEDCODE ||
ps->statuscode == PEER_SESS_SC_TRYAGAIN)) {
/* The resync timeout is not expired and
* connection never tried
* or previous peer connection was successfully established
* or previous tcp connect succeeded but init state incomplete
* or during previous connect, peer replies a try again statuscode */
/* connect to the local peer if we must push a local sync */
if (peers->flags & PEERS_F_DONOTSTOP) {
peer_session_create(peers, ps);
if (!tick_is_expired(ps->reconnect, now_ms)) {
/* reconnection timer is not expired. reschedule task for reconnect */
task->expire = tick_first(task->expire, ps->reconnect);
}
else {
/* connect to the local peer if we must push a local sync */
if (peers->flags & PEERS_F_DONOTSTOP) {
peer_session_create(peers, ps);
}
}
}
else {