mirror of
http://git.haproxy.org/git/haproxy.git/
synced 2025-02-07 14:01:54 +00:00
1dad3843dc
While experimenting with potentially improved fairness and latency using ticket locks on a Ryzen 16-thread/8-core, a very strange situation happened a lot for some levels of traffic. Around 300k connections per second, no more connections would be accepted on the multi-threaded listener but all others would continue to work fine. All attempts to trace showed that the threads were all in the trylock in the fd cache, or in the spinlock of fd_update_events(), or in the one of fd_may_recv(). But as indicated this was not a deadlock since the process continues to work fine. After quite some investigation it appeared that the issue is caused by a lack of fairness between the fdcache's trylock and these functions' spin locks above. In fact, regardless of the success or failure of the fdcache's attempt at grabbing the lock, the poller was calling fd_update_events() which locks the FD once for something that can be done with a CAS, and then calls fd_may_recv() with another lock for something that most often didn't change. The high contention on these spinlocks leaves no chance to any other thread to grab the lock using trylock(), and once this happens, there is no thread left to process incoming connection events nor to stop polling on the FD, leaving all threads at 100% CPU but partially operational. This patch addresses the issue by using bit-test-and-set instead of the OR in fd_may_recv() / fd_may_send() so that nothing is done if the FD was already configured as expected. It does the same in fd_update_events() using a CAS to check if the FD's events need to be changed at all or not. With this patch applied, it became impossible to reproduce the issue, and now there's no way to saturate all 16 CPUs with the load used for testing, as no more than 1350-1400 were noticed at 300+kcps vs 1600. Ideally this patch should go further and try to remove the remaining incarnations of the fdlock as this seems possible, but it's difficult enough to be done in a distinct patch that will not have to be backported. It is possible that workloads involving a high connection rate may slightly benefit from this patch and observe a slightly lower CPU usage even when the service doesn't misbehave. This patch must be backported to 2.0 and 1.9. |
||
|---|---|---|
| .. | ||
| acl.h | ||
| action.h | ||
| activity.h | ||
| applet.h | ||
| arg.h | ||
| auth.h | ||
| backend.h | ||
| capture.h | ||
| channel.h | ||
| checks.h | ||
| cli.h | ||
| compression.h | ||
| connection.h | ||
| counters.h | ||
| dict.h | ||
| dns.h | ||
| fd.h | ||
| filters.h | ||
| freq_ctr.h | ||
| global.h | ||
| hdr_idx.h | ||
| hlua.h | ||
| http_htx.h | ||
| lb_chash.h | ||
| lb_fas.h | ||
| lb_fwlc.h | ||
| lb_fwrr.h | ||
| lb_map.h | ||
| listener.h | ||
| log.h | ||
| mailers.h | ||
| map.h | ||
| obj_type.h | ||
| pattern.h | ||
| peers.h | ||
| pipe.h | ||
| port_range.h | ||
| proto_http.h | ||
| proto_udp.h | ||
| protocol_buffers.h | ||
| protocol.h | ||
| proxy.h | ||
| queue.h | ||
| sample.h | ||
| server.h | ||
| session.h | ||
| shctx.h | ||
| signal.h | ||
| spoe.h | ||
| ssl_sock.h | ||
| stats.h | ||
| stick_table.h | ||
| stream_interface.h | ||
| stream.h | ||
| task.h | ||
| template.h | ||
| vars.h | ||