62e8aaa1bd
In github issue #822, user @ngaugler reported some performance problems when dealing with many concurrent SSL connections on restarts, after migrating from 1.6 to 2.2, indicating a long time required to re-establish connections. The Run_queue metric in the traces showed an abnormally high number of tasks in the run queue, likely indicating we were accepting faster than we could process. And this is indeed one of the differences between 1.6 and 2.2, the accept I/O loop and the TLS handshakes are totally independent, so much that they can even run on different threads. In 1.6 the SSL handshake was handled almost immediately after the accept(), so this was limiting the input rate. With large maxconn values, as long as there are incoming connections, new I/Os are scheduled and many of them pass before the handshake, being tagged for low latency processing. The result is that handshakes get postponed, and are further postponed as new connections are accepted. When they are finally able to be processed, some of them fail as the client is gone, and the client had already queued new ones. This causes an excess number of apparent connections and total number of handshakes to be processed, just because we were accepting connections on a temporarily saturated machine. The solution is to temporarily pause new incoming connections when the load already indicates that more tasks are already queued than will be handled in a poll loop. The difficulty with this usually is to be able to come back to re-enable the operation, but given that the metric is the run queue, we just have to queue the global_listener_queue task so that it gets picked by any thread once the run queues get flushed. Before this patch, injecting with SSL reneg with 10000 concurrent connections resulted in 350k tasks in the run queue, and a majority of handshake timeouts noticed by the client. With the patch, the run queue fluctuates between 1-3x runqueue-depth, the process is constantly busy, the accept rate is maximized and clients observe no error anymore. It would be desirable to backport this patch to 2.3 and 2.2 after some more testing, provided the accept loop there is compatible. |
||
|---|---|---|
| .github | ||
| contrib | ||
| doc | ||
| examples | ||
| include | ||
| reg-tests | ||
| scripts | ||
| src | ||
| tests | ||
| .cirrus.yml | ||
| .gitattributes | ||
| .gitignore | ||
| .travis.yml | ||
| BRANCHES | ||
| CHANGELOG | ||
| CONTRIBUTING | ||
| INSTALL | ||
| LICENSE | ||
| MAINTAINERS | ||
| Makefile | ||
| README | ||
| ROADMAP | ||
| SUBVERS | ||
| VERDATE | ||
| VERSION | ||
README
The HAProxy documentation has been split into a number of different files for ease of use. Please refer to the following files depending on what you're looking for : - INSTALL for instructions on how to build and install HAProxy - BRANCHES to understand the project's life cycle and what version to use - LICENSE for the project's license - CONTRIBUTING for the process to follow to submit contributions The more detailed documentation is located into the doc/ directory : - doc/intro.txt for a quick introduction on HAProxy - doc/configuration.txt for the configuration's reference manual - doc/lua.txt for the Lua's reference manual - doc/SPOE.txt for how to use the SPOE engine - doc/network-namespaces.txt for how to use network namespaces under Linux - doc/management.txt for the management guide - doc/regression-testing.txt for how to use the regression testing suite - doc/peers.txt for the peers protocol reference - doc/coding-style.txt for how to adopt HAProxy's coding style - doc/internals for developer-specific documentation (not all up to date)