RepoMirrors/haproxy
1
0
Fork 0
mirror of http://git.haproxy.org/git/haproxy.git/ synced 2024-12-25 06:02:08 +00:00
Code Issues Packages Projects Releases Wiki Activity
4c7e4b7738
haproxy/include/proto/task.h

647 lines
21 KiB
C
Raw Normal View History

[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
/*
[MEDIUM] signals: add support for registering functions and tasks The two new functions below make it possible to register any number of functions or tasks to a system signal. They will be called in the registration order when the signal is received. struct sig_handler *signal_register_fct(int sig, void (*fct)(struct sig_handler *), int arg); struct sig_handler *signal_register_task(int sig, struct task *task, int reason);
2010-08-27 15:56:48 +00:00
* include/proto/task.h
* Functions for task management.
*
* Copyright (C) 2000-2010 Willy Tarreau - w@1wt.eu
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation, version 2.1
* exclusively.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
#ifndef _PROTO_TASK_H
#define _PROTO_TASK_H
#include <sys/time.h>
[CLEANUP] included common/version.h everywhere
2006-06-29 16:54:54 +00:00
REORG: include: update all files to use haproxy/api.h or api-t.h if needed All files that were including one of the following include files have been updated to only include haproxy/api.h or haproxy/api-t.h once instead: - common/config.h - common/compat.h - common/compiler.h - common/defaults.h - common/initcall.h - common/tools.h The choice is simple: if the file only requires type definitions, it includes api-t.h, otherwise it includes the full api.h. In addition, in these files, explicit includes for inttypes.h and limits.h were dropped since these are now covered by api.h and api-t.h. No other change was performed, given that this patch is large and affects 201 files. At least one (tools.h) was already freestanding and didn't get the new one added.
2020-05-27 10:58:42 +00:00
#include <haproxy/api.h>
[CLEANUP] renamed include/haproxy to include/common
2006-06-29 15:53:05 +00:00
#include <common/memory.h>
[MAJOR] replaced rbtree with ul2tree. The rbtree-based wait queue consumes a lot of CPU. Use the ul2tree instead. Lots of cleanups and code reorganizations made it possible to reduce the task struct and simplify the code a bit.
2007-04-29 08:41:56 +00:00
#include <common/mini-clist.h>
#include <common/standard.h>
[CLEANUP] task: distinguish between clock ticks and timers Timers are unsigned and used as tree positions. Ticks are signed and used as absolute date within current time frame. While the two are normally equal (except zero), it's important not to confuse them in the code as they are not interchangeable. We add two inline functions to turn each one into the other. The comments have also been moved to the proper location, as it was not easy to understand what was a tick and what was a timer unit.
2009-03-08 14:53:06 +00:00
#include <common/ticks.h>
MAJOR: threads/task: handle multithread on task scheduler 2 global locks have been added to protect, respectively, the run queue and the wait queue. And a process mask has been added on each task. Like for FDs, this mask is used to know which threads are allowed to process a task. For many tasks, all threads are granted. And this must be your first intension when you create a new task, else you have a good reason to make a task sticky on some threads. This is then the responsibility to the process callback to lock what have to be locked in the task context. Nevertheless, all tasks linked to a session must be sticky on the thread creating the session. It is important that I/O handlers processing session FDs and these tasks run on the same thread to avoid conflicts.
2017-09-27 12:59:38 +00:00
#include <common/hathreads.h>
REORG: ebtree: move the include files from ebtree to include/import/ This is where other imported components are located. All files which used to directly include ebtree were touched to update their include path so that "import/" is now prefixed before the ebtree-related files. The ebtree.h file was slightly adjusted to read compiler.h from the common/ subdirectory (this is the only change). A build issue was encountered when eb32sctree.h is loaded before eb32tree.h because only the former checks for the latter before defining type u32. This was addressed by adding the reverse ifdef in eb32tree.h. No further cleanup was done yet in order to keep changes minimal.
2020-05-27 08:58:19 +00:00
#include <import/eb32sctree.h>
#include <import/eb32tree.h>
[MAJOR] replaced rbtree with ul2tree. The rbtree-based wait queue consumes a lot of CPU. Use the ul2tree instead. Lots of cleanups and code reorganizations made it possible to reduce the task struct and simplify the code a bit.
2007-04-29 08:41:56 +00:00
MINOR: task: release the task pool when stopping When we're stopping, we're not going to create new tasks anymore, so let's release the task pool upon each task_free() in order to reduce memory fragmentation.
2014-11-13 15:57:19 +00:00
#include <types/global.h>
[CLEANUP] included common/version.h everywhere
2006-06-29 16:54:54 +00:00
#include <types/task.h>
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
BUG/MEDIUM: tasklets: Make sure we're waking the target thread if it sleeps. Now that we can wake tasklet for other threads, make sure that if the thread is sleeping, we wake it up, or the tasklet won't be executed until it's done sleeping. That also means that, before going to sleep, and after we put our bit in sleeping_thread_mask, we have to check that nobody added a tasklet for us, just checking for global_tasks_mask isn't enough anymore.
2019-09-24 12:55:28 +00:00
#include <proto/fd.h>
[CLEANUP] task: distinguish between clock ticks and timers Timers are unsigned and used as tree positions. Ticks are signed and used as absolute date within current time frame. While the two are normally equal (except zero), it's important not to confuse them in the code as they are not interchangeable. We add two inline functions to turn each one into the other. The comments have also been moved to the proper location, as it was not easy to understand what was a tick and what was a timer unit.
2009-03-08 14:53:06 +00:00
/* Principle of the wait queue.
*
* We want to be able to tell whether an expiration date is before of after the
* current time <now>. We KNOW that expiration dates are never too far apart,
* because they are measured in ticks (milliseconds). We also know that almost
* all dates will be in the future, and that a very small part of them will be
* in the past, they are the ones which have expired since last time we checked
* them. Using ticks, we know if a date is in the future or in the past, but we
* cannot use that to store sorted information because that reference changes
* all the time.
*
[MEDIUM] scheduler: get rid of the 4 trees thanks and use ebtree v4.1 Since we're now able to search from a precise expiration date in the timer tree using ebtree 4.1, we don't need to maintain 4 trees anymore. Not only does this simplify the code a lot, but it also ensures that we can always look 24 days back and ahead, which doubles the ability of the previous scheduler. Indeed, while based on absolute values, the timer tree is now relative to <now> as we can always search from <now>-31 bits. The run queue uses the exact same principle now, and is now simpler and a bit faster to process. With these changes alone, an overall 0.5% performance gain was observed. Tests were performed on the few wrapping cases and everything works as expected.
2009-03-21 09:01:42 +00:00
* We'll use the fact that the time wraps to sort timers. Timers above <now>
* are in the future, timers below <now> are in the past. Here, "above" and
* "below" are to be considered modulo 2^31.
[CLEANUP] task: distinguish between clock ticks and timers Timers are unsigned and used as tree positions. Ticks are signed and used as absolute date within current time frame. While the two are normally equal (except zero), it's important not to confuse them in the code as they are not interchangeable. We add two inline functions to turn each one into the other. The comments have also been moved to the proper location, as it was not easy to understand what was a tick and what was a timer unit.
2009-03-08 14:53:06 +00:00
*
[MEDIUM] scheduler: get rid of the 4 trees thanks and use ebtree v4.1 Since we're now able to search from a precise expiration date in the timer tree using ebtree 4.1, we don't need to maintain 4 trees anymore. Not only does this simplify the code a lot, but it also ensures that we can always look 24 days back and ahead, which doubles the ability of the previous scheduler. Indeed, while based on absolute values, the timer tree is now relative to <now> as we can always search from <now>-31 bits. The run queue uses the exact same principle now, and is now simpler and a bit faster to process. With these changes alone, an overall 0.5% performance gain was observed. Tests were performed on the few wrapping cases and everything works as expected.
2009-03-21 09:01:42 +00:00
* Timers are stored sorted in an ebtree. We use the new ability for ebtrees to
* lookup values starting from X to only expire tasks between <now> - 2^31 and
* <now>. If the end of the tree is reached while walking over it, we simply
* loop back to the beginning. That way, we have no problem keeping sorted
* wrapping timers in a tree, between (now - 24 days) and (now + 24 days). The
* keys in the tree always reflect their real position, none can be infinite.
* This reduces the number of checks to be performed.
[CLEANUP] task: distinguish between clock ticks and timers Timers are unsigned and used as tree positions. Ticks are signed and used as absolute date within current time frame. While the two are normally equal (except zero), it's important not to confuse them in the code as they are not interchangeable. We add two inline functions to turn each one into the other. The comments have also been moved to the proper location, as it was not easy to understand what was a tick and what was a timer unit.
2009-03-08 14:53:06 +00:00
*
* Another nice optimisation is to allow a timer to stay at an old place in the
* queue as long as it's not further than the real expiration date. That way,
* we use the tree as a place holder for a minorant of the real expiration
* date. Since we have a very low chance of hitting a timeout anyway, we can
* bounce the nodes to their right place when we scan the tree if we encounter
* a misplaced node once in a while. This even allows us not to remove the
* infinite timers from the wait queue.
*
* So, to summarize, we have :
* - node->key always defines current position in the wait queue
* - timer is the real expiration date (possibly infinite)
[MEDIUM] scheduler: get rid of the 4 trees thanks and use ebtree v4.1 Since we're now able to search from a precise expiration date in the timer tree using ebtree 4.1, we don't need to maintain 4 trees anymore. Not only does this simplify the code a lot, but it also ensures that we can always look 24 days back and ahead, which doubles the ability of the previous scheduler. Indeed, while based on absolute values, the timer tree is now relative to <now> as we can always search from <now>-31 bits. The run queue uses the exact same principle now, and is now simpler and a bit faster to process. With these changes alone, an overall 0.5% performance gain was observed. Tests were performed on the few wrapping cases and everything works as expected.
2009-03-21 09:01:42 +00:00
* - node->key is always before or equal to timer
[CLEANUP] task: distinguish between clock ticks and timers Timers are unsigned and used as tree positions. Ticks are signed and used as absolute date within current time frame. While the two are normally equal (except zero), it's important not to confuse them in the code as they are not interchangeable. We add two inline functions to turn each one into the other. The comments have also been moved to the proper location, as it was not easy to understand what was a tick and what was a timer unit.
2009-03-08 14:53:06 +00:00
*
* The run queue works similarly to the wait queue except that the current date
* is replaced by an insertion counter which can also wrap without any problem.
*/
[MEDIUM] scheduler: get rid of the 4 trees thanks and use ebtree v4.1 Since we're now able to search from a precise expiration date in the timer tree using ebtree 4.1, we don't need to maintain 4 trees anymore. Not only does this simplify the code a lot, but it also ensures that we can always look 24 days back and ahead, which doubles the ability of the previous scheduler. Indeed, while based on absolute values, the timer tree is now relative to <now> as we can always search from <now>-31 bits. The run queue uses the exact same principle now, and is now simpler and a bit faster to process. With these changes alone, an overall 0.5% performance gain was observed. Tests were performed on the few wrapping cases and everything works as expected.
2009-03-21 09:01:42 +00:00
/* The farthest we can look back in a timer tree */
#define TIMER_LOOK_BACK (1U << 31)
[CLEANUP] task: distinguish between clock ticks and timers Timers are unsigned and used as tree positions. Ticks are signed and used as absolute date within current time frame. While the two are normally equal (except zero), it's important not to confuse them in the code as they are not interchangeable. We add two inline functions to turn each one into the other. The comments have also been moved to the proper location, as it was not easy to understand what was a tick and what was a timer unit.
2009-03-08 14:53:06 +00:00
/* a few exported variables */
[MINOR] task: keep a task count and clean up task creators It's sometimes useful at least for statistics to keep a task count. It's easy to do by forcing the rare task creators to always use the same functions to create/destroy a task.
2009-03-21 17:13:21 +00:00
extern unsigned int nb_tasks; /* total number of tasks */
MINOR: task: export global_task_mask It will be used in debugging functions and must be exported.
2019-05-16 15:37:27 +00:00
extern volatile unsigned long global_tasks_mask; /* Mask of threads with tasks in the global runqueue */
MINOR: task: Rename run_queue and run_queue_cur counters <run_queue> is used to track the number of task in the run queue and <run_queue_cur> is a copy used for the reporting purpose. These counters has been renamed, respectively, <tasks_run_queue> and <tasks_run_queue_cur>. So the naming is consistent between tasks and applets. [wt: needed for next fixes, backport to 1.7 and 1.6]
2016-12-06 08:15:30 +00:00
extern unsigned int tasks_run_queue; /* run queue size */
extern unsigned int tasks_run_queue_cur;
[MINOR] stats: report number of tasks (active and running) It may be useful for statistics purposes to report the number of tasks.
2009-03-21 17:33:52 +00:00
extern unsigned int nb_tasks_cur;
[MEDIUM] introduce task->nice and boot access to statistics The run queue scheduler now considers task->nice to queue a task and to pick a task out of the queue. This makes it possible to boost the access to statistics (both via HTTP and UNIX socket). The UNIX socket receives twice as much a boost as the HTTP socket because it is more sensible.
2008-06-30 05:51:00 +00:00
extern unsigned int niced_tasks; /* number of niced tasks in the run queue */
CLEANUP: pools: rename all pool functions and pointers to remove this "2" During the migration to the second version of the pools, the new functions and pool pointers were all called "pool_something2()" and "pool2_something". Now there's no more pool v1 code and it's a real pain to still have to deal with this. Let's clean this up now by removing the "2" everywhere, and by renaming the pool heads "pool_head_something".
2017-11-24 16:34:44 +00:00
extern struct pool_head *pool_head_task;
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
extern struct pool_head *pool_head_tasklet;
CLEANUP: pools: rename all pool functions and pointers to remove this "2" During the migration to the second version of the pools, the new functions and pool pointers were all called "pool_something2()" and "pool2_something". Now there's no more pool v1 code and it's a real pain to still have to deal with this. Let's clean this up now by removing the "2" everywhere, and by renaming the pool heads "pool_head_something".
2017-11-24 16:34:44 +00:00
extern struct pool_head *pool_head_notification;
MINOR: task: introduce a thread-local "sched" variable for local scheduler stuff The aim is to rassemble all scheduler information related to the current thread. It simply points to task_per_thread[tid] without having to perform the operation at each time. We save around 1.2 kB of code on performance sensitive paths and increase the request rate by almost 1%.
2019-09-24 06:25:15 +00:00
extern THREAD_LOCAL struct task_per_thread *sched; /* current's thread scheduler context */
MINOR: tasks: Don't define rqueue if we're building without threads. To make sure we don't inadvertently insert task in the global runqueue, while only the local runqueue is used without threads, make its definition and usage conditional on USE_THREAD.
2018-06-06 12:22:03 +00:00
#ifdef USE_THREAD
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
extern struct eb_root timers; /* sorted timers tree, global */
MAJOR: tasks: Create a per-thread runqueue. A lot of tasks are run on one thread only, so instead of having them all in the global runqueue, create a per-thread runqueue which doesn't require any locking, and add all tasks belonging to only one thread to the corresponding runqueue. The global runqueue is still used for non-local tasks, and is visited by each thread when checking its own runqueue. The nice parameter is thus used both in the global runqueue and in the local ones. The rare tasks that are bound to multiple threads will have their nice value used twice (once for the global queue, once for the thread-local one).
2018-05-18 16:38:23 +00:00
extern struct eb_root rqueue; /* tree constituting the run queue */
BUG/MEDIUM: tasks: make __task_unlink_rq responsible for the rqueue size. As __task_wakeup() is responsible for increasing rqueue_local[tid]/global_rqueue_size, make __task_unlink_rq responsible for decreasing it, as process_runnable_tasks() isn't the only one that removes tasks from runqueues.
2018-07-26 13:59:38 +00:00
extern int global_rqueue_size; /* Number of element sin the global runqueue */
MINOR: tasks: Don't define rqueue if we're building without threads. To make sure we don't inadvertently insert task in the global runqueue, while only the local runqueue is used without threads, make its definition and usage conditional on USE_THREAD.
2018-06-06 12:22:03 +00:00
#endif
BUG/MEDIUM: tasks: make __task_unlink_rq responsible for the rqueue size. As __task_wakeup() is responsible for increasing rqueue_local[tid]/global_rqueue_size, make __task_unlink_rq responsible for decreasing it, as process_runnable_tasks() isn't the only one that removes tasks from runqueues.
2018-07-26 13:59:38 +00:00
OPTIM: tasks: group all tree roots per cache line Currently we have per-thread arrays of trees and counts, but these ones unfortunately share cache lines and are accessed very often. This patch moves the task-specific stuff into a structure taking a multiple of a cache line, and has one such per thread. Just doing this has reduced the cache miss ratio from 19.2% to 18.7% and increased the 12-thread test performance by 3%. It starts to become visible that we really need a process-wide per-thread storage area that would cover more than just these parts of the tasks. The code was arranged so that it's easy to move the pieces elsewhere if needed.
2018-10-15 14:12:48 +00:00
extern struct task_per_thread task_per_thread[MAX_THREADS];
MINOR: threads: Use __decl_hathreads to declare locks This macro should be used to declare variables or struct members depending on the USE_THREAD compile option. It avoids the encapsulation of such declarations between #ifdef/#endif. It is used to declare all lock variables.
2017-11-13 09:34:01 +00:00
__decl_hathreads(extern HA_SPINLOCK_T rq_lock); /* spin lock related to run queue */
MINOR: task: turn the WQ lock to an RW_LOCK For now it's exclusively used as a write lock though, thus it remains 100% equivalent to the spinlock it replaces.
2019-05-28 16:48:07 +00:00
__decl_hathreads(extern HA_RWLOCK_T wq_lock); /* RW lock related to the wait queue */
[MAJOR] migrated task, tree64 and session to pool2 task and tree64 are already very close in size and are merged together. Overall performance gained slightly by this simple change.
2007-05-13 17:43:47 +00:00
BUG/MEDIUM: tasks: Make sure we switch wait queues in task_set_affinity(). In task_set_affinity(), leave the wait_queue if any before changing the affinity, and re-enter a wait queue once it is done. If we don't do that, the task may stay in the wait queue of another thread, and we later may end up modifying that wait queue while holding no lock, which could lead to memory corruption. THis should be backported to 2.1, 2.0 and 1.9.
2019-12-05 14:11:19 +00:00
static inline struct task *task_unlink_wq(struct task *t);
static inline void task_queue(struct task *task);
MINOR: tasks: Allow tasklet_wakeup() to wakeup a task. Modify tasklet_wakeup() so that it handles a task as well, and inserts it directly into the tasklet list, making it effectively a tasklet. This should make future developments easier.
2018-08-01 13:58:44 +00:00
[OPTIM] task: don't unlink a task from a wait queue when waking it up In many situations, we wake a task on an I/O event, then queue it exactly where it was. This is a real waste because we delete/insert tasks into the wait queue for nothing. The only reason for this is that there was only one tree node in the task struct. By adding another tree node, we can have one tree for the timers (wait queue) and one tree for the priority (run queue). That way, we can have a task both in the run queue and wait queue at the same time. The wait queue now really holds timers, which is what it was designed for. The net gain is at least 1 delete/insert cycle per session, and up to 2-3 depending on the workload, since we save one cycle each time the expiration date is not changed during a wake up.
2009-03-07 16:25:21 +00:00
/* return 0 if task is in run queue, otherwise non-zero */
static inline int task_in_rq(struct task *t)
{
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
/* Check if leaf_p is NULL, in case he's not in the runqueue, and if
* it's not 0x1, which would mean it's in the tasklet list.
*/
MEDIUM: tasks: No longer use rq.node.leaf_p as a lock. Now that we have the warranty that a task won't be added in the runqueue while the TASK_QUEUED or the TASK_RUNNING flag is set, don't bother trying to lock the task by setting leaf_p to 0x1 while inserting it in the runqueue or having it in the tasklet_list, as nobody else will attempt to add it.
2019-04-17 17:13:07 +00:00
return t->rq.node.leaf_p != NULL;
[OPTIM] task: don't unlink a task from a wait queue when waking it up In many situations, we wake a task on an I/O event, then queue it exactly where it was. This is a real waste because we delete/insert tasks into the wait queue for nothing. The only reason for this is that there was only one tree node in the task struct. By adding another tree node, we can have one tree for the timers (wait queue) and one tree for the priority (run queue). That way, we can have a task both in the run queue and wait queue at the same time. The wait queue now really holds timers, which is what it was designed for. The net gain is at least 1 delete/insert cycle per session, and up to 2-3 depending on the workload, since we save one cycle each time the expiration date is not changed during a wake up.
2009-03-07 16:25:21 +00:00
}
/* return 0 if task is in wait queue, otherwise non-zero */
static inline int task_in_wq(struct task *t)
{
return t->wq.node.leaf_p != NULL;
}
[MEDIUM] indicate a reason for a task wakeup It's very frequent to require some information about the reason why a task is running. Some flags have been added so that a task now knows if it got woken up due to I/O completion, timeout, etc...
2008-08-29 16:19:04 +00:00
/* puts the task <t> in run queue with reason flags <f>, and returns <t> */
MAJOR: tasks: Create a per-thread runqueue. A lot of tasks are run on one thread only, so instead of having them all in the global runqueue, create a per-thread runqueue which doesn't require any locking, and add all tasks belonging to only one thread to the corresponding runqueue. The global runqueue is still used for non-local tasks, and is visited by each thread when checking its own runqueue. The nice parameter is thus used both in the global runqueue and in the local ones. The rare tasks that are bound to multiple threads will have their nice value used twice (once for the global queue, once for the thread-local one).
2018-05-18 16:38:23 +00:00
/* This will put the task in the local runqueue if the task is only runnable
* by the current thread, in the global runqueue otherwies.
*/
void __task_wakeup(struct task *t, struct eb_root *);
static inline void task_wakeup(struct task *t, unsigned int f)
[OPTIM] reduce the number of calls to task_wakeup() A test has shown that more than 16% of the calls to task_wakeup() could be avoided because the task is already woken up. So make it inline and move the test to the inline part.
2008-08-29 13:26:14 +00:00
{
MAJOR: tasks: Create a per-thread runqueue. A lot of tasks are run on one thread only, so instead of having them all in the global runqueue, create a per-thread runqueue which doesn't require any locking, and add all tasks belonging to only one thread to the corresponding runqueue. The global runqueue is still used for non-local tasks, and is visited by each thread when checking its own runqueue. The nice parameter is thus used both in the global runqueue and in the local ones. The rare tasks that are bound to multiple threads will have their nice value used twice (once for the global queue, once for the thread-local one).
2018-05-18 16:38:23 +00:00
unsigned short state;
#ifdef USE_THREAD
struct eb_root *root;
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
if (t->thread_mask == tid_bit || global.nbthread == 1)
MINOR: task: introduce a thread-local "sched" variable for local scheduler stuff The aim is to rassemble all scheduler information related to the current thread. It simply points to task_per_thread[tid] without having to perform the operation at each time. We save around 1.2 kB of code on performance sensitive paths and increase the request rate by almost 1%.
2019-09-24 06:25:15 +00:00
root = &sched->rqueue;
MAJOR: tasks: Create a per-thread runqueue. A lot of tasks are run on one thread only, so instead of having them all in the global runqueue, create a per-thread runqueue which doesn't require any locking, and add all tasks belonging to only one thread to the corresponding runqueue. The global runqueue is still used for non-local tasks, and is visited by each thread when checking its own runqueue. The nice parameter is thus used both in the global runqueue and in the local ones. The rare tasks that are bound to multiple threads will have their nice value used twice (once for the global queue, once for the thread-local one).
2018-05-18 16:38:23 +00:00
else
root = &rqueue;
#else
MINOR: task: introduce a thread-local "sched" variable for local scheduler stuff The aim is to rassemble all scheduler information related to the current thread. It simply points to task_per_thread[tid] without having to perform the operation at each time. We save around 1.2 kB of code on performance sensitive paths and increase the request rate by almost 1%.
2019-09-24 06:25:15 +00:00
struct eb_root *root = &sched->rqueue;
MAJOR: tasks: Create a per-thread runqueue. A lot of tasks are run on one thread only, so instead of having them all in the global runqueue, create a per-thread runqueue which doesn't require any locking, and add all tasks belonging to only one thread to the corresponding runqueue. The global runqueue is still used for non-local tasks, and is visited by each thread when checking its own runqueue. The nice parameter is thus used both in the global runqueue and in the local ones. The rare tasks that are bound to multiple threads will have their nice value used twice (once for the global queue, once for the thread-local one).
2018-05-18 16:38:23 +00:00
#endif
BUG/MEDIUM: tasks: Make sure we set TASK_QUEUED before adding a task to the rq. Make sure we set TASK_QUEUED in every case before adding the task to the run queue. task_wakeup() now checks if either TASK_QUEUED or TASK_RUNNING is set, and if neither is set, add TASK_QUEUED and effectively add the task to the runqueue. No longer use __task_wakeup() anywhere except in task_wakeup(), always use task_wakeup() instead. With the old code, process_runnable_task() may re-add a task in the runqueue without setting the TASK_QUEUED flag, and there were race conditions that could lead to a task having the TASK_QUEUED flag but not in the runqueue, thus being unschedulable. This should be backported to 1.9.
2019-04-17 09:47:18 +00:00
state = _HA_ATOMIC_OR(&t->state, f);
while (!(state & (TASK_RUNNING | TASK_QUEUED))) {
MINOR: task/thread: factor out a wake-up condition The wakeup condition in task_wakeup() is redundant as it is already validated by the CAS. Better move the __task_wakeup() call there, it also has the merit of being easier to audit this way. This also reduces the code size by around 1.8 kB : $ size haproxy-? text data bss dec hex filename 2153806 100208 1307676 3561690 3658da haproxy-1 2152094 100208 1307676 3559978 36522a haproxy-2
2019-04-17 18:52:51 +00:00
if (_HA_ATOMIC_CAS(&t->state, &state, state | TASK_QUEUED)) {
__task_wakeup(t, root);
BUG/MEDIUM: tasks: Make sure we set TASK_QUEUED before adding a task to the rq. Make sure we set TASK_QUEUED in every case before adding the task to the run queue. task_wakeup() now checks if either TASK_QUEUED or TASK_RUNNING is set, and if neither is set, add TASK_QUEUED and effectively add the task to the runqueue. No longer use __task_wakeup() anywhere except in task_wakeup(), always use task_wakeup() instead. With the old code, process_runnable_task() may re-add a task in the runqueue without setting the TASK_QUEUED flag, and there were race conditions that could lead to a task having the TASK_QUEUED flag but not in the runqueue, thus being unschedulable. This should be backported to 1.9.
2019-04-17 09:47:18 +00:00
break;
MINOR: task/thread: factor out a wake-up condition The wakeup condition in task_wakeup() is redundant as it is already validated by the CAS. Better move the __task_wakeup() call there, it also has the merit of being easier to audit this way. This also reduces the code size by around 1.8 kB : $ size haproxy-? text data bss dec hex filename 2153806 100208 1307676 3561690 3658da haproxy-1 2152094 100208 1307676 3559978 36522a haproxy-2
2019-04-17 18:52:51 +00:00
}
BUG/MEDIUM: tasks: Make sure we set TASK_QUEUED before adding a task to the rq. Make sure we set TASK_QUEUED in every case before adding the task to the run queue. task_wakeup() now checks if either TASK_QUEUED or TASK_RUNNING is set, and if neither is set, add TASK_QUEUED and effectively add the task to the runqueue. No longer use __task_wakeup() anywhere except in task_wakeup(), always use task_wakeup() instead. With the old code, process_runnable_task() may re-add a task in the runqueue without setting the TASK_QUEUED flag, and there were race conditions that could lead to a task having the TASK_QUEUED flag but not in the runqueue, thus being unschedulable. This should be backported to 1.9.
2019-04-17 09:47:18 +00:00
}
[OPTIM] reduce the number of calls to task_wakeup() A test has shown that more than 16% of the calls to task_wakeup() could be avoided because the task is already woken up. So make it inline and move the test to the inline part.
2008-08-29 13:26:14 +00:00
}
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
BUG/MAJOR: task: add a new TASK_SHARED_WQ flag to fix foreing requeuing Since 1.9 with commit b20aa9eef3 ("MAJOR: tasks: create per-thread wait queues") a task bound to a single thread will not use locks when being queued or dequeued because the wait queue is assumed to be the owner thread's. But there exists a rare situation where this is not true: the health check tasks may be running on one thread waiting for a response, and may in parallel be requeued by another thread calling health_adjust() after a detecting a response error in traffic when "observe l7" is set, and "fastinter" is lower than "inter", requiring to shorten the running check's timeout. In this case, the task being requeued was present in another thread's wait queue, thus opening a race during task_unlink_wq(), and gets requeued into the calling thread's wait queue instead of the running one's, opening a second race here. This patch aims at protecting against the risk of calling task_unlink_wq() from one thread while the task is queued on another thread, hence unlocked, by introducing a new TASK_SHARED_WQ flag. This new flag indicates that a task's position in the wait queue may be adjusted by other threads than then one currently executing it. This means that such WQ manipulations must be performed under a lock. There are two types of such tasks: - the global ones, using the global wait queue (technically speaking, those whose thread_mask has at least 2 bits set). - some local ones, which for now will be placed into the global wait queue as well in order to benefit from its lock. The flag is automatically set on initialization if the task's thread mask indicates more than one thread. The caller must also set it if it intends to let other threads update the task's expiration delay (e.g. delegated I/Os), or if it intends to change the task's affinity over time as this could lead to the same situation. Right now only the situation described above seems to be affected by this issue, and it is very difficult to trigger, and even then, will often have no visible effect beyond stopping the checks for example once the race is met. On my laptop it is feasible with the following config, chained to httpterm: global maxconn 400 # provoke FD errors, calling health_adjust() defaults mode http timeout client 10s timeout server 10s timeout connect 10s listen px bind :8001 option httpchk /?t=50 server sback 127.0.0.1:8000 backup server-template s 0-999 127.0.0.1:8000 check port 8001 inter 100 fastinter 10 observe layer7 This patch will automatically address the case for the checks because check tasks are created with multiple threads bound and will get the TASK_SHARED_WQ flag set. If in the future more tasks need to rely on this (multi-threaded muxes for example) and the use of the global wait queue becomes a bottleneck again, then it should not be too difficult to place locks on the local wait queues and queue the task on its bound thread. This patch needs to be backported to 2.1, 2.0 and 1.9. It depends on previous patch "MINOR: task: only check TASK_WOKEN_ANY to decide to requeue a task". Many thanks to William Dauchy for providing detailed traces allowing to spot the problem.
2019-12-19 06:39:06 +00:00
/* change the thread affinity of a task to <thread_mask>.
* This may only be done from within the running task itself or during its
* initialization. It will unqueue and requeue the task from the wait queue
* if it was in it. This is safe against a concurrent task_queue() call because
* task_queue() itself will unlink again if needed after taking into account
* the new thread_mask.
*/
MAJOR: threads/task: handle multithread on task scheduler 2 global locks have been added to protect, respectively, the run queue and the wait queue. And a process mask has been added on each task. Like for FDs, this mask is used to know which threads are allowed to process a task. For many tasks, all threads are granted. And this must be your first intension when you create a new task, else you have a good reason to make a task sticky on some threads. This is then the responsibility to the process callback to lock what have to be locked in the task context. Nevertheless, all tasks linked to a session must be sticky on the thread creating the session. It is important that I/O handlers processing session FDs and these tasks run on the same thread to avoid conflicts.
2017-09-27 12:59:38 +00:00
static inline void task_set_affinity(struct task *t, unsigned long thread_mask)
{
BUG/MINOR: tasks: only requeue a task if it was already in the queue Commit 0742c314c3 ("BUG/MEDIUM: tasks: Make sure we switch wait queues in task_set_affinity().") had a slight side effect on expired timeouts, which is that when used before a timeout is updated, it will cause an existing task to be requeued earlier than its expected timeout when done before being updated, resulting in the next poll wakup timeout too early or even instantly if the previous wake up was done on a timeout. This is visible in strace when health checks are enabled because there are two poll calls, one of which has a short or zero delay. The correct solution is to only requeue a task if it was already in the queue. This can be backported to all branches having the fix above.
2019-12-11 08:11:58 +00:00
if (unlikely(task_in_wq(t))) {
BUG/MEDIUM: tasks: Make sure we switch wait queues in task_set_affinity(). In task_set_affinity(), leave the wait_queue if any before changing the affinity, and re-enter a wait queue once it is done. If we don't do that, the task may stay in the wait queue of another thread, and we later may end up modifying that wait queue while holding no lock, which could lead to memory corruption. THis should be backported to 2.1, 2.0 and 1.9.
2019-12-05 14:11:19 +00:00
task_unlink_wq(t);
BUG/MINOR: tasks: only requeue a task if it was already in the queue Commit 0742c314c3 ("BUG/MEDIUM: tasks: Make sure we switch wait queues in task_set_affinity().") had a slight side effect on expired timeouts, which is that when used before a timeout is updated, it will cause an existing task to be requeued earlier than its expected timeout when done before being updated, resulting in the next poll wakup timeout too early or even instantly if the previous wake up was done on a timeout. This is visible in strace when health checks are enabled because there are two poll calls, one of which has a short or zero delay. The correct solution is to only requeue a task if it was already in the queue. This can be backported to all branches having the fix above.
2019-12-11 08:11:58 +00:00
t->thread_mask = thread_mask;
BUG/MEDIUM: tasks: Make sure we switch wait queues in task_set_affinity(). In task_set_affinity(), leave the wait_queue if any before changing the affinity, and re-enter a wait queue once it is done. If we don't do that, the task may stay in the wait queue of another thread, and we later may end up modifying that wait queue while holding no lock, which could lead to memory corruption. THis should be backported to 2.1, 2.0 and 1.9.
2019-12-05 14:11:19 +00:00
task_queue(t);
BUG/MINOR: tasks: only requeue a task if it was already in the queue Commit 0742c314c3 ("BUG/MEDIUM: tasks: Make sure we switch wait queues in task_set_affinity().") had a slight side effect on expired timeouts, which is that when used before a timeout is updated, it will cause an existing task to be requeued earlier than its expected timeout when done before being updated, resulting in the next poll wakup timeout too early or even instantly if the previous wake up was done on a timeout. This is visible in strace when health checks are enabled because there are two poll calls, one of which has a short or zero delay. The correct solution is to only requeue a task if it was already in the queue. This can be backported to all branches having the fix above.
2019-12-11 08:11:58 +00:00
}
else
t->thread_mask = thread_mask;
MAJOR: threads/task: handle multithread on task scheduler 2 global locks have been added to protect, respectively, the run queue and the wait queue. And a process mask has been added on each task. Like for FDs, this mask is used to know which threads are allowed to process a task. For many tasks, all threads are granted. And this must be your first intension when you create a new task, else you have a good reason to make a task sticky on some threads. This is then the responsibility to the process callback to lock what have to be locked in the task context. Nevertheless, all tasks linked to a session must be sticky on the thread creating the session. It is important that I/O handlers processing session FDs and these tasks run on the same thread to avoid conflicts.
2017-09-27 12:59:38 +00:00
}
CLEANUP: threads: rename process_mask to thread_mask It was a leftover from the last cleaning session; this mask applies to threads and calling it process_mask is a bit confusing. It's the same in fd, task and applets.
2017-10-31 15:06:06 +00:00
[OPTIM] task: don't unlink a task from a wait queue when waking it up In many situations, we wake a task on an I/O event, then queue it exactly where it was. This is a real waste because we delete/insert tasks into the wait queue for nothing. The only reason for this is that there was only one tree node in the task struct. By adding another tree node, we can have one tree for the timers (wait queue) and one tree for the priority (run queue). That way, we can have a task both in the run queue and wait queue at the same time. The wait queue now really holds timers, which is what it was designed for. The net gain is at least 1 delete/insert cycle per session, and up to 2-3 depending on the workload, since we save one cycle each time the expiration date is not changed during a wake up.
2009-03-07 16:25:21 +00:00
/*
* Unlink the task from the wait queue, and possibly update the last_timer
* pointer. A pointer to the task itself is returned. The task *must* already
* be in the wait queue before calling this function. If unsure, use the safer
* task_unlink_wq() function.
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
*/
[OPTIM] task: don't unlink a task from a wait queue when waking it up In many situations, we wake a task on an I/O event, then queue it exactly where it was. This is a real waste because we delete/insert tasks into the wait queue for nothing. The only reason for this is that there was only one tree node in the task struct. By adding another tree node, we can have one tree for the timers (wait queue) and one tree for the priority (run queue). That way, we can have a task both in the run queue and wait queue at the same time. The wait queue now really holds timers, which is what it was designed for. The net gain is at least 1 delete/insert cycle per session, and up to 2-3 depending on the workload, since we save one cycle each time the expiration date is not changed during a wake up.
2009-03-07 16:25:21 +00:00
static inline struct task *__task_unlink_wq(struct task *t)
{
eb32_delete(&t->wq);
return t;
}
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
/* remove a task from its wait queue. It may either be the local wait queue if
BUG/MAJOR: task: add a new TASK_SHARED_WQ flag to fix foreing requeuing Since 1.9 with commit b20aa9eef3 ("MAJOR: tasks: create per-thread wait queues") a task bound to a single thread will not use locks when being queued or dequeued because the wait queue is assumed to be the owner thread's. But there exists a rare situation where this is not true: the health check tasks may be running on one thread waiting for a response, and may in parallel be requeued by another thread calling health_adjust() after a detecting a response error in traffic when "observe l7" is set, and "fastinter" is lower than "inter", requiring to shorten the running check's timeout. In this case, the task being requeued was present in another thread's wait queue, thus opening a race during task_unlink_wq(), and gets requeued into the calling thread's wait queue instead of the running one's, opening a second race here. This patch aims at protecting against the risk of calling task_unlink_wq() from one thread while the task is queued on another thread, hence unlocked, by introducing a new TASK_SHARED_WQ flag. This new flag indicates that a task's position in the wait queue may be adjusted by other threads than then one currently executing it. This means that such WQ manipulations must be performed under a lock. There are two types of such tasks: - the global ones, using the global wait queue (technically speaking, those whose thread_mask has at least 2 bits set). - some local ones, which for now will be placed into the global wait queue as well in order to benefit from its lock. The flag is automatically set on initialization if the task's thread mask indicates more than one thread. The caller must also set it if it intends to let other threads update the task's expiration delay (e.g. delegated I/Os), or if it intends to change the task's affinity over time as this could lead to the same situation. Right now only the situation described above seems to be affected by this issue, and it is very difficult to trigger, and even then, will often have no visible effect beyond stopping the checks for example once the race is met. On my laptop it is feasible with the following config, chained to httpterm: global maxconn 400 # provoke FD errors, calling health_adjust() defaults mode http timeout client 10s timeout server 10s timeout connect 10s listen px bind :8001 option httpchk /?t=50 server sback 127.0.0.1:8000 backup server-template s 0-999 127.0.0.1:8000 check port 8001 inter 100 fastinter 10 observe layer7 This patch will automatically address the case for the checks because check tasks are created with multiple threads bound and will get the TASK_SHARED_WQ flag set. If in the future more tasks need to rely on this (multi-threaded muxes for example) and the use of the global wait queue becomes a bottleneck again, then it should not be too difficult to place locks on the local wait queues and queue the task on its bound thread. This patch needs to be backported to 2.1, 2.0 and 1.9. It depends on previous patch "MINOR: task: only check TASK_WOKEN_ANY to decide to requeue a task". Many thanks to William Dauchy for providing detailed traces allowing to spot the problem.
2019-12-19 06:39:06 +00:00
* the task is bound to a single thread or the global queue. If the task uses a
* shared wait queue, the global wait queue lock is used.
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
*/
[OPTIM] task: don't unlink a task from a wait queue when waking it up In many situations, we wake a task on an I/O event, then queue it exactly where it was. This is a real waste because we delete/insert tasks into the wait queue for nothing. The only reason for this is that there was only one tree node in the task struct. By adding another tree node, we can have one tree for the timers (wait queue) and one tree for the priority (run queue). That way, we can have a task both in the run queue and wait queue at the same time. The wait queue now really holds timers, which is what it was designed for. The net gain is at least 1 delete/insert cycle per session, and up to 2-3 depending on the workload, since we save one cycle each time the expiration date is not changed during a wake up.
2009-03-07 16:25:21 +00:00
static inline struct task *task_unlink_wq(struct task *t)
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
{
BUG/MAJOR: fd/threads, task/threads: ensure all spin locks are unlocked Calculate if the fd or task should be locked once, before locking, and reuse the calculation when determing when to unlock. Fixes a race condition added in 87d54a9a for fds, and b20aa9ee for tasks, released in 1.9-dev4. When one thread modifies thread_mask to be a single thread for a task or fd while a second thread has locked or is waiting on a lock for that task or fd, the second thread will not unlock it. For FDs, this is observable when a listener is polled by multiple threads, and is closed while those threads have events pending. For tasks, this seems possible, where task_set_affinity is called, but I did not observe it. This must be backported to 1.9.
2019-02-20 20:43:45 +00:00
unsigned long locked;
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
if (likely(task_in_wq(t))) {
BUG/MAJOR: task: add a new TASK_SHARED_WQ flag to fix foreing requeuing Since 1.9 with commit b20aa9eef3 ("MAJOR: tasks: create per-thread wait queues") a task bound to a single thread will not use locks when being queued or dequeued because the wait queue is assumed to be the owner thread's. But there exists a rare situation where this is not true: the health check tasks may be running on one thread waiting for a response, and may in parallel be requeued by another thread calling health_adjust() after a detecting a response error in traffic when "observe l7" is set, and "fastinter" is lower than "inter", requiring to shorten the running check's timeout. In this case, the task being requeued was present in another thread's wait queue, thus opening a race during task_unlink_wq(), and gets requeued into the calling thread's wait queue instead of the running one's, opening a second race here. This patch aims at protecting against the risk of calling task_unlink_wq() from one thread while the task is queued on another thread, hence unlocked, by introducing a new TASK_SHARED_WQ flag. This new flag indicates that a task's position in the wait queue may be adjusted by other threads than then one currently executing it. This means that such WQ manipulations must be performed under a lock. There are two types of such tasks: - the global ones, using the global wait queue (technically speaking, those whose thread_mask has at least 2 bits set). - some local ones, which for now will be placed into the global wait queue as well in order to benefit from its lock. The flag is automatically set on initialization if the task's thread mask indicates more than one thread. The caller must also set it if it intends to let other threads update the task's expiration delay (e.g. delegated I/Os), or if it intends to change the task's affinity over time as this could lead to the same situation. Right now only the situation described above seems to be affected by this issue, and it is very difficult to trigger, and even then, will often have no visible effect beyond stopping the checks for example once the race is met. On my laptop it is feasible with the following config, chained to httpterm: global maxconn 400 # provoke FD errors, calling health_adjust() defaults mode http timeout client 10s timeout server 10s timeout connect 10s listen px bind :8001 option httpchk /?t=50 server sback 127.0.0.1:8000 backup server-template s 0-999 127.0.0.1:8000 check port 8001 inter 100 fastinter 10 observe layer7 This patch will automatically address the case for the checks because check tasks are created with multiple threads bound and will get the TASK_SHARED_WQ flag set. If in the future more tasks need to rely on this (multi-threaded muxes for example) and the use of the global wait queue becomes a bottleneck again, then it should not be too difficult to place locks on the local wait queues and queue the task on its bound thread. This patch needs to be backported to 2.1, 2.0 and 1.9. It depends on previous patch "MINOR: task: only check TASK_WOKEN_ANY to decide to requeue a task". Many thanks to William Dauchy for providing detailed traces allowing to spot the problem.
2019-12-19 06:39:06 +00:00
locked = t->state & TASK_SHARED_WQ;
BUG/MAJOR: fd/threads, task/threads: ensure all spin locks are unlocked Calculate if the fd or task should be locked once, before locking, and reuse the calculation when determing when to unlock. Fixes a race condition added in 87d54a9a for fds, and b20aa9ee for tasks, released in 1.9-dev4. When one thread modifies thread_mask to be a single thread for a task or fd while a second thread has locked or is waiting on a lock for that task or fd, the second thread will not unlock it. For FDs, this is observable when a listener is polled by multiple threads, and is closed while those threads have events pending. For tasks, this seems possible, where task_set_affinity is called, but I did not observe it. This must be backported to 1.9.
2019-02-20 20:43:45 +00:00
if (locked)
MINOR: task: turn the WQ lock to an RW_LOCK For now it's exclusively used as a write lock though, thus it remains 100% equivalent to the spinlock it replaces.
2019-05-28 16:48:07 +00:00
HA_RWLOCK_WRLOCK(TASK_WQ_LOCK, &wq_lock);
[OPTIM] task: don't unlink a task from a wait queue when waking it up In many situations, we wake a task on an I/O event, then queue it exactly where it was. This is a real waste because we delete/insert tasks into the wait queue for nothing. The only reason for this is that there was only one tree node in the task struct. By adding another tree node, we can have one tree for the timers (wait queue) and one tree for the priority (run queue). That way, we can have a task both in the run queue and wait queue at the same time. The wait queue now really holds timers, which is what it was designed for. The net gain is at least 1 delete/insert cycle per session, and up to 2-3 depending on the workload, since we save one cycle each time the expiration date is not changed during a wake up.
2009-03-07 16:25:21 +00:00
__task_unlink_wq(t);
BUG/MAJOR: fd/threads, task/threads: ensure all spin locks are unlocked Calculate if the fd or task should be locked once, before locking, and reuse the calculation when determing when to unlock. Fixes a race condition added in 87d54a9a for fds, and b20aa9ee for tasks, released in 1.9-dev4. When one thread modifies thread_mask to be a single thread for a task or fd while a second thread has locked or is waiting on a lock for that task or fd, the second thread will not unlock it. For FDs, this is observable when a listener is polled by multiple threads, and is closed while those threads have events pending. For tasks, this seems possible, where task_set_affinity is called, but I did not observe it. This must be backported to 1.9.
2019-02-20 20:43:45 +00:00
if (locked)
MINOR: task: turn the WQ lock to an RW_LOCK For now it's exclusively used as a write lock though, thus it remains 100% equivalent to the spinlock it replaces.
2019-05-28 16:48:07 +00:00
HA_RWLOCK_WRUNLOCK(TASK_WQ_LOCK, &wq_lock);
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
}
[MAJOR] replaced rbtree with ul2tree. The rbtree-based wait queue consumes a lot of CPU. Use the ul2tree instead. Lots of cleanups and code reorganizations made it possible to reduce the task struct and simplify the code a bit.
2007-04-29 08:41:56 +00:00
return t;
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
}
/*
MINOR: task: Rename run_queue and run_queue_cur counters <run_queue> is used to track the number of task in the run queue and <run_queue_cur> is a copy used for the reporting purpose. These counters has been renamed, respectively, <tasks_run_queue> and <tasks_run_queue_cur>. So the naming is consistent between tasks and applets. [wt: needed for next fixes, backport to 1.7 and 1.6]
2016-12-06 08:15:30 +00:00
* Unlink the task from the run queue. The tasks_run_queue size and number of
* niced tasks are updated too. A pointer to the task itself is returned. The
* task *must* already be in the run queue before calling this function. If
* unsure, use the safer task_unlink_rq() function. Note that the pointer to the
* next run queue entry is neither checked nor updated.
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
*/
[OPTIM] task: don't unlink a task from a wait queue when waking it up In many situations, we wake a task on an I/O event, then queue it exactly where it was. This is a real waste because we delete/insert tasks into the wait queue for nothing. The only reason for this is that there was only one tree node in the task struct. By adding another tree node, we can have one tree for the timers (wait queue) and one tree for the priority (run queue). That way, we can have a task both in the run queue and wait queue at the same time. The wait queue now really holds timers, which is what it was designed for. The net gain is at least 1 delete/insert cycle per session, and up to 2-3 depending on the workload, since we save one cycle each time the expiration date is not changed during a wake up.
2009-03-07 16:25:21 +00:00
static inline struct task *__task_unlink_rq(struct task *t)
{
MEDIUM: task: Use the new _HA_ATOMIC_* macros. Use the new _HA_ATOMIC_* macros and add barriers where needed.
2019-03-08 17:48:47 +00:00
_HA_ATOMIC_SUB(&tasks_run_queue, 1);
BUG/MEDIUM: tasks: make __task_unlink_rq responsible for the rqueue size. As __task_wakeup() is responsible for increasing rqueue_local[tid]/global_rqueue_size, make __task_unlink_rq responsible for decreasing it, as process_runnable_tasks() isn't the only one that removes tasks from runqueues.
2018-07-26 13:59:38 +00:00
#ifdef USE_THREAD
if (t->state & TASK_GLOBAL) {
MEDIUM: task: Use the new _HA_ATOMIC_* macros. Use the new _HA_ATOMIC_* macros and add barriers where needed.
2019-03-08 17:48:47 +00:00
_HA_ATOMIC_AND(&t->state, ~TASK_GLOBAL);
BUG/MEDIUM: tasks: make __task_unlink_rq responsible for the rqueue size. As __task_wakeup() is responsible for increasing rqueue_local[tid]/global_rqueue_size, make __task_unlink_rq responsible for decreasing it, as process_runnable_tasks() isn't the only one that removes tasks from runqueues.
2018-07-26 13:59:38 +00:00
global_rqueue_size--;
} else
#endif
MINOR: task: introduce a thread-local "sched" variable for local scheduler stuff The aim is to rassemble all scheduler information related to the current thread. It simply points to task_per_thread[tid] without having to perform the operation at each time. We save around 1.2 kB of code on performance sensitive paths and increase the request rate by almost 1%.
2019-09-24 06:25:15 +00:00
sched->rqueue_size--;
BUG/MEDIUM: tasks: make __task_unlink_rq responsible for the rqueue size. As __task_wakeup() is responsible for increasing rqueue_local[tid]/global_rqueue_size, make __task_unlink_rq responsible for decreasing it, as process_runnable_tasks() isn't the only one that removes tasks from runqueues.
2018-07-26 13:59:38 +00:00
eb32sc_delete(&t->rq);
[OPTIM] task: don't unlink a task from a wait queue when waking it up In many situations, we wake a task on an I/O event, then queue it exactly where it was. This is a real waste because we delete/insert tasks into the wait queue for nothing. The only reason for this is that there was only one tree node in the task struct. By adding another tree node, we can have one tree for the timers (wait queue) and one tree for the priority (run queue). That way, we can have a task both in the run queue and wait queue at the same time. The wait queue now really holds timers, which is what it was designed for. The net gain is at least 1 delete/insert cycle per session, and up to 2-3 depending on the workload, since we save one cycle each time the expiration date is not changed during a wake up.
2009-03-07 16:25:21 +00:00
if (likely(t->nice))
MEDIUM: task: Use the new _HA_ATOMIC_* macros. Use the new _HA_ATOMIC_* macros and add barriers where needed.
2019-03-08 17:48:47 +00:00
_HA_ATOMIC_SUB(&niced_tasks, 1);
[OPTIM] task: don't unlink a task from a wait queue when waking it up In many situations, we wake a task on an I/O event, then queue it exactly where it was. This is a real waste because we delete/insert tasks into the wait queue for nothing. The only reason for this is that there was only one tree node in the task struct. By adding another tree node, we can have one tree for the timers (wait queue) and one tree for the priority (run queue). That way, we can have a task both in the run queue and wait queue at the same time. The wait queue now really holds timers, which is what it was designed for. The net gain is at least 1 delete/insert cycle per session, and up to 2-3 depending on the workload, since we save one cycle each time the expiration date is not changed during a wake up.
2009-03-07 16:25:21 +00:00
return t;
}
MEDIUM: task: always ensure that the run queue is consistent As found by Thierry Fournier, if a task manages to kill another one and if this other task is the next one in the run queue, we can do whatever including crashing, because the scheduler restarts from the saved next task. For now, there is no such concept of a task killing another one, but with Lua it will come. A solution consists in always performing the lookup of the first task in the scheduler's loop, but it's expensive and costs around 2% of the performance. Another solution consists in keeping a global next run queue node and ensuring that when this task gets removed, it updates this pointer to the next one. This allows to simplify the code a bit and in the end to slightly increase the performance (0.3-0.5%). The mechanism might still be usable if we later migrate to a multi-threaded scheduler.
2015-02-23 15:07:01 +00:00
/* This function unlinks task <t> from the run queue if it is in it. It also
* takes care of updating the next run queue task if it was this task.
*/
[OPTIM] task: don't unlink a task from a wait queue when waking it up In many situations, we wake a task on an I/O event, then queue it exactly where it was. This is a real waste because we delete/insert tasks into the wait queue for nothing. The only reason for this is that there was only one tree node in the task struct. By adding another tree node, we can have one tree for the timers (wait queue) and one tree for the priority (run queue). That way, we can have a task both in the run queue and wait queue at the same time. The wait queue now really holds timers, which is what it was designed for. The net gain is at least 1 delete/insert cycle per session, and up to 2-3 depending on the workload, since we save one cycle each time the expiration date is not changed during a wake up.
2009-03-07 16:25:21 +00:00
static inline struct task *task_unlink_rq(struct task *t)
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
{
BUG/MAJOR: tasks: Use the TASK_GLOBAL flag to know if we're in the global rq. In task_unlink_rq, to decide if we should logk the global runqueue lock, use the TASK_GLOBAL flag instead of relying on t->thread_mask being tid_bit, as it could be so while still being in the global runqueue if another thread woke that task for us. This should be backported to 1.9.
2019-03-14 15:14:04 +00:00
int is_global = t->state & TASK_GLOBAL;
if (is_global)
MAJOR: tasks: Create a per-thread runqueue. A lot of tasks are run on one thread only, so instead of having them all in the global runqueue, create a per-thread runqueue which doesn't require any locking, and add all tasks belonging to only one thread to the corresponding runqueue. The global runqueue is still used for non-local tasks, and is visited by each thread when checking its own runqueue. The nice parameter is thus used both in the global runqueue and in the local ones. The rare tasks that are bound to multiple threads will have their nice value used twice (once for the global queue, once for the thread-local one).
2018-05-18 16:38:23 +00:00
HA_SPIN_LOCK(TASK_RQ_LOCK, &rq_lock);
CLEANUP: task: do not export rq_next anymore This one hasn't been used anymore since the scheduler changes after 1.8 but it kept being exported and maintained up to date while it's always reset when scanning the trees. Let's stop exporting it and updating it.
2019-04-12 14:10:55 +00:00
if (likely(task_in_rq(t)))
[OPTIM] task: don't unlink a task from a wait queue when waking it up In many situations, we wake a task on an I/O event, then queue it exactly where it was. This is a real waste because we delete/insert tasks into the wait queue for nothing. The only reason for this is that there was only one tree node in the task struct. By adding another tree node, we can have one tree for the timers (wait queue) and one tree for the priority (run queue). That way, we can have a task both in the run queue and wait queue at the same time. The wait queue now really holds timers, which is what it was designed for. The net gain is at least 1 delete/insert cycle per session, and up to 2-3 depending on the workload, since we save one cycle each time the expiration date is not changed during a wake up.
2009-03-07 16:25:21 +00:00
__task_unlink_rq(t);
BUG/MAJOR: tasks: Use the TASK_GLOBAL flag to know if we're in the global rq. In task_unlink_rq, to decide if we should logk the global runqueue lock, use the TASK_GLOBAL flag instead of relying on t->thread_mask being tid_bit, as it could be so while still being in the global runqueue if another thread woke that task for us. This should be backported to 1.9.
2019-03-14 15:14:04 +00:00
if (is_global)
MAJOR: tasks: Create a per-thread runqueue. A lot of tasks are run on one thread only, so instead of having them all in the global runqueue, create a per-thread runqueue which doesn't require any locking, and add all tasks belonging to only one thread to the corresponding runqueue. The global runqueue is still used for non-local tasks, and is visited by each thread when checking its own runqueue. The nice parameter is thus used both in the global runqueue and in the local ones. The rare tasks that are bound to multiple threads will have their nice value used twice (once for the global queue, once for the thread-local one).
2018-05-18 16:38:23 +00:00
HA_SPIN_UNLOCK(TASK_RQ_LOCK, &rq_lock);
[OPTIM] task_queue: assume most consecutive timers are equal When queuing a timer, it's very likely that an expiration date is equal to that of the previously queued timer, due to time rounding to the millisecond. Optimizing for this case provides a noticeable 1% performance boost.
2008-07-05 16:16:19 +00:00
return t;
}
[MAJOR] replace ultree with ebtree in wait-queues The ultree code has been removed in favor of a simpler and cleaner ebtree implementation. The eternity queue does not need to exist anymore, and the pool_tree64 has been removed. The ebtree node is stored in the task itself. The qlist list header is still used by the run-queue, but will be able to disappear once the run-queue uses ebtree too.
2008-06-24 06:17:16 +00:00
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
static inline void tasklet_wakeup(struct tasklet *tl)
{
BUG/MEDIUM: task: make tasklets either local or shared but not both at once Tasklets may be woken up to run on the calling thread or by a specific thread (the owner). But since we use a non-thread safe mechanism when the calling thread is also the for the owner, there may sometimes be collisions when two threads decide to wake the same tasklet up at the same time and one of them is the owner. This is more of a matter of usage than code, in that a tasklet usually is designed to be woken up and executed on the calling thread only (most cases) or on a specific thread. Thus it is a property of the tasklet itself as this solely depends how the code is constructed around it. This patch performs a small change to address this. By default tasklet_new() creates a "local" tasklet, which will run on the calling thread, like in 2.0. This is done by setting tl->tid to a negative value. If the caller wants the tasklet to run exclusively on a specific thread, it just has to set tl->tid, which is already what shared tasklet callers do anyway. No backport is needed.
2019-10-18 04:43:53 +00:00
if (likely(tl->tid < 0)) {
/* this tasklet runs on the caller thread */
MEDIUM: task: Split the tasklet list into two lists. As using an mt_list for the tasklet list is costly, instead use a regular list, but add an mt_list for tasklet woken up by other threads, to be run on the current thread. At the beginning of process_runnable_tasks(), we just take the new list, and merge it into the task_list. This should give us performances comparable to before we started using a mt_list, but allow us to use tasklet_wakeup() from other threads.
2019-10-11 14:35:01 +00:00
if (LIST_ISEMPTY(&tl->list)) {
MINOR: task: permanently flag tasklets waking themselves up Commit a17664d829 ("MEDIUM: tasks: automatically requeue into the bulk queue an already running tasklet") tried to inflict a penalty to self-requeuing tasks/tasklets which correspond to those involved in large, high-latency data transfers, for the benefit of all other processing which requires a low latency. However, it turns out that while it ought to do this on a case-by-case basis, basing itself on the RUNNING flag isn't accurate because this flag doesn't leave for tasklets, so we'd rather need a distinct flag to tag such tasklets. This commit introduces TASK_SELF_WAKING to mark tasklets acting like this. For now it's still set when TASK_RUNNING is present but this will have to change. The flag is kept across wakeups.
2020-01-31 09:48:10 +00:00
if (tl->state & TASK_SELF_WAKING) {
MEDIUM: tasks: automatically requeue into the bulk queue an already running tasklet When a tasklet re-runs itself such as in this chain: si_cs_io_cb -> si_cs_process -> si_notify -> si_chk_rcv then we know it can easily clobber the run queue and harm latency. Now what the scheduler does when it detects this is that such a tasklet is automatically placed into the bulk list so that it's processed with the remaining CPU bandwidth only. Thanks to this the CLI becomes instantly responsive again even under heavy stress at 50 Gbps over 40kcon and 100% CPU on 16 threads.
2020-01-30 17:59:43 +00:00
LIST_ADDQ(&task_per_thread[tid].tasklets[TL_BULK], &tl->list);
MINOR: task: permanently flag tasklets waking themselves up Commit a17664d829 ("MEDIUM: tasks: automatically requeue into the bulk queue an already running tasklet") tried to inflict a penalty to self-requeuing tasks/tasklets which correspond to those involved in large, high-latency data transfers, for the benefit of all other processing which requires a low latency. However, it turns out that while it ought to do this on a case-by-case basis, basing itself on the RUNNING flag isn't accurate because this flag doesn't leave for tasklets, so we'd rather need a distinct flag to tag such tasklets. This commit introduces TASK_SELF_WAKING to mark tasklets acting like this. For now it's still set when TASK_RUNNING is present but this will have to change. The flag is kept across wakeups.
2020-01-31 09:48:10 +00:00
}
MINOR: task: detect self-wakeups on tl==sched->current instead of TASK_RUNNING This is exactly what we want to detect (a task/tasklet waking itself), so let's use the proper condition for this.
2020-01-31 15:37:34 +00:00
else if ((struct task *)tl == sched->current) {
MINOR: task: permanently flag tasklets waking themselves up Commit a17664d829 ("MEDIUM: tasks: automatically requeue into the bulk queue an already running tasklet") tried to inflict a penalty to self-requeuing tasks/tasklets which correspond to those involved in large, high-latency data transfers, for the benefit of all other processing which requires a low latency. However, it turns out that while it ought to do this on a case-by-case basis, basing itself on the RUNNING flag isn't accurate because this flag doesn't leave for tasklets, so we'd rather need a distinct flag to tag such tasklets. This commit introduces TASK_SELF_WAKING to mark tasklets acting like this. For now it's still set when TASK_RUNNING is present but this will have to change. The flag is kept across wakeups.
2020-01-31 09:48:10 +00:00
_HA_ATOMIC_OR(&tl->state, TASK_SELF_WAKING);
LIST_ADDQ(&task_per_thread[tid].tasklets[TL_BULK], &tl->list);
}
else {
MEDIUM: tasks: automatically requeue into the bulk queue an already running tasklet When a tasklet re-runs itself such as in this chain: si_cs_io_cb -> si_cs_process -> si_notify -> si_chk_rcv then we know it can easily clobber the run queue and harm latency. Now what the scheduler does when it detects this is that such a tasklet is automatically placed into the bulk list so that it's processed with the remaining CPU bandwidth only. Thanks to this the CLI becomes instantly responsive again even under heavy stress at 50 Gbps over 40kcon and 100% CPU on 16 threads.
2020-01-30 17:59:43 +00:00
LIST_ADDQ(&task_per_thread[tid].tasklets[TL_URGENT], &tl->list);
MINOR: task: permanently flag tasklets waking themselves up Commit a17664d829 ("MEDIUM: tasks: automatically requeue into the bulk queue an already running tasklet") tried to inflict a penalty to self-requeuing tasks/tasklets which correspond to those involved in large, high-latency data transfers, for the benefit of all other processing which requires a low latency. However, it turns out that while it ought to do this on a case-by-case basis, basing itself on the RUNNING flag isn't accurate because this flag doesn't leave for tasklets, so we'd rather need a distinct flag to tag such tasklets. This commit introduces TASK_SELF_WAKING to mark tasklets acting like this. For now it's still set when TASK_RUNNING is present but this will have to change. The flag is kept across wakeups.
2020-01-31 09:48:10 +00:00
}
MEDIUM: task: Split the tasklet list into two lists. As using an mt_list for the tasklet list is costly, instead use a regular list, but add an mt_list for tasklet woken up by other threads, to be run on the current thread. At the beginning of process_runnable_tasks(), we just take the new list, and merge it into the task_list. This should give us performances comparable to before we started using a mt_list, but allow us to use tasklet_wakeup() from other threads.
2019-10-11 14:35:01 +00:00
_HA_ATOMIC_ADD(&tasks_run_queue, 1);
}
} else {
BUG/MEDIUM: task: make tasklets either local or shared but not both at once Tasklets may be woken up to run on the calling thread or by a specific thread (the owner). But since we use a non-thread safe mechanism when the calling thread is also the for the owner, there may sometimes be collisions when two threads decide to wake the same tasklet up at the same time and one of them is the owner. This is more of a matter of usage than code, in that a tasklet usually is designed to be woken up and executed on the calling thread only (most cases) or on a specific thread. Thus it is a property of the tasklet itself as this solely depends how the code is constructed around it. This patch performs a small change to address this. By default tasklet_new() creates a "local" tasklet, which will run on the calling thread, like in 2.0. This is done by setting tl->tid to a negative value. If the caller wants the tasklet to run exclusively on a specific thread, it just has to set tl->tid, which is already what shared tasklet callers do anyway. No backport is needed.
2019-10-18 04:43:53 +00:00
/* this tasklet runs on a specific thread */
MEDIUM: task: Split the tasklet list into two lists. As using an mt_list for the tasklet list is costly, instead use a regular list, but add an mt_list for tasklet woken up by other threads, to be run on the current thread. At the beginning of process_runnable_tasks(), we just take the new list, and merge it into the task_list. This should give us performances comparable to before we started using a mt_list, but allow us to use tasklet_wakeup() from other threads.
2019-10-11 14:35:01 +00:00
if (MT_LIST_ADDQ(&task_per_thread[tl->tid].shared_tasklet_list, (struct mt_list *)&tl->list) == 1) {
_HA_ATOMIC_ADD(&tasks_run_queue, 1);
BUG/MEDIUM: tasklet: properly compute the sleeping threads mask in tasklet_wakeup() The use of ~(1 << tid) to compute the sleeping_mask in tasklet_wakeup() will result in breakage above 32 threads, because (1<<31) = 0xFFFFFFFF8000000, and upper values will lead to theorically undefined results, but practically will wrap over 0x1 to 0x80000000 again and indicate wrong sleeping masks. It seems that the main visible effect maybe extra latency on some threads or short CPU loops on others. No backport is needed.
2019-10-18 06:45:41 +00:00
if (sleeping_thread_mask & (1UL << tl->tid)) {
_HA_ATOMIC_AND(&sleeping_thread_mask, ~(1UL << tl->tid));
MEDIUM: task: Split the tasklet list into two lists. As using an mt_list for the tasklet list is costly, instead use a regular list, but add an mt_list for tasklet woken up by other threads, to be run on the current thread. At the beginning of process_runnable_tasks(), we just take the new list, and merge it into the task_list. This should give us performances comparable to before we started using a mt_list, but allow us to use tasklet_wakeup() from other threads.
2019-10-11 14:35:01 +00:00
wake_thread(tl->tid);
}
BUG/MEDIUM: tasklets: Make sure we're waking the target thread if it sleeps. Now that we can wake tasklet for other threads, make sure that if the thread is sleeping, we wake it up, or the tasklet won't be executed until it's done sleeping. That also means that, before going to sleep, and after we put our bit in sleeping_thread_mask, we have to check that nobody added a tasklet for us, just checking for global_tasks_mask isn't enough anymore.
2019-09-24 12:55:28 +00:00
}
}
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
}
BUG: tasks: fix bug introduced by latest scheduler cleanup In commit 86eded6c6 ("CLEANUP: tasks: rename task_remove_from_tasklet_list() to tasklet_remove_*") which consisted in removing the casts between tasks and tasklet, I was a bit too fast to believe that we only saw tasklets in this function since process_runnable_tasks() also uses it with tasks under a cast. So removing the bookkeeping on task_list_size was not appropriate. Bah, the joy of casts which hide the real thing... This patch does two things at once to address this mess once for all: - it restores the decrement of task_list_size when it's a real task, but moves it to process_runnable_task() since it's the only place where it's allowed to call it with a task - it moves the increment there as well and renames task_insert_into_tasklet_list() to tasklet_insert_into_tasklet_list() of obvious consistency reasons. This way the increment/decrement of task_list_size is made at the only places where the cast is enforced, so it has less risks to be missed. The comments on top of these functions were updated to reflect that they are only supposed to be used with tasklets and that the caller is responsible for keeping task_list_size up to date if it decides to enforce a task there. Now we don't have to worry anymore about how these functions work outside of the scheduler, which is better longterm-wise. Thanks to Christopher for spotting this mistake. No backport is needed.
2019-06-14 16:05:54 +00:00
/* Insert a tasklet into the tasklet list. If used with a plain task instead,
* the caller must update the task_list_size.
*/
MEDIUM: tasks: implement 3 different tasklet classes with their own queues We used to mix high latency tasks and low latency tasklets in the same list, and to even refill bulk tasklets there, causing some unfairness in certain situations (e.g. poll-less transfers between many connections saturating the machine with similarly-sized in and out network interfaces). This patch changes the mechanism to split the load into 3 lists depending on the task/tasklet's desired classes : - URGENT: this is mainly for tasklets used as deferred callbacks - NORMAL: this is for regular tasks - BULK: this is for bulk tasks/tasklets Arbitrary ratios of max_processed are picked from each of these lists in turn, with the ability to complete in one list from what was not picked in the previous one. After some quick tests, the following setup gave apparently good results both for raw TCP with splicing and for H2-to-H1 request rate: - 0 to 75% for urgent - 12 to 50% for normal - 12 to what remains for bulk Bulk is not used yet.
2020-01-30 17:37:28 +00:00
static inline void tasklet_insert_into_tasklet_list(struct list *list, struct tasklet *tl)
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
{
MEDIUM: task: Split the tasklet list into two lists. As using an mt_list for the tasklet list is costly, instead use a regular list, but add an mt_list for tasklet woken up by other threads, to be run on the current thread. At the beginning of process_runnable_tasks(), we just take the new list, and merge it into the task_list. This should give us performances comparable to before we started using a mt_list, but allow us to use tasklet_wakeup() from other threads.
2019-10-11 14:35:01 +00:00
_HA_ATOMIC_ADD(&tasks_run_queue, 1);
MEDIUM: tasks: implement 3 different tasklet classes with their own queues We used to mix high latency tasks and low latency tasklets in the same list, and to even refill bulk tasklets there, causing some unfairness in certain situations (e.g. poll-less transfers between many connections saturating the machine with similarly-sized in and out network interfaces). This patch changes the mechanism to split the load into 3 lists depending on the task/tasklet's desired classes : - URGENT: this is mainly for tasklets used as deferred callbacks - NORMAL: this is for regular tasks - BULK: this is for bulk tasks/tasklets Arbitrary ratios of max_processed are picked from each of these lists in turn, with the ability to complete in one list from what was not picked in the previous one. After some quick tests, the following setup gave apparently good results both for raw TCP with splicing and for H2-to-H1 request rate: - 0 to 75% for urgent - 12 to 50% for normal - 12 to what remains for bulk Bulk is not used yet.
2020-01-30 17:37:28 +00:00
LIST_ADDQ(list, &tl->list);
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
}
BUG: tasks: fix bug introduced by latest scheduler cleanup In commit 86eded6c6 ("CLEANUP: tasks: rename task_remove_from_tasklet_list() to tasklet_remove_*") which consisted in removing the casts between tasks and tasklet, I was a bit too fast to believe that we only saw tasklets in this function since process_runnable_tasks() also uses it with tasks under a cast. So removing the bookkeeping on task_list_size was not appropriate. Bah, the joy of casts which hide the real thing... This patch does two things at once to address this mess once for all: - it restores the decrement of task_list_size when it's a real task, but moves it to process_runnable_task() since it's the only place where it's allowed to call it with a task - it moves the increment there as well and renames task_insert_into_tasklet_list() to tasklet_insert_into_tasklet_list() of obvious consistency reasons. This way the increment/decrement of task_list_size is made at the only places where the cast is enforced, so it has less risks to be missed. The comments on top of these functions were updated to reflect that they are only supposed to be used with tasklets and that the caller is responsible for keeping task_list_size up to date if it decides to enforce a task there. Now we don't have to worry anymore about how these functions work outside of the scheduler, which is better longterm-wise. Thanks to Christopher for spotting this mistake. No backport is needed.
2019-06-14 16:05:54 +00:00
/* Remove the tasklet from the tasklet list. The tasklet MUST already be there.
* If unsure, use tasklet_remove_from_tasklet_list() instead. If used with a
* plain task, the caller must update the task_list_size.
MEDIUM: task: Split the tasklet list into two lists. As using an mt_list for the tasklet list is costly, instead use a regular list, but add an mt_list for tasklet woken up by other threads, to be run on the current thread. At the beginning of process_runnable_tasks(), we just take the new list, and merge it into the task_list. This should give us performances comparable to before we started using a mt_list, but allow us to use tasklet_wakeup() from other threads.
2019-10-11 14:35:01 +00:00
* This should only be used by the thread that owns the tasklet, any other
* thread should use tasklet_cancel().
BUG/MEDIUM: task/h2: add an idempotent task removal fucntion Previous commit 3ea351368 ("BUG/MEDIUM: h2: Remove the tasklet from the task list if unsubscribing.") uncovered an issue which needs to be addressed in the scheduler's API. The function task_remove_from_task_list() was initially designed to remove a task from the running tasklet list from within the scheduler, and had to be used in h2 to abort pending I/O events. However this function was not designed to be idempotent, occasionally causing a double removal from the tasklet list, with the second doing nothing but affecting the apparent tasks count and making haproxy use 100% CPU on some tests consisting in stopping the client during some transfers. The h2_unsubscribe() function can sometimes be called upon stream exit after an error where the tasklet was possibly already removed, so it. This patch does 2 things : - it renames task_remove_from_task_list() to __task_remove_from_tasklet_list() to discourage users from calling it. Also note the fix in the naming since it's a tasklet list and not a task list. This function is still uesd from the scheduler. - it adds a new, idempotent, task_remove_from_tasklet_list() function which does nothing if the task is already not in the tasklet list. This patch will need to be backported where the commit above is backported.
2019-03-25 17:02:54 +00:00
*/
CLEANUP: tasks: rename task_remove_from_tasklet_list() to tasklet_remove_* The function really only operates on tasklets, its arguments are always tasklets cast as tasks to match the function's type, to be cast back to a struct tasklet. Let's rename it to tasklet_remove_from_tasklet_list(), take a struct tasklet, and get rid of the undesired task casts.
2019-06-14 12:47:49 +00:00
static inline void __tasklet_remove_from_tasklet_list(struct tasklet *t)
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
{
MEDIUM: task: Split the tasklet list into two lists. As using an mt_list for the tasklet list is costly, instead use a regular list, but add an mt_list for tasklet woken up by other threads, to be run on the current thread. At the beginning of process_runnable_tasks(), we just take the new list, and merge it into the task_list. This should give us performances comparable to before we started using a mt_list, but allow us to use tasklet_wakeup() from other threads.
2019-10-11 14:35:01 +00:00
LIST_DEL_INIT(&t->list);
_HA_ATOMIC_SUB(&tasks_run_queue, 1);
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
}
CLEANUP: tasks: rename task_remove_from_tasklet_list() to tasklet_remove_* The function really only operates on tasklets, its arguments are always tasklets cast as tasks to match the function's type, to be cast back to a struct tasklet. Let's rename it to tasklet_remove_from_tasklet_list(), take a struct tasklet, and get rid of the undesired task casts.
2019-06-14 12:47:49 +00:00
static inline void tasklet_remove_from_tasklet_list(struct tasklet *t)
BUG/MEDIUM: task/h2: add an idempotent task removal fucntion Previous commit 3ea351368 ("BUG/MEDIUM: h2: Remove the tasklet from the task list if unsubscribing.") uncovered an issue which needs to be addressed in the scheduler's API. The function task_remove_from_task_list() was initially designed to remove a task from the running tasklet list from within the scheduler, and had to be used in h2 to abort pending I/O events. However this function was not designed to be idempotent, occasionally causing a double removal from the tasklet list, with the second doing nothing but affecting the apparent tasks count and making haproxy use 100% CPU on some tests consisting in stopping the client during some transfers. The h2_unsubscribe() function can sometimes be called upon stream exit after an error where the tasklet was possibly already removed, so it. This patch does 2 things : - it renames task_remove_from_task_list() to __task_remove_from_tasklet_list() to discourage users from calling it. Also note the fix in the naming since it's a tasklet list and not a task list. This function is still uesd from the scheduler. - it adds a new, idempotent, task_remove_from_tasklet_list() function which does nothing if the task is already not in the tasklet list. This patch will need to be backported where the commit above is backported.
2019-03-25 17:02:54 +00:00
{
BUG/MEDIUM: tasks: Make tasklet_remove_from_tasklet_list() no matter the tasklet. In tasklet_remove_from_tasket_list(), we can be called for a tasklet that is either in the private task list, or in the shared tasklet list. Take that into account and always use MT_LIST_DEL() to remove it, otherwise if we're in the shared list and another thread attempts to add a tasklet in it, bad things will happen. __tasklet_remove_from_tasklet_list() is left unchanged, it's only supposed to be used by process_runnable_task() to remove task/tasklets from the private tast list. This should not be backported. This should fix github issue #357.
2019-11-08 14:41:55 +00:00
if (MT_LIST_DEL((struct mt_list *)&t->list))
_HA_ATOMIC_SUB(&tasks_run_queue, 1);
BUG/MEDIUM: task/h2: add an idempotent task removal fucntion Previous commit 3ea351368 ("BUG/MEDIUM: h2: Remove the tasklet from the task list if unsubscribing.") uncovered an issue which needs to be addressed in the scheduler's API. The function task_remove_from_task_list() was initially designed to remove a task from the running tasklet list from within the scheduler, and had to be used in h2 to abort pending I/O events. However this function was not designed to be idempotent, occasionally causing a double removal from the tasklet list, with the second doing nothing but affecting the apparent tasks count and making haproxy use 100% CPU on some tests consisting in stopping the client during some transfers. The h2_unsubscribe() function can sometimes be called upon stream exit after an error where the tasklet was possibly already removed, so it. This patch does 2 things : - it renames task_remove_from_task_list() to __task_remove_from_tasklet_list() to discourage users from calling it. Also note the fix in the naming since it's a tasklet list and not a task list. This function is still uesd from the scheduler. - it adds a new, idempotent, task_remove_from_tasklet_list() function which does nothing if the task is already not in the tasklet list. This patch will need to be backported where the commit above is backported.
2019-03-25 17:02:54 +00:00
}
[MAJOR] replace ultree with ebtree in wait-queues The ultree code has been removed in favor of a simpler and cleaner ebtree implementation. The eternity queue does not need to exist anymore, and the pool_tree64 has been removed. The ebtree node is stored in the task itself. The qlist list header is still used by the run-queue, but will be able to disappear once the run-queue uses ebtree too.
2008-06-24 06:17:16 +00:00
/*
[MINOR] task: keep a task count and clean up task creators It's sometimes useful at least for statistics to keep a task count. It's easy to do by forcing the rare task creators to always use the same functions to create/destroy a task.
2009-03-21 17:13:21 +00:00
* Initialize a new task. The bare minimum is performed (queue pointers and
* state). The task is returned. This function should not be used outside of
BUG/MAJOR: task: add a new TASK_SHARED_WQ flag to fix foreing requeuing Since 1.9 with commit b20aa9eef3 ("MAJOR: tasks: create per-thread wait queues") a task bound to a single thread will not use locks when being queued or dequeued because the wait queue is assumed to be the owner thread's. But there exists a rare situation where this is not true: the health check tasks may be running on one thread waiting for a response, and may in parallel be requeued by another thread calling health_adjust() after a detecting a response error in traffic when "observe l7" is set, and "fastinter" is lower than "inter", requiring to shorten the running check's timeout. In this case, the task being requeued was present in another thread's wait queue, thus opening a race during task_unlink_wq(), and gets requeued into the calling thread's wait queue instead of the running one's, opening a second race here. This patch aims at protecting against the risk of calling task_unlink_wq() from one thread while the task is queued on another thread, hence unlocked, by introducing a new TASK_SHARED_WQ flag. This new flag indicates that a task's position in the wait queue may be adjusted by other threads than then one currently executing it. This means that such WQ manipulations must be performed under a lock. There are two types of such tasks: - the global ones, using the global wait queue (technically speaking, those whose thread_mask has at least 2 bits set). - some local ones, which for now will be placed into the global wait queue as well in order to benefit from its lock. The flag is automatically set on initialization if the task's thread mask indicates more than one thread. The caller must also set it if it intends to let other threads update the task's expiration delay (e.g. delegated I/Os), or if it intends to change the task's affinity over time as this could lead to the same situation. Right now only the situation described above seems to be affected by this issue, and it is very difficult to trigger, and even then, will often have no visible effect beyond stopping the checks for example once the race is met. On my laptop it is feasible with the following config, chained to httpterm: global maxconn 400 # provoke FD errors, calling health_adjust() defaults mode http timeout client 10s timeout server 10s timeout connect 10s listen px bind :8001 option httpchk /?t=50 server sback 127.0.0.1:8000 backup server-template s 0-999 127.0.0.1:8000 check port 8001 inter 100 fastinter 10 observe layer7 This patch will automatically address the case for the checks because check tasks are created with multiple threads bound and will get the TASK_SHARED_WQ flag set. If in the future more tasks need to rely on this (multi-threaded muxes for example) and the use of the global wait queue becomes a bottleneck again, then it should not be too difficult to place locks on the local wait queues and queue the task on its bound thread. This patch needs to be backported to 2.1, 2.0 and 1.9. It depends on previous patch "MINOR: task: only check TASK_WOKEN_ANY to decide to requeue a task". Many thanks to William Dauchy for providing detailed traces allowing to spot the problem.
2019-12-19 06:39:06 +00:00
* task_new(). If the thread mask contains more than one thread, TASK_SHARED_WQ
* is set.
[MAJOR] replace ultree with ebtree in wait-queues The ultree code has been removed in favor of a simpler and cleaner ebtree implementation. The eternity queue does not need to exist anymore, and the pool_tree64 has been removed. The ebtree node is stored in the task itself. The qlist list header is still used by the run-queue, but will be able to disappear once the run-queue uses ebtree too.
2008-06-24 06:17:16 +00:00
*/
MAJOR: threads/task: handle multithread on task scheduler 2 global locks have been added to protect, respectively, the run queue and the wait queue. And a process mask has been added on each task. Like for FDs, this mask is used to know which threads are allowed to process a task. For many tasks, all threads are granted. And this must be your first intension when you create a new task, else you have a good reason to make a task sticky on some threads. This is then the responsibility to the process callback to lock what have to be locked in the task context. Nevertheless, all tasks linked to a session must be sticky on the thread creating the session. It is important that I/O handlers processing session FDs and these tasks run on the same thread to avoid conflicts.
2017-09-27 12:59:38 +00:00
static inline struct task *task_init(struct task *t, unsigned long thread_mask)
[MAJOR] replace ultree with ebtree in wait-queues The ultree code has been removed in favor of a simpler and cleaner ebtree implementation. The eternity queue does not need to exist anymore, and the pool_tree64 has been removed. The ebtree node is stored in the task itself. The qlist list header is still used by the run-queue, but will be able to disappear once the run-queue uses ebtree too.
2008-06-24 06:17:16 +00:00
{
[OPTIM] task: don't unlink a task from a wait queue when waking it up In many situations, we wake a task on an I/O event, then queue it exactly where it was. This is a real waste because we delete/insert tasks into the wait queue for nothing. The only reason for this is that there was only one tree node in the task struct. By adding another tree node, we can have one tree for the timers (wait queue) and one tree for the priority (run queue). That way, we can have a task both in the run queue and wait queue at the same time. The wait queue now really holds timers, which is what it was designed for. The net gain is at least 1 delete/insert cycle per session, and up to 2-3 depending on the workload, since we save one cycle each time the expiration date is not changed during a wake up.
2009-03-07 16:25:21 +00:00
t->wq.node.leaf_p = NULL;
t->rq.node.leaf_p = NULL;
MAJOR: tasks: Create a per-thread runqueue. A lot of tasks are run on one thread only, so instead of having them all in the global runqueue, create a per-thread runqueue which doesn't require any locking, and add all tasks belonging to only one thread to the corresponding runqueue. The global runqueue is still used for non-local tasks, and is visited by each thread when checking its own runqueue. The nice parameter is thus used both in the global runqueue and in the local ones. The rare tasks that are bound to multiple threads will have their nice value used twice (once for the global queue, once for the thread-local one).
2018-05-18 16:38:23 +00:00
t->state = TASK_SLEEPING;
CLEANUP: threads: rename process_mask to thread_mask It was a leftover from the last cleaning session; this mask applies to threads and calling it process_mask is a bit confusing. It's the same in fd, task and applets.
2017-10-31 15:06:06 +00:00
t->thread_mask = thread_mask;
BUG/MAJOR: task: add a new TASK_SHARED_WQ flag to fix foreing requeuing Since 1.9 with commit b20aa9eef3 ("MAJOR: tasks: create per-thread wait queues") a task bound to a single thread will not use locks when being queued or dequeued because the wait queue is assumed to be the owner thread's. But there exists a rare situation where this is not true: the health check tasks may be running on one thread waiting for a response, and may in parallel be requeued by another thread calling health_adjust() after a detecting a response error in traffic when "observe l7" is set, and "fastinter" is lower than "inter", requiring to shorten the running check's timeout. In this case, the task being requeued was present in another thread's wait queue, thus opening a race during task_unlink_wq(), and gets requeued into the calling thread's wait queue instead of the running one's, opening a second race here. This patch aims at protecting against the risk of calling task_unlink_wq() from one thread while the task is queued on another thread, hence unlocked, by introducing a new TASK_SHARED_WQ flag. This new flag indicates that a task's position in the wait queue may be adjusted by other threads than then one currently executing it. This means that such WQ manipulations must be performed under a lock. There are two types of such tasks: - the global ones, using the global wait queue (technically speaking, those whose thread_mask has at least 2 bits set). - some local ones, which for now will be placed into the global wait queue as well in order to benefit from its lock. The flag is automatically set on initialization if the task's thread mask indicates more than one thread. The caller must also set it if it intends to let other threads update the task's expiration delay (e.g. delegated I/Os), or if it intends to change the task's affinity over time as this could lead to the same situation. Right now only the situation described above seems to be affected by this issue, and it is very difficult to trigger, and even then, will often have no visible effect beyond stopping the checks for example once the race is met. On my laptop it is feasible with the following config, chained to httpterm: global maxconn 400 # provoke FD errors, calling health_adjust() defaults mode http timeout client 10s timeout server 10s timeout connect 10s listen px bind :8001 option httpchk /?t=50 server sback 127.0.0.1:8000 backup server-template s 0-999 127.0.0.1:8000 check port 8001 inter 100 fastinter 10 observe layer7 This patch will automatically address the case for the checks because check tasks are created with multiple threads bound and will get the TASK_SHARED_WQ flag set. If in the future more tasks need to rely on this (multi-threaded muxes for example) and the use of the global wait queue becomes a bottleneck again, then it should not be too difficult to place locks on the local wait queues and queue the task on its bound thread. This patch needs to be backported to 2.1, 2.0 and 1.9. It depends on previous patch "MINOR: task: only check TASK_WOKEN_ANY to decide to requeue a task". Many thanks to William Dauchy for providing detailed traces allowing to spot the problem.
2019-12-19 06:39:06 +00:00
if (atleast2(thread_mask))
t->state |= TASK_SHARED_WQ;
[MEDIUM] introduce task->nice and boot access to statistics The run queue scheduler now considers task->nice to queue a task and to pick a task out of the queue. This makes it possible to boost the access to statistics (both via HTTP and UNIX socket). The UNIX socket receives twice as much a boost as the HTTP socket because it is more sensible.
2008-06-30 05:51:00 +00:00
t->nice = 0;
[MINOR] show sess: report number of calls to each task For debugging purposes, it can be useful to know how many times each task has been called.
2009-03-28 16:54:35 +00:00
t->calls = 0;
MEDIUM: tasks: collect per-task CPU time and latency Right now we measure for each task the cumulated time spent waiting for the CPU and using it. The timestamp uses a 64-bit integer to report a nanosecond-level date. This is only enabled when "profiling.tasks" is enabled, and consumes less than 1% extra CPU on x86_64 when enabled. The cumulated processing time and wait time are reported in "show sess". The task's counters are also reset when an HTTP transaction is reset since the HTTP part pretends to restart on a fresh new stream. This will make sure we always report correct numbers for each request in the logs.
2018-05-31 12:48:54 +00:00
t->call_date = 0;
t->cpu_time = 0;
t->lat_time = 0;
MINOR: task: always preinitialize the task's timeout in task_init() task_init() is called exclusively by task_new() which is the only way to create a task. Most callers set t->expire to TICK_ETERNITY, some set it to another value and a few like Lua don't set it at all as they don't need a timeout, causing random values to be used in case the task gets queued. Let's always set t->expire to TICK_ETERNITY in task_init() so that all tasks are now initialized in a clean state. This patch can be backported as it will definitely make the code more robust (at least the Lua code, possibly other places).
2017-07-24 15:52:58 +00:00
t->expire = TICK_ETERNITY;
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
return t;
}
BUG/MEDIUM: task: make tasklets either local or shared but not both at once Tasklets may be woken up to run on the calling thread or by a specific thread (the owner). But since we use a non-thread safe mechanism when the calling thread is also the for the owner, there may sometimes be collisions when two threads decide to wake the same tasklet up at the same time and one of them is the owner. This is more of a matter of usage than code, in that a tasklet usually is designed to be woken up and executed on the calling thread only (most cases) or on a specific thread. Thus it is a property of the tasklet itself as this solely depends how the code is constructed around it. This patch performs a small change to address this. By default tasklet_new() creates a "local" tasklet, which will run on the calling thread, like in 2.0. This is done by setting tl->tid to a negative value. If the caller wants the tasklet to run exclusively on a specific thread, it just has to set tl->tid, which is already what shared tasklet callers do anyway. No backport is needed.
2019-10-18 04:43:53 +00:00
/* Initialize a new tasklet. It's identified as a tasklet by ->nice=-32768. It
* is expected to run on the calling thread by default, it's up to the caller
* to change ->tid if it wants to own it.
*/
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
static inline void tasklet_init(struct tasklet *t)
{
t->nice = -32768;
t->calls = 0;
t->state = 0;
MINOR: tasklet: Set process to NULL. Some consumers expect the process to be NULL when a tasklet it created, so do so.
2018-07-19 14:02:16 +00:00
t->process = NULL;
BUG/MEDIUM: task: make tasklets either local or shared but not both at once Tasklets may be woken up to run on the calling thread or by a specific thread (the owner). But since we use a non-thread safe mechanism when the calling thread is also the for the owner, there may sometimes be collisions when two threads decide to wake the same tasklet up at the same time and one of them is the owner. This is more of a matter of usage than code, in that a tasklet usually is designed to be woken up and executed on the calling thread only (most cases) or on a specific thread. Thus it is a property of the tasklet itself as this solely depends how the code is constructed around it. This patch performs a small change to address this. By default tasklet_new() creates a "local" tasklet, which will run on the calling thread, like in 2.0. This is done by setting tl->tid to a negative value. If the caller wants the tasklet to run exclusively on a specific thread, it just has to set tl->tid, which is already what shared tasklet callers do anyway. No backport is needed.
2019-10-18 04:43:53 +00:00
t->tid = -1;
MEDIUM: task: Split the tasklet list into two lists. As using an mt_list for the tasklet list is costly, instead use a regular list, but add an mt_list for tasklet woken up by other threads, to be run on the current thread. At the beginning of process_runnable_tasks(), we just take the new list, and merge it into the task_list. This should give us performances comparable to before we started using a mt_list, but allow us to use tasklet_wakeup() from other threads.
2019-10-11 14:35:01 +00:00
LIST_INIT(&t->list);
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
}
BUG/MEDIUM: task: make tasklets either local or shared but not both at once Tasklets may be woken up to run on the calling thread or by a specific thread (the owner). But since we use a non-thread safe mechanism when the calling thread is also the for the owner, there may sometimes be collisions when two threads decide to wake the same tasklet up at the same time and one of them is the owner. This is more of a matter of usage than code, in that a tasklet usually is designed to be woken up and executed on the calling thread only (most cases) or on a specific thread. Thus it is a property of the tasklet itself as this solely depends how the code is constructed around it. This patch performs a small change to address this. By default tasklet_new() creates a "local" tasklet, which will run on the calling thread, like in 2.0. This is done by setting tl->tid to a negative value. If the caller wants the tasklet to run exclusively on a specific thread, it just has to set tl->tid, which is already what shared tasklet callers do anyway. No backport is needed.
2019-10-18 04:43:53 +00:00
/* Allocate and initialize a new tasklet, local to the thread by default. The
CLEANUP: assorted typo fixes in the code and comments These are mostly comments in the code. A few error messages were fixed and are of low enough importance not to deserve a backport. Some regtests were also fixed.
2020-03-10 07:06:11 +00:00
* caller may assign its tid if it wants to own the tasklet.
BUG/MEDIUM: task: make tasklets either local or shared but not both at once Tasklets may be woken up to run on the calling thread or by a specific thread (the owner). But since we use a non-thread safe mechanism when the calling thread is also the for the owner, there may sometimes be collisions when two threads decide to wake the same tasklet up at the same time and one of them is the owner. This is more of a matter of usage than code, in that a tasklet usually is designed to be woken up and executed on the calling thread only (most cases) or on a specific thread. Thus it is a property of the tasklet itself as this solely depends how the code is constructed around it. This patch performs a small change to address this. By default tasklet_new() creates a "local" tasklet, which will run on the calling thread, like in 2.0. This is done by setting tl->tid to a negative value. If the caller wants the tasklet to run exclusively on a specific thread, it just has to set tl->tid, which is already what shared tasklet callers do anyway. No backport is needed.
2019-10-18 04:43:53 +00:00
*/
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
static inline struct tasklet *tasklet_new(void)
{
struct tasklet *t = pool_alloc(pool_head_tasklet);
if (t) {
tasklet_init(t);
}
return t;
}
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
/*
[MINOR] task: keep a task count and clean up task creators It's sometimes useful at least for statistics to keep a task count. It's easy to do by forcing the rare task creators to always use the same functions to create/destroy a task.
2009-03-21 17:13:21 +00:00
* Allocate and initialise a new task. The new task is returned, or NULL in
* case of lack of memory. The task count is incremented. Tasks should only
* be allocated this way, and must be freed using task_free().
*/
MAJOR: threads/task: handle multithread on task scheduler 2 global locks have been added to protect, respectively, the run queue and the wait queue. And a process mask has been added on each task. Like for FDs, this mask is used to know which threads are allowed to process a task. For many tasks, all threads are granted. And this must be your first intension when you create a new task, else you have a good reason to make a task sticky on some threads. This is then the responsibility to the process callback to lock what have to be locked in the task context. Nevertheless, all tasks linked to a session must be sticky on the thread creating the session. It is important that I/O handlers processing session FDs and these tasks run on the same thread to avoid conflicts.
2017-09-27 12:59:38 +00:00
static inline struct task *task_new(unsigned long thread_mask)
[MINOR] task: keep a task count and clean up task creators It's sometimes useful at least for statistics to keep a task count. It's easy to do by forcing the rare task creators to always use the same functions to create/destroy a task.
2009-03-21 17:13:21 +00:00
{
CLEANUP: pools: rename all pool functions and pointers to remove this "2" During the migration to the second version of the pools, the new functions and pool pointers were all called "pool_something2()" and "pool2_something". Now there's no more pool v1 code and it's a real pain to still have to deal with this. Let's clean this up now by removing the "2" everywhere, and by renaming the pool heads "pool_head_something".
2017-11-24 16:34:44 +00:00
struct task *t = pool_alloc(pool_head_task);
[MINOR] task: keep a task count and clean up task creators It's sometimes useful at least for statistics to keep a task count. It's easy to do by forcing the rare task creators to always use the same functions to create/destroy a task.
2009-03-21 17:13:21 +00:00
if (t) {
MEDIUM: task: Use the new _HA_ATOMIC_* macros. Use the new _HA_ATOMIC_* macros and add barriers where needed.
2019-03-08 17:48:47 +00:00
_HA_ATOMIC_ADD(&nb_tasks, 1);
MAJOR: threads/task: handle multithread on task scheduler 2 global locks have been added to protect, respectively, the run queue and the wait queue. And a process mask has been added on each task. Like for FDs, this mask is used to know which threads are allowed to process a task. For many tasks, all threads are granted. And this must be your first intension when you create a new task, else you have a good reason to make a task sticky on some threads. This is then the responsibility to the process callback to lock what have to be locked in the task context. Nevertheless, all tasks linked to a session must be sticky on the thread creating the session. It is important that I/O handlers processing session FDs and these tasks run on the same thread to avoid conflicts.
2017-09-27 12:59:38 +00:00
task_init(t, thread_mask);
[MINOR] task: keep a task count and clean up task creators It's sometimes useful at least for statistics to keep a task count. It's easy to do by forcing the rare task creators to always use the same functions to create/destroy a task.
2009-03-21 17:13:21 +00:00
}
return t;
}
/*
MINOR: task: always reset curr_task when freeing a task or tasklet With the thread debugger it becomes visible that we can leave some wandering pointers for a while in curr_task, which is inappropriate. This patch addresses this by resetting curr_task to NULL before really freeing the area. This way it becomes safe even regarding signals.
2019-05-17 12:16:51 +00:00
* Free a task. Its context must have been freed since it will be lost. The
* task count is decremented. It it is the current task, this one is reset.
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
*/
BUG/MEDIUM: task: Don't free a task that is about to be run. While running a task, we may try to delete and free a task that is about to be run, because it's part of the local tasks list, or because rq_next points to it. So flag any task that is in the local tasks list to be deleted, instead of run, by setting t->process to NULL, and re-make rq_next a global, thread-local variable, that is modified if we attempt to delete that task. Many thanks to PiBa-NL for reporting this and analysing the problem. This should be backported to 1.8.
2018-05-04 13:46:16 +00:00
static inline void __task_free(struct task *t)
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
{
MINOR: task: introduce a thread-local "sched" variable for local scheduler stuff The aim is to rassemble all scheduler information related to the current thread. It simply points to task_per_thread[tid] without having to perform the operation at each time. We save around 1.2 kB of code on performance sensitive paths and increase the request rate by almost 1%.
2019-09-24 06:25:15 +00:00
if (t == sched->current) {
sched->current = NULL;
MINOR: task: always reset curr_task when freeing a task or tasklet With the thread debugger it becomes visible that we can leave some wandering pointers for a while in curr_task, which is inappropriate. This patch addresses this by resetting curr_task to NULL before really freeing the area. This way it becomes safe even regarding signals.
2019-05-17 12:16:51 +00:00
__ha_barrier_store();
}
CLEANUP: pools: rename all pool functions and pointers to remove this "2" During the migration to the second version of the pools, the new functions and pool pointers were all called "pool_something2()" and "pool2_something". Now there's no more pool v1 code and it's a real pain to still have to deal with this. Let's clean this up now by removing the "2" everywhere, and by renaming the pool heads "pool_head_something".
2017-11-24 16:34:44 +00:00
pool_free(pool_head_task, t);
MINOR: task: release the task pool when stopping When we're stopping, we're not going to create new tasks anymore, so let's release the task pool upon each task_free() in order to reduce memory fragmentation.
2014-11-13 15:57:19 +00:00
if (unlikely(stopping))
CLEANUP: pools: rename all pool functions and pointers to remove this "2" During the migration to the second version of the pools, the new functions and pool pointers were all called "pool_something2()" and "pool2_something". Now there's no more pool v1 code and it's a real pain to still have to deal with this. Let's clean this up now by removing the "2" everywhere, and by renaming the pool heads "pool_head_something".
2017-11-24 16:34:44 +00:00
pool_flush(pool_head_task);
MEDIUM: task: Use the new _HA_ATOMIC_* macros. Use the new _HA_ATOMIC_* macros and add barriers where needed.
2019-03-08 17:48:47 +00:00
_HA_ATOMIC_SUB(&nb_tasks, 1);
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
}
CLEANUP: task: remove unneeded tests before task_destroy() Since previous commit it's not needed anymore to test a task pointer before calling task_destory() so let's just remove these tests from the various callers before they become confusing. The function's arguments were also documented. The same should probably be done with tasklet_free() which involves a test in roughly half of the call places.
2019-05-07 17:05:35 +00:00
/* Destroys a task : it's unlinked from the wait queues and is freed if it's
* the current task or not queued otherwise it's marked to be freed by the
* scheduler. It does nothing if <t> is NULL.
*/
MEDIUM: tasks: Merge task_delete() and task_free() into task_destroy(). task_delete() was never used without calling task_free() just after, and task_free() was only used on error pathes to destroy a just-created task, so merge them into task_destroy(), that will remove the task from the wait queue, and make sure the task is either destroyed immediately if it's not in the run queue, or destroyed when it's supposed to run.
2019-04-17 20:51:06 +00:00
static inline void task_destroy(struct task *t)
BUG/MEDIUM: task: Don't free a task that is about to be run. While running a task, we may try to delete and free a task that is about to be run, because it's part of the local tasks list, or because rq_next points to it. So flag any task that is in the local tasks list to be deleted, instead of run, by setting t->process to NULL, and re-make rq_next a global, thread-local variable, that is modified if we attempt to delete that task. Many thanks to PiBa-NL for reporting this and analysing the problem. This should be backported to 1.8.
2018-05-04 13:46:16 +00:00
{
BUG/MEDIUM: tasks: fix possible segfault on task_destroy() Commit 3f795f7 ("MEDIUM: tasks: Merge task_delete() and task_free() into task_destroy().") replaced task_delete() and task_free() with a single function named task_destroy(). This patch adds a check for struct task* argument in function task_destroy() to prevent a possible segfault on NULL and also to make the function safer for use in other cases.
2019-05-07 13:25:25 +00:00
if (!t)
return;
MEDIUM: tasks: Merge task_delete() and task_free() into task_destroy(). task_delete() was never used without calling task_free() just after, and task_free() was only used on error pathes to destroy a just-created task, so merge them into task_destroy(), that will remove the task from the wait queue, and make sure the task is either destroyed immediately if it's not in the run queue, or destroyed when it's supposed to run.
2019-04-17 20:51:06 +00:00
task_unlink_wq(t);
CLEANUP: assorted typo fixes in the code and comments These are mostly comments in the code. A few error messages were fixed and are of low enough importance not to deserve a backport. Some regtests were also fixed.
2020-03-10 07:06:11 +00:00
/* We don't have to explicitly remove from the run queue.
MEDIUM: tasks: Merge task_delete() and task_free() into task_destroy(). task_delete() was never used without calling task_free() just after, and task_free() was only used on error pathes to destroy a just-created task, so merge them into task_destroy(), that will remove the task from the wait queue, and make sure the task is either destroyed immediately if it's not in the run queue, or destroyed when it's supposed to run.
2019-04-17 20:51:06 +00:00
* If we are in the runqueue, the test below will set t->process
* to NULL, and the task will be free'd when it'll be its turn
* to run.
*/
BUG/MEDIUM: task: Don't free a task that is about to be run. While running a task, we may try to delete and free a task that is about to be run, because it's part of the local tasks list, or because rq_next points to it. So flag any task that is in the local tasks list to be deleted, instead of run, by setting t->process to NULL, and re-make rq_next a global, thread-local variable, that is modified if we attempt to delete that task. Many thanks to PiBa-NL for reporting this and analysing the problem. This should be backported to 1.8.
2018-05-04 13:46:16 +00:00
/* There's no need to protect t->state with a lock, as the task
* has to run on the current thread.
*/
MINOR: task: introduce a thread-local "sched" variable for local scheduler stuff The aim is to rassemble all scheduler information related to the current thread. It simply points to task_per_thread[tid] without having to perform the operation at each time. We save around 1.2 kB of code on performance sensitive paths and increase the request rate by almost 1%.
2019-09-24 06:25:15 +00:00
if (t == sched->current || !(t->state & (TASK_QUEUED | TASK_RUNNING)))
BUG/MEDIUM: task: Don't free a task that is about to be run. While running a task, we may try to delete and free a task that is about to be run, because it's part of the local tasks list, or because rq_next points to it. So flag any task that is in the local tasks list to be deleted, instead of run, by setting t->process to NULL, and re-make rq_next a global, thread-local variable, that is modified if we attempt to delete that task. Many thanks to PiBa-NL for reporting this and analysing the problem. This should be backported to 1.8.
2018-05-04 13:46:16 +00:00
__task_free(t);
else
t->process = NULL;
}
MEDIUM: task: Split the tasklet list into two lists. As using an mt_list for the tasklet list is costly, instead use a regular list, but add an mt_list for tasklet woken up by other threads, to be run on the current thread. At the beginning of process_runnable_tasks(), we just take the new list, and merge it into the task_list. This should give us performances comparable to before we started using a mt_list, but allow us to use tasklet_wakeup() from other threads.
2019-10-11 14:35:01 +00:00
/* Should only be called by the thread responsible for the tasklet */
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
static inline void tasklet_free(struct tasklet *tl)
{
BUG/MEDIUM: tasks: Use the MT macros in tasklet_free(). In tasklet_free(), to attempt to remove ourself, use MT_LIST_DEL, we can't just use LIST_DEL(), as we theorically could be in the shared tasklet list. This should be backported to 2.1.
2020-01-10 15:46:48 +00:00
if (MT_LIST_DEL((struct mt_list *)&tl->list))
MEDIUM: task: Split the tasklet list into two lists. As using an mt_list for the tasklet list is costly, instead use a regular list, but add an mt_list for tasklet woken up by other threads, to be run on the current thread. At the beginning of process_runnable_tasks(), we just take the new list, and merge it into the task_list. This should give us performances comparable to before we started using a mt_list, but allow us to use tasklet_wakeup() from other threads.
2019-10-11 14:35:01 +00:00
_HA_ATOMIC_SUB(&tasks_run_queue, 1);
MINOR: tasks: Make sure we correctly init and deinit a tasklet. Up until now, a tasklet couldn't be free'd while it was in the list, it is no longer the case, so make sure we remove it from the list before freeing it. To do so, we have to make sure we correctly initialize it, so use LIST_INIT, instead of setting the pointers to NULL.
2018-06-08 15:08:19 +00:00
MAJOR: tasks: Introduce tasklets. Introduce tasklets, lightweight tasks. They have no notion of priority, they are just run as soon as possible, and will probably be used for I/O later. For the moment they're used to replace the temporary thread-local list that was used in the scheduler. The first part of the struct is common with tasks so that tasks can be cast to tasklets and queued in this list. Once a task is in the tasklet list, it has its leaf_p set to 0x1 so that it cannot accidently be confused as not in the queue. Pure tasklets are identifiable by their nice value of -32768 (which is normally not possible).
2018-05-18 16:45:28 +00:00
pool_free(pool_head_tasklet, tl);
if (unlikely(stopping))
pool_flush(pool_head_tasklet);
}
MEDIUM: tasklets: Make the tasklet list a struct mt_list. Change the tasklet code so that the tasklet list is now a mt_list. That means that tasklet now do have an associated tid, for the thread it is expected to run on, and any thread can now call tasklet_wakeup() for that tasklet. One can change the associated tid with tasklet_set_tid().
2019-09-20 15:18:35 +00:00
static inline void tasklet_set_tid(struct tasklet *tl, int tid)
{
tl->tid = tid;
}
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
void __task_queue(struct task *task, struct eb_root *wq);
[OPTIM] task: don't unlink a task from a wait queue when waking it up In many situations, we wake a task on an I/O event, then queue it exactly where it was. This is a real waste because we delete/insert tasks into the wait queue for nothing. The only reason for this is that there was only one tree node in the task struct. By adding another tree node, we can have one tree for the timers (wait queue) and one tree for the priority (run queue). That way, we can have a task both in the run queue and wait queue at the same time. The wait queue now really holds timers, which is what it was designed for. The net gain is at least 1 delete/insert cycle per session, and up to 2-3 depending on the workload, since we save one cycle each time the expiration date is not changed during a wake up.
2009-03-07 16:25:21 +00:00
/* Place <task> into the wait queue, where it may already be. If the expiration
[OPTIM] task: reduce the number of calls to task_queue() Most of the time, task_queue() will immediately return. By extracting the preliminary checks and putting them in an inline function, we can significantly reduce the number of calls to the function itself, and most of the tests can be optimized away due to the caller's context. Another minor improvement in process_runnable_tasks() consisted in taking benefit from the processor's branch prediction unit by making a special case of the process_session() callback which is by far the most common one. All this improved performance by about 1%, mainly during the call from process_runnable_tasks().
2009-03-08 15:35:27 +00:00
* timer is infinite, do nothing and rely on wake_expired_task to clean up.
BUG/MAJOR: task: add a new TASK_SHARED_WQ flag to fix foreing requeuing Since 1.9 with commit b20aa9eef3 ("MAJOR: tasks: create per-thread wait queues") a task bound to a single thread will not use locks when being queued or dequeued because the wait queue is assumed to be the owner thread's. But there exists a rare situation where this is not true: the health check tasks may be running on one thread waiting for a response, and may in parallel be requeued by another thread calling health_adjust() after a detecting a response error in traffic when "observe l7" is set, and "fastinter" is lower than "inter", requiring to shorten the running check's timeout. In this case, the task being requeued was present in another thread's wait queue, thus opening a race during task_unlink_wq(), and gets requeued into the calling thread's wait queue instead of the running one's, opening a second race here. This patch aims at protecting against the risk of calling task_unlink_wq() from one thread while the task is queued on another thread, hence unlocked, by introducing a new TASK_SHARED_WQ flag. This new flag indicates that a task's position in the wait queue may be adjusted by other threads than then one currently executing it. This means that such WQ manipulations must be performed under a lock. There are two types of such tasks: - the global ones, using the global wait queue (technically speaking, those whose thread_mask has at least 2 bits set). - some local ones, which for now will be placed into the global wait queue as well in order to benefit from its lock. The flag is automatically set on initialization if the task's thread mask indicates more than one thread. The caller must also set it if it intends to let other threads update the task's expiration delay (e.g. delegated I/Os), or if it intends to change the task's affinity over time as this could lead to the same situation. Right now only the situation described above seems to be affected by this issue, and it is very difficult to trigger, and even then, will often have no visible effect beyond stopping the checks for example once the race is met. On my laptop it is feasible with the following config, chained to httpterm: global maxconn 400 # provoke FD errors, calling health_adjust() defaults mode http timeout client 10s timeout server 10s timeout connect 10s listen px bind :8001 option httpchk /?t=50 server sback 127.0.0.1:8000 backup server-template s 0-999 127.0.0.1:8000 check port 8001 inter 100 fastinter 10 observe layer7 This patch will automatically address the case for the checks because check tasks are created with multiple threads bound and will get the TASK_SHARED_WQ flag set. If in the future more tasks need to rely on this (multi-threaded muxes for example) and the use of the global wait queue becomes a bottleneck again, then it should not be too difficult to place locks on the local wait queues and queue the task on its bound thread. This patch needs to be backported to 2.1, 2.0 and 1.9. It depends on previous patch "MINOR: task: only check TASK_WOKEN_ANY to decide to requeue a task". Many thanks to William Dauchy for providing detailed traces allowing to spot the problem.
2019-12-19 06:39:06 +00:00
* If the task uses a shared wait queue, it's queued into the global wait queue,
* protected by the global wq_lock, otherwise by it necessarily belongs to the
* current thread'sand is queued without locking.
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
*/
[OPTIM] task: reduce the number of calls to task_queue() Most of the time, task_queue() will immediately return. By extracting the preliminary checks and putting them in an inline function, we can significantly reduce the number of calls to the function itself, and most of the tests can be optimized away due to the caller's context. Another minor improvement in process_runnable_tasks() consisted in taking benefit from the processor's branch prediction unit by making a special case of the process_session() callback which is by far the most common one. All this improved performance by about 1%, mainly during the call from process_runnable_tasks().
2009-03-08 15:35:27 +00:00
static inline void task_queue(struct task *task)
{
/* If we already have a place in the wait queue no later than the
* timeout we're trying to set, we'll stay there, because it is very
* unlikely that we will reach the timeout anyway. If the timeout
* has been disabled, it's useless to leave the queue as well. We'll
* rely on wake_expired_tasks() to catch the node and move it to the
* proper place should it ever happen. Finally we only add the task
* to the queue if it was not there or if it was further than what
* we want.
*/
if (!tick_isset(task->expire))
return;
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
#ifdef USE_THREAD
BUG/MAJOR: task: add a new TASK_SHARED_WQ flag to fix foreing requeuing Since 1.9 with commit b20aa9eef3 ("MAJOR: tasks: create per-thread wait queues") a task bound to a single thread will not use locks when being queued or dequeued because the wait queue is assumed to be the owner thread's. But there exists a rare situation where this is not true: the health check tasks may be running on one thread waiting for a response, and may in parallel be requeued by another thread calling health_adjust() after a detecting a response error in traffic when "observe l7" is set, and "fastinter" is lower than "inter", requiring to shorten the running check's timeout. In this case, the task being requeued was present in another thread's wait queue, thus opening a race during task_unlink_wq(), and gets requeued into the calling thread's wait queue instead of the running one's, opening a second race here. This patch aims at protecting against the risk of calling task_unlink_wq() from one thread while the task is queued on another thread, hence unlocked, by introducing a new TASK_SHARED_WQ flag. This new flag indicates that a task's position in the wait queue may be adjusted by other threads than then one currently executing it. This means that such WQ manipulations must be performed under a lock. There are two types of such tasks: - the global ones, using the global wait queue (technically speaking, those whose thread_mask has at least 2 bits set). - some local ones, which for now will be placed into the global wait queue as well in order to benefit from its lock. The flag is automatically set on initialization if the task's thread mask indicates more than one thread. The caller must also set it if it intends to let other threads update the task's expiration delay (e.g. delegated I/Os), or if it intends to change the task's affinity over time as this could lead to the same situation. Right now only the situation described above seems to be affected by this issue, and it is very difficult to trigger, and even then, will often have no visible effect beyond stopping the checks for example once the race is met. On my laptop it is feasible with the following config, chained to httpterm: global maxconn 400 # provoke FD errors, calling health_adjust() defaults mode http timeout client 10s timeout server 10s timeout connect 10s listen px bind :8001 option httpchk /?t=50 server sback 127.0.0.1:8000 backup server-template s 0-999 127.0.0.1:8000 check port 8001 inter 100 fastinter 10 observe layer7 This patch will automatically address the case for the checks because check tasks are created with multiple threads bound and will get the TASK_SHARED_WQ flag set. If in the future more tasks need to rely on this (multi-threaded muxes for example) and the use of the global wait queue becomes a bottleneck again, then it should not be too difficult to place locks on the local wait queues and queue the task on its bound thread. This patch needs to be backported to 2.1, 2.0 and 1.9. It depends on previous patch "MINOR: task: only check TASK_WOKEN_ANY to decide to requeue a task". Many thanks to William Dauchy for providing detailed traces allowing to spot the problem.
2019-12-19 06:39:06 +00:00
if (task->state & TASK_SHARED_WQ) {
MINOR: task: turn the WQ lock to an RW_LOCK For now it's exclusively used as a write lock though, thus it remains 100% equivalent to the spinlock it replaces.
2019-05-28 16:48:07 +00:00
HA_RWLOCK_WRLOCK(TASK_WQ_LOCK, &wq_lock);
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
if (!task_in_wq(task) || tick_is_lt(task->expire, task->wq.key))
__task_queue(task, &timers);
MINOR: task: turn the WQ lock to an RW_LOCK For now it's exclusively used as a write lock though, thus it remains 100% equivalent to the spinlock it replaces.
2019-05-28 16:48:07 +00:00
HA_RWLOCK_WRUNLOCK(TASK_WQ_LOCK, &wq_lock);
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
} else
#endif
{
BUG/MAJOR: task: add a new TASK_SHARED_WQ flag to fix foreing requeuing Since 1.9 with commit b20aa9eef3 ("MAJOR: tasks: create per-thread wait queues") a task bound to a single thread will not use locks when being queued or dequeued because the wait queue is assumed to be the owner thread's. But there exists a rare situation where this is not true: the health check tasks may be running on one thread waiting for a response, and may in parallel be requeued by another thread calling health_adjust() after a detecting a response error in traffic when "observe l7" is set, and "fastinter" is lower than "inter", requiring to shorten the running check's timeout. In this case, the task being requeued was present in another thread's wait queue, thus opening a race during task_unlink_wq(), and gets requeued into the calling thread's wait queue instead of the running one's, opening a second race here. This patch aims at protecting against the risk of calling task_unlink_wq() from one thread while the task is queued on another thread, hence unlocked, by introducing a new TASK_SHARED_WQ flag. This new flag indicates that a task's position in the wait queue may be adjusted by other threads than then one currently executing it. This means that such WQ manipulations must be performed under a lock. There are two types of such tasks: - the global ones, using the global wait queue (technically speaking, those whose thread_mask has at least 2 bits set). - some local ones, which for now will be placed into the global wait queue as well in order to benefit from its lock. The flag is automatically set on initialization if the task's thread mask indicates more than one thread. The caller must also set it if it intends to let other threads update the task's expiration delay (e.g. delegated I/Os), or if it intends to change the task's affinity over time as this could lead to the same situation. Right now only the situation described above seems to be affected by this issue, and it is very difficult to trigger, and even then, will often have no visible effect beyond stopping the checks for example once the race is met. On my laptop it is feasible with the following config, chained to httpterm: global maxconn 400 # provoke FD errors, calling health_adjust() defaults mode http timeout client 10s timeout server 10s timeout connect 10s listen px bind :8001 option httpchk /?t=50 server sback 127.0.0.1:8000 backup server-template s 0-999 127.0.0.1:8000 check port 8001 inter 100 fastinter 10 observe layer7 This patch will automatically address the case for the checks because check tasks are created with multiple threads bound and will get the TASK_SHARED_WQ flag set. If in the future more tasks need to rely on this (multi-threaded muxes for example) and the use of the global wait queue becomes a bottleneck again, then it should not be too difficult to place locks on the local wait queues and queue the task on its bound thread. This patch needs to be backported to 2.1, 2.0 and 1.9. It depends on previous patch "MINOR: task: only check TASK_WOKEN_ANY to decide to requeue a task". Many thanks to William Dauchy for providing detailed traces allowing to spot the problem.
2019-12-19 06:39:06 +00:00
BUG_ON((task->thread_mask & tid_bit) == 0); // should have TASK_SHARED_WQ
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
if (!task_in_wq(task) || tick_is_lt(task->expire, task->wq.key))
MINOR: task: introduce a thread-local "sched" variable for local scheduler stuff The aim is to rassemble all scheduler information related to the current thread. It simply points to task_per_thread[tid] without having to perform the operation at each time. We save around 1.2 kB of code on performance sensitive paths and increase the request rate by almost 1%.
2019-09-24 06:25:15 +00:00
__task_queue(task, &sched->timers);
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
}
[OPTIM] task: reduce the number of calls to task_queue() Most of the time, task_queue() will immediately return. By extracting the preliminary checks and putting them in an inline function, we can significantly reduce the number of calls to the function itself, and most of the tests can be optimized away due to the caller's context. Another minor improvement in process_runnable_tasks() consisted in taking benefit from the processor's branch prediction unit by making a special case of the process_session() callback which is by far the most common one. All this improved performance by about 1%, mainly during the call from process_runnable_tasks().
2009-03-08 15:35:27 +00:00
}
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
[MINOR] task: new function task_schedule() to schedule a wake up This function is used when a task should be woken up at most at a given date. This will be used with rate shapers.
2011-07-25 12:30:42 +00:00
/* Ensure <task> will be woken up at most at <when>. If the task is already in
* the run queue (but not running), nothing is done. It may be used that way
* with a delay : task_schedule(task, tick_add(now_ms, delay));
*/
static inline void task_schedule(struct task *task, int when)
{
MAJOR: threads/task: handle multithread on task scheduler 2 global locks have been added to protect, respectively, the run queue and the wait queue. And a process mask has been added on each task. Like for FDs, this mask is used to know which threads are allowed to process a task. For many tasks, all threads are granted. And this must be your first intension when you create a new task, else you have a good reason to make a task sticky on some threads. This is then the responsibility to the process callback to lock what have to be locked in the task context. Nevertheless, all tasks linked to a session must be sticky on the thread creating the session. It is important that I/O handlers processing session FDs and these tasks run on the same thread to avoid conflicts.
2017-09-27 12:59:38 +00:00
/* TODO: mthread, check if there is no tisk with this test */
[MINOR] task: new function task_schedule() to schedule a wake up This function is used when a task should be woken up at most at a given date. This will be used with rate shapers.
2011-07-25 12:30:42 +00:00
if (task_in_rq(task))
return;
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
#ifdef USE_THREAD
BUG/MAJOR: task: add a new TASK_SHARED_WQ flag to fix foreing requeuing Since 1.9 with commit b20aa9eef3 ("MAJOR: tasks: create per-thread wait queues") a task bound to a single thread will not use locks when being queued or dequeued because the wait queue is assumed to be the owner thread's. But there exists a rare situation where this is not true: the health check tasks may be running on one thread waiting for a response, and may in parallel be requeued by another thread calling health_adjust() after a detecting a response error in traffic when "observe l7" is set, and "fastinter" is lower than "inter", requiring to shorten the running check's timeout. In this case, the task being requeued was present in another thread's wait queue, thus opening a race during task_unlink_wq(), and gets requeued into the calling thread's wait queue instead of the running one's, opening a second race here. This patch aims at protecting against the risk of calling task_unlink_wq() from one thread while the task is queued on another thread, hence unlocked, by introducing a new TASK_SHARED_WQ flag. This new flag indicates that a task's position in the wait queue may be adjusted by other threads than then one currently executing it. This means that such WQ manipulations must be performed under a lock. There are two types of such tasks: - the global ones, using the global wait queue (technically speaking, those whose thread_mask has at least 2 bits set). - some local ones, which for now will be placed into the global wait queue as well in order to benefit from its lock. The flag is automatically set on initialization if the task's thread mask indicates more than one thread. The caller must also set it if it intends to let other threads update the task's expiration delay (e.g. delegated I/Os), or if it intends to change the task's affinity over time as this could lead to the same situation. Right now only the situation described above seems to be affected by this issue, and it is very difficult to trigger, and even then, will often have no visible effect beyond stopping the checks for example once the race is met. On my laptop it is feasible with the following config, chained to httpterm: global maxconn 400 # provoke FD errors, calling health_adjust() defaults mode http timeout client 10s timeout server 10s timeout connect 10s listen px bind :8001 option httpchk /?t=50 server sback 127.0.0.1:8000 backup server-template s 0-999 127.0.0.1:8000 check port 8001 inter 100 fastinter 10 observe layer7 This patch will automatically address the case for the checks because check tasks are created with multiple threads bound and will get the TASK_SHARED_WQ flag set. If in the future more tasks need to rely on this (multi-threaded muxes for example) and the use of the global wait queue becomes a bottleneck again, then it should not be too difficult to place locks on the local wait queues and queue the task on its bound thread. This patch needs to be backported to 2.1, 2.0 and 1.9. It depends on previous patch "MINOR: task: only check TASK_WOKEN_ANY to decide to requeue a task". Many thanks to William Dauchy for providing detailed traces allowing to spot the problem.
2019-12-19 06:39:06 +00:00
if (task->state & TASK_SHARED_WQ) {
MINOR: task: turn the WQ lock to an RW_LOCK For now it's exclusively used as a write lock though, thus it remains 100% equivalent to the spinlock it replaces.
2019-05-28 16:48:07 +00:00
/* FIXME: is it really needed to lock the WQ during the check ? */
HA_RWLOCK_WRLOCK(TASK_WQ_LOCK, &wq_lock);
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
if (task_in_wq(task))
when = tick_first(when, task->expire);
task->expire = when;
if (!task_in_wq(task) || tick_is_lt(task->expire, task->wq.key))
__task_queue(task, &timers);
MINOR: task: turn the WQ lock to an RW_LOCK For now it's exclusively used as a write lock though, thus it remains 100% equivalent to the spinlock it replaces.
2019-05-28 16:48:07 +00:00
HA_RWLOCK_WRUNLOCK(TASK_WQ_LOCK, &wq_lock);
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
} else
#endif
{
BUG/MAJOR: task: add a new TASK_SHARED_WQ flag to fix foreing requeuing Since 1.9 with commit b20aa9eef3 ("MAJOR: tasks: create per-thread wait queues") a task bound to a single thread will not use locks when being queued or dequeued because the wait queue is assumed to be the owner thread's. But there exists a rare situation where this is not true: the health check tasks may be running on one thread waiting for a response, and may in parallel be requeued by another thread calling health_adjust() after a detecting a response error in traffic when "observe l7" is set, and "fastinter" is lower than "inter", requiring to shorten the running check's timeout. In this case, the task being requeued was present in another thread's wait queue, thus opening a race during task_unlink_wq(), and gets requeued into the calling thread's wait queue instead of the running one's, opening a second race here. This patch aims at protecting against the risk of calling task_unlink_wq() from one thread while the task is queued on another thread, hence unlocked, by introducing a new TASK_SHARED_WQ flag. This new flag indicates that a task's position in the wait queue may be adjusted by other threads than then one currently executing it. This means that such WQ manipulations must be performed under a lock. There are two types of such tasks: - the global ones, using the global wait queue (technically speaking, those whose thread_mask has at least 2 bits set). - some local ones, which for now will be placed into the global wait queue as well in order to benefit from its lock. The flag is automatically set on initialization if the task's thread mask indicates more than one thread. The caller must also set it if it intends to let other threads update the task's expiration delay (e.g. delegated I/Os), or if it intends to change the task's affinity over time as this could lead to the same situation. Right now only the situation described above seems to be affected by this issue, and it is very difficult to trigger, and even then, will often have no visible effect beyond stopping the checks for example once the race is met. On my laptop it is feasible with the following config, chained to httpterm: global maxconn 400 # provoke FD errors, calling health_adjust() defaults mode http timeout client 10s timeout server 10s timeout connect 10s listen px bind :8001 option httpchk /?t=50 server sback 127.0.0.1:8000 backup server-template s 0-999 127.0.0.1:8000 check port 8001 inter 100 fastinter 10 observe layer7 This patch will automatically address the case for the checks because check tasks are created with multiple threads bound and will get the TASK_SHARED_WQ flag set. If in the future more tasks need to rely on this (multi-threaded muxes for example) and the use of the global wait queue becomes a bottleneck again, then it should not be too difficult to place locks on the local wait queues and queue the task on its bound thread. This patch needs to be backported to 2.1, 2.0 and 1.9. It depends on previous patch "MINOR: task: only check TASK_WOKEN_ANY to decide to requeue a task". Many thanks to William Dauchy for providing detailed traces allowing to spot the problem.
2019-12-19 06:39:06 +00:00
BUG_ON((task->thread_mask & tid_bit) == 0); // should have TASK_SHARED_WQ
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
if (task_in_wq(task))
when = tick_first(when, task->expire);
[MINOR] task: new function task_schedule() to schedule a wake up This function is used when a task should be woken up at most at a given date. This will be used with rate shapers.
2011-07-25 12:30:42 +00:00
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
task->expire = when;
if (!task_in_wq(task) || tick_is_lt(task->expire, task->wq.key))
MINOR: task: introduce a thread-local "sched" variable for local scheduler stuff The aim is to rassemble all scheduler information related to the current thread. It simply points to task_per_thread[tid] without having to perform the operation at each time. We save around 1.2 kB of code on performance sensitive paths and increase the request rate by almost 1%.
2019-09-24 06:25:15 +00:00
__task_queue(task, &sched->timers);
MAJOR: tasks: create per-thread wait queues Now we still have a main contention point with the timers in the main wait queue, but the vast majority of the tasks are pinned to a single thread. This patch creates a per-thread wait queue and queues a task to the local wait queue without any locking if the task is bound to a single thread (the current one) otherwise to the shared queue using locking. This significantly reduces contention on the wait queue. A test with 12 threads showed 11 ms spent in the WQ lock compared to 4.7 seconds in the same test without this change. The cache miss ratio decreased from 19.7% to 19.2% on the 12-thread test, and its performance increased by 1.5%. Another indirect benefit is that the average queue size is divided by the number of threads, which roughly removes log(nbthreads) levels in the tree and further speeds up lookups.
2018-10-15 12:52:21 +00:00
}
[MINOR] task: new function task_schedule() to schedule a wake up This function is used when a task should be woken up at most at a given date. This will be used with rate shapers.
2011-07-25 12:30:42 +00:00
}
MINOR: tasks: Move Lua notification from Lua to tasks These notification management function and structs are generic and it will be better to move in common parts. The notification management functions and structs have names containing some "lua" references because it was written for the Lua. This patch removes also these references.
2017-07-12 12:31:10 +00:00
/* This function register a new signal. "lua" is the current lua
* execution context. It contains a pointer to the associated task.
* "link" is a list head attached to an other task that must be wake
* the lua task if an event occurs. This is useful with external
CLEANUP: assorted typo fixes in the code and comments These are mostly comments in the code. A few error messages were fixed and are of low enough importance not to deserve a backport. Some regtests were also fixed.
2020-03-10 07:06:11 +00:00
* events like TCP I/O or sleep functions. This function allocate
MINOR: tasks: Move Lua notification from Lua to tasks These notification management function and structs are generic and it will be better to move in common parts. The notification management functions and structs have names containing some "lua" references because it was written for the Lua. This patch removes also these references.
2017-07-12 12:31:10 +00:00
* memory for the signal.
*/
static inline struct notification *notification_new(struct list *purge, struct list *event, struct task *wakeup)
{
CLEANUP: pools: rename all pool functions and pointers to remove this "2" During the migration to the second version of the pools, the new functions and pool pointers were all called "pool_something2()" and "pool2_something". Now there's no more pool v1 code and it's a real pain to still have to deal with this. Let's clean this up now by removing the "2" everywhere, and by renaming the pool heads "pool_head_something".
2017-11-24 16:34:44 +00:00
struct notification *com = pool_alloc(pool_head_notification);
MINOR: tasks: Move Lua notification from Lua to tasks These notification management function and structs are generic and it will be better to move in common parts. The notification management functions and structs have names containing some "lua" references because it was written for the Lua. This patch removes also these references.
2017-07-12 12:31:10 +00:00
if (!com)
return NULL;
LIST_ADDQ(purge, &com->purge_me);
LIST_ADDQ(event, &com->wake_me);
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 09:42:54 +00:00
HA_SPIN_INIT(&com->lock);
MINOR: tasks: Move Lua notification from Lua to tasks These notification management function and structs are generic and it will be better to move in common parts. The notification management functions and structs have names containing some "lua" references because it was written for the Lua. This patch removes also these references.
2017-07-12 12:31:10 +00:00
com->task = wakeup;
return com;
}
/* This function purge all the pending signals when the LUA execution
* is finished. This prevent than a coprocess try to wake a deleted
* task. This function remove the memory associated to the signal.
DOC: notifications: add precisions about thread usage Precise the terms of use the notification functions.
2017-12-10 16:14:07 +00:00
* The purge list is not locked because it is owned by only one
* process. before browsing this list, the caller must ensure to be
* the only one browser.
MINOR: tasks: Move Lua notification from Lua to tasks These notification management function and structs are generic and it will be better to move in common parts. The notification management functions and structs have names containing some "lua" references because it was written for the Lua. This patch removes also these references.
2017-07-12 12:31:10 +00:00
*/
static inline void notification_purge(struct list *purge)
{
struct notification *com, *back;
/* Delete all pending communication signals. */
list_for_each_entry_safe(com, back, purge, purge_me) {
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 09:42:54 +00:00
HA_SPIN_LOCK(NOTIF_LOCK, &com->lock);
MINOR: tasks: Move Lua notification from Lua to tasks These notification management function and structs are generic and it will be better to move in common parts. The notification management functions and structs have names containing some "lua" references because it was written for the Lua. This patch removes also these references.
2017-07-12 12:31:10 +00:00
LIST_DEL(&com->purge_me);
MEDIUM: threads/tasks: Add lock around notifications This patch add lock around some notification calls
2017-07-16 22:14:07 +00:00
if (!com->task) {
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 09:42:54 +00:00
HA_SPIN_UNLOCK(NOTIF_LOCK, &com->lock);
CLEANUP: pools: rename all pool functions and pointers to remove this "2" During the migration to the second version of the pools, the new functions and pool pointers were all called "pool_something2()" and "pool2_something". Now there's no more pool v1 code and it's a real pain to still have to deal with this. Let's clean this up now by removing the "2" everywhere, and by renaming the pool heads "pool_head_something".
2017-11-24 16:34:44 +00:00
pool_free(pool_head_notification, com);
MEDIUM: threads/tasks: Add lock around notifications This patch add lock around some notification calls
2017-07-16 22:14:07 +00:00
continue;
}
com->task = NULL;
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 09:42:54 +00:00
HA_SPIN_UNLOCK(NOTIF_LOCK, &com->lock);
MINOR: tasks: Move Lua notification from Lua to tasks These notification management function and structs are generic and it will be better to move in common parts. The notification management functions and structs have names containing some "lua" references because it was written for the Lua. This patch removes also these references.
2017-07-12 12:31:10 +00:00
}
}
BUG/MEDIUM: lua/notification: memory leak The thread patches adds refcount for notifications. The notifications are used with the Lua cosocket. These refcount free the notifications when the session is cleared. In the Lua task case, it not have sessions, so the nofications are never cleraed. This patch adds a garbage collector for signals. The garbage collector just clean the notifications for which the end point is disconnected. This patch should be backported in 1.8
2017-12-10 16:10:57 +00:00
/* In some cases, the disconnected notifications must be cleared.
CLEANUP: assorted typo fixes in the code and comments These are mostly comments in the code. A few error messages were fixed and are of low enough importance not to deserve a backport. Some regtests were also fixed.
2020-03-10 07:06:11 +00:00
* This function just release memory blocks. The purge list is not
BUG/MEDIUM: lua/notification: memory leak The thread patches adds refcount for notifications. The notifications are used with the Lua cosocket. These refcount free the notifications when the session is cleared. In the Lua task case, it not have sessions, so the nofications are never cleraed. This patch adds a garbage collector for signals. The garbage collector just clean the notifications for which the end point is disconnected. This patch should be backported in 1.8
2017-12-10 16:10:57 +00:00
* locked because it is owned by only one process. Before browsing
* this list, the caller must ensure to be the only one browser.
* The "com" is not locked because when com->task is NULL, the
* notification is no longer used.
*/
static inline void notification_gc(struct list *purge)
{
struct notification *com, *back;
/* Delete all pending communication signals. */
list_for_each_entry_safe (com, back, purge, purge_me) {
if (com->task)
continue;
LIST_DEL(&com->purge_me);
pool_free(pool_head_notification, com);
}
}
MINOR: tasks: Move Lua notification from Lua to tasks These notification management function and structs are generic and it will be better to move in common parts. The notification management functions and structs have names containing some "lua" references because it was written for the Lua. This patch removes also these references.
2017-07-12 12:31:10 +00:00
/* This function sends signals. It wakes all the tasks attached
* to a list head, and remove the signal, and free the used
DOC: notifications: add precisions about thread usage Precise the terms of use the notification functions.
2017-12-10 16:14:07 +00:00
* memory. The wake list is not locked because it is owned by
* only one process. before browsing this list, the caller must
* ensure to be the only one browser.
MINOR: tasks: Move Lua notification from Lua to tasks These notification management function and structs are generic and it will be better to move in common parts. The notification management functions and structs have names containing some "lua" references because it was written for the Lua. This patch removes also these references.
2017-07-12 12:31:10 +00:00
*/
static inline void notification_wake(struct list *wake)
{
struct notification *com, *back;
/* Wake task and delete all pending communication signals. */
list_for_each_entry_safe(com, back, wake, wake_me) {
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 09:42:54 +00:00
HA_SPIN_LOCK(NOTIF_LOCK, &com->lock);
MINOR: tasks: Move Lua notification from Lua to tasks These notification management function and structs are generic and it will be better to move in common parts. The notification management functions and structs have names containing some "lua" references because it was written for the Lua. This patch removes also these references.
2017-07-12 12:31:10 +00:00
LIST_DEL(&com->wake_me);
MEDIUM: threads/tasks: Add lock around notifications This patch add lock around some notification calls
2017-07-16 22:14:07 +00:00
if (!com->task) {
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 09:42:54 +00:00
HA_SPIN_UNLOCK(NOTIF_LOCK, &com->lock);
CLEANUP: pools: rename all pool functions and pointers to remove this "2" During the migration to the second version of the pools, the new functions and pool pointers were all called "pool_something2()" and "pool2_something". Now there's no more pool v1 code and it's a real pain to still have to deal with this. Let's clean this up now by removing the "2" everywhere, and by renaming the pool heads "pool_head_something".
2017-11-24 16:34:44 +00:00
pool_free(pool_head_notification, com);
MEDIUM: threads/tasks: Add lock around notifications This patch add lock around some notification calls
2017-07-16 22:14:07 +00:00
continue;
}
MINOR: tasks: Move Lua notification from Lua to tasks These notification management function and structs are generic and it will be better to move in common parts. The notification management functions and structs have names containing some "lua" references because it was written for the Lua. This patch removes also these references.
2017-07-12 12:31:10 +00:00
task_wakeup(com->task, TASK_WOKEN_MSG);
MEDIUM: threads/tasks: Add lock around notifications This patch add lock around some notification calls
2017-07-16 22:14:07 +00:00
com->task = NULL;
BUILD: threads: Rename SPIN/RWLOCK macros using HA_ prefix This remove any name conflicts, especially on Solaris.
2017-11-07 09:42:54 +00:00
HA_SPIN_UNLOCK(NOTIF_LOCK, &com->lock);
MINOR: tasks: Move Lua notification from Lua to tasks These notification management function and structs are generic and it will be better to move in common parts. The notification management functions and structs have names containing some "lua" references because it was written for the Lua. This patch removes also these references.
2017-07-12 12:31:10 +00:00
}
}
MINOR: task/notification: Is notifications registered ? This function returns true is some notifications are registered. This function is usefull for the following patch BUG/MEDIUM: lua/socket: Sheduling error on write: may dead-lock It should be backported in 1.6, 1.7 and 1.8
2018-05-30 09:40:08 +00:00
/* This function returns true is some notification are pending
*/
static inline int notification_registered(struct list *wake)
{
return !LIST_ISEMPTY(wake);
}
MEDIUM: tasks: Get rid of active_tasks_mask. Remove the active_tasks_mask variable, we can deduce if we've work to do by other means, and it is costly to maintain. Instead, introduce a new function, thread_has_tasks(), that returns non-zero if there's tasks scheduled for the thread, zero otherwise.
2019-05-29 17:22:43 +00:00
static inline int thread_has_tasks(void)
{
return (!!(global_tasks_mask & tid_bit) |
MINOR: task: introduce a thread-local "sched" variable for local scheduler stuff The aim is to rassemble all scheduler information related to the current thread. It simply points to task_per_thread[tid] without having to perform the operation at each time. We save around 1.2 kB of code on performance sensitive paths and increase the request rate by almost 1%.
2019-09-24 06:25:15 +00:00
(sched->rqueue_size > 0) |
MEDIUM: tasks: implement 3 different tasklet classes with their own queues We used to mix high latency tasks and low latency tasklets in the same list, and to even refill bulk tasklets there, causing some unfairness in certain situations (e.g. poll-less transfers between many connections saturating the machine with similarly-sized in and out network interfaces). This patch changes the mechanism to split the load into 3 lists depending on the task/tasklet's desired classes : - URGENT: this is mainly for tasklets used as deferred callbacks - NORMAL: this is for regular tasks - BULK: this is for bulk tasks/tasklets Arbitrary ratios of max_processed are picked from each of these lists in turn, with the ability to complete in one list from what was not picked in the previous one. After some quick tests, the following setup gave apparently good results both for raw TCP with splicing and for H2-to-H1 request rate: - 0 to 75% for urgent - 12 to 50% for normal - 12 to what remains for bulk Bulk is not used yet.
2020-01-30 17:37:28 +00:00
!LIST_ISEMPTY(&sched->tasklets[TL_URGENT]) |
!LIST_ISEMPTY(&sched->tasklets[TL_NORMAL]) |
!LIST_ISEMPTY(&sched->tasklets[TL_BULK]) |
!MT_LIST_ISEMPTY(&sched->shared_tasklet_list));
MEDIUM: tasks: Get rid of active_tasks_mask. Remove the active_tasks_mask variable, we can deduce if we've work to do by other means, and it is costly to maintain. Instead, introduce a new function, thread_has_tasks(), that returns non-zero if there's tasks scheduled for the thread, zero otherwise.
2019-05-29 17:22:43 +00:00
}
MINOR: task: introduce work lists Sometimes we need to delegate some list processing to a function running on another thread. In this case the list element will simply be queued into a dedicated self-locked list and the task responsible for this list will be woken up, calling the associated function which will run over the list. This is what work_list does. Such lists will be dedicated to a limited type of work but will significantly ease such remote handling. A function is provided to create these per-thread lists, their tasks and to properly bind each task to a distinct thread, so that the caller only has to store the resulting pointer to the start of the structure. These structures should not be abused though as each head will consume 4 pointers per thread, hence 32 bytes per thread or 2 kB for 64 threads.
2019-07-12 06:31:17 +00:00
/* adds list item <item> to work list <work> and wake up the associated task */
MEDIUM: list: Separate "locked" list from regular list. Instead of using the same type for regular linked lists and "autolocked" linked lists, use a separate type, "struct mt_list", for the autolocked one, and introduce a set of macros, similar to the LIST_* macros, with the MT_ prefix. When we use the same entry for both regular list and autolocked list, as is done for the "list" field in struct connection, we know have to explicitely cast it to struct mt_list when using MT_ macros.
2019-08-08 13:47:21 +00:00
static inline void work_list_add(struct work_list *work, struct mt_list *item)
MINOR: task: introduce work lists Sometimes we need to delegate some list processing to a function running on another thread. In this case the list element will simply be queued into a dedicated self-locked list and the task responsible for this list will be woken up, calling the associated function which will run over the list. This is what work_list does. Such lists will be dedicated to a limited type of work but will significantly ease such remote handling. A function is provided to create these per-thread lists, their tasks and to properly bind each task to a distinct thread, so that the caller only has to store the resulting pointer to the start of the structure. These structures should not be abused though as each head will consume 4 pointers per thread, hence 32 bytes per thread or 2 kB for 64 threads.
2019-07-12 06:31:17 +00:00
{
MEDIUM: list: Separate "locked" list from regular list. Instead of using the same type for regular linked lists and "autolocked" linked lists, use a separate type, "struct mt_list", for the autolocked one, and introduce a set of macros, similar to the LIST_* macros, with the MT_ prefix. When we use the same entry for both regular list and autolocked list, as is done for the "list" field in struct connection, we know have to explicitely cast it to struct mt_list when using MT_ macros.
2019-08-08 13:47:21 +00:00
MT_LIST_ADDQ(&work->head, item);
MINOR: task: introduce work lists Sometimes we need to delegate some list processing to a function running on another thread. In this case the list element will simply be queued into a dedicated self-locked list and the task responsible for this list will be woken up, calling the associated function which will run over the list. This is what work_list does. Such lists will be dedicated to a limited type of work but will significantly ease such remote handling. A function is provided to create these per-thread lists, their tasks and to properly bind each task to a distinct thread, so that the caller only has to store the resulting pointer to the start of the structure. These structures should not be abused though as each head will consume 4 pointers per thread, hence 32 bytes per thread or 2 kB for 64 threads.
2019-07-12 06:31:17 +00:00
task_wakeup(work->task, TASK_WOKEN_OTHER);
}
struct work_list *work_list_create(int nbthread,
struct task *(*fct)(struct task *, void *, unsigned short),
void *arg);
void work_list_destroy(struct work_list *work, int nbthread);
MINOR: task: export run_tasks_from_list This will help refine debug traces.
2020-03-03 13:59:28 +00:00
int run_tasks_from_list(struct list *list, int max);
MINOR: task: introduce work lists Sometimes we need to delegate some list processing to a function running on another thread. In this case the list element will simply be queued into a dedicated self-locked list and the task responsible for this list will be woken up, calling the associated function which will run over the list. This is what work_list does. Such lists will be dedicated to a limited type of work but will significantly ease such remote handling. A function is provided to create these per-thread lists, their tasks and to properly bind each task to a distinct thread, so that the caller only has to store the resulting pointer to the start of the structure. These structures should not be abused though as each head will consume 4 pointers per thread, hence 32 bytes per thread or 2 kB for 64 threads.
2019-07-12 06:31:17 +00:00
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
/*
[MAJOR] proxy: finally get rid of maintain_proxies() This function is finally not needed anymore, as it has been replaced with a per-proxy task that is scheduled when some limits are encountered on incoming connections or when the process is stopping. The savings should be noticeable on configs with a large number of proxies. The most important point is that the rate limiting is now enforced in a clean and solid way.
2011-07-25 14:33:49 +00:00
* This does 3 things :
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
* - wake up all expired tasks
* - call all runnable tasks
[MAJOR] replaced all timeouts with struct timeval The timeout functions were difficult to manipulate because they were rounding results to the millisecond. Thus, it was difficult to compare and to check what expired and what did not. Also, the comparison functions were heavy with multiplies and divides by 1000. Now, all timeouts are stored in timevals, reducing the number of operations for updates and leading to cleaner and more efficient code.
2007-05-12 20:35:00 +00:00
* - return the date of next event in <next> or eternity.
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
*/
MAJOR: poll: only rely on wake_expired_tasks() to compute the wait delay Actually, HAProxy uses the function "process_runnable_tasks" and "wake_expired_tasks" to get the next task which can expires. If a task is added with "task_schedule" or other method during the execution of an other task, the expiration of this new task is not taken into account, and the execution of this task can be too late. Actualy, HAProxy seems to be no sensitive to this bug. This fix moves the call to process_runnable_tasks() before the timeout calculation and ensures that all wakeups are processed together. Only wake_expired_tasks() needs to return a timeout now.
2014-12-15 12:26:01 +00:00
void process_runnable_tasks();
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
[MAJOR] use an ebtree instead of a list for the run queue We now insert tasks in a certain sequence in the run queue. The sorting key currently is the arrival order. It will now be possible to apply a "nice" value to any task so that it goes forwards or backwards in the run queue. The calls to wake_expired_tasks() and maintain_proxies() have been moved to the main run_poll_loop(), because they had nothing to do in process_runnable_tasks(). The task_wakeup() function is not inlined anymore, as it was only used at one place. The qlist member of the task structure has been removed now. The run_queue list has been replaced for an integer indicating the number of tasks in the run queue.
2008-06-29 20:40:23 +00:00
/*
* Extract all expired timers from the timer queue, and wakes up all
MINOR: tasks: split wake_expired_tasks() in two parts to avoid useless wakeups We used to have wake_expired_tasks() wake up tasks and return the next expiration delay. The problem this causes is that we have to call it just before poll() in order to consider latest timers, but this also means that we don't wake up all newly expired tasks upon return from poll(), which thus systematically requires a second poll() round. This is visible when running any scheduled task like a health check, as there are systematically two poll() calls, one with the interval, nothing is done after it, and another one with a zero delay, and the task is called: listen test bind *:8001 server s1 127.0.0.1:1111 check 09:37:38.200959 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8696843}) = 0 09:37:38.200967 epoll_wait(3, [], 200, 1000) = 0 09:37:39.202459 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8712467}) = 0 >> nothing run here, as the expired task was not woken up yet. 09:37:39.202497 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8715766}) = 0 09:37:39.202505 epoll_wait(3, [], 200, 0) = 0 09:37:39.202513 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8719064}) = 0 >> now the expired task was woken up 09:37:39.202522 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 7 09:37:39.202537 fcntl(7, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 09:37:39.202565 setsockopt(7, SOL_TCP, TCP_NODELAY, [1], 4) = 0 09:37:39.202577 setsockopt(7, SOL_TCP, TCP_QUICKACK, [0], 4) = 0 09:37:39.202585 connect(7, {sa_family=AF_INET, sin_port=htons(1111), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress) 09:37:39.202659 epoll_ctl(3, EPOLL_CTL_ADD, 7, {EPOLLOUT, {u32=7, u64=7}}) = 0 09:37:39.202673 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8814713}) = 0 09:37:39.202683 epoll_wait(3, [{EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=7, u64=7}}], 200, 1000) = 1 09:37:39.202693 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8818617}) = 0 09:37:39.202701 getsockopt(7, SOL_SOCKET, SO_ERROR, [111], [4]) = 0 09:37:39.202715 close(7) = 0 Let's instead split the function in two parts: - the first part, wake_expired_tasks(), called just before process_runnable_tasks(), wakes up all expired tasks; it doesn't compute any timeout. - the second part, next_timer_expiry(), called just before poll(), only computes the next timeout for the current thread. Thanks to this, all expired tasks are properly woken up when leaving poll, and each poll call's timeout remains up to date: 09:41:16.270449 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=10223556}) = 0 09:41:16.270457 epoll_wait(3, [], 200, 999) = 0 09:41:17.270130 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=10238572}) = 0 09:41:17.270157 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 7 09:41:17.270194 fcntl(7, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 09:41:17.270204 setsockopt(7, SOL_TCP, TCP_NODELAY, [1], 4) = 0 09:41:17.270216 setsockopt(7, SOL_TCP, TCP_QUICKACK, [0], 4) = 0 09:41:17.270224 connect(7, {sa_family=AF_INET, sin_port=htons(1111), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress) 09:41:17.270299 epoll_ctl(3, EPOLL_CTL_ADD, 7, {EPOLLOUT, {u32=7, u64=7}}) = 0 09:41:17.270314 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=10337841}) = 0 09:41:17.270323 epoll_wait(3, [{EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=7, u64=7}}], 200, 1000) = 1 09:41:17.270332 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=10341860}) = 0 09:41:17.270340 getsockopt(7, SOL_SOCKET, SO_ERROR, [111], [4]) = 0 09:41:17.270367 close(7) = 0 This may be backported to 2.1 and 2.0 though it's unlikely to bring any user-visible improvement except to clarify debugging.
2019-12-11 07:12:23 +00:00
* associated tasks.
[MAJOR] use an ebtree instead of a list for the run queue We now insert tasks in a certain sequence in the run queue. The sorting key currently is the arrival order. It will now be possible to apply a "nice" value to any task so that it goes forwards or backwards in the run queue. The calls to wake_expired_tasks() and maintain_proxies() have been moved to the main run_poll_loop(), because they had nothing to do in process_runnable_tasks(). The task_wakeup() function is not inlined anymore, as it was only used at one place. The qlist member of the task structure has been removed now. The run_queue list has been replaced for an integer indicating the number of tasks in the run queue.
2008-06-29 20:40:23 +00:00
*/
MINOR: tasks: split wake_expired_tasks() in two parts to avoid useless wakeups We used to have wake_expired_tasks() wake up tasks and return the next expiration delay. The problem this causes is that we have to call it just before poll() in order to consider latest timers, but this also means that we don't wake up all newly expired tasks upon return from poll(), which thus systematically requires a second poll() round. This is visible when running any scheduled task like a health check, as there are systematically two poll() calls, one with the interval, nothing is done after it, and another one with a zero delay, and the task is called: listen test bind *:8001 server s1 127.0.0.1:1111 check 09:37:38.200959 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8696843}) = 0 09:37:38.200967 epoll_wait(3, [], 200, 1000) = 0 09:37:39.202459 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8712467}) = 0 >> nothing run here, as the expired task was not woken up yet. 09:37:39.202497 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8715766}) = 0 09:37:39.202505 epoll_wait(3, [], 200, 0) = 0 09:37:39.202513 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8719064}) = 0 >> now the expired task was woken up 09:37:39.202522 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 7 09:37:39.202537 fcntl(7, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 09:37:39.202565 setsockopt(7, SOL_TCP, TCP_NODELAY, [1], 4) = 0 09:37:39.202577 setsockopt(7, SOL_TCP, TCP_QUICKACK, [0], 4) = 0 09:37:39.202585 connect(7, {sa_family=AF_INET, sin_port=htons(1111), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress) 09:37:39.202659 epoll_ctl(3, EPOLL_CTL_ADD, 7, {EPOLLOUT, {u32=7, u64=7}}) = 0 09:37:39.202673 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8814713}) = 0 09:37:39.202683 epoll_wait(3, [{EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=7, u64=7}}], 200, 1000) = 1 09:37:39.202693 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8818617}) = 0 09:37:39.202701 getsockopt(7, SOL_SOCKET, SO_ERROR, [111], [4]) = 0 09:37:39.202715 close(7) = 0 Let's instead split the function in two parts: - the first part, wake_expired_tasks(), called just before process_runnable_tasks(), wakes up all expired tasks; it doesn't compute any timeout. - the second part, next_timer_expiry(), called just before poll(), only computes the next timeout for the current thread. Thanks to this, all expired tasks are properly woken up when leaving poll, and each poll call's timeout remains up to date: 09:41:16.270449 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=10223556}) = 0 09:41:16.270457 epoll_wait(3, [], 200, 999) = 0 09:41:17.270130 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=10238572}) = 0 09:41:17.270157 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 7 09:41:17.270194 fcntl(7, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 09:41:17.270204 setsockopt(7, SOL_TCP, TCP_NODELAY, [1], 4) = 0 09:41:17.270216 setsockopt(7, SOL_TCP, TCP_QUICKACK, [0], 4) = 0 09:41:17.270224 connect(7, {sa_family=AF_INET, sin_port=htons(1111), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress) 09:41:17.270299 epoll_ctl(3, EPOLL_CTL_ADD, 7, {EPOLLOUT, {u32=7, u64=7}}) = 0 09:41:17.270314 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=10337841}) = 0 09:41:17.270323 epoll_wait(3, [{EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=7, u64=7}}], 200, 1000) = 1 09:41:17.270332 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=10341860}) = 0 09:41:17.270340 getsockopt(7, SOL_SOCKET, SO_ERROR, [111], [4]) = 0 09:41:17.270367 close(7) = 0 This may be backported to 2.1 and 2.0 though it's unlikely to bring any user-visible improvement except to clarify debugging.
2019-12-11 07:12:23 +00:00
void wake_expired_tasks();
/* Checks the next timer for the current thread by looking into its own timer
* list and the global one. It may return TICK_ETERNITY if no timer is present.
CLEANUP: assorted typo fixes in the code and comments These are mostly comments in the code. A few error messages were fixed and are of low enough importance not to deserve a backport. Some regtests were also fixed.
2020-03-10 07:06:11 +00:00
* Note that the next timer might very well be slightly in the past.
MINOR: tasks: split wake_expired_tasks() in two parts to avoid useless wakeups We used to have wake_expired_tasks() wake up tasks and return the next expiration delay. The problem this causes is that we have to call it just before poll() in order to consider latest timers, but this also means that we don't wake up all newly expired tasks upon return from poll(), which thus systematically requires a second poll() round. This is visible when running any scheduled task like a health check, as there are systematically two poll() calls, one with the interval, nothing is done after it, and another one with a zero delay, and the task is called: listen test bind *:8001 server s1 127.0.0.1:1111 check 09:37:38.200959 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8696843}) = 0 09:37:38.200967 epoll_wait(3, [], 200, 1000) = 0 09:37:39.202459 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8712467}) = 0 >> nothing run here, as the expired task was not woken up yet. 09:37:39.202497 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8715766}) = 0 09:37:39.202505 epoll_wait(3, [], 200, 0) = 0 09:37:39.202513 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8719064}) = 0 >> now the expired task was woken up 09:37:39.202522 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 7 09:37:39.202537 fcntl(7, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 09:37:39.202565 setsockopt(7, SOL_TCP, TCP_NODELAY, [1], 4) = 0 09:37:39.202577 setsockopt(7, SOL_TCP, TCP_QUICKACK, [0], 4) = 0 09:37:39.202585 connect(7, {sa_family=AF_INET, sin_port=htons(1111), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress) 09:37:39.202659 epoll_ctl(3, EPOLL_CTL_ADD, 7, {EPOLLOUT, {u32=7, u64=7}}) = 0 09:37:39.202673 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8814713}) = 0 09:37:39.202683 epoll_wait(3, [{EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=7, u64=7}}], 200, 1000) = 1 09:37:39.202693 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=8818617}) = 0 09:37:39.202701 getsockopt(7, SOL_SOCKET, SO_ERROR, [111], [4]) = 0 09:37:39.202715 close(7) = 0 Let's instead split the function in two parts: - the first part, wake_expired_tasks(), called just before process_runnable_tasks(), wakes up all expired tasks; it doesn't compute any timeout. - the second part, next_timer_expiry(), called just before poll(), only computes the next timeout for the current thread. Thanks to this, all expired tasks are properly woken up when leaving poll, and each poll call's timeout remains up to date: 09:41:16.270449 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=10223556}) = 0 09:41:16.270457 epoll_wait(3, [], 200, 999) = 0 09:41:17.270130 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=10238572}) = 0 09:41:17.270157 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 7 09:41:17.270194 fcntl(7, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 09:41:17.270204 setsockopt(7, SOL_TCP, TCP_NODELAY, [1], 4) = 0 09:41:17.270216 setsockopt(7, SOL_TCP, TCP_QUICKACK, [0], 4) = 0 09:41:17.270224 connect(7, {sa_family=AF_INET, sin_port=htons(1111), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress) 09:41:17.270299 epoll_ctl(3, EPOLL_CTL_ADD, 7, {EPOLLOUT, {u32=7, u64=7}}) = 0 09:41:17.270314 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=10337841}) = 0 09:41:17.270323 epoll_wait(3, [{EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=7, u64=7}}], 200, 1000) = 1 09:41:17.270332 clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=10341860}) = 0 09:41:17.270340 getsockopt(7, SOL_SOCKET, SO_ERROR, [111], [4]) = 0 09:41:17.270367 close(7) = 0 This may be backported to 2.1 and 2.0 though it's unlikely to bring any user-visible improvement except to clarify debugging.
2019-12-11 07:12:23 +00:00
*/
int next_timer_expiry();
[MAJOR] use an ebtree instead of a list for the run queue We now insert tasks in a certain sequence in the run queue. The sorting key currently is the arrival order. It will now be possible to apply a "nice" value to any task so that it goes forwards or backwards in the run queue. The calls to wake_expired_tasks() and maintain_proxies() have been moved to the main run_poll_loop(), because they had nothing to do in process_runnable_tasks(). The task_wakeup() function is not inlined anymore, as it was only used at one place. The qlist member of the task structure has been removed now. The run_queue list has been replaced for an integer indicating the number of tasks in the run queue.
2008-06-29 20:40:23 +00:00
BUG/MEDIUM: mworker: stop every tasks in the master The master is not supposed to run (at the moment) any task before the polling loop, the created tasks should be run only in the workers but in the master they should be disabled or removed. No backport needed.
2018-12-06 13:05:20 +00:00
/*
* Delete every tasks before running the master polling loop
*/
void mworker_cleantasks();
[BIGMOVE] exploded the monolithic haproxy.c file into multiple files. The files are now stored under : - include/haproxy for the generic includes - include/types.h for the structures needed within prototypes - include/proto.h for function prototypes and inline functions - src/*.c for the C files Most include files are now covered by LGPL. A last move still needs to be done to put inline functions under GPL and not LGPL. Version has been set to 1.3.0 in the code but some control still needs to be done before releasing.
2006-06-26 00:48:02 +00:00
#endif /* _PROTO_TASK_H */
/*
* Local variables:
* c-indent-level: 8
* c-basic-offset: 8
* End:
*/
Reference in New Issue Copy Permalink