RepoMirrors
/
haproxy
mirror of http://git.haproxy.org/git/haproxy.git/
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
haproxy/include/common/buffer.h

219 lines
6.4 KiB
C
Raw Normal View History

REORG: buffers: split buffers into chunk,buffer,channel Many parts of the channel definition still make use of the "buffer" word.
2012-08-24 17:22:53 +00:00
/*
* include/common/buffer.h
* Buffer management definitions, macros and inline functions.
*
* Copyright (C) 2000-2012 Willy Tarreau - w@1wt.eu
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation, version 2.1
* exclusively.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
#ifndef _COMMON_BUFFER_H
#define _COMMON_BUFFER_H
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
MINOR: buffer: implement a new file for low-level buffer manipulation functions The buffer code currently depends on pools and other stuff and is not really autonomous anymore. The rewrite of the new API is an opportunity to clean this up. This patch creates a new file (buf.h) which does not depend on other elements and which will only contain what is needed to perform the most basic buffer operations. The new API will be introduced in this file and the conversion will be finished once buffer.h is empty. The definition of struct buffer was moved to this new file, using more explicity stdint types for the sizes and offsets. Most new functions will be implemented in two variants : __b_something() : unchecked variant, no wrapping is expected b_something() : wrapping-checked variant This way callers will be able to select which one to use depending on the use cases.
2018-07-11 07:39:05 +00:00
#include <common/buf.h>
MINOR: buffers: add a few functions to write chars, strings and blocks bo_put{chr,blk,str,chk} are used to write data on the output of a buffer. Output is truncated if the buffer is not large enough.
2012-09-28 14:02:48 +00:00
#include <common/chunk.h>
REORG: buffers: split buffers into chunk,buffer,channel Many parts of the channel definition still make use of the "buffer" word.
2012-08-24 17:22:53 +00:00
#include <common/config.h>
MINOR: buffer: add a function to match against string patterns In order to match known patterns in wrapping buffer, we'll introduce new string manipulation functions for buffers. The new function b_isteq() relies on an ist string for the pattern and compares it against any location in the buffer relative to <p>. The second function bi_eat() is specially designed to match input contents.
2017-09-22 13:02:54 +00:00
#include <common/ist.h>
MINOR: buffer: add a new file for ist + buffer manipulation functions The new file istbuf.h links the indirect strings (ist) with the buffers. The purpose is to encourage addition of more standard buffer manipulation functions that rely on this in order to improve the overall ease of use along all the code. Just like ist.h and buf.h, this new file is not expected to depend on anything beyond these two files. A few functions were added and/or converted from buffer.h : - b_isteq() : indicates if a buffer and a string match - b_isteat() : consumes a string from the buffer if it matches - b_istput() : appends a small string to a buffer (all or none) - b_putist() : appends part of a large string to a buffer The equivalent functions were removed from buffer.h and changed at the various call places.
2018-07-12 07:02:47 +00:00
#include <common/istbuf.h>
MAJOR: channel: replace the struct buffer with a pointer to a buffer With this commit, we now separate the channel from the buffer. This will allow us to replace buffers on the fly without touching the channel. Since nobody is supposed to keep a reference to a buffer anymore, doing so is not a problem and will also permit some copy-less data manipulation. Interestingly, these changes have shown a 2% performance increase on some workloads, probably due to a better cache placement of data.
2012-10-12 21:49:43 +00:00
#include <common/memory.h>
REORG: buffers: split buffers into chunk,buffer,channel Many parts of the channel definition still make use of the "buffer" word.
2012-08-24 17:22:53 +00:00
MINOR: activity: report the number of failed pool/buffer allocations Haproxy is designed to be able to continue to run even under very low memory conditions. However this can sometimes have a serious impact on performance that it hard to diagnose. Let's report counters of failed pool and buffer allocations per thread in show activity.
2019-05-28 15:04:16 +00:00
#include <proto/activity.h>
REORG: buffers: split buffers into chunk,buffer,channel Many parts of the channel definition still make use of the "buffer" word.
2012-08-24 17:22:53 +00:00
BUG/MAJOR: Fix how the list of entities waiting for a buffer is handled When an entity tries to get a buffer, if it cannot be allocted, for example because the number of buffers which may be allocated per process is limited, this entity is added in a list (called <buffer_wq>) and wait for an available buffer. Historically, the <buffer_wq> list was logically attached to streams because it were the only entities likely to be added in it. Now, applets can also be waiting for a free buffer. And with filters, we could imagine to have more other entities waiting for a buffer. So it make sense to have a generic list. Anyway, with the current design there is a bug. When an applet failed to get a buffer, it will wait. But we add the stream attached to the applet in <buffer_wq>, instead of the applet itself. So when a buffer is available, we wake up the stream and not the waiting applet. So, it is possible to have waiting applets and never awakened. So, now, <buffer_wq> is independant from streams. And we really add the waiting entity in <buffer_wq>. To be generic, the entity is responsible to define the callback used to awaken it. In addition, applets will still request an input buffer when they become active. But they will not be sleeped anymore if no buffer are available. So this is the responsibility to the applet I/O handler to check if this buffer is allocated or not. This way, an applet can decide if this buffer is required or not and can do additional processing if not. [wt: backport to 1.7 and 1.6]
2016-12-09 16:30:18 +00:00
/* an element of the <buffer_wq> list. It represents an object that need to
* acquire a buffer to continue its process. */
struct buffer_wait {
void *target; /* The waiting object that should be woken up */
int (*wakeup_cb)(void *); /* The function used to wake up the <target>, passed as argument */
MEDIUM: buffer: remove the buffer_wq lock This lock was only needed to protect the buffer_wq list, but now we have the mt_list for this. This patch simply turns the buffer_wq list to an mt_list and gets rid of the lock. It's worth noting that the whole buffer_wait thing still looks totally wrong especially in a threaded context: the wakeup_cb() callback is called synchronously from any thread and may end up calling some connection code that was not expected to run on a given thread. The whole thing should probably be reworked to use tasklets instead and be a bit more centralized.
2020-02-26 09:39:36 +00:00
struct mt_list list; /* Next element in the <buffer_wq> list */
BUG/MAJOR: Fix how the list of entities waiting for a buffer is handled When an entity tries to get a buffer, if it cannot be allocted, for example because the number of buffers which may be allocated per process is limited, this entity is added in a list (called <buffer_wq>) and wait for an available buffer. Historically, the <buffer_wq> list was logically attached to streams because it were the only entities likely to be added in it. Now, applets can also be waiting for a free buffer. And with filters, we could imagine to have more other entities waiting for a buffer. So it make sense to have a generic list. Anyway, with the current design there is a bug. When an applet failed to get a buffer, it will wait. But we add the stream attached to the applet in <buffer_wq>, instead of the applet itself. So when a buffer is available, we wake up the stream and not the waiting applet. So, it is possible to have waiting applets and never awakened. So, now, <buffer_wq> is independant from streams. And we really add the waiting entity in <buffer_wq>. To be generic, the entity is responsible to define the callback used to awaken it. In addition, applets will still request an input buffer when they become active. But they will not be sleeped anymore if no buffer are available. So this is the responsibility to the applet I/O handler to check if this buffer is allocated or not. This way, an applet can decide if this buffer is required or not and can do additional processing if not. [wt: backport to 1.7 and 1.6]
2016-12-09 16:30:18 +00:00
};
CLEANUP: pools: rename all pool functions and pointers to remove this "2" During the migration to the second version of the pools, the new functions and pool pointers were all called "pool_something2()" and "pool2_something". Now there's no more pool v1 code and it's a real pain to still have to deal with this. Let's clean this up now by removing the "2" everywhere, and by renaming the pool heads "pool_head_something".
2017-11-24 16:34:44 +00:00
extern struct pool_head *pool_head_buffer;
MEDIUM: buffer: remove the buffer_wq lock This lock was only needed to protect the buffer_wq list, but now we have the mt_list for this. This patch simply turns the buffer_wq list to an mt_list and gets rid of the lock. It's worth noting that the whole buffer_wait thing still looks totally wrong especially in a threaded context: the wakeup_cb() callback is called synchronously from any thread and may end up calling some connection code that was not expected to run on a given thread. The whole thing should probably be reworked to use tasklets instead and be a bit more centralized.
2020-02-26 09:39:36 +00:00
extern struct mt_list buffer_wq;
BUG/MINOR: threads: don't drop "extern" on the lock in include files Commit 9dcf9b6 ("MINOR: threads: Use __decl_hathreads to declare locks") accidently lost a few "extern" in certain lock declarations, possibly causing certain entries to be declared at multiple places. Apparently it hasn't caused any harm though. The offending ones were : - fdtab_lock - fdcache_lock - poll_lock - buffer_wq_lock
2017-11-26 10:00:37 +00:00
__decl_hathreads(extern HA_SPINLOCK_T buffer_wq_lock);
REORG: buffers: split buffers into chunk,buffer,channel Many parts of the channel definition still make use of the "buffer" word.
2012-08-24 17:22:53 +00:00
MAJOR: channel: replace the struct buffer with a pointer to a buffer With this commit, we now separate the channel from the buffer. This will allow us to replace buffers on the fly without touching the channel. Since nobody is supposed to keep a reference to a buffer anymore, doing so is not a problem and will also permit some copy-less data manipulation. Interestingly, these changes have shown a 2% performance increase on some workloads, probably due to a better cache placement of data.
2012-10-12 21:49:43 +00:00
int init_buffer();
REORG: buffers: split buffers into chunk,buffer,channel Many parts of the channel definition still make use of the "buffer" word.
2012-08-24 17:22:53 +00:00
void buffer_dump(FILE *o, struct buffer *b, int from, int to);
/*****************************************************************/
/* These functions are used to compute various buffer area sizes */
/*****************************************************************/
/* Return 1 if the buffer has less than 1/4 of its capacity free, otherwise 0 */
static inline int buffer_almost_full(const struct buffer *buf)
{
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
if (b_is_null(buf))
MEDIUM: channel: do not report full when buf_empty is present on a channel Till now we'd consider a buffer full even if it had size==0 due to pointing to buf.size. Now we change this : if buf_wanted is present, it means that we have already tried to allocate a buffer but failed. Thus the buffer must be considered full so that we stop trying to poll for reads on it. Otherwise if it's empty, it's buf_empty and we report !full since we may allocate it on the fly.
2014-11-28 19:54:13 +00:00
return 0;
MINOR: buffer: add a few basic functions for the new API Here's the list of newly introduced functions : - b_data(), returning the total amount of data in the buffer (currently i+o) - b_orig(), returning the origin of the storage area, that is, the place of position 0. - b_wrap(), pointer to wrapping point (currently data+size) - b_size(), returning the size of the buffer - b_room(), returning the amount of bytes left available - b_full(), returning true if the buffer is full, otherwise false - b_stop(), pointer to end of data mark (currently p+i), used to compute distances or a stop pointer for a loop. - b_peek(), this one will help make the transition to the new buffer model. It returns a pointer to a position in the buffer known from an offest relative to the beginning of the data in the buffer. Thus, we can replace the following occurrences : bo_ptr(b) => b_peek(b, 0); bo_end(b) => b_peek(b, b->o); bi_ptr(b) => b_peek(b, b->o); bi_end(b) => b_peek(b, b->i + b->o); b_ptr(b, ofs) => b_peek(b, b->o + ofs); - b_head(), pointer to the beginning of data (currently bo_ptr()) - b_tail(), pointer to first free place (currently bi_ptr()) - b_next() / b_next_ofs(), pointer to the next byte, taking wrapping into account. - b_dist(), returning the distance between two pointers belonging to a buffer - b_reset(), which resets the buffer - b_space_wraps(), indicating if the free space wraps around the buffer - b_almost_full(), indicating if 3/4 or more of the buffer are used Some of these are provided with the unchecked variants using the "__" prefix, or with the "_ofs" suffix indicating they return a relative position to the buffer's origin instead of a pointer. Cc: Olivier Houchard <ohouchard@haproxy.com>
2018-06-06 12:30:50 +00:00
return b_almost_full(buf);
REORG: buffers: split buffers into chunk,buffer,channel Many parts of the channel definition still make use of the "buffer" word.
2012-08-24 17:22:53 +00:00
}
CLEANUP: buffer: minor cleanups to buffer.h Remove a few unused functions and add some comments to split the file parts in sections.
2018-07-10 08:35:02 +00:00
/**************************************************/
/* Functions below are used for buffer allocation */
/**************************************************/
REORG: channel: move buffer_{replace,insert_line}* to buffer.{c,h} These functions do not depend on the channel flags anymore thus they're much better suited to be used on plain buffers. Move them from channel to buffer.
2012-08-27 20:08:00 +00:00
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
/* Allocates a buffer and assigns it to *buf. If no memory is available,
* ((char *)1) is assigned instead with a zero size. No control is made to
* check if *buf already pointed to another buffer. The allocated buffer is
* returned, or NULL in case no memory is available.
MEDIUM: buffer: use b_alloc() to allocate and initialize a buffer b_alloc() now allocates a buffer and initializes it to the size specified in the pool minus the size of the struct buffer itself. This ensures that callers do not need to care about buffer details anymore. Also this never applies memory poisonning, which is slow and useless on buffers.
2014-11-24 10:30:16 +00:00
*/
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
static inline struct buffer *b_alloc(struct buffer *buf)
MEDIUM: buffer: use b_alloc() to allocate and initialize a buffer b_alloc() now allocates a buffer and initializes it to the size specified in the pool minus the size of the struct buffer itself. This ensures that callers do not need to care about buffer details anymore. Also this never applies memory poisonning, which is slow and useless on buffers.
2014-11-24 10:30:16 +00:00
{
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
char *area;
*buf = BUF_WANTED;
area = pool_alloc_dirty(pool_head_buffer);
MINOR: activity: report the number of failed pool/buffer allocations Haproxy is designed to be able to continue to run even under very low memory conditions. However this can sometimes have a serious impact on performance that it hard to diagnose. Let's report counters of failed pool and buffer allocations per thread in show activity.
2019-05-28 15:04:16 +00:00
if (unlikely(!area)) {
activity[tid].buf_wait++;
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
return NULL;
MINOR: activity: report the number of failed pool/buffer allocations Haproxy is designed to be able to continue to run even under very low memory conditions. However this can sometimes have a serious impact on performance that it hard to diagnose. Let's report counters of failed pool and buffer allocations per thread in show activity.
2019-05-28 15:04:16 +00:00
}
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
buf->area = area;
buf->size = pool_head_buffer->size;
return buf;
MEDIUM: buffer: use b_alloc() to allocate and initialize a buffer b_alloc() now allocates a buffer and initializes it to the size specified in the pool minus the size of the struct buffer itself. This ensures that callers do not need to care about buffer details anymore. Also this never applies memory poisonning, which is slow and useless on buffers.
2014-11-24 10:30:16 +00:00
}
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
/* Allocates a buffer and assigns it to *buf. If no memory is available,
* ((char *)1) is assigned instead with a zero size. No control is made to
* check if *buf already pointed to another buffer. The allocated buffer is
* returned, or NULL in case no memory is available. The difference with
* b_alloc() is that this function only picks from the pool and never calls
* malloc(), so it can fail even if some memory is available.
MINOR: buffer: implement b_alloc_fast() This function allocates a buffer and replaces *buf with this buffer. If no memory is available, &buf_wanted is used instead. No control is made to check if *buf already pointed to another buffer. The allocated buffer is returned, or NULL in case no memory is available. The difference with b_alloc() is that this function only picks from the pool and never calls malloc(), so it can fail even if some memory is available. It is the caller's job to refill the buffer pool if needed.
2014-12-08 15:37:26 +00:00
*/
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
static inline struct buffer *b_alloc_fast(struct buffer *buf)
MINOR: buffer: implement b_alloc_fast() This function allocates a buffer and replaces *buf with this buffer. If no memory is available, &buf_wanted is used instead. No control is made to check if *buf already pointed to another buffer. The allocated buffer is returned, or NULL in case no memory is available. The difference with b_alloc() is that this function only picks from the pool and never calls malloc(), so it can fail even if some memory is available. It is the caller's job to refill the buffer pool if needed.
2014-12-08 15:37:26 +00:00
{
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
char *area;
*buf = BUF_WANTED;
area = pool_get_first(pool_head_buffer);
if (unlikely(!area))
return NULL;
buf->area = area;
buf->size = pool_head_buffer->size;
return buf;
MINOR: buffer: implement b_alloc_fast() This function allocates a buffer and replaces *buf with this buffer. If no memory is available, &buf_wanted is used instead. No control is made to check if *buf already pointed to another buffer. The allocated buffer is returned, or NULL in case no memory is available. The difference with b_alloc() is that this function only picks from the pool and never calls malloc(), so it can fail even if some memory is available. It is the caller's job to refill the buffer pool if needed.
2014-12-08 15:37:26 +00:00
}
BUG/MINOR: buffers/threads: always clear a buffer's head before releasing it A small race exists in buffers with "show sess all". This one wants to show some information grabbed from the buffer (especially in HTX mode). But the thread owning this buffer might just be releasing its area, right after a free() or munmap() call, resulting in a head that is not seen as empty yet though the area was released. It may then be dereferenced by "show sess all" causing a crash. Note that in practice it only happens in debug mode with UAF enabled, but it's tricky enough to fix it right now. This should be backported to stable versions which support threads and a store barrier. It's worth noting that by performing the clearing first, b_free() and b_drop() now become two exact equivalent.
2019-08-08 05:53:20 +00:00
/* Releases buffer <buf> (no check of emptiness). The buffer's head is marked
* empty.
*/
CLEANUP: buffer: replace b_drop() with b_free() Since last commit there's no point anymore in having two variants of the same function, let's switch to b_free() only. __b_drop() was renamed to __b_free() for obvious consistency reasons.
2019-08-08 06:06:27 +00:00
static inline void __b_free(struct buffer *buf)
MINOR: buffer: only use b_free to release buffers We don't call pool_free2(pool2_buffers) anymore, we only call b_free() to do the job. This ensures that we can start to centralize the releasing of buffers.
2014-11-25 18:45:11 +00:00
{
BUG/MINOR: buffers/threads: always clear a buffer's head before releasing it A small race exists in buffers with "show sess all". This one wants to show some information grabbed from the buffer (especially in HTX mode). But the thread owning this buffer might just be releasing its area, right after a free() or munmap() call, resulting in a head that is not seen as empty yet though the area was released. It may then be dereferenced by "show sess all" causing a crash. Note that in practice it only happens in debug mode with UAF enabled, but it's tricky enough to fix it right now. This should be backported to stable versions which support threads and a store barrier. It's worth noting that by performing the clearing first, b_free() and b_drop() now become two exact equivalent.
2019-08-08 05:53:20 +00:00
char *area = buf->area;
/* let's first clear the area to save an occasional "show sess all"
* glancing over our shoulder from getting a dangling pointer.
*/
*buf = BUF_NULL;
__ha_barrier_store();
pool_free(pool_head_buffer, area);
MINOR: buffer: only use b_free to release buffers We don't call pool_free2(pool2_buffers) anymore, we only call b_free() to do the job. This ensures that we can start to centralize the releasing of buffers.
2014-11-25 18:45:11 +00:00
}
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
/* Releases buffer <buf> if allocated, and marks it empty. */
static inline void b_free(struct buffer *buf)
MEDIUM: buffer: always assign a dummy empty buffer to channels Channels are now created with a valid pointer to a buffer before the buffer is allocated. This buffer is a global one called "buf_empty" and of size zero. Thus it prevents any activity from being performed on the buffer and still ensures that chn->buf may always be dereferenced. b_free() also resets the buffer to &buf_empty, and was split into b_drop() which does not reset the buffer.
2014-11-24 10:39:34 +00:00
{
CLEANUP: buffer: replace b_drop() with b_free() Since last commit there's no point anymore in having two variants of the same function, let's switch to b_free() only. __b_drop() was renamed to __b_free() for obvious consistency reasons.
2019-08-08 06:06:27 +00:00
if (buf->size)
__b_free(buf);
MEDIUM: buffer: always assign a dummy empty buffer to channels Channels are now created with a valid pointer to a buffer before the buffer is allocated. This buffer is a global one called "buf_empty" and of size zero. Thus it prevents any activity from being performed on the buffer and still ensures that chn->buf may always be dereferenced. b_free() also resets the buffer to &buf_empty, and was split into b_drop() which does not reset the buffer.
2014-11-24 10:39:34 +00:00
}
MEDIUM: buffer: implement b_alloc_margin() This function is used to allocate a buffer and ensure that we leave some margin after it in the pool. The function is not obvious. While we allocate only one buffer, we want to ensure that at least two remain available after our allocation. The purpose is to ensure we'll never enter a deadlock where all sessions allocate exactly one buffer, and none of them will be able to allocate the second buffer needed to build a response in order to release the first one. We also take care of remaining fast in the the fast path by first checking whether or not there is enough margin, in which case we only rely on b_alloc_fast() which is guaranteed to succeed. Otherwise we take the slow path using pool_refill_alloc().
2014-12-02 12:54:01 +00:00
/* Ensures that <buf> is allocated. If an allocation is needed, it ensures that
* there are still at least <margin> buffers available in the pool after this
* allocation so that we don't leave the pool in a condition where a session or
* a response buffer could not be allocated anymore, resulting in a deadlock.
* This means that we sometimes need to try to allocate extra entries even if
* only one buffer is needed.
BUG/MINOR: buffers: Fix b_alloc_margin to be "fonctionnaly" thread-safe b_alloc_margin is, strickly speeking, thread-safe. It will not crash HAproxy. But its contract is not respected anymore in a multithreaded environment. In this function, we need to be sure to have <margin> buffers available in the pool after the allocation. So to have this guarantee, we must lock the memory pool during all the operation. This also means, we must call internal and lockless memory functions (prefixed with '__'). For the record, this patch fixes a pernicious bug happens after a soft reload where some streams can be blocked infinitly, waiting for a buffer in the buffer_wq list. This happens because, during a soft reload, pool_gc2 is called, making some calls to b_alloc_fast fail. This is specific to threads, no backport is needed.
2017-11-10 09:39:16 +00:00
*
* We need to lock the pool here to be sure to have <margin> buffers available
* after the allocation, regardless how many threads that doing it in the same
* time. So, we use internal and lockless memory functions (prefixed with '__').
MEDIUM: buffer: implement b_alloc_margin() This function is used to allocate a buffer and ensure that we leave some margin after it in the pool. The function is not obvious. While we allocate only one buffer, we want to ensure that at least two remain available after our allocation. The purpose is to ensure we'll never enter a deadlock where all sessions allocate exactly one buffer, and none of them will be able to allocate the second buffer needed to build a response in order to release the first one. We also take care of remaining fast in the the fast path by first checking whether or not there is enough margin, in which case we only rely on b_alloc_fast() which is guaranteed to succeed. Otherwise we take the slow path using pool_refill_alloc().
2014-12-02 12:54:01 +00:00
*/
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
static inline struct buffer *b_alloc_margin(struct buffer *buf, int margin)
MEDIUM: buffer: implement b_alloc_margin() This function is used to allocate a buffer and ensure that we leave some margin after it in the pool. The function is not obvious. While we allocate only one buffer, we want to ensure that at least two remain available after our allocation. The purpose is to ensure we'll never enter a deadlock where all sessions allocate exactly one buffer, and none of them will be able to allocate the second buffer needed to build a response in order to release the first one. We also take care of remaining fast in the the fast path by first checking whether or not there is enough margin, in which case we only rely on b_alloc_fast() which is guaranteed to succeed. Otherwise we take the slow path using pool_refill_alloc().
2014-12-02 12:54:01 +00:00
{
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
char *area;
MEDIUM: pools: implement a thread-local cache for pool entries Each thread now keeps the last ~512 kB of freed objects into a local cache. There are some heuristics involved so that a specific pool cannot use more than 1/8 of the total cache in number of objects. Tests have shown that 512 kB is an optimal size on a 24-thread test running on a dual-socket machine, resulting in an overall 7.5% performance increase and a cache miss ratio reducing from 19.2 to 17.7%. Anyway it seems pointless to keep more than an L2 cache, which probably explains why sizes between 256 and 512 kB are optimal. Cached objects appear in two lists, one per pool and one LRU to help with fair eviction. Currently there is no way to check each thread's cache state nor to flush it. This cache cannot be disabled and is enabled as soon as the lockless pools are enabled (i.e.: threads are enabled, no pool debugging is in use and the CPU supports a double word CAS).
2018-10-16 08:28:54 +00:00
ssize_t idx;
unsigned int cached;
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
if (buf->size)
return buf;
MEDIUM: buffer: implement b_alloc_margin() This function is used to allocate a buffer and ensure that we leave some margin after it in the pool. The function is not obvious. While we allocate only one buffer, we want to ensure that at least two remain available after our allocation. The purpose is to ensure we'll never enter a deadlock where all sessions allocate exactly one buffer, and none of them will be able to allocate the second buffer needed to build a response in order to release the first one. We also take care of remaining fast in the the fast path by first checking whether or not there is enough margin, in which case we only rely on b_alloc_fast() which is guaranteed to succeed. Otherwise we take the slow path using pool_refill_alloc().
2014-12-02 12:54:01 +00:00
MEDIUM: pools: implement a thread-local cache for pool entries Each thread now keeps the last ~512 kB of freed objects into a local cache. There are some heuristics involved so that a specific pool cannot use more than 1/8 of the total cache in number of objects. Tests have shown that 512 kB is an optimal size on a 24-thread test running on a dual-socket machine, resulting in an overall 7.5% performance increase and a cache miss ratio reducing from 19.2 to 17.7%. Anyway it seems pointless to keep more than an L2 cache, which probably explains why sizes between 256 and 512 kB are optimal. Cached objects appear in two lists, one per pool and one LRU to help with fair eviction. Currently there is no way to check each thread's cache state nor to flush it. This cache cannot be disabled and is enabled as soon as the lockless pools are enabled (i.e.: threads are enabled, no pool debugging is in use and the CPU supports a double word CAS).
2018-10-16 08:28:54 +00:00
cached = 0;
idx = pool_get_index(pool_head_buffer);
if (idx >= 0)
MEDIUM: memory: make the pool cache an array and not a thread_local Having a thread_local for the pool cache is messy as we need to initialize all elements upon startup, but we can't until the threads are created, and once created it's too late. For this reason, the allocation code used to check for the pool's initialization, and it was the release code which used to detect the first call and to initialize the cache on the fly, which is not exactly optimal. Now that we have initcalls, let's turn this into a per-thread array. This array is initialized very early in the boot process (STG_PREPARE) so that pools are always safe to use. This allows to remove the tests from the alloc/free calls. Doing just this has removed 2.5 kB of code on all cumulated pool_alloc() and pool_free() paths.
2018-11-26 16:09:46 +00:00
cached = pool_cache[tid][idx].count;
MEDIUM: pools: implement a thread-local cache for pool entries Each thread now keeps the last ~512 kB of freed objects into a local cache. There are some heuristics involved so that a specific pool cannot use more than 1/8 of the total cache in number of objects. Tests have shown that 512 kB is an optimal size on a 24-thread test running on a dual-socket machine, resulting in an overall 7.5% performance increase and a cache miss ratio reducing from 19.2 to 17.7%. Anyway it seems pointless to keep more than an L2 cache, which probably explains why sizes between 256 and 512 kB are optimal. Cached objects appear in two lists, one per pool and one LRU to help with fair eviction. Currently there is no way to check each thread's cache state nor to flush it. This cache cannot be disabled and is enabled as soon as the lockless pools are enabled (i.e.: threads are enabled, no pool debugging is in use and the CPU supports a double word CAS).
2018-10-16 08:28:54 +00:00
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
*buf = BUF_WANTED;
MEDIUM: buffer: implement b_alloc_margin() This function is used to allocate a buffer and ensure that we leave some margin after it in the pool. The function is not obvious. While we allocate only one buffer, we want to ensure that at least two remain available after our allocation. The purpose is to ensure we'll never enter a deadlock where all sessions allocate exactly one buffer, and none of them will be able to allocate the second buffer needed to build a response in order to release the first one. We also take care of remaining fast in the the fast path by first checking whether or not there is enough margin, in which case we only rely on b_alloc_fast() which is guaranteed to succeed. Otherwise we take the slow path using pool_refill_alloc().
2014-12-02 12:54:01 +00:00
BUG/MINOR: pools/threads: don't ignore DEBUG_UAF on double-word CAS capable archs Since commit cf975d4 ("MINOR: pools/threads: Implement lockless memory pools."), we support lockless pools. However the parts dedicated to detecting use-after-free are not present in this part, making DEBUG_UAF useless in this situation. The present patch sets a new define CONFIG_HAP_LOCKLESS_POOLS when such a compatible architecture is detected, and when pool debugging is not requested, then makes use of this everywhere in pools and buffers functions. This way enabling DEBUG_UAF will automatically disable the lockless version. No backport is needed as this is purely 1.9-dev.
2018-02-22 13:05:55 +00:00
#ifndef CONFIG_HAP_LOCKLESS_POOLS
CLEANUP: pools: rename all pool functions and pointers to remove this "2" During the migration to the second version of the pools, the new functions and pool pointers were all called "pool_something2()" and "pool2_something". Now there's no more pool v1 code and it's a real pain to still have to deal with this. Let's clean this up now by removing the "2" everywhere, and by renaming the pool heads "pool_head_something".
2017-11-24 16:34:44 +00:00
HA_SPIN_LOCK(POOL_LOCK, &pool_head_buffer->lock);
MINOR: pools/threads: Implement lockless memory pools. On CPUs that support a double-width compare-and-swap, implement lockless pools.
2018-01-24 17:38:31 +00:00
#endif
BUG/MINOR: buffers: Fix b_alloc_margin to be "fonctionnaly" thread-safe b_alloc_margin is, strickly speeking, thread-safe. It will not crash HAproxy. But its contract is not respected anymore in a multithreaded environment. In this function, we need to be sure to have <margin> buffers available in the pool after the allocation. So to have this guarantee, we must lock the memory pool during all the operation. This also means, we must call internal and lockless memory functions (prefixed with '__'). For the record, this patch fixes a pernicious bug happens after a soft reload where some streams can be blocked infinitly, waiting for a buffer in the buffer_wq list. This happens because, during a soft reload, pool_gc2 is called, making some calls to b_alloc_fast fail. This is specific to threads, no backport is needed.
2017-11-10 09:39:16 +00:00
MEDIUM: buffer: implement b_alloc_margin() This function is used to allocate a buffer and ensure that we leave some margin after it in the pool. The function is not obvious. While we allocate only one buffer, we want to ensure that at least two remain available after our allocation. The purpose is to ensure we'll never enter a deadlock where all sessions allocate exactly one buffer, and none of them will be able to allocate the second buffer needed to build a response in order to release the first one. We also take care of remaining fast in the the fast path by first checking whether or not there is enough margin, in which case we only rely on b_alloc_fast() which is guaranteed to succeed. Otherwise we take the slow path using pool_refill_alloc().
2014-12-02 12:54:01 +00:00
/* fast path */
MEDIUM: pools: implement a thread-local cache for pool entries Each thread now keeps the last ~512 kB of freed objects into a local cache. There are some heuristics involved so that a specific pool cannot use more than 1/8 of the total cache in number of objects. Tests have shown that 512 kB is an optimal size on a 24-thread test running on a dual-socket machine, resulting in an overall 7.5% performance increase and a cache miss ratio reducing from 19.2 to 17.7%. Anyway it seems pointless to keep more than an L2 cache, which probably explains why sizes between 256 and 512 kB are optimal. Cached objects appear in two lists, one per pool and one LRU to help with fair eviction. Currently there is no way to check each thread's cache state nor to flush it. This cache cannot be disabled and is enabled as soon as the lockless pools are enabled (i.e.: threads are enabled, no pool debugging is in use and the CPU supports a double word CAS).
2018-10-16 08:28:54 +00:00
if ((pool_head_buffer->allocated - pool_head_buffer->used + cached) > margin) {
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
area = __pool_get_first(pool_head_buffer);
if (likely(area)) {
BUG/MINOR: pools/threads: don't ignore DEBUG_UAF on double-word CAS capable archs Since commit cf975d4 ("MINOR: pools/threads: Implement lockless memory pools."), we support lockless pools. However the parts dedicated to detecting use-after-free are not present in this part, making DEBUG_UAF useless in this situation. The present patch sets a new define CONFIG_HAP_LOCKLESS_POOLS when such a compatible architecture is detected, and when pool debugging is not requested, then makes use of this everywhere in pools and buffers functions. This way enabling DEBUG_UAF will automatically disable the lockless version. No backport is needed as this is purely 1.9-dev.
2018-02-22 13:05:55 +00:00
#ifndef CONFIG_HAP_LOCKLESS_POOLS
CLEANUP: pools: rename all pool functions and pointers to remove this "2" During the migration to the second version of the pools, the new functions and pool pointers were all called "pool_something2()" and "pool2_something". Now there's no more pool v1 code and it's a real pain to still have to deal with this. Let's clean this up now by removing the "2" everywhere, and by renaming the pool heads "pool_head_something".
2017-11-24 16:34:44 +00:00
HA_SPIN_UNLOCK(POOL_LOCK, &pool_head_buffer->lock);
MINOR: pools/threads: Implement lockless memory pools. On CPUs that support a double-width compare-and-swap, implement lockless pools.
2018-01-24 17:38:31 +00:00
#endif
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
goto done;
BUG/MINOR: buffers: Fix b_alloc_margin to be "fonctionnaly" thread-safe b_alloc_margin is, strickly speeking, thread-safe. It will not crash HAproxy. But its contract is not respected anymore in a multithreaded environment. In this function, we need to be sure to have <margin> buffers available in the pool after the allocation. So to have this guarantee, we must lock the memory pool during all the operation. This also means, we must call internal and lockless memory functions (prefixed with '__'). For the record, this patch fixes a pernicious bug happens after a soft reload where some streams can be blocked infinitly, waiting for a buffer in the buffer_wq list. This happens because, during a soft reload, pool_gc2 is called, making some calls to b_alloc_fast fail. This is specific to threads, no backport is needed.
2017-11-10 09:39:16 +00:00
}
}
MEDIUM: buffer: implement b_alloc_margin() This function is used to allocate a buffer and ensure that we leave some margin after it in the pool. The function is not obvious. While we allocate only one buffer, we want to ensure that at least two remain available after our allocation. The purpose is to ensure we'll never enter a deadlock where all sessions allocate exactly one buffer, and none of them will be able to allocate the second buffer needed to build a response in order to release the first one. We also take care of remaining fast in the the fast path by first checking whether or not there is enough margin, in which case we only rely on b_alloc_fast() which is guaranteed to succeed. Otherwise we take the slow path using pool_refill_alloc().
2014-12-02 12:54:01 +00:00
BUG/MINOR: buffers: Fix b_alloc_margin to be "fonctionnaly" thread-safe b_alloc_margin is, strickly speeking, thread-safe. It will not crash HAproxy. But its contract is not respected anymore in a multithreaded environment. In this function, we need to be sure to have <margin> buffers available in the pool after the allocation. So to have this guarantee, we must lock the memory pool during all the operation. This also means, we must call internal and lockless memory functions (prefixed with '__'). For the record, this patch fixes a pernicious bug happens after a soft reload where some streams can be blocked infinitly, waiting for a buffer in the buffer_wq list. This happens because, during a soft reload, pool_gc2 is called, making some calls to b_alloc_fast fail. This is specific to threads, no backport is needed.
2017-11-10 09:39:16 +00:00
/* slow path, uses malloc() */
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
area = __pool_refill_alloc(pool_head_buffer, margin);
MEDIUM: buffer: implement b_alloc_margin() This function is used to allocate a buffer and ensure that we leave some margin after it in the pool. The function is not obvious. While we allocate only one buffer, we want to ensure that at least two remain available after our allocation. The purpose is to ensure we'll never enter a deadlock where all sessions allocate exactly one buffer, and none of them will be able to allocate the second buffer needed to build a response in order to release the first one. We also take care of remaining fast in the the fast path by first checking whether or not there is enough margin, in which case we only rely on b_alloc_fast() which is guaranteed to succeed. Otherwise we take the slow path using pool_refill_alloc().
2014-12-02 12:54:01 +00:00
BUG/MINOR: pools/threads: don't ignore DEBUG_UAF on double-word CAS capable archs Since commit cf975d4 ("MINOR: pools/threads: Implement lockless memory pools."), we support lockless pools. However the parts dedicated to detecting use-after-free are not present in this part, making DEBUG_UAF useless in this situation. The present patch sets a new define CONFIG_HAP_LOCKLESS_POOLS when such a compatible architecture is detected, and when pool debugging is not requested, then makes use of this everywhere in pools and buffers functions. This way enabling DEBUG_UAF will automatically disable the lockless version. No backport is needed as this is purely 1.9-dev.
2018-02-22 13:05:55 +00:00
#ifndef CONFIG_HAP_LOCKLESS_POOLS
CLEANUP: pools: rename all pool functions and pointers to remove this "2" During the migration to the second version of the pools, the new functions and pool pointers were all called "pool_something2()" and "pool2_something". Now there's no more pool v1 code and it's a real pain to still have to deal with this. Let's clean this up now by removing the "2" everywhere, and by renaming the pool heads "pool_head_something".
2017-11-24 16:34:44 +00:00
HA_SPIN_UNLOCK(POOL_LOCK, &pool_head_buffer->lock);
MINOR: pools/threads: Implement lockless memory pools. On CPUs that support a double-width compare-and-swap, implement lockless pools.
2018-01-24 17:38:31 +00:00
#endif
BUG/MINOR: buffers: Fix b_alloc_margin to be "fonctionnaly" thread-safe b_alloc_margin is, strickly speeking, thread-safe. It will not crash HAproxy. But its contract is not respected anymore in a multithreaded environment. In this function, we need to be sure to have <margin> buffers available in the pool after the allocation. So to have this guarantee, we must lock the memory pool during all the operation. This also means, we must call internal and lockless memory functions (prefixed with '__'). For the record, this patch fixes a pernicious bug happens after a soft reload where some streams can be blocked infinitly, waiting for a buffer in the buffer_wq list. This happens because, during a soft reload, pool_gc2 is called, making some calls to b_alloc_fast fail. This is specific to threads, no backport is needed.
2017-11-10 09:39:16 +00:00
MINOR: activity: report the number of failed pool/buffer allocations Haproxy is designed to be able to continue to run even under very low memory conditions. However this can sometimes have a serious impact on performance that it hard to diagnose. Let's report counters of failed pool and buffer allocations per thread in show activity.
2019-05-28 15:04:16 +00:00
if (unlikely(!area)) {
activity[tid].buf_wait++;
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
return NULL;
MINOR: activity: report the number of failed pool/buffer allocations Haproxy is designed to be able to continue to run even under very low memory conditions. However this can sometimes have a serious impact on performance that it hard to diagnose. Let's report counters of failed pool and buffer allocations per thread in show activity.
2019-05-28 15:04:16 +00:00
}
MAJOR: buffer: finalize buffer detachment Now the buffers only contain the header and a pointer to the storage area which can be anywhere. This will significantly simplify buffer swapping and will make it possible to map chunks on buffers as well. The buf_empty variable was removed, as now it's enough to have size==0 and area==NULL to designate the empty buffer (thus a non-allocated head is the empty buffer by default). buf_wanted for now is indicated by size==0 and area==(void *)1. The channels and the checks now embed the buffer's head, and the only pointer is to the storage area. This slightly increases the unallocated buffer size (3 extra ints for the empty buffer) but considerably simplifies dynamic buffer management. It will also later permit to detach unused checks. The way the struct buffer is arranged has proven quite efficient on a number of tests, which makes sense given that size is always accessed and often first, followed by the othe ones.
2018-07-10 15:43:27 +00:00
done:
buf->area = area;
buf->size = pool_head_buffer->size;
return buf;
MEDIUM: buffer: implement b_alloc_margin() This function is used to allocate a buffer and ensure that we leave some margin after it in the pool. The function is not obvious. While we allocate only one buffer, we want to ensure that at least two remain available after our allocation. The purpose is to ensure we'll never enter a deadlock where all sessions allocate exactly one buffer, and none of them will be able to allocate the second buffer needed to build a response in order to release the first one. We also take care of remaining fast in the the fast path by first checking whether or not there is enough margin, in which case we only rely on b_alloc_fast() which is guaranteed to succeed. Otherwise we take the slow path using pool_refill_alloc().
2014-12-02 12:54:01 +00:00
}
BUG/MAJOR: Fix how the list of entities waiting for a buffer is handled When an entity tries to get a buffer, if it cannot be allocted, for example because the number of buffers which may be allocated per process is limited, this entity is added in a list (called <buffer_wq>) and wait for an available buffer. Historically, the <buffer_wq> list was logically attached to streams because it were the only entities likely to be added in it. Now, applets can also be waiting for a free buffer. And with filters, we could imagine to have more other entities waiting for a buffer. So it make sense to have a generic list. Anyway, with the current design there is a bug. When an applet failed to get a buffer, it will wait. But we add the stream attached to the applet in <buffer_wq>, instead of the applet itself. So when a buffer is available, we wake up the stream and not the waiting applet. So, it is possible to have waiting applets and never awakened. So, now, <buffer_wq> is independant from streams. And we really add the waiting entity in <buffer_wq>. To be generic, the entity is responsible to define the callback used to awaken it. In addition, applets will still request an input buffer when they become active. But they will not be sleeped anymore if no buffer are available. So this is the responsibility to the applet I/O handler to check if this buffer is allocated or not. This way, an applet can decide if this buffer is required or not and can do additional processing if not. [wt: backport to 1.7 and 1.6]
2016-12-09 16:30:18 +00:00
DOC: buffers: clarify the purpose of the <from> pointer in offer_buffers() This one is only used to compare pointers and NULL is permitted though this is far from being clear.
2018-03-02 09:27:12 +00:00
/* Offer a buffer currently belonging to target <from> to whoever needs one.
* Any pointer is valid for <from>, including NULL. Its purpose is to avoid
* passing a buffer to oneself in case of failed allocations (e.g. need two
* buffers, get one, fail, release it and wake up self again). In case of
* normal buffer release where it is expected that the caller is not waiting
* for a buffer, NULL is fine.
*/
BUG/MAJOR: Fix how the list of entities waiting for a buffer is handled When an entity tries to get a buffer, if it cannot be allocted, for example because the number of buffers which may be allocated per process is limited, this entity is added in a list (called <buffer_wq>) and wait for an available buffer. Historically, the <buffer_wq> list was logically attached to streams because it were the only entities likely to be added in it. Now, applets can also be waiting for a free buffer. And with filters, we could imagine to have more other entities waiting for a buffer. So it make sense to have a generic list. Anyway, with the current design there is a bug. When an applet failed to get a buffer, it will wait. But we add the stream attached to the applet in <buffer_wq>, instead of the applet itself. So when a buffer is available, we wake up the stream and not the waiting applet. So, it is possible to have waiting applets and never awakened. So, now, <buffer_wq> is independant from streams. And we really add the waiting entity in <buffer_wq>. To be generic, the entity is responsible to define the callback used to awaken it. In addition, applets will still request an input buffer when they become active. But they will not be sleeped anymore if no buffer are available. So this is the responsibility to the applet I/O handler to check if this buffer is allocated or not. This way, an applet can decide if this buffer is required or not and can do additional processing if not. [wt: backport to 1.7 and 1.6]
2016-12-09 16:30:18 +00:00
void __offer_buffer(void *from, unsigned int threshold);
static inline void offer_buffers(void *from, unsigned int threshold)
{
MEDIUM: buffer: remove the buffer_wq lock This lock was only needed to protect the buffer_wq list, but now we have the mt_list for this. This patch simply turns the buffer_wq list to an mt_list and gets rid of the lock. It's worth noting that the whole buffer_wait thing still looks totally wrong especially in a threaded context: the wakeup_cb() callback is called synchronously from any thread and may end up calling some connection code that was not expected to run on a given thread. The whole thing should probably be reworked to use tasklets instead and be a bit more centralized.
2020-02-26 09:39:36 +00:00
if (!MT_LIST_ISEMPTY(&buffer_wq))
MEDIUM: buffers: relax the buffer lock a little bit In lock profiles it's visible that there is a huge contention on the buffer lock. The reason is that when offer_buffers() is called, it systematically takes the lock before verifying if there is any waiter. However doing so doesn't protect against races since a waiter can happen just after we release the lock as well. Similarly in h2 we take the lock every time an h2c is going to be released, even without checking that the h2c belongs to a wait list. These two have now been addressed by verifying non-emptiness of the list prior to taking the lock.
2019-05-28 15:21:18 +00:00
__offer_buffer(from, threshold);
BUG/MAJOR: Fix how the list of entities waiting for a buffer is handled When an entity tries to get a buffer, if it cannot be allocted, for example because the number of buffers which may be allocated per process is limited, this entity is added in a list (called <buffer_wq>) and wait for an available buffer. Historically, the <buffer_wq> list was logically attached to streams because it were the only entities likely to be added in it. Now, applets can also be waiting for a free buffer. And with filters, we could imagine to have more other entities waiting for a buffer. So it make sense to have a generic list. Anyway, with the current design there is a bug. When an applet failed to get a buffer, it will wait. But we add the stream attached to the applet in <buffer_wq>, instead of the applet itself. So when a buffer is available, we wake up the stream and not the waiting applet. So, it is possible to have waiting applets and never awakened. So, now, <buffer_wq> is independant from streams. And we really add the waiting entity in <buffer_wq>. To be generic, the entity is responsible to define the callback used to awaken it. In addition, applets will still request an input buffer when they become active. But they will not be sleeped anymore if no buffer are available. So this is the responsibility to the applet I/O handler to check if this buffer is allocated or not. This way, an applet can decide if this buffer is required or not and can do additional processing if not. [wt: backport to 1.7 and 1.6]
2016-12-09 16:30:18 +00:00
}
MINOR: buffer: add two functions to inject data into buffers bi_istput() injects the ist string into the input region of the buffer, it will be used to feed small data chunks into the conn_stream. bo_istput() does the same into the output region of the buffer, it will be used to send data via the transport layer and assumes there's no input data.
2017-09-22 13:47:51 +00:00
REORG: buffers: split buffers into chunk,buffer,channel Many parts of the channel definition still make use of the "buffer" word.
2012-08-24 17:22:53 +00:00
#endif /* _COMMON_BUFFER_H */
/*
* Local variables:
* c-indent-level: 8
* c-basic-offset: 8
* End:
*/