RepoMirrors/haproxy
1
0
Fork 0
mirror of http://git.haproxy.org/git/haproxy.git/ synced 2024-12-12 14:35:14 +00:00
Code Issues Packages Projects Releases Wiki Activity
ff97edac3e
haproxy/include/import/ist.h

833 lines
25 KiB
C
Raw Normal View History

MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
/*
REORG: include: move ist.h from common/ to import/ Fortunately that file wasn't made dependent upon haproxy since it was integrated, better isolate it before it's too late. Its dependency on api.h was the result of the change from config.h, which in turn wasn't correct. It was changed back to stddef.h for size_t and sys/types.h for ssize_t. The recently added reference to MAX() was changed as it was placed only to avoid a zero length in the non-free-standing version and was causing a build warning in the hpack encoder.
2020-05-27 14:21:26 +00:00
* include/import/ist.h
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
* Very simple indirect string manipulation functions.
*
REORG: include: move ist.h from common/ to import/ Fortunately that file wasn't made dependent upon haproxy since it was integrated, better isolate it before it's too late. Its dependency on api.h was the result of the change from config.h, which in turn wasn't correct. It was changed back to stddef.h for size_t and sys/types.h for ssize_t. The recently added reference to MAX() was changed as it was placed only to avoid a zero length in the non-free-standing version and was causing a build warning in the hpack encoder.
2020-05-27 14:21:26 +00:00
* Copyright (C) 2014-2020 Willy Tarreau - w@1wt.eu
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
*
* Permission is hereby granted, free of charge, to any person obtaining
* a copy of this software and associated documentation files (the
* "Software"), to deal in the Software without restriction, including
* without limitation the rights to use, copy, modify, merge, publish,
* distribute, sublicense, and/or sell copies of the Software, and to
* permit persons to whom the Software is furnished to do so, subject to
* the following conditions:
*
* The above copyright notice and this permission notice shall be
* included in all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
* OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
* HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
* OTHER DEALINGS IN THE SOFTWARE.
*/
REORG: include: move ist.h from common/ to import/ Fortunately that file wasn't made dependent upon haproxy since it was integrated, better isolate it before it's too late. Its dependency on api.h was the result of the change from config.h, which in turn wasn't correct. It was changed back to stddef.h for size_t and sys/types.h for ssize_t. The recently added reference to MAX() was changed as it was placed only to avoid a zero length in the non-free-standing version and was causing a build warning in the hpack encoder.
2020-05-27 14:21:26 +00:00
#ifndef _IMPORT_IST_H
#define _IMPORT_IST_H
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
REORG: include: move ist.h from common/ to import/ Fortunately that file wasn't made dependent upon haproxy since it was integrated, better isolate it before it's too late. Its dependency on api.h was the result of the change from config.h, which in turn wasn't correct. It was changed back to stddef.h for size_t and sys/types.h for ssize_t. The recently added reference to MAX() was changed as it was placed only to avoid a zero length in the non-free-standing version and was causing a build warning in the hpack encoder.
2020-05-27 14:21:26 +00:00
#include <sys/types.h>
MINOR: ist: Add the function isteqi This new function does the same as isteq, but ignoring the case.
2018-06-06 14:33:53 +00:00
#include <ctype.h>
REORG: include: move ist.h from common/ to import/ Fortunately that file wasn't made dependent upon haproxy since it was integrated, better isolate it before it's too late. Its dependency on api.h was the result of the change from config.h, which in turn wasn't correct. It was changed back to stddef.h for size_t and sys/types.h for ssize_t. The recently added reference to MAX() was changed as it was placed only to avoid a zero length in the non-free-standing version and was causing a build warning in the hpack encoder.
2020-05-27 14:21:26 +00:00
#include <stddef.h>
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
#include <string.h>
MINOR: ist: Add `struct ist istalloc(size_t)` and `void istfree(struct ist*)` `istalloc` allocates memory and returns an `ist` with the size `0` that points to this allocation. `istfree` frees the pointed memory and clears the pointer.
2020-03-05 16:56:32 +00:00
#ifndef IST_FREESTANDING
#include <stdlib.h>
#endif
MEDIUM: ist: use local conversion arrays to case conversion Calling tolower/toupper for each character is slow, a lookup into a 256-byte table is cheaper, especially for common characters used in header field names which all fit into a cache line. Let's create these two variables marked weak so that they're included only once.
2018-12-07 08:40:01 +00:00
/* ASCII to lower case conversion table */
BUILD: ist: turn the lower/upper case tables to literal on obsolete linkers Gil Bahat reported build issues on Cygwin starting with 1.9 due to a difference in the way the linker handles the weak symbols there, causing multiple declarations of ist_lc[] and ist_uc[]. It's likely that this issue could also happen on any older or non-ELF linker. This patch addresses this by using literals instead on such platforms, leaving it to the compiler to merge the constants when it can. On other platforms the resulting executable is slightly larger due to strings that could not be merged but this is a minor detail compared to not being able to build at all. If this change alone is confirmed to fix these issues, it's safe to backport to 1.9.
2019-05-15 14:07:36 +00:00
#define _IST_LC ((const unsigned char[256]){ \
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, \
0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, \
0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, \
0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, \
0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, \
0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, \
0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, \
0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, \
0x40, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, \
0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, \
0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, \
0x78, 0x79, 0x7a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, \
0x60, 0x61, 0x62, 0x63, 0x64, 0x65, 0x66, 0x67, \
0x68, 0x69, 0x6a, 0x6b, 0x6c, 0x6d, 0x6e, 0x6f, \
0x70, 0x71, 0x72, 0x73, 0x74, 0x75, 0x76, 0x77, \
0x78, 0x79, 0x7a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, \
0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, \
0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, \
0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, \
0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f, \
0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, \
0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, \
0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, \
0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, \
0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, \
0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, \
0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, \
0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf, \
0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, \
0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, \
0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, \
0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff, \
})
MEDIUM: ist: use local conversion arrays to case conversion Calling tolower/toupper for each character is slow, a lookup into a 256-byte table is cheaper, especially for common characters used in header field names which all fit into a cache line. Let's create these two variables marked weak so that they're included only once.
2018-12-07 08:40:01 +00:00
/* ASCII to upper case conversion table */
BUILD: ist: turn the lower/upper case tables to literal on obsolete linkers Gil Bahat reported build issues on Cygwin starting with 1.9 due to a difference in the way the linker handles the weak symbols there, causing multiple declarations of ist_lc[] and ist_uc[]. It's likely that this issue could also happen on any older or non-ELF linker. This patch addresses this by using literals instead on such platforms, leaving it to the compiler to merge the constants when it can. On other platforms the resulting executable is slightly larger due to strings that could not be merged but this is a minor detail compared to not being able to build at all. If this change alone is confirmed to fix these issues, it's safe to backport to 1.9.
2019-05-15 14:07:36 +00:00
#define _IST_UC ((const unsigned char[256]){ \
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, \
0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, \
0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, \
0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, \
0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, \
0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, \
0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, \
0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, \
0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, \
0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, \
0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, \
0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, \
0x60, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, \
0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, \
0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, \
0x58, 0x59, 0x5a, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, \
0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, \
0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, \
0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, \
0x98, 0x99, 0x9a, 0x9b, 0x9c, 0x9d, 0x9e, 0x9f, \
0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, \
0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, \
0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, \
0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, \
0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, \
0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, \
0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0xd6, 0xd7, \
0xd8, 0xd9, 0xda, 0xdb, 0xdc, 0xdd, 0xde, 0xdf, \
0xe0, 0xe1, 0xe2, 0xe3, 0xe4, 0xe5, 0xe6, 0xe7, \
0xe8, 0xe9, 0xea, 0xeb, 0xec, 0xed, 0xee, 0xef, \
0xf0, 0xf1, 0xf2, 0xf3, 0xf4, 0xf5, 0xf6, 0xf7, \
0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff, \
})
#ifdef USE_OBSOLETE_LINKER
/* some old linkers and some non-ELF platforms have issues with the weak
* attribute so we turn these arrays to literals there.
*/
#define ist_lc _IST_LC
#define ist_uc _IST_UC
#else
const unsigned char ist_lc[256] __attribute__((weak)) = _IST_LC;
const unsigned char ist_uc[256] __attribute__((weak)) = _IST_UC;
#endif
MEDIUM: ist: use local conversion arrays to case conversion Calling tolower/toupper for each character is slow, a lookup into a 256-byte table is cheaper, especially for common characters used in header field names which all fit into a cache line. Let's create these two variables marked weak so that they're included only once.
2018-12-07 08:40:01 +00:00
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
/* This string definition will most often be used to represent a read-only
* string returned from a function, based on the starting point and its length
* in bytes. No storage is provided, only a pointer and a length. The types
* here are important as we only want to have 2 native machine words there so
* that on modern architectures the compiler is capable of efficiently
* returning a register pair without having to allocate stack room from the
* caller. This is done with -freg-struct which is often enabled by default.
*/
struct ist {
char *ptr;
size_t len;
};
MINOR: ist: add a macro to ease const array initialization It's not possible to use strlen() in const arrays even with const strings, but we can use sizeof-1 via a macro. Let's provide this in the IST() macro, as it saves the developer from having to count the characters.
2017-09-21 13:24:10 +00:00
/* makes a constant ist from a constant string, for use in array declarations */
#define IST(str) { .ptr = str "", .len = (sizeof str "") - 1 }
MINOR: ist: Add `IST_NULL` macro `IST_NULL` is equivalent to an `struct ist` with `.ptr = NULL` and `.len = 0`.
2020-03-05 16:56:30 +00:00
/* IST_NULL is equivalent to an `ist` with `.ptr = NULL` and `.len = 0` */
#define IST_NULL ((const struct ist){ .ptr = 0, .len = 0 })
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
/* makes an ist from a regular zero terminated string. Null has length 0.
* Constants are detected and replaced with constant initializers. Other values
* are measured by hand without strlen() as it's much cheaper and inlinable on
* small strings. The construct is complex because we must never call
* __builtin_strlen() with an expression otherwise it involves a real
* measurement.
*/
#if __GNUC__ >= 4
// gcc >= 4 detects constant propagation of str through __x and resolves the
// length of constant strings easily.
#define ist(str) ({ \
char *__x = (void *)(str); \
(struct ist){ \
.ptr = __x, \
.len = __builtin_constant_p(str) ? \
((void *)str == (void *)0) ? 0 : \
__builtin_strlen(__x) : \
({ \
size_t __l = 0; \
if (__x) for (__l--; __x[++__l]; ) ; \
__l; \
}) \
}; \
})
#else
// gcc < 4 can't do this, and the side effect is a warning each time a NULL is
// passed to ist() due to the check on __builtin_strlen(). It doesn't have the
// ability to know that this code is never called.
#define ist(str) ({ \
char *__x = (void *)(str); \
(struct ist){ \
.ptr = __x, \
.len = __builtin_constant_p(str) ? \
((void *)str == (void *)0) ? 0 : \
__builtin_strlen(str) : \
({ \
size_t __l = 0; \
if (__x) for (__l--; __x[++__l]; ) ; \
__l; \
}) \
}; \
})
#endif
/* makes an ist struct from a string and a length */
static inline struct ist ist2(const void *ptr, size_t len)
{
return (struct ist){ .ptr = (char *)ptr, .len = len };
}
MINOR: ist: Add `int isttest(const struct ist)` `isttest` returns whether the `.ptr` is non-null.
2020-03-05 16:56:31 +00:00
/* returns the result of `ist.ptr != NULL` */
static inline int isttest(const struct ist ist)
{
return ist.ptr != NULL;
}
MINOR: ist: add ist0() to add a trailing zero to a string. This function modifies the string to add a zero after the end, and returns the start pointer. The purpose is to use it on strings extracted by parsers from larger strings cut with delimiters that are not important and can be destroyed. It allows any such string to be used with regular string functions. It's also convenient to use with printf() to show data extracted from writable areas.
2017-10-19 04:28:23 +00:00
/* This function MODIFIES the string to add a zero AFTER the end, and returns
* the start pointer. The purpose is to use it on strings extracted by parsers
* from larger strings cut with delimiters that are not important and can be
* destroyed. It allows any such string to be used with regular string
* functions. It's also convenient to use with printf() to show data extracted
* from writable areas. The caller is obviously responsible for ensuring that
* the string is valid and that the first byte past the end is writable. If
* these conditions cannot be satisfied, use istpad() below instead.
*/
static inline char *ist0(struct ist ist)
{
ist.ptr[ist.len] = 0;
return ist.ptr;
}
MINOR: ist: Add a function to retrieve the ist pointer There is already the istlen() function to get the ist length. Now, it is possible to call istptr() to get the ist pointer.
2020-04-21 08:46:43 +00:00
/* returns the pointer of the string */
static inline char *istptr(const struct ist ist)
{
return ist.ptr;
}
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
/* returns the length of the string */
static inline size_t istlen(const struct ist ist)
{
return ist.len;
}
MINOR: ist: Add istend() function to return a pointer to the end of the string istend() is a shortcut to istptr() + istlen().
2020-10-22 12:37:12 +00:00
/* returns the pointer to the end the string */
static inline char *istend(const struct ist ist)
{
return (ist.ptr + ist.len);
}
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
/* skips to next character in the string, always stops at the end */
static inline struct ist istnext(const struct ist ist)
{
struct ist ret = ist;
if (ret.len) {
ret.len--;
ret.ptr++;
}
return ret;
}
/* copies the contents from string <ist> to buffer <buf> and adds a trailing
* zero. The caller must ensure <buf> is large enough.
*/
static inline struct ist istpad(void *buf, const struct ist ist)
{
struct ist ret = { .ptr = buf, .len = ist.len };
for (ret.len = 0; ret.len < ist.len; ret.len++)
ret.ptr[ret.len] = ist.ptr[ret.len];
ret.ptr[ret.len] = 0;
return ret;
}
/* trims string <ist> to no more than <size> characters. The string is
* returned.
*/
static inline struct ist isttrim(const struct ist ist, size_t size)
{
struct ist ret = ist;
if (ret.len > size)
ret.len = size;
return ret;
}
/* trims string <ist> to no more than <size>-1 characters and ensures that a
* zero is placed after <ist.len> (possibly reduced by one) and before <size>,
* unless <size> is already zero. The string is returned. This is mostly aimed
* at building printable strings that need to be zero-terminated.
*/
static inline struct ist istzero(const struct ist ist, size_t size)
{
struct ist ret = ist;
if (!size)
ret.len = 0;
else {
if (ret.len > size - 1)
ret.len = size - 1;
ret.ptr[ret.len] = 0;
}
return ret;
}
/* returns the ordinal difference between two strings :
* < 0 if ist1 < ist2
* = 0 if ist1 == ist2
* > 0 if ist1 > ist2
*/
static inline int istdiff(const struct ist ist1, const struct ist ist2)
{
struct ist l = ist1;
struct ist r = ist2;
do {
if (!l.len--)
return -r.len;
if (!r.len--)
return 1;
} while (*l.ptr++ == *r.ptr++);
return *(unsigned char *)(l.ptr - 1) - *(unsigned char *)(r.ptr - 1);
}
/* returns non-zero if <ist1> starts like <ist2> (empty strings do match) */
static inline int istmatch(const struct ist ist1, const struct ist ist2)
{
struct ist l = ist1;
struct ist r = ist2;
if (l.len < r.len)
return 0;
while (r.len--) {
if (*l.ptr++ != *r.ptr++)
return 0;
}
return 1;
}
MINOR: ist: Add a case insensitive istmatch function Add a helper function that checks if a string starts with another string while ignoring case.
2020-10-28 10:35:14 +00:00
/* returns non-zero if <ist1> starts like <ist2>, ignoring the case (empty strings do match) */
static inline int istmatchi(const struct ist ist1, const struct ist ist2)
{
struct ist l = ist1;
struct ist r = ist2;
if (l.len < r.len)
return 0;
while (r.len--) {
if (*l.ptr != *r.ptr &&
ist_lc[(unsigned char)*l.ptr] != ist_lc[(unsigned char)*r.ptr])
return 0;
l.ptr++;
r.ptr++;
}
return 1;
}
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
/* returns non-zero if <ist1> starts like <ist2> on the first <count>
* characters (empty strings do match).
*/
static inline int istnmatch(const struct ist ist1, const struct ist ist2, size_t count)
{
struct ist l = ist1;
struct ist r = ist2;
if (l.len > count)
l.len = count;
if (r.len > count)
r.len = count;
return istmatch(l, r);
}
/* returns non-zero if <ist1> equals <ist2> (empty strings are equal) */
static inline int isteq(const struct ist ist1, const struct ist ist2)
{
struct ist l = ist1;
struct ist r = ist2;
if (l.len != r.len)
return 0;
while (l.len--) {
if (*l.ptr++ != *r.ptr++)
return 0;
}
return 1;
}
MINOR: ist: Add the function isteqi This new function does the same as isteq, but ignoring the case.
2018-06-06 14:33:53 +00:00
/* returns non-zero if <ist1> equals <ist2>, ignoring the case (empty strings are equal) */
static inline int isteqi(const struct ist ist1, const struct ist ist2)
{
struct ist l = ist1;
struct ist r = ist2;
if (l.len != r.len)
return 0;
while (l.len--) {
MEDIUM: ist: use local conversion arrays to case conversion Calling tolower/toupper for each character is slow, a lookup into a 256-byte table is cheaper, especially for common characters used in header field names which all fit into a cache line. Let's create these two variables marked weak so that they're included only once.
2018-12-07 08:40:01 +00:00
if (*l.ptr != *r.ptr &&
ist_lc[(unsigned char)*l.ptr] != ist_lc[(unsigned char)*r.ptr])
MINOR: ist: Add the function isteqi This new function does the same as isteq, but ignoring the case.
2018-06-06 14:33:53 +00:00
return 0;
MEDIUM: ist: use local conversion arrays to case conversion Calling tolower/toupper for each character is slow, a lookup into a 256-byte table is cheaper, especially for common characters used in header field names which all fit into a cache line. Let's create these two variables marked weak so that they're included only once.
2018-12-07 08:40:01 +00:00
MINOR: ist: Add the function isteqi This new function does the same as isteq, but ignoring the case.
2018-06-06 14:33:53 +00:00
l.ptr++;
r.ptr++;
}
return 1;
}
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
/* returns non-zero if <ist1> equals <ist2> on the first <count> characters
* (empty strings are equal).
*/
static inline int istneq(const struct ist ist1, const struct ist ist2, size_t count)
{
struct ist l = ist1;
struct ist r = ist2;
if (l.len > count)
l.len = count;
if (r.len > count)
r.len = count;
return isteq(l, r);
}
/* copies <src> over <dst> for a maximum of <count> bytes. Returns the number
* of characters copied (src.len), or -1 if it does not fit. In all cases, the
* contents are copied prior to reporting an error, so that the destination
* at least contains a valid but truncated string.
*/
static inline ssize_t istcpy(struct ist *dst, const struct ist src, size_t count)
{
dst->len = 0;
if (count > src.len)
count = src.len;
while (dst->len < count) {
dst->ptr[dst->len] = src.ptr[dst->len];
dst->len++;
}
if (dst->len == src.len)
return src.len;
return -1;
}
/* copies <src> over <dst> for a maximum of <count> bytes. Returns the number
* of characters copied, or -1 if it does not fit. A (possibly truncated) valid
* copy of <src> is always left into <dst>, and a trailing \0 is appended as
* long as <count> is not null, even if that results in reducing the string by
* one character.
*/
static inline ssize_t istscpy(struct ist *dst, const struct ist src, size_t count)
{
dst->len = 0;
if (!count)
goto fail;
if (count > src.len)
count = src.len + 1;
while (dst->len < count - 1) {
dst->ptr[dst->len] = src.ptr[dst->len];
dst->len++;
}
dst->ptr[dst->len] = 0;
if (dst->len == src.len)
return src.len;
fail:
return -1;
}
/* appends <src> after <dst> for a maximum of <count> total bytes in <dst> after
* the copy. <dst> is assumed to be <count> or less before the call. The new
* string's length is returned, or -1 if a truncation happened. In all cases,
* the contents are copied prior to reporting an error, so that the destination
* at least contains a valid but truncated string.
*/
static inline ssize_t istcat(struct ist *dst, const struct ist src, size_t count)
{
const char *s = src.ptr;
while (dst->len < count && s != src.ptr + src.len)
dst->ptr[dst->len++] = *s++;
if (s == src.ptr + src.len)
return dst->len;
return -1;
}
/* appends <src> after <dst> for a maximum of <count> total bytes in <dst> after
* the copy. <dst> is assumed to be <count> or less before the call. The new
* string's length is returned, or -1 if a truncation happened. In all cases,
* the contents are copied prior to reporting an error, so that the destination
* at least contains a valid but truncated string.
*/
static inline ssize_t istscat(struct ist *dst, const struct ist src, size_t count)
{
const char *s = src.ptr;
if (!count)
goto fail;
while (dst->len < count - 1 && s != src.ptr + src.len) {
dst->ptr[dst->len++] = *s++;
}
dst->ptr[dst->len] = 0;
if (s == src.ptr + src.len)
return dst->len;
fail:
return -1;
}
MINOR: ist: add functions to copy/uppercase/lowercase into a buffer or string The ist functions were missing functions to copy an IST into a target buffer, making some code have to resort to memcpy(), which tends to be overkill for small strings, that the compiler cannot guess. In addition sometimes there is a need to turn a string to lower or upper case so it had to be overwritten after the operation. This patch adds 6 functions to copy an ist to a buffer, as binary or as a string (i.e. a zero is or is not appended), and optionally to apply a lower case or upper case transformation on the fly. A number of tests were performed to optimize the processing for small strings. The loops are marked unlikely to dissuade the compilers from over-optimizing them and switching to SIMD instructions. The lower case or upper case transformations used to rely on external functions for each character and to crappify the code due to clobbered registers, which is not acceptable when we know that only a certain class of chars has to be transformed, so the test was open-coded.
2018-12-07 07:35:07 +00:00
/* copies the entire <src> over <dst>, which must be allocated large enough to
* hold the whole contents. No trailing zero is appended, this is mainly used
* for protocol processing where the frame length has already been checked. An
* ist made of the output and its length are returned. The destination is not
* touched if src.len is null.
*/
static inline struct ist ist2bin(char *dst, const struct ist src)
{
size_t ofs = 0;
/* discourage the compiler from trying to optimize for large strings,
* but tell it that most of our strings are not empty.
*/
if (__builtin_expect(ofs < src.len, 1)) {
do {
dst[ofs] = src.ptr[ofs];
ofs++;
} while (__builtin_expect(ofs < src.len, 0));
}
return ist2(dst, ofs);
}
/* copies the entire <src> over <dst>, which must be allocated large enough to
* hold the whole contents as well as a trailing zero which is always appended.
* This is mainly used for protocol conversions where the frame length has
* already been checked. An ist made of the output and its length (not counting
* the trailing zero) are returned.
*/
static inline struct ist ist2str(char *dst, const struct ist src, size_t count)
{
size_t ofs = 0;
/* discourage the compiler from trying to optimize for large strings,
* but tell it that most of our strings are not empty.
*/
if (__builtin_expect(ofs < src.len, 1)) {
do {
dst[ofs] = src.ptr[ofs];
ofs++;
} while (__builtin_expect(ofs < src.len, 0));
}
dst[ofs] = 0;
return ist2(dst, ofs);
}
/* makes a lower case copy of the entire <src> into <dst>, which must have been
* allocated large enough to hold the whole contents. No trailing zero is
* appended, this is mainly used for protocol processing where the frame length
* has already been checked. An ist made of the output and its length are
* returned. The destination is not touched if src.len is null.
*/
static inline struct ist ist2bin_lc(char *dst, const struct ist src)
{
size_t ofs = 0;
/* discourage the compiler from trying to optimize for large strings,
* but tell it that most of our strings are not empty.
*/
if (__builtin_expect(ofs < src.len, 1)) {
do {
MEDIUM: ist: use local conversion arrays to case conversion Calling tolower/toupper for each character is slow, a lookup into a 256-byte table is cheaper, especially for common characters used in header field names which all fit into a cache line. Let's create these two variables marked weak so that they're included only once.
2018-12-07 08:40:01 +00:00
dst[ofs] = ist_lc[(unsigned char)src.ptr[ofs]];
MINOR: ist: add functions to copy/uppercase/lowercase into a buffer or string The ist functions were missing functions to copy an IST into a target buffer, making some code have to resort to memcpy(), which tends to be overkill for small strings, that the compiler cannot guess. In addition sometimes there is a need to turn a string to lower or upper case so it had to be overwritten after the operation. This patch adds 6 functions to copy an ist to a buffer, as binary or as a string (i.e. a zero is or is not appended), and optionally to apply a lower case or upper case transformation on the fly. A number of tests were performed to optimize the processing for small strings. The loops are marked unlikely to dissuade the compilers from over-optimizing them and switching to SIMD instructions. The lower case or upper case transformations used to rely on external functions for each character and to crappify the code due to clobbered registers, which is not acceptable when we know that only a certain class of chars has to be transformed, so the test was open-coded.
2018-12-07 07:35:07 +00:00
ofs++;
} while (__builtin_expect(ofs < src.len, 0));
}
return ist2(dst, ofs);
}
/* makes a lower case copy of the entire <src> into <dst>, which must have been
* allocated large enough to hold the whole contents as well as a trailing zero
* which is always appended. This is mainly used for protocol conversions where
* the frame length has already been checked. An ist made of the output and its
* length (not counting the trailing zero) are returned.
*/
static inline struct ist ist2str_lc(char *dst, const struct ist src, size_t count)
{
size_t ofs = 0;
/* discourage the compiler from trying to optimize for large strings,
* but tell it that most of our strings are not empty.
*/
if (__builtin_expect(ofs < src.len, 1)) {
do {
MEDIUM: ist: use local conversion arrays to case conversion Calling tolower/toupper for each character is slow, a lookup into a 256-byte table is cheaper, especially for common characters used in header field names which all fit into a cache line. Let's create these two variables marked weak so that they're included only once.
2018-12-07 08:40:01 +00:00
dst[ofs] = ist_lc[(unsigned char)src.ptr[ofs]];
MINOR: ist: add functions to copy/uppercase/lowercase into a buffer or string The ist functions were missing functions to copy an IST into a target buffer, making some code have to resort to memcpy(), which tends to be overkill for small strings, that the compiler cannot guess. In addition sometimes there is a need to turn a string to lower or upper case so it had to be overwritten after the operation. This patch adds 6 functions to copy an ist to a buffer, as binary or as a string (i.e. a zero is or is not appended), and optionally to apply a lower case or upper case transformation on the fly. A number of tests were performed to optimize the processing for small strings. The loops are marked unlikely to dissuade the compilers from over-optimizing them and switching to SIMD instructions. The lower case or upper case transformations used to rely on external functions for each character and to crappify the code due to clobbered registers, which is not acceptable when we know that only a certain class of chars has to be transformed, so the test was open-coded.
2018-12-07 07:35:07 +00:00
ofs++;
} while (__builtin_expect(ofs < src.len, 0));
}
dst[ofs] = 0;
return ist2(dst, ofs);
}
/* makes an upper case copy of the entire <src> into <dst>, which must have
* been allocated large enough to hold the whole contents. No trailing zero is
* appended, this is mainly used for protocol processing where the frame length
* has already been checked. An ist made of the output and its length are
* returned. The destination is not touched if src.len is null.
*/
static inline struct ist ist2bin_uc(char *dst, const struct ist src)
{
size_t ofs = 0;
/* discourage the compiler from trying to optimize for large strings,
* but tell it that most of our strings are not empty.
*/
if (__builtin_expect(ofs < src.len, 1)) {
do {
MEDIUM: ist: use local conversion arrays to case conversion Calling tolower/toupper for each character is slow, a lookup into a 256-byte table is cheaper, especially for common characters used in header field names which all fit into a cache line. Let's create these two variables marked weak so that they're included only once.
2018-12-07 08:40:01 +00:00
dst[ofs] = ist_uc[(unsigned char)src.ptr[ofs]];
MINOR: ist: add functions to copy/uppercase/lowercase into a buffer or string The ist functions were missing functions to copy an IST into a target buffer, making some code have to resort to memcpy(), which tends to be overkill for small strings, that the compiler cannot guess. In addition sometimes there is a need to turn a string to lower or upper case so it had to be overwritten after the operation. This patch adds 6 functions to copy an ist to a buffer, as binary or as a string (i.e. a zero is or is not appended), and optionally to apply a lower case or upper case transformation on the fly. A number of tests were performed to optimize the processing for small strings. The loops are marked unlikely to dissuade the compilers from over-optimizing them and switching to SIMD instructions. The lower case or upper case transformations used to rely on external functions for each character and to crappify the code due to clobbered registers, which is not acceptable when we know that only a certain class of chars has to be transformed, so the test was open-coded.
2018-12-07 07:35:07 +00:00
ofs++;
} while (__builtin_expect(ofs < src.len, 0));
}
return ist2(dst, ofs);
}
/* makes an upper case copy of the entire <src> into <dst>, which must have been
* allocated large enough to hold the whole contents as well as a trailing zero
* which is always appended. This is mainly used for protocol conversions where
* the frame length has already been checked. An ist made of the output and its
* length (not counting the trailing zero) are returned.
*/
static inline struct ist ist2str_uc(char *dst, const struct ist src, size_t count)
{
size_t ofs = 0;
/* discourage the compiler from trying to optimize for large strings,
* but tell it that most of our strings are not empty.
*/
if (__builtin_expect(ofs < src.len, 1)) {
do {
MEDIUM: ist: use local conversion arrays to case conversion Calling tolower/toupper for each character is slow, a lookup into a 256-byte table is cheaper, especially for common characters used in header field names which all fit into a cache line. Let's create these two variables marked weak so that they're included only once.
2018-12-07 08:40:01 +00:00
dst[ofs] = ist_uc[(unsigned char)src.ptr[ofs]];
MINOR: ist: add functions to copy/uppercase/lowercase into a buffer or string The ist functions were missing functions to copy an IST into a target buffer, making some code have to resort to memcpy(), which tends to be overkill for small strings, that the compiler cannot guess. In addition sometimes there is a need to turn a string to lower or upper case so it had to be overwritten after the operation. This patch adds 6 functions to copy an ist to a buffer, as binary or as a string (i.e. a zero is or is not appended), and optionally to apply a lower case or upper case transformation on the fly. A number of tests were performed to optimize the processing for small strings. The loops are marked unlikely to dissuade the compilers from over-optimizing them and switching to SIMD instructions. The lower case or upper case transformations used to rely on external functions for each character and to crappify the code due to clobbered registers, which is not acceptable when we know that only a certain class of chars has to be transformed, so the test was open-coded.
2018-12-07 07:35:07 +00:00
ofs++;
} while (__builtin_expect(ofs < src.len, 0));
}
dst[ofs] = 0;
return ist2(dst, ofs);
}
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
/* looks for first occurrence of character <chr> in string <ist>. Returns the
* pointer if found, or NULL if not found.
*/
static inline char *istchr(const struct ist ist, char chr)
{
char *s = ist.ptr;
do {
if (s >= ist.ptr + ist.len)
return NULL;
} while (*s++ != chr);
return s - 1;
}
MINOR: ist: add ist_find_ctl() This new function looks for the first control character in a string (a char whose value is between 0x00 and 0x1F included) and returns it, or NULL if there is none. It is optimized for quickly evicting non-matching strings and scans ~0.43 bytes per cycle. It can be used as an accelerator when it's needed to look up several of these characters (e.g. CR/LF/NUL).
2019-11-22 14:58:53 +00:00
/* Returns a pointer to the first control character found in <ist>, or NULL if
* none is present. A control character is defined as a byte whose value is
* between 0x00 and 0x1F included. The function is optimized for strings having
* no CTL chars by processing up to sizeof(long) bytes at once on architectures
* supporting efficient unaligned accesses. Despite this it is not very fast
* (~0.43 byte/cycle) and should mostly be used on low match probability when
* it can save a call to a much slower function.
*/
static inline const char *ist_find_ctl(const struct ist ist)
{
const union { unsigned long v; } __attribute__((packed)) *u;
const char *curr = (void *)ist.ptr - sizeof(long);
const char *last = curr + ist.len;
unsigned long l1, l2;
do {
curr += sizeof(long);
if (curr > last)
break;
u = (void *)curr;
/* subtract 0x202020...20 to the value to generate a carry in
* the lower byte if the byte contains a lower value. If we
* generate a bit 7 that was not there, it means the byte was
* within 0x00..0x1F.
*/
l2 = u->v;
l1 = ~l2 & ((~0UL / 255) * 0x80); /* 0x808080...80 */
l2 -= (~0UL / 255) * 0x20; /* 0x202020...20 */
} while ((l1 & l2) == 0);
last += sizeof(long);
if (__builtin_expect(curr < last, 0)) {
do {
REORG: include: move ist.h from common/ to import/ Fortunately that file wasn't made dependent upon haproxy since it was integrated, better isolate it before it's too late. Its dependency on api.h was the result of the change from config.h, which in turn wasn't correct. It was changed back to stddef.h for size_t and sys/types.h for ssize_t. The recently added reference to MAX() was changed as it was placed only to avoid a zero length in the non-free-standing version and was causing a build warning in the hpack encoder.
2020-05-27 14:21:26 +00:00
if ((unsigned char)*curr < 0x20)
MINOR: ist: add ist_find_ctl() This new function looks for the first control character in a string (a char whose value is between 0x00 and 0x1F included) and returns it, or NULL if there is none. It is optimized for quickly evicting non-matching strings and scans ~0.43 bytes per cycle. It can be used as an accelerator when it's needed to look up several of these characters (e.g. CR/LF/NUL).
2019-11-22 14:58:53 +00:00
return curr;
curr++;
} while (curr < last);
}
return NULL;
}
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
/* looks for first occurrence of character <chr> in string <ist> and returns
* the tail of the string starting with this character, or (ist.end,0) if not
* found.
*/
static inline struct ist istfind(const struct ist ist, char chr)
{
struct ist ret = ist;
while (ret.len--) {
if (*ret.ptr++ == chr)
return ist2(ret.ptr - 1, ret.len + 1);
}
return ist2(ret.ptr, 0);
}
/* looks for first occurrence of character different from <chr> in string <ist>
* and returns the tail of the string starting at this character, or (ist_end,0)
* if not found.
*/
static inline struct ist istskip(const struct ist ist, char chr)
{
struct ist ret = ist;
while (ret.len--) {
if (*ret.ptr++ != chr)
return ist2(ret.ptr - 1, ret.len + 1);
}
return ist2(ret.ptr, 0);
}
/* looks for first occurrence of string <pat> in string <ist> and returns the
* tail of the string starting at this position, or (NULL,0) if not found. The
* empty pattern is found everywhere.
*/
static inline struct ist istist(const struct ist ist, const struct ist pat)
{
struct ist ret = ist;
size_t pos;
if (!pat.len)
return ret;
while (1) {
loop:
ret = istfind(ret, *pat.ptr);
if (ret.len < pat.len)
break;
/* ret.len >= 1, pat.len >= 1 and *ret.ptr == *pat.ptr */
ret = istnext(ret);
for (pos = 0; pos < pat.len - 1; ) {
++pos;
if (ret.ptr[pos - 1] != pat.ptr[pos])
goto loop;
}
return ist2(ret.ptr - 1, ret.len + 1);
}
MINOR: ist: Add `IST_NULL` macro `IST_NULL` is equivalent to an `struct ist` with `.ptr = NULL` and `.len = 0`.
2020-03-05 16:56:30 +00:00
return IST_NULL;
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
}
MINOR: ist: add an iststop() function Add a function that finds a character in an ist and returns an updated ist with the length of the portion of the original string that doesn't contain the char. Might be backported to 2.1
2020-02-21 09:33:12 +00:00
/*
CLEANUP: assorted typo fixes in the code and comments These are mostly comments in the code. A few error messages were fixed and are of low enough importance not to deserve a backport. Some regtests were also fixed.
2020-03-10 07:06:11 +00:00
* looks for the first occurrence of <chr> in string <ist> and returns a shorter
MINOR: ist: add an iststop() function Add a function that finds a character in an ist and returns an updated ist with the length of the portion of the original string that doesn't contain the char. Might be backported to 2.1
2020-02-21 09:33:12 +00:00
* ist if char is found.
*/
static inline struct ist iststop(const struct ist ist, char chr)
{
size_t len = 0;
while (len++ < ist.len && ist.ptr[len - 1] != chr)
;
return ist2(ist.ptr, len - 1);
}
MINOR: ist: Add `struct ist istalloc(size_t)` and `void istfree(struct ist*)` `istalloc` allocates memory and returns an `ist` with the size `0` that points to this allocation. `istfree` frees the pointed memory and clears the pointer.
2020-03-05 16:56:32 +00:00
MINOR: ist: add istadv() function The purpose of istadv() function is to move forward <.ptr> by <nb> characters. It is very useful when parsing a payload.
2020-03-29 06:10:16 +00:00
/*
* advance <.ptr> by <nb> characters.
* If <ist> is too short, (ist.end,0) is returned.
*/
static inline struct ist istadv(const struct ist ist, const size_t nb)
{
if (ist.len < nb)
return ist2(ist.ptr + ist.len, 0);
return ist2(ist.ptr + nb, ist.len - nb);
}
MINOR: ist: add istissame() function The istissame() function takes 2 ist and compare their <.ptr> and <.len> values respectively. It returns non-zero if they are the same.
2020-03-29 07:10:27 +00:00
/*
* compare 2 ists and return non-zero if they are the same
*/
static inline int istissame(const struct ist ist1, const struct ist ist2)
{
return ((ist1.ptr == ist2.ptr) && (ist1.len == ist2.len));
}
MINOR: ist: Add `struct ist istalloc(size_t)` and `void istfree(struct ist*)` `istalloc` allocates memory and returns an `ist` with the size `0` that points to this allocation. `istfree` frees the pointed memory and clears the pointer.
2020-03-05 16:56:32 +00:00
#ifndef IST_FREESTANDING
/* This function allocates <size> bytes and returns an `ist` pointing to
* the allocated area with size `0`.
*
* If this function fails to allocate memory the return value is equivalent
* to IST_NULL.
*/
static inline struct ist istalloc(const size_t size)
{
return ist2(malloc(size), 0);
}
/* This function performs the equivalent of free() on the given <ist>.
*
* After this function returns the value of the given <ist> will be
* modified to be equivalent to IST_NULL.
*/
static inline void istfree(struct ist *ist)
{
free(ist->ptr);
*ist = IST_NULL;
}
MINOR: ist: Add `struct ist istdup(const struct ist)` istdup() performs the equivalent of strdup() on a `struct ist`.
2020-03-05 16:56:34 +00:00
/* This function performs the equivalent of strdup() on the given <src>.
*
* If this function fails to allocate memory the return value is equivalent
* to IST_NULL.
*/
static inline struct ist istdup(const struct ist src)
{
const size_t src_size = src.len;
/* Allocate at least 1 byte to allow duplicating an empty string with
* malloc implementations that return NULL for a 0-size allocation.
*/
REORG: include: move ist.h from common/ to import/ Fortunately that file wasn't made dependent upon haproxy since it was integrated, better isolate it before it's too late. Its dependency on api.h was the result of the change from config.h, which in turn wasn't correct. It was changed back to stddef.h for size_t and sys/types.h for ssize_t. The recently added reference to MAX() was changed as it was placed only to avoid a zero length in the non-free-standing version and was causing a build warning in the hpack encoder.
2020-05-27 14:21:26 +00:00
struct ist dst = istalloc(src_size ? src_size : 1);
MINOR: ist: Add `struct ist istdup(const struct ist)` istdup() performs the equivalent of strdup() on a `struct ist`.
2020-03-05 16:56:34 +00:00
if (isttest(dst)) {
istcpy(&dst, src, src_size);
}
return dst;
}
MINOR: ist: Add `struct ist istalloc(size_t)` and `void istfree(struct ist*)` `istalloc` allocates memory and returns an `ist` with the size `0` that points to this allocation. `istfree` frees the pointed memory and clears the pointer.
2020-03-05 16:56:32 +00:00
#endif
MINOR: ist: implement very simple indirect strings For HPACK we'll need to perform a lot of string manipulation between the dynamic headers table and the output stream, and we need an efficient way to deal with that, considering that the zero character is not an end of string marker here. It turns out that gcc supports returning structs from functions and is able to place up to two words directly in registers when -freg-struct is used, which is the case by default on x86 and armv8. On other architectures the caller reserves some stack space where the callee can write, which is equivalent to passing a pointer to the return value. So let's implement a few functions to deal with this as the resulting code will be optimized on certain architectures where retrieving the length of a string will simply consist in reading one of the two returned registers. Extreme care was taken to ensure that the compiler gets maximum opportunities to optimize out every bit of unused code. This is also the reason why no call to regular string functions (such as strlen(), memcmp(), memcpy() etc) were used. The code involving them is often larger than when they are open coded. Given that strings are usually very small, especially when manipulating headers, the time spent calling a function optimized for large vectors often ends up being higher than the few cycles needed to count a few bytes. An issue was met with __builtin_strlen() which can automatically convert a constant string to its constant length. It doesn't accept NULLs and there is no way to hide them using expressions as the check is made before the optimizer is called. On gcc 4 and above, using an intermediary variable is enough to hide it. On older versions, calls to ist() with an explicit NULL argument will issue a warning. There is normally no reason to do this but taking care of it the best possible still seems important.
2017-05-30 15:49:36 +00:00
#endif
Reference in New Issue Copy Permalink