RepoMirrors/haproxy
1
0
Fork 0
mirror of http://git.haproxy.org/git/haproxy.git/ synced 2025-02-17 11:06:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

MEDIUM: vars: add a new "set-var-fmt" action

Browse Source 
The set-var() action is convenient because it preserves the input type
but it's a pain to deal with when trying to concatenate values. The
most recurring example is when it's needed to build a variable composed
of the source address and the source port. Usually it ends up like this:

    tcp-request session set-var(sess.port) src_port
    tcp-request session set-var(sess.addr) src,concat(":",sess.port)

This is even worse when trying to aggregate multiple fields from stick-table
data for example. Due to this a lot of users instead abuse headers from HTTP
rules:

    http-request set-header(x-addr) %[src]:%[src_port]

But this requires some careful cleanups to make sure they won't leak, and
it's significantly more expensive to deal with. And generally speaking it's
not clean. Plus it must be performed for each and every request, which is
expensive for this common case of ip+port that doesn't change for the whole
session.

This patch addresses this limitation by implementing a new "set-var-fmt"
action which performs the same work as "set-var" but takes a format string
in argument instead of an expression. This way it becomes pretty simple to
just write:

    tcp-request session set-var-fmt(sess.addr) %[src]:%[src_port]

It is usable in all rulesets that already support the "set-var" action.
It is not yet implemented for the global "set-var" directive (which already
takes a string) and the CLI's "set var" command, which would definitely
benefit from it but currently uses its own parser and engine, thus it
must be reworked.

The doc and regtests were updated.
This commit is contained in:
Willy Tarreau 2021-09-02 21:00:38 +02:00
parent e7267120d5
commit 9a621ae76d
4 changed files with 144 additions and 34 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

60
doc/configuration.txt
View File

@ -5282,7 +5282,7 @@ http-after-response set-status <status> [reason <str>]
http-response set-status 503 reason "Slow Down"
http-after-response set-var(<var-name>) <expr> [ { if | unless } <condition> ]
http-after-response set-var-fmt(<var-name>) <fmt> [ { if | unless } <condition> ]
This is used to set the contents of a variable. The variable is declared
inline.
@ -5304,8 +5304,12 @@ http-after-response set-var(<var-name>) <expr> [ { if | unless } <condition> ]
<expr> Is a standard HAProxy expression formed by a sample-fetch
followed by some converters.
Example:
http-after-response set-var(sess.last_redir) res.hdr(location)
<fmt> This is the value expressed using log-format rules (see Custom
Log Format in section 8.2.4).
Examples:
http-after-response set-var(sess.last_redir) res.hdr(location)
http-after-response set-var-fmt(sess.last_be_addr) %[bi]:%[bp]
http-after-response strict-mode { on | off }
@ -5753,6 +5757,7 @@ http-check send-state
http-check set-var(<var-name>) <expr>
http-check set-var-fmt(<var-name>) <fmt>
This operation sets the content of a variable. The variable is declared inline.
May be used in sections: defaults | frontend | listen | backend
yes | no | yes | yes
@ -5769,8 +5774,12 @@ http-check set-var(<var-name>) <expr>
<expr> Is a sample-fetch expression potentially followed by converters.
<fmt> This is the value expressed using log-format rules (see Custom
Log Format in section 8.2.4).
Examples :
http-check set-var(check.port) int(1234)
http-check set-var-fmt(check.port) "name=%H"
http-check unset-var(<var-name>)
@ -6746,6 +6755,7 @@ http-request set-uri <fmt> [ { if | unless } <condition> ]
See also "http-request set-path" and "http-request set-query".
http-request set-var(<var-name>) <expr> [ { if | unless } <condition> ]
http-request set-var-fmt(<var-name>) <fmt> [ { if | unless } <condition> ]
This is used to set the contents of a variable. The variable is declared
inline.
@ -6768,8 +6778,13 @@ http-request set-var(<var-name>) <expr> [ { if | unless } <condition> ]
<expr> Is a standard HAProxy expression formed by a sample-fetch
followed by some converters.
<fmt> This is the value expressed using log-format rules (see Custom
Log Format in section 8.2.4).
Example:
http-request set-var(req.my_var) req.fhdr(user-agent),lower
http-request set-var-fmt(txn.from) %[src]:%[src_port]
http-request send-spoe-group <engine-name> <group-name>
[ { if | unless } <condition> ]
@ -7316,6 +7331,7 @@ http-response set-tos <tos> [ { if | unless } <condition> ]
See RFC 2474, 2597, 3260 and 4594 for more information.
http-response set-var(<var-name>) <expr> [ { if | unless } <condition> ]
http-response set-var-fmt(<var-name>) <fmt> [ { if | unless } <condition> ]
This is used to set the contents of a variable. The variable is declared
inline.
@ -7338,8 +7354,12 @@ http-response set-var(<var-name>) <expr> [ { if | unless } <condition> ]
<expr> Is a standard HAProxy expression formed by a sample-fetch
followed by some converters.
Example:
http-response set-var(sess.last_redir) res.hdr(location)
<fmt> This is the value expressed using log-format rules (see Custom
Log Format in section 8.2.4).
Examples:
http-response set-var(sess.last_redir) res.hdr(location)
http-response set-var-fmt(sess.last_be_addr) %[bi]:%[bp]
http-response silent-drop [ { if | unless } <condition> ]
@ -11883,6 +11903,7 @@ tcp-check send-binary-lf <hexfmt> [comment <msg>]
tcp-check set-var(<var-name>) <expr>
tcp-check set-var-fmt(<var-name>) <fmt>
This operation sets the content of a variable. The variable is declared inline.
May be used in sections: defaults | frontend | listen | backend
yes | no | yes | yes
@ -11899,8 +11920,12 @@ tcp-check set-var(<var-name>) <expr>
<expr> Is a sample-fetch expression potentially followed by converters.
<fmt> This is the value expressed using log-format rules (see Custom
Log Format in section 8.2.4).
Examples :
tcp-check set-var(check.port) int(1234)
tcp-check set-var-fmt(check.name) "%H"
tcp-check unset-var(<var-name>)
@ -12266,6 +12291,7 @@ tcp-request content <action> [{if | unless} <condition>]
- set-nice <nice>
- set-tos <tos>
- set-var(<var-name>) <expr>
- set-var-fmt(<var-name>) <fmt>
- switch-mode http [ proto <name> ]
- unset-var(<var-name>)
- silent-drop
@ -12327,9 +12353,11 @@ tcp-request content <action> [{if | unless} <condition>]
The "set-tos" is used to set the TOS or DSCP field value of packets sent to
the client. More information on how to use it at "http-request set-tos".
The "set-var" is used to set the content of a variable. The variable is
declared inline. For "tcp-request session" rules, only session-level
variables can be used, without any layer7 contents.
The "set-var" and "set-var-fmt" are used to set the contents of a variable.
The variable is declared inline. For "tcp-request session" rules, only
session-level variables can be used, without any layer7 contents. The
"set-var" action takes a regular expression while "set-var-fmt" takes a
format string.
<var-name> The name of the variable starts with an indication about
its scope. The scopes allowed are:
@ -12348,6 +12376,9 @@ tcp-request content <action> [{if | unless} <condition>]
<expr> Is a standard HAProxy expression formed by a sample-fetch
followed by some converters.
<fmt> This is the value expressed using log-format rules (see Custom
Log Format in section 8.2.4).
The "switch-mode" is used to perform a connection upgrade. Only HTTP
upgrades are supported for now. The protocol may optionally be
specified. This action is only available for a proxy with the frontend
@ -12405,6 +12436,7 @@ tcp-request content <action> [{if | unless} <condition>]
Example:
tcp-request content set-var(sess.my_var) src
tcp-request content set-var-fmt(sess.from) %[src]:%[src_port]
tcp-request content unset-var(sess.my_var2)
Example:
@ -12589,6 +12621,9 @@ tcp-response content <action> [{if | unless} <condition>]
- set-var(<var-name>) <expr>
Sets a variable from an expression.
- set-var-fmt(<var-name>) <fmt>
Sets a variable from a log format string.
- unset-var(<var-name>)
Unsets a variable.
@ -12681,6 +12716,9 @@ tcp-response content <action> [{if | unless} <condition>]
<expr> Is a standard HAProxy expression formed by a sample-fetch
followed by some converters.
<fmt> This is the value expressed using log-format rules (see Custom
Log Format in section 8.2.4).
The "unset-var" is used to unset a variable. See above for details about
<var-name>.
@ -12751,6 +12789,7 @@ tcp-request session <action> [{if | unless} <condition>]
- set-src-port <expr>
- set-tos <tos>
- set-var(<var-name>) <expr>
- set-var-fmt(<var-name>) <fmt>
- unset-var(<var-name>)
- silent-drop
@ -16153,13 +16192,16 @@ concat([<start>],[<var>],[<end>])
other variables, such as colon-delimited values. If commas or closing
parenthesis are needed as delimiters, they must be protected by quotes or
backslashes, themselves protected so that they are not stripped by the first
level parser. See examples below.
level parser. This is often used to build composite variables from other
ones, but sometimes using a format string with multiple fields may be more
convenient. See examples below.
Example:
tcp-request session set-var(sess.src) src
tcp-request session set-var(sess.dn) ssl_c_s_dn
tcp-request session set-var(txn.sig) str(),concat(<ip=,sess.ip,>),concat(<dn=,sess.dn,>)
tcp-request session set-var(txn.ipport) "str(),concat('addr=(',sess.ip),concat(',',sess.port,')')"
tcp-request session set-var-fmt(txn.ipport) "addr=(%[sess.ip],%[sess.port])" ## does the same
http-request set-header x-hap-sig %[var(txn.sig)]
cpl

1
include/haproxy/action-t.h
View File

@ -162,6 +162,7 @@ struct act_rule {
} timeout;
struct hlua_rule *hlua_rule;
struct {
struct list fmt; /* log-format compatible expression */
struct sample_expr *expr;
const char *name;
enum vars_scope scope;

21
reg-tests/sample_fetches/vars.vtc
View File

@ -1,5 +1,5 @@
varnishtest "Test a few set-var() in global, tcp and http rule sets, at different scopes"
#REQUIRE_VERSION=2.4
feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev5)'"
feature ignore_unknown_macro
@ -7,7 +7,9 @@ haproxy h1 -conf {
global
set-var proc.int12 int(12)
set-var proc.int5 str(60),div(proc.int12)
set-var proc.str str("this is")
set-var proc.str1 str("this is")
set-var proc.str2 str("a string")
set-var proc.str var(proc.str1)
set-var proc.str str(""),concat("",proc.str," a string")
set-var proc.uuid uuid()
@ -19,11 +21,14 @@ haproxy h1 -conf {
frontend fe1
bind "fd@${fe1}"
tcp-request session set-var-fmt(sess.str3) "%[var(proc.str1)] %[var(proc.str2)]"
tcp-request session set-var(sess.int5) var(proc.int5)
tcp-request session set-var(proc.int5) var(proc.int5),add(sess.int5) ## proc. becomes 10
tcp-request content set-var-fmt(req.str4) "%[var(sess.str3),regsub(is a,is also a)]"
http-request set-var-fmt(txn.str5) "%[var(req.str4)]"
http-request set-var(req.int5) var(sess.int5)
http-request set-var(sess.int5) var(sess.int5),add(req.int5) ## sess. becomes 10 first time, then 15...
http-request return status 200 hdr x-var "proc=%[var(proc.int5)] sess=%[var(sess.int5)] req=%[var(req.int5)] str=%[var(proc.str)] uuid=%[var(proc.uuid)]"
http-request return status 200 hdr x-var "proc=%[var(proc.int5)] sess=%[var(sess.int5)] req=%[var(req.int5)] str=%[var(proc.str)] str5=%[var(txn.str5)] uuid=%[var(proc.uuid)]"
} -start
haproxy h1 -cli {
@ -35,12 +40,12 @@ client c1 -connect ${h1_fe1_sock} {
txreq -req GET -url /req1_1
rxresp
expect resp.status == 200
expect resp.http.x-var ~ "proc=10 sess=10 req=5 str=this is a string uuid=[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*"
expect resp.http.x-var ~ "proc=10 sess=10 req=5 str=this is a string str5=this is also a string uuid=[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*"
txreq -req GET -url /req1_2
rxresp
expect resp.status == 200
expect resp.http.x-var ~ "proc=10 sess=20 req=10 str=this is a string uuid=[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*"
expect resp.http.x-var ~ "proc=10 sess=20 req=10 str=this is a string str5=this is also a string uuid=[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*"
} -run
haproxy h1 -cli {
@ -52,12 +57,12 @@ client c2 -connect ${h1_fe1_sock} {
txreq -req GET -url /req2_1
rxresp
expect resp.status == 200
expect resp.http.x-var ~ "proc=20 sess=20 req=10 str=this is a string uuid=[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*"
expect resp.http.x-var ~ "proc=20 sess=20 req=10 str=this is a string str5=this is also a string uuid=[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*"
txreq -req GET -url /req2_2
rxresp
expect resp.status == 200
expect resp.http.x-var ~ "proc=20 sess=40 req=20 str=this is a string uuid=[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*"
expect resp.http.x-var ~ "proc=20 sess=40 req=20 str=this is a string str5=this is also a string uuid=[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*"
} -run
haproxy h1 -cli {
@ -74,5 +79,5 @@ client c3 -connect ${h1_fe1_sock} {
txreq -req GET -url /req3_1
rxresp
expect resp.status == 200
expect resp.http.x-var ~ "proc=40 sess=40 req=20 str=updated uuid=[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*"
expect resp.http.x-var ~ "proc=40 sess=40 req=20 str=updated str5=this is also a string uuid=[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*"
} -run

96
src/vars.c
View File

@ -649,6 +649,7 @@ int vars_get_by_desc(const struct var_desc *var_desc, struct sample *smp)
static enum act_return action_store(struct act_rule *rule, struct proxy *px,
struct session *sess, struct stream *s, int flags)
{
struct buffer *fmtstr = NULL;
struct sample smp;
int dir;
@ -670,12 +671,39 @@ static enum act_return action_store(struct act_rule *rule, struct proxy *px,
/* Process the expression. */
memset(&smp, 0, sizeof(smp));
if (!sample_process(px, sess, s, dir|SMP_OPT_FINAL,
rule->arg.vars.expr, &smp))
return ACT_RET_CONT;
if (!LIST_ISEMPTY(&rule->arg.vars.fmt)) {
/* a format-string is used */
fmtstr = alloc_trash_chunk();
if (!fmtstr) {
send_log(px, LOG_ERR, "Vars: memory allocation failure while processing store rule.");
if (!(global.mode & MODE_QUIET) || (global.mode & MODE_VERBOSE))
ha_alert("Vars: memory allocation failure while processing store rule.\n");
return ACT_RET_CONT;
}
/* execute the log-format expression */
fmtstr->data = sess_build_logline(sess, s, fmtstr->area, fmtstr->size, &rule->arg.vars.fmt);
/* convert it to a sample of type string as it's what the vars
* API consumes, and store it.
*/
smp_set_owner(&smp, px, sess, s, 0);
smp.data.type = SMP_T_STR;
smp.data.u.str = *fmtstr;
sample_store_stream(rule->arg.vars.name, rule->arg.vars.scope, &smp);
}
else {
/* an expression is used */
if (!sample_process(px, sess, s, dir|SMP_OPT_FINAL,
rule->arg.vars.expr, &smp))
return ACT_RET_CONT;
}
/* Store the sample, and ignore errors. */
sample_store_stream(rule->arg.vars.name, rule->arg.vars.scope, &smp);
free_trash_chunk(fmtstr);
return ACT_RET_CONT;
}
@ -695,6 +723,14 @@ static enum act_return action_clear(struct act_rule *rule, struct proxy *px,
static void release_store_rule(struct act_rule *rule)
{
struct logformat_node *lf, *lfb;
list_for_each_entry_safe(lf, lfb, &rule->arg.http.fmt, list) {
LIST_DELETE(&lf->list);
release_sample_expr(lf->expr);
free(lf->arg);
free(lf);
}
release_sample_expr(rule->arg.vars.expr);
}
@ -720,6 +756,7 @@ static int conv_check_var(struct arg *args, struct sample_conv *conv,
/* This function is a common parser for using variables. It understands
* the format:
*
* set-var-fmt(<variable-name>) <format-string>
* set-var(<variable-name>) <expression>
* unset-var(<variable-name>)
*
@ -734,9 +771,13 @@ static enum act_parse_ret parse_store(const char **args, int *arg, struct proxy
const char *var_name = args[*arg-1];
int var_len;
const char *kw_name;
int flags, set_var = 0;
int flags, set_var = 0; /* 0=unset-var, 1=set-var, 2=set-var-fmt */
if (strncmp(var_name, "set-var", 7) == 0) {
if (strncmp(var_name, "set-var-fmt", 11) == 0) {
var_name += 11;
set_var = 2;
}
else if (strncmp(var_name, "set-var", 7) == 0) {
var_name += 7;
set_var = 1;
}
@ -746,7 +787,7 @@ static enum act_parse_ret parse_store(const char **args, int *arg, struct proxy
}
if (*var_name != '(') {
memprintf(err, "invalid or incomplete action '%s'. Expects 'set-var(<var-name>)' or 'unset-var(<var-name>)'",
memprintf(err, "invalid or incomplete action '%s'. Expects 'set-var(<var-name>)', 'set-var-fmt(<var-name>)' or 'unset-var(<var-name>)'",
args[*arg-1]);
return ACT_RET_PRS_ERR;
}
@ -754,11 +795,12 @@ static enum act_parse_ret parse_store(const char **args, int *arg, struct proxy
var_len = strlen(var_name);
var_len--; /* remove the ')' */
if (var_name[var_len] != ')') {
memprintf(err, "incomplete expression after action '%s'. Expects 'set-var(<var-name>)' or 'unset-var(<var-name>)'",
memprintf(err, "incomplete argument after action '%s'. Expects 'set-var(<var-name>)', 'set-var-fmt(<var-name>)' or 'unset-var(<var-name>)'",
args[*arg-1]);
return ACT_RET_PRS_ERR;
}
LIST_INIT(&rule->arg.vars.fmt);
rule->arg.vars.name = register_name(var_name, var_len, &rule->arg.vars.scope, 1, err);
if (!rule->arg.vars.name)
return ACT_RET_PRS_ERR;
@ -814,17 +856,30 @@ static enum act_parse_ret parse_store(const char **args, int *arg, struct proxy
return ACT_RET_PRS_ERR;
}
rule->arg.vars.expr = sample_parse_expr((char **)args, arg, px->conf.args.file,
px->conf.args.line, err, &px->conf.args, NULL);
if (!rule->arg.vars.expr)
return ACT_RET_PRS_ERR;
if (set_var == 2) { /* set-var-fmt */
if (!parse_logformat_string(args[*arg], px, &rule->arg.vars.fmt, 0, flags, err))
return ACT_RET_PRS_ERR;
if (!(rule->arg.vars.expr->fetch->val & flags)) {
memprintf(err,
"fetch method '%s' extracts information from '%s', none of which is available here",
kw_name, sample_src_names(rule->arg.vars.expr->fetch->use));
free(rule->arg.vars.expr);
return ACT_RET_PRS_ERR;
(*arg)++;
/* for late error reporting */
free(px->conf.lfs_file);
px->conf.lfs_file = strdup(px->conf.args.file);
px->conf.lfs_line = px->conf.args.line;
} else {
/* set-var */
rule->arg.vars.expr = sample_parse_expr((char **)args, arg, px->conf.args.file,
px->conf.args.line, err, &px->conf.args, NULL);
if (!rule->arg.vars.expr)
return ACT_RET_PRS_ERR;
if (!(rule->arg.vars.expr->fetch->val & flags)) {
memprintf(err,
"fetch method '%s' extracts information from '%s', none of which is available here",
kw_name, sample_src_names(rule->arg.vars.expr->fetch->use));
free(rule->arg.vars.expr);
return ACT_RET_PRS_ERR;
}
}
rule->action = ACT_CUSTOM;
@ -1085,6 +1140,7 @@ static struct sample_conv_kw_list sample_conv_kws = {ILH, {
INITCALL1(STG_REGISTER, sample_register_convs, &sample_conv_kws);
static struct action_kw_list tcp_req_sess_kws = { { }, {
{ "set-var-fmt", parse_store, KWF_MATCH_PREFIX },
{ "set-var", parse_store, KWF_MATCH_PREFIX },
{ "unset-var", parse_store, KWF_MATCH_PREFIX },
{ /* END */ }
@ -1093,6 +1149,7 @@ static struct action_kw_list tcp_req_sess_kws = { { }, {
INITCALL1(STG_REGISTER, tcp_req_sess_keywords_register, &tcp_req_sess_kws);
static struct action_kw_list tcp_req_cont_kws = { { }, {
{ "set-var-fmt", parse_store, KWF_MATCH_PREFIX },
{ "set-var", parse_store, KWF_MATCH_PREFIX },
{ "unset-var", parse_store, KWF_MATCH_PREFIX },
{ /* END */ }
@ -1101,6 +1158,7 @@ static struct action_kw_list tcp_req_cont_kws = { { }, {
INITCALL1(STG_REGISTER, tcp_req_cont_keywords_register, &tcp_req_cont_kws);
static struct action_kw_list tcp_res_kws = { { }, {
{ "set-var-fmt", parse_store, KWF_MATCH_PREFIX },
{ "set-var", parse_store, KWF_MATCH_PREFIX },
{ "unset-var", parse_store, KWF_MATCH_PREFIX },
{ /* END */ }
@ -1109,6 +1167,7 @@ static struct action_kw_list tcp_res_kws = { { }, {
INITCALL1(STG_REGISTER, tcp_res_cont_keywords_register, &tcp_res_kws);
static struct action_kw_list tcp_check_kws = {ILH, {
{ "set-var-fmt", parse_store, KWF_MATCH_PREFIX },
{ "set-var", parse_store, KWF_MATCH_PREFIX },
{ "unset-var", parse_store, KWF_MATCH_PREFIX },
{ /* END */ }
@ -1117,6 +1176,7 @@ static struct action_kw_list tcp_check_kws = {ILH, {
INITCALL1(STG_REGISTER, tcp_check_keywords_register, &tcp_check_kws);
static struct action_kw_list http_req_kws = { { }, {
{ "set-var-fmt", parse_store, KWF_MATCH_PREFIX },
{ "set-var", parse_store, KWF_MATCH_PREFIX },
{ "unset-var", parse_store, KWF_MATCH_PREFIX },
{ /* END */ }
@ -1125,6 +1185,7 @@ static struct action_kw_list http_req_kws = { { }, {
INITCALL1(STG_REGISTER, http_req_keywords_register, &http_req_kws);
static struct action_kw_list http_res_kws = { { }, {
{ "set-var-fmt", parse_store, KWF_MATCH_PREFIX },
{ "set-var", parse_store, KWF_MATCH_PREFIX },
{ "unset-var", parse_store, KWF_MATCH_PREFIX },
{ /* END */ }
@ -1133,6 +1194,7 @@ static struct action_kw_list http_res_kws = { { }, {
INITCALL1(STG_REGISTER, http_res_keywords_register, &http_res_kws);
static struct action_kw_list http_after_res_kws = { { }, {
{ "set-var-fmt", parse_store, KWF_MATCH_PREFIX },
{ "set-var", parse_store, KWF_MATCH_PREFIX },
{ "unset-var", parse_store, KWF_MATCH_PREFIX },
{ /* END */ }